Skip to content
H

hirs_tpw_orbital

The Glue Code

After package delivery we can access package information from ipython using the delivered_software method of the flo.sw.lib.glutil module:

from flo.sw.lib.glutil import delivered_software
delivered_software.lookup('hirstpw_L2', delivery_id='20181107-1')

which gives the output

Delivery(id='20181107-1', name='hirstpw_L2', version='3.0', path='/mnt/deliveredcode/deliveries/hirstpw_L2/20181107-1')

We also need to install the following packages:

pip install -i https://sips.ssec.wisc.edu/eggs -U flo rados
pip install -i https://sips.ssec.wisc.edu/eggs -U sipsprod glutil
pip install -i https://sips.ssec.wisc.edu/eggs -U timeutil
pip install -i https://sips.ssec.wisc.edu/eggs -U simple

Local Deployment of python glue code

The glue code for hirs_tpw_orbital can be started by creating the file source/flo/__init__.py in ~/code/PeateScience/packages/hirs_tpw_orbital. The glue code can then be linked into the local execution directory:

cd ~/code/PeateScience/local/dist/hirs_tpw_orbital
ln -s ../../../packages/hirs_tpw_orbital/source/flo ./

Local Processing

repo=$HOME'/git'
work_dir='/data/'$USER'/HIRS_processing/Work/local_processing/hirs_tpw_orbital'

satellite='metop-b'

str_import_hirs_tpw_orbital="import os; from timeutil import TimeInterval, datetime, timedelta; os.chdir('$repo/hirs_tpw_orbital'); import example_local_prepare; os.chdir('$work_dir')"

hirs2nc_id='20180410-1'
hirs_avhrr_id='20180505-1'
hirs_csrb_daily_id='20180714-1'
hirs_csrb_monthly_id='20180516-1'
hirs_ctp_orbital_id='20180730-1'
hirs_ctp_daily_id='20180802-1'
hirs_ctp_monthly_id='20180803-1'
hirs_tpw_orbital_id='20190722-1'

python -c "$str_import_hirs_tpw_orbital; granule = datetime(2002, 7, 7, 13, 52); interval = TimeInterval(granule, granule+timedelta(hours=0)); example_local_prepare.print_contexts(interval, '$satellite', '$hirs2nc_id', '$hirs_avhrr_id', '$hirs_csrb_daily_id', '$hirs_csrb_monthly_id', '$hirs_ctp_orbital_id', '$hirs_ctp_daily_id', '$hirs_ctp_monthly_id', '$hirs_tpw_orbital_id', verbosity=2)"

python -c "$str_import_hirs_tpw_orbital; granule = datetime(2002, 7, 7, 13, 52); interval = TimeInterval(granule, granule+timedelta(hours=0)); example_local_prepare.local_execute_example(interval, '$satellite', '$hirs2nc_id', '$hirs_avhrr_id', '$hirs_csrb_daily_id', '$hirs_csrb_monthly_id', '$hirs_ctp_orbital_id', '$hirs_ctp_daily_id', '$hirs_ctp_monthly_id', '$hirs_tpw_orbital_id', skip_prepare=False, skip_execute=False, verbosity=2)"

Downloading results from the cluster

This code will download orbitals into a directory structure

└── metop-a
    ├── 2007
    │   ├── 2007_06_29_180
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S0058.E0241.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S0240.E0424.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S0424.E0606.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S0606.E0749.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S0748.E0930.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S0929.E1109.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S1109.E1250.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S1250.E1430.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S1429.E1610.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S1609.E1751.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S1750.E1932.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S1931.E2113.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07180.S2112.E2254.nc
    │   │   └── hirs_tpw_orbital_metop-a_D07180.S2253.E0037.nc
    │   ├── 2007_06_30_181
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S0037.E0220.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S0219.E0403.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S0402.E0545.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S0545.E0728.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S0727.E0909.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S0908.E1050.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S1049.E1229.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S1229.E1409.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S1408.E1550.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S1549.E1730.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S1729.E1911.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S1910.E2052.nc
    │   │   ├── hirs_tpw_orbital_metop-a_D07181.S2051.E2233.nc
    │   │   └── hirs_tpw_orbital_metop-a_D07181.S2232.E0016.nc
    ...
satellite='metop-a'
sat_dir='/data/geoffc/HIRS_processing/results/hirs_tpw_20190722-1/orbitals/'$satellite

# Get string representations of the begin and end datetime objects
dt_start=$(psql $flo_user -tA -c "select context from stored_products where computation='flo.sw.hirs2nc:HIRS2NC' and context->'satellite'='''$satellite''' order by context;" | head -n 1 | gawk -F'"datetime.' '{print $NF}' | gawk -F'"' '{print $1}')
dt_end=$(psql $flo_user -tA -c "select context from stored_products where computation='flo.sw.hirs2nc:HIRS2NC' and context->'satellite'='''$satellite''' order by context;" | tail -n 1 | gawk -F'"datetime.' '{print $NF}' | gawk -F'"' '{print $1}')
# ...or set start and end times explicitly...
dt_start='datetime(2007, 5, 1, 0, 0)'
dt_end='datetime(2017, 4, 30, 23, 59)'

day_dt=$dt_start

mkdir -p $sat_dir
cd $sat_dir

while : ; \
do \
    echo "day_dt = "$day_dt; \

    # Compute various dates
    year=$(python -c "from datetime import datetime; print($day_dt.utctimetuple().tm_year)"); \
    jday=$(python -c "from datetime import datetime; print($day_dt.utctimetuple().tm_yday)"); \
    day_dir=$(python -c "from datetime import datetime; print($day_dt.strftime('%Y_%m_%d_%j'))"); \
    psql_date=$(python -c 'from datetime import datetime; print(datetime.strptime("'$year$jday'","%Y%j").strftime("%Y-%m-%d"))'); \

    # Create the required directory
    echo "Creating directory "$year/$day_dir; \
    mkdir -p $year/$day_dir; \
    cd $year/$day_dir; \

    # Retrieve the data from this date.
    echo "Retrieving data from "$psql_date; \
    psql $flo_user -tA -c "select job from stored_products where context->'satellite'='''$satellite''' and computation='flo.sw.hirs_tpw_orbital:HIRS_TPW_ORBITAL' and date_trunc('days',pydt(context->'granule'))='$psql_date' order by file_name" | xargs -n 1 flo_fetch -j ; \
    cd $sat_dir

    # Compute next day_dt
    day_dt=$(python -c "from datetime import datetime,timedelta; print(repr($day_dt + timedelta(days=1)).split('.')[1])"); \

    # Check if the new date is off the end of the date range.
    dt_test=$(python -c "from datetime import datetime; print(int(($dt_end - $day_dt).total_seconds()))"); \
    if (( dt_test < 0 )); \
    then \
        break; \
    fi ; \
done

cd ..