Skip to content
H

hirs_ctp_daily

The Glue Code

After package delivery we can access package information from ipython using the delivered_software method of the flo.sw.lib.glutil module:

from flo.sw.lib.glutil import delivered_software
delivered_software.lookup('hirs_ctp_daily', delivery_id='20180802-1')

which gives the output

Delivery(id='20180802-1', name='hirs_ctp_daily', version='7.4.1', path='/mnt/deliveredcode/deliveries/hirs_ctp_daily/20180802-1')

We also need to install the following packages:

pip install -i https://sips.ssec.wisc.edu/eggs -U flo rados
pip install -i https://sips.ssec.wisc.edu/eggs -U sipsprod glutil
pip install -i https://sips.ssec.wisc.edu/eggs -U timeutil
pip install -i https://sips.ssec.wisc.edu/eggs -U simple

Local Deployment of python glue code

The glue code for hirs_ctp_daily can be started by creating the file source/flo/__init__.py in ~/code/PeateScience/packages/hirs_ctp_daily. The glue code can then be linked into the local execution directory:

cd ~/code/PeateScience/packages/hirs_ctp_daily
ln -s ../../../packages/hirs_ctp_daily/source/flo ./

Local Processing

repo=$HOME'/git'
work_dir='/data/'$USER'/HIRS_processing/Work/local_processing/hirs_ctp_daily'

satellite='metop-b'

str_import_hirs_ctp_daily="import os; from timeutil import TimeInterval, datetime, timedelta; wedge=timedelta(seconds=1); day=timedelta(days=1); os.chdir('$repo/hirs_ctp_daily'); import example_local_prepare; os.chdir('$work_dir')"

hirs2nc_id='20180410-1'
hirs_avhrr_id='20180505-1'
hirs_csrb_daily_id='20180714-1'
hirs_csrb_monthly_id='20180516-1'
hirs_ctp_orbital_id='20180730-1'
hirs_ctp_daily_id='20180802-1'

python -c "$str_import_hirs_ctp_daily; granule =  datetime(2015,5,14, 0); interval = TimeInterval(granule, granule+day-wedge); example_local_prepare.local_execute_example(interval, '$satellite', hirs2nc_id, hirs_avhrr_id, hirs_csrb_daily_id, hirs_csrb_monthly_id, hirs_ctp_orbital_id, hirs_ctp_daily_id, skip_prepare=False, skip_execute=True, verbosity=2)"

python -c "$str_import_hirs_ctp_daily; granule = datetime(2015,5,14, 0); interval = TimeInterval(granule, granule+day-wedge); example_local_prepare.print_contexts(interval, '$satellite', hirs2nc_id, hirs_avhrr_id, hirs_csrb_daily_id, hirs_csrb_monthly_id, hirs_ctp_orbital_id, hirs_ctp_daily_id, verbosity=2)"

Downloading results from the cluster

satellite='metop-a'
sat_dir='/data/geoffc/HIRS_processing/results/hirs_ctp_20180730-1/dailies/'$satellite

# Get string representations of the begin and end datetime objects
dt_start=$(psql $flo_user -tA -c "select context from stored_products where computation='flo.sw.hirs2nc:HIRS2NC' and context->'satellite'='''$satellite''' order by context;" | head -n 1 | gawk -F'"datetime.' '{print $NF}' | gawk -F'"' '{print $1}')
dt_end=$(psql $flo_user -tA -c "select context from stored_products where computation='flo.sw.hirs2nc:HIRS2NC' and context->'satellite'='''$satellite''' order by context;" | tail -n 1 | gawk -F'"datetime.' '{print $NF}' | gawk -F'"' '{print $1}')
# ...or set start and end times explicitly...
dt_start='datetime(2007, 5, 1, 0, 0)'
dt_end='datetime(2017, 4, 30, 23, 59)'

year_dt=$dt_start

mkdir -p $sat_dir
cd $sat_dir

while : ; \
do \
    echo "year_dt = "$year_dt; \

    # Compute various dates
    year=$(python -c "from datetime import datetime; print($year_dt.utctimetuple().tm_year)"); \
    psql_date=$(python -c 'from datetime import datetime; print(datetime.strptime("'$year'","%Y").strftime("%Y-%m-%d"))'); \

    # Create the required directory
    echo "Creating directory "$year; \
    mkdir -p $year; \
    cd $year; \

    # Retrieve the data from this date.
    echo "Retrieving data from "$psql_date; \
    psql $flo_user -tA -c "select job from stored_products where context->'satellite'='''$satellite''' and computation='flo.sw.hirs_ctp_daily:HIRS_CTP_DAILY' and date_trunc('years',pydt(context->'granule'))='$psql_date' order by file_name" | xargs -n 1 flo_fetch -j ; \
    cd $sat_dir

    # Compute next year_dt
    year_dt=$(python -c "from datetime import datetime,timedelta; print(repr($year_dt + timedelta(years=1)).split('.')[1])"); \

    # Check if the new date is off the end of the date range.
    dt_test=$(python -c "from datetime import datetime; print(int(($dt_end - $year_dt).total_seconds()))"); \
    if (( dt_test < 0 )); \
    then \
        break; \
    fi ; \
done

cd ..