hirs_csrb_monthly

The Glue Code

After package delivery we can access package information from ipython using the delivered_software method of the flo.sw.lib.glutil module:

from flo.sw.lib.glutil import delivered_software
delivered_software.lookup('hirs_csrb_monthly', delivery_id='20180516-1')

which gives the output

Delivery(id='20180516-1', name='hirs_csrb_monthly', version='7.4.1', path='/mnt/deliveredcode/deliveries/hirs_csrb_monthly/20180516-1')

We also need to install the following packages:

pip install -i https://sips.ssec.wisc.edu/eggs -U flo rados
pip install -i https://sips.ssec.wisc.edu/eggs -U sipsprod glutil
pip install -i https://sips.ssec.wisc.edu/eggs -U timeutil
pip install -i https://sips.ssec.wisc.edu/eggs -U simple

Local Deployment of python glue code

The glue code for hirs_csrb_monthly can be started by creating the file source/flo/__init__.py in ~/code/PeateScience/packages/hirs_csrb_monthly. The glue code can then be linked into the local execution directory:

cd ~/code/PeateScience/local/dist/hirs_csrb_monthly
ln -s ../../../packages/hirs_csrb_monthly/source/flo ./

Local Processing

repo=$HOME'/git'
work_dir='/data/'$USER'/HIRS_processing/Work/local_processing/hirs_csrb_monthly'

satellite='metop-b'

str_import_hirs_csrb_monthly="import os, calendar; from timeutil import TimeInterval, datetime, timedelta; wedge=timedelta(seconds=1); day=timedelta(days=1); os.chdir('$repo/hirs_csrb_monthly'); import example_local_prepare; os.chdir('$work_dir')"

hirs2nc_id='20180410-1'
hirs_avhrr_id='20180505-1'
hirs_csrb_daily_id='20180714-1'
hirs_csrb_monthly_id='20180516-1'

python -c "$str_import_hirs_csrb_monthly; year,month=2015,7; interval = TimeInterval(datetime(year, month, 1, 0), datetime(year, month, calendar.monthrange(year,month)[1])+day-wedge); example_local_prepare.local_execute_example(interval, '$satellite', '$hirs2nc_id', '$hirs_avhrr_id', '$hirs_csrb_daily_id', '$hirs_csrb_monthly_id', skip_prepare=False, skip_execute=True, verbosity=3)"

python -c "$str_import_hirs_csrb_monthly; year,month=2015,7; interval = TimeInterval(datetime(year, month, 1, 0), datetime(year, month, calendar.monthrange(year,month)[1])+day-wedge); example_local_prepare.print_contexts(interval, '$satellite', '$hirs2nc_id', '$hirs_avhrr_id', '$hirs_csrb_daily_id', '$hirs_csrb_monthly_id', verbosity=3)"

Cluster Processing

hirs_csrb_monthly files generated on the cluster can be downloaded using rados...

satellite='metop-b'

flo_user='-d postgresql://flo3@ratchet.sips/flo3'

psql $flo_user -tA -c "SELECT format ('flo3/%s/%s %s',job,file_name,file_name) FROM stored_products WHERE  computation='flo.sw.hirs_csrb_monthly:HIRS_CSRB_MONTHLY' and context->'satellite'='''$satellite'''  order by file_name ;"  | xargs -n1 -IXX echo rados -p dev --id flo get XX > csrb_monthly_zmeans.sh

sh csrb_monthly_zmeans.sh

Downloading Results from the Cluster

export satellite="noaa-19"
flo_dbase='postgresql://flo3@ratchet.sips/flo3'
satellite='noaa-17'; psql -d $flo_dbase -tA -c "select job from stored_products where context->'satellite'='''$satellite''' and computation='flo.sw.hirs_csrb_monthly:HIRS_CSRB_MONTHLY' and output='zonal_means' order by context" | xargs -n 1 flo_fetch -o 'zonal_means' -j

satellite='noaa-17'; psql -d $flo_dbase -tA -c "select job from stored_products where context->'satellite'='''$satellite''' and computation='flo.sw.hirs_ctp_monthly:HIRS_CTP_MONTHLY' order by context" | xargs -n 1 flo_fetch -j