H
hirs_tpw_orbital
The Glue Code
After package delivery we can access package information from ipython
using the delivered_software
method of the
flo.sw.lib.glutil
module:
from flo.sw.lib.glutil import delivered_software
delivered_software.lookup('hirstpw_L2', delivery_id='20181107-1')
which gives the output
Delivery(id='20181107-1', name='hirstpw_L2', version='3.0', path='/mnt/deliveredcode/deliveries/hirstpw_L2/20181107-1')
We also need to install the following packages:
pip install -i https://sips.ssec.wisc.edu/eggs -U flo rados
pip install -i https://sips.ssec.wisc.edu/eggs -U sipsprod glutil
pip install -i https://sips.ssec.wisc.edu/eggs -U timeutil
pip install -i https://sips.ssec.wisc.edu/eggs -U simple
Local Deployment of python glue code
The glue code for hirs_tpw_orbital
can be started by creating the file source/flo/__init__.py
in ~/code/PeateScience/packages/hirs_tpw_orbital
. The glue code can then be linked into the local execution directory:
cd ~/code/PeateScience/local/dist/hirs_tpw_orbital
ln -s ../../../packages/hirs_tpw_orbital/source/flo ./
Local Processing
repo=$HOME'/git'
work_dir='/data/'$USER'/HIRS_processing/Work/local_processing/hirs_tpw_orbital'
satellite='metop-b'
str_import_hirs_tpw_orbital="import os; from timeutil import TimeInterval, datetime, timedelta; os.chdir('$repo/hirs_tpw_orbital'); import example_local_prepare; os.chdir('$work_dir')"
hirs2nc_id='20180410-1'
hirs_avhrr_id='20180505-1'
hirs_csrb_daily_id='20180714-1'
hirs_csrb_monthly_id='20180516-1'
hirs_ctp_orbital_id='20180730-1'
hirs_ctp_daily_id='20180802-1'
hirs_ctp_monthly_id='20180803-1'
hirs_tpw_orbital_id='20190722-1'
python -c "$str_import_hirs_tpw_orbital; granule = datetime(2002, 7, 7, 13, 52); interval = TimeInterval(granule, granule+timedelta(hours=0)); example_local_prepare.print_contexts(interval, '$satellite', '$hirs2nc_id', '$hirs_avhrr_id', '$hirs_csrb_daily_id', '$hirs_csrb_monthly_id', '$hirs_ctp_orbital_id', '$hirs_ctp_daily_id', '$hirs_ctp_monthly_id', '$hirs_tpw_orbital_id', verbosity=2)"
python -c "$str_import_hirs_tpw_orbital; granule = datetime(2002, 7, 7, 13, 52); interval = TimeInterval(granule, granule+timedelta(hours=0)); example_local_prepare.local_execute_example(interval, '$satellite', '$hirs2nc_id', '$hirs_avhrr_id', '$hirs_csrb_daily_id', '$hirs_csrb_monthly_id', '$hirs_ctp_orbital_id', '$hirs_ctp_daily_id', '$hirs_ctp_monthly_id', '$hirs_tpw_orbital_id', skip_prepare=False, skip_execute=False, verbosity=2)"
Downloading results from the cluster
This code will download orbitals into a directory structure
└── metop-a
├── 2007
│ ├── 2007_06_29_180
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S0058.E0241.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S0240.E0424.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S0424.E0606.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S0606.E0749.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S0748.E0930.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S0929.E1109.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S1109.E1250.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S1250.E1430.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S1429.E1610.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S1609.E1751.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S1750.E1932.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S1931.E2113.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07180.S2112.E2254.nc
│ │ └── hirs_tpw_orbital_metop-a_D07180.S2253.E0037.nc
│ ├── 2007_06_30_181
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S0037.E0220.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S0219.E0403.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S0402.E0545.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S0545.E0728.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S0727.E0909.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S0908.E1050.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S1049.E1229.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S1229.E1409.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S1408.E1550.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S1549.E1730.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S1729.E1911.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S1910.E2052.nc
│ │ ├── hirs_tpw_orbital_metop-a_D07181.S2051.E2233.nc
│ │ └── hirs_tpw_orbital_metop-a_D07181.S2232.E0016.nc
...
satellite='metop-a'
sat_dir='/data/geoffc/HIRS_processing/results/hirs_tpw_20190722-1/orbitals/'$satellite
# Get string representations of the begin and end datetime objects
dt_start=$(psql $flo_user -tA -c "select context from stored_products where computation='flo.sw.hirs2nc:HIRS2NC' and context->'satellite'='''$satellite''' order by context;" | head -n 1 | gawk -F'"datetime.' '{print $NF}' | gawk -F'"' '{print $1}')
dt_end=$(psql $flo_user -tA -c "select context from stored_products where computation='flo.sw.hirs2nc:HIRS2NC' and context->'satellite'='''$satellite''' order by context;" | tail -n 1 | gawk -F'"datetime.' '{print $NF}' | gawk -F'"' '{print $1}')
# ...or set start and end times explicitly...
dt_start='datetime(2007, 5, 1, 0, 0)'
dt_end='datetime(2017, 4, 30, 23, 59)'
day_dt=$dt_start
mkdir -p $sat_dir
cd $sat_dir
while : ; \
do \
echo "day_dt = "$day_dt; \
# Compute various dates
year=$(python -c "from datetime import datetime; print($day_dt.utctimetuple().tm_year)"); \
jday=$(python -c "from datetime import datetime; print($day_dt.utctimetuple().tm_yday)"); \
day_dir=$(python -c "from datetime import datetime; print($day_dt.strftime('%Y_%m_%d_%j'))"); \
psql_date=$(python -c 'from datetime import datetime; print(datetime.strptime("'$year$jday'","%Y%j").strftime("%Y-%m-%d"))'); \
# Create the required directory
echo "Creating directory "$year/$day_dir; \
mkdir -p $year/$day_dir; \
cd $year/$day_dir; \
# Retrieve the data from this date.
echo "Retrieving data from "$psql_date; \
psql $flo_user -tA -c "select job from stored_products where context->'satellite'='''$satellite''' and computation='flo.sw.hirs_tpw_orbital:HIRS_TPW_ORBITAL' and date_trunc('days',pydt(context->'granule'))='$psql_date' order by file_name" | xargs -n 1 flo_fetch -j ; \
cd $sat_dir
# Compute next day_dt
day_dt=$(python -c "from datetime import datetime,timedelta; print(repr($day_dt + timedelta(days=1)).split('.')[1])"); \
# Check if the new date is off the end of the date range.
dt_test=$(python -c "from datetime import datetime; print(int(($dt_end - $day_dt).total_seconds()))"); \
if (( dt_test < 0 )); \
then \
break; \
fi ; \
done
cd ..