... | ... | @@ -312,6 +312,8 @@ Delivery successful |
|
|
Contents of delivery available at /deliveredcode/deliveries/fusion_matlab/20170920-1
|
|
|
```
|
|
|
|
|
|
## Local Processing
|
|
|
|
|
|
### The Glue Code
|
|
|
After package delivery we can access package information from `ipython` using the `delivered_software` method of the
|
|
|
`flo.sw.lib.glutil` module:
|
... | ... | @@ -337,36 +339,27 @@ pip install -i https://sips.ssec.wisc.edu/eggs/simple |
|
|
pip install -i https://sips.ssec.wisc.edu/eggs timeutil
|
|
|
pip install -i https://sips.ssec.wisc.edu/eggs glutil
|
|
|
```
|
|
|
#### Local Deployment of python glue code
|
|
|
The glue code for `fusion_matlab` can be started by creating the files `deploy/deploy.py` and `source/flo/__init__.py`
|
|
|
in `~/code/PeateScience/packages/fusion_matlab`.
|
|
|
|
|
|
- `deploy/deploy.py`: which copies the python code in `~/code/PeateScience/packages/fusion_matlab/flo` to
|
|
|
`~/code/PeateScience/local/dist/fusion_matlab/flo/`
|
|
|
|
|
|
```python
|
|
|
import os
|
|
|
from flo_deploy.packagelib import *
|
|
|
|
|
|
class fusion_matlab(Package):
|
|
|
def deploy_package(self):
|
|
|
pass
|
|
|
```
|
|
|
|
|
|
after which we can deploy the `fusion_matlab` package...
|
|
|
### Local Deployment of python glue code
|
|
|
The glue code for `fusion_matlab` can be started by creating the file `source/flo/__init__.py`
|
|
|
in `~/code/PeateScience/packages/fusion_matlab`. The glue code can then be linked into the local execution directory:
|
|
|
```bash
|
|
|
cd ~/code/PeateScience/packages/fusion_matlab/deploy
|
|
|
flo_deploy
|
|
|
cd ~/code/PeateScience/local/dist/fusion_matlab
|
|
|
ln -s ../../../packages/fusion_matlab/source/flo ./
|
|
|
```
|
|
|
|
|
|
### Running the glue code and package locally
|
|
|
```bash
|
|
|
repo=$HOME'/git'
|
|
|
work_dir='/data/'$USER'/fusion_matlab/work/local_processing'
|
|
|
|
|
|
### Locally execute the binary package using the python interface code
|
|
|
We can either run the local execution script from a bash shell:
|
|
|
```bash
|
|
|
str_import="import os; from timeutil import TimeInterval, datetime, timedelta; os.chdir('/home/geoffc/code/PeateScience/packages/fusion_matlab'); import example_local_prepare; os.chdir('/mnt/sdata/geoffc/fusion_matlab/work');"
|
|
|
str_exe="satellite='snpp'; granule = datetime(2014, 10, 15, 8, 42); interval = TimeInterval(granule, granule+timedelta(minutes=0)); example_local_prepare.local_execute_example(interval, satellite, '1.0dev1', skip_prepare=False, skip_execute=False)"
|
|
|
python -c "$str_import $str_exe"
|
|
|
str_import_fusion_matlab="import os; from timeutil import TimeInterval, datetime, timedelta; os.chdir('/home/geoffc/code/PeateScience/packages/fusion_matlab'); import example_local_prepare; os.chdir('/data/geoffc/fusion_matlab/work');"
|
|
|
str_import_fusion_matlab_ql="import os; from timeutil import TimeInterval, datetime, timedelta; os.chdir('/home/geoffc/code/PeateScience/packages/fusion_matlab'); import example_local_prepare_ql; os.chdir('/data/geoffc/fusion_matlab/work');"
|
|
|
|
|
|
python -c "$str_import granule = datetime(2015, 4, 17, 14, 36); interval = TimeInterval(granule, granule+timedelta(minutes=0)); example_local_prepare.local_execute_example(interval, 'snpp', '1.0dev1', skip_prepare=True, skip_execute=False)"
|
|
|
python -c "$str_import_fusion_matlab granule = datetime(2015, 4, 17, 14, 36); interval = TimeInterval(granule, granule+timedelta(minutes=0)); example_local_prepare.local_execute_example(interval, 'snpp', '1.0dev2', skip_prepare=True, skip_execute=False)"
|
|
|
python -c "$str_import_fusion_matlab_ql granule = datetime(2015, 4, 17, 14, 36); interval = TimeInterval(granule, granule+timedelta(minutes=0)); example_local_prepare_ql.local_execute_example(interval, 'snpp', '1.0dev3', skip_prepare=True, skip_execute=False)"
|
|
|
```
|
|
|
or from a python shell
|
|
|
```python
|
... | ... | @@ -592,23 +585,79 @@ rados -p dev --id flo get flo3/91069111/VNP02FSN.A2015091.0000.001.2018025170339 |
|
|
|
|
|
## Running in Forward Stream
|
|
|
|
|
|
Bruce has sumbmitted the computation `'flo.sw.fusion_matlab:FUSION_MATLAB'` to the forward stream submitter, for `v1.0dev1`.
|
|
|
The details o failed jobs can be found from
|
|
|
Job parameters for `FUSION_MATLAB` can be found from
|
|
|
```
|
|
|
psql $flo_user -c "select * from failed_jobs where head_computation = 'flo.sw.fusion_matlab:FUSION_MATLAB' and context->'version'='''1.0dev1''' and timestamp > '2018-01-30';"
|
|
|
flo3=> select * from forward_streams where name = 'FusionMatlab';
|
|
|
id | name | offset_start | offset_end | find_contexts_arguments | workflow_head | workflow_targets | workflow_download_onlies | job_mods | output_volume | num_retries | expiration
|
|
|
----+--------------+--------------+------------+-----------------------------------------------+------------------------------------+------------------------------------------------+--------------------------+-------------------------------+---------------+-------------+------------
|
|
|
43 | FusionMatlab | -4 days | 00:00:00 | "version"=>"'1.0dev3'", "satellite"=>"'snpp'" | flo.sw.fusion_matlab:FUSION_MATLAB | {flo.sw.fusion_matlab:FUSION_MATLAB;fused_l1b} | {} | "requests"=>"['Memory=8000']" | | |
|
|
|
(1 row)
|
|
|
```
|
|
|
Job parameters can be found from
|
|
|
and for `FUSION_MATLAB_QL`:
|
|
|
```
|
|
|
psql $flo_user -c "select * from forward_streams where name = 'FusionMatlab';"
|
|
|
flo3=> select * from forward_streams where name = 'FusionMatlabDailyQL';
|
|
|
id | name | offset_start | offset_end | find_contexts_arguments | workflow_head | workflow_targets | workflow_download_onlies | job_mods | output_volume | num_retries | expiration
|
|
|
----+---------------------+--------------+------------+-----------------------------------------------+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------------------------+---------------+-------------+------------
|
|
|
53 | FusionMatlabDailyQL | -6 days | -2 days | "version"=>"'1.0dev2'", "satellite"=>"'snpp'" | flo.sw.fusion_matlab:FUSION_MATLAB_QL | {flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band27_asc,flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band27_desc,flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band33_asc,flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band33_desc} | {} | "classads"=>"['HookKeyword=SCRATCH']", "requests"=>"['Scratch=3','Memory=8000']", "requirements"=>"['TARGET.Scratch >= RequestScratch']" | | |
|
|
|
(1 row)
|
|
|
```
|
|
|
giving
|
|
|
To preview the submission of the Fusion Matlab level-1b files to Forward Stream, we enter the following in the `psql` shell:
|
|
|
```
|
|
|
explain INSERT INTO forward_streams (
|
|
|
name, offset_start, offset_end, find_contexts_arguments,
|
|
|
workflow_head, workflow_targets, job_mods
|
|
|
)
|
|
|
VALUES (
|
|
|
'FusionMatlab',
|
|
|
'-4 Days'::interval,
|
|
|
'00:00:00'::interval,
|
|
|
'version=>"''1.0dev4''", satellite=>"''snpp''"'::hstore,
|
|
|
'flo.sw.fusion_matlab:FUSION_MATLAB',
|
|
|
'{flo.sw.fusion_matlab:FUSION_MATLAB;fused_l1b}'::text[],
|
|
|
'requests=>"[''Memory=8000'']"'::hstore
|
|
|
)
|
|
|
;
|
|
|
```
|
|
|
id | name | offset_start | offset_end | find_contexts_arguments | workflow_head | workflow_targets | workflow_download_onlies | job_mods | output_volume | num_retries
|
|
|
----+--------------+--------------+------------+-----------------------------------------------+------------------------------------+------------------------------------------------+--------------------------+-------------------------------+---------------+-------------
|
|
|
43 | FusionMatlab | -4 days | 00:00:00 | "version"=>"'1.0dev0'", "satellite"=>"'snpp'" | flo.sw.fusion_matlab:FUSION_MATLAB | {flo.sw.fusion_matlab:FUSION_MATLAB;fused_l1b} | {} | "requests"=>"['Memory=8000']" | |
|
|
|
giving
|
|
|
```
|
|
|
QUERY PLAN
|
|
|
---------------------------------------------------------------
|
|
|
Insert on forward_streams (cost=0.00..0.01 rows=1 width=288)
|
|
|
-> Result (cost=0.00..0.01 rows=1 width=288)
|
|
|
(2 rows)
|
|
|
```
|
|
|
To preview the submission of the Fusion Matlab Quicklooks to Forward Stream, we enter the following in the `psql` shell:
|
|
|
```
|
|
|
explain INSERT INTO forward_streams (
|
|
|
name, offset_start, offset_end, find_contexts_arguments,
|
|
|
workflow_head, workflow_targets, job_mods
|
|
|
)
|
|
|
VALUES (
|
|
|
'FusionMatlabDailyQL',
|
|
|
'-6 Days'::interval,
|
|
|
'-2 Days'::interval,
|
|
|
'version=>"''1.0dev3''", satellite=>"''snpp''"'::hstore,
|
|
|
'flo.sw.fusion_matlab:FUSION_MATLAB_QL',
|
|
|
'{flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band27_asc,flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band27_desc,flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band33_asc,flo.sw.fusion_matlab:FUSION_MATLAB_QL;fused_l1b_ql_band33_desc}'::text[],
|
|
|
'requirements=>"[''TARGET.Scratch >= RequestScratch'']", requests=>"[''Scratch=3'',''Memory=8000'']", classads=>"[''HookKeyword=SCRATCH'']"'::hstore
|
|
|
)
|
|
|
;
|
|
|
```
|
|
|
which outputs
|
|
|
```
|
|
|
QUERY PLAN
|
|
|
---------------------------------------------------------------
|
|
|
Insert on forward_streams (cost=0.00..0.01 rows=1 width=288)
|
|
|
-> Result (cost=0.00..0.01 rows=1 width=288)
|
|
|
(2 rows)
|
|
|
```
|
|
|
To actually submit the task, remove the `explain` keyword from the above invocation.
|
|
|
|
|
|
## Examining log files of failed jobs
|
|
|
The details o failed jobs can be found from
|
|
|
```
|
|
|
psql $flo_user -c "select * from failed_jobs where head_computation = 'flo.sw.fusion_matlab:FUSION_MATLAB' and context->'version'='''1.0dev1''' and timestamp > '2018-01-30';"
|
|
|
```
|
|
|
Generate a list of jobnumbers for failed jobs:
|
|
|
```
|
|
|
psql $flo_user -c "SELECT job, context FROM failed_jobs WHERE head_computation='flo.sw.fusion_matlab:FUSION_MATLAB' and context->'version'='''1.0dev1''' and timestamp > '2018-01-30' order by context;" | grep granule | gawk '{print $1}' > fusion_matlab_v1.0dev1_failed_granules.txt
|
... | ... | @@ -685,4 +734,4 @@ for stderr_file in [glob(dir+'-stdout') for dir in [job_number_to_dir('/scratch/ |
|
|
pass
|
|
|
|
|
|
logfile_obj.close()
|
|
|
``` |
|
|
\ No newline at end of file |
|
|
``` |