Packaging CSPP-GEO

The generic packaging scripts are in a git repo, and can be obtained by...

cd ~/GEOCAT/repos/git
git clone git@gitlab.ssec.wisc.edu:scottm/infra.git

creating the directory infra.

There are three main packaging json files associated with CSPP-GEO, cspp_geocat_l2.json, cspp_geocat_l2_cache.json and cspp_geocat_l2_static.json...

infra/configs/GEOCAT/cspp_geocat_l2.json

This is the packaging setup file concerning the application component of the package, which includes the python scripting.

infra/configs/GEOCAT/cspp_geocat_l2_cache.json

This is the packaging setup file concerning the cache component of the package.

infra/configs/GEOCAT/cspp_geocat_l2_static.json

This is the packaging setup file concerning the vendor component of the package, primarily geocat and associated files.

These files all have the sections...

a_home: The path of the dir (e.g.: ~/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0b3) that will be packaged.
a_repo_home: The root SVN repository (e.g.: repos).
copy_from_app:
dereferenced_links: A collection of lists [file,link], which each contain a file or dir, and the location of a link to it. Might include links to third-party binary packages that want to be included (say, like, ShellB3).
links_in_package: A collection of lists [file,link], which each contain a file or dir, and the location of a link to it (both relative to a_home). Links which help enforce a packaging scheme which differs from the physical file locations.
links_from_repos: Links to files in repositories which are being actively developed.
dirs_in_package: A list of dirs to create
ignores: A list of files and dirs which are ingnored by the tar command or rpath.
trace:
repo_urls: A list of version control urls which contribute files to the package.

Once we have updated the json files, in infra/tools we have the following files...

prj-repos.bash: generate a script to pull the repos.
prj-create.bash : This will create your project directory with links.
prj-rpath.bash : do the rpathes
prj-trace.bash : pull out binaries
prj-tar.bash : generates script to create tarball.

which take the command line args...

-H project home.  ~/GEOCAT
-c "path to json config"

We first run prj-repos.bash to populate our work environment with the correct revisions from the desired repos. If we have been doing this by hand (most likely), then we can skip this step.

Running prj-create.bash...

verbosity="";
export PKG_ROOT=/data/users/geoffc/CSPP_GEO_dev/GEOCAT
export INFRA_HOME=$PKG_ROOT/repos/git/infra
export PKG_NAME=cspp-geo-geocat-1.0a1

cd $PKG_ROOT
rm -rf $PKG_ROOT/CSPP_GEOCAT/$PKG_NAME ;
$INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json $verbosity ; echo $'\n';
$INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json $verbosity; echo $'\n';
$INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json $verbosity

builds the software tree cspp-geo-geocat-1.0-rc1 under /data/users/geoffc/CSPP_GEO_dev/GEOCAT. The cspp_geocat_l2*.json files contain the entry "a_home": "CSPP_GEOCAT/cspp-geo-geocat-1.0-rc1", and so create a package dir tree under /data/users/geoffc/CSPP_GEO_dev/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc1. The script prj-create.bash can be run over and over again until the correct package dir tree is achieved, without the need to remove previous attempts (although this may be desirable).

Running prj-rpath.bash...

$INFRA_HOME/tools/prj-rpath.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -v

which inserts the correct RPATH into the listed executables.

Running prj-trace.bash is not needed for this project.

Running prj-tar.bash for the main software package config cspp_geocat_l2.json...

cd $PKG_ROOT
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -t $PKG_NAME

generates these files in the $PKG_ROOT/prj_scripts directory...

cspp-geo-geocat-1.0-rc1_links.bash
cspp-geo-geocat-1.0-rc1_deref.bash
cspp-geo-geocat-1.0-rc1_ignores.bash
cspp-geo-geocat-1.0-rc1_tar.bash

Running prj-tar.bash for the static data config cspp_geocat_l2_static.json...

cd $PKG_ROOT
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json -t $PKG_NAME"_static"

generates these files in the $PKG_ROOT/prj_scripts directory...

cspp-geo-geocat-1.0-rc1_static_links.bash
cspp-geo-geocat-1.0-rc1_static_deref.bash
cspp-geo-geocat-1.0-rc1_static_ignores.bash
cspp-geo-geocat-1.0-rc1_static_tar.bash

Running prj-tar.bash for the cache config cspp_geocat_l2_cache.json...

cd $PKG_ROOT
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json -t $PKG_NAME"_cache"

generates these files in the $PKG_ROOT/prj_scripts directory...

cspp-geo-geocat-1.0-rc1_cache.test_deref.bash
cspp-geo-geocat-1.0-rc1_cache.test_ignores.bash
cspp-geo-geocat-1.0-rc1_cache.test_links.bash
cspp-geo-geocat-1.0-rc1_cache.test_tar.bash

Run the scripts generated in steps 5-7 in $PKG_ROOT/prj_scripts to generate the tarballs...

cd $PKG_ROOT/prj_scripts/
./$PKG_NAME"_tar.bash"
./$PKG_NAME"_static_tar.bash"
./$PKG_NAME"_cache_tar.bash"

Consolidated commands...

Build the new development environment...

verbosity="";
export PKG_ROOT=$HOME/SSEC/CSPP-GEO/docker_test/GEOCAT
export INFRA_HOME=$PKG_ROOT/repos/git/infra
export PKG_NAME=cspp-geo-geocat-1.0a1

cd $PKG_ROOT
rm -rf $PKG_ROOT/CSPP_GEOCAT/$PKG_NAME ;
$INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json $verbosity ; echo $'\n';
$INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json $verbosity; echo $'\n';
$INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json $verbosity

$INFRA_HOME/tools/prj-rpath.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -v

cd $PKG_ROOT/CSPP_GEOCAT
ln -s -f $PKG_NAME cspp-geo-geocat

ensuring that the JSON files have their attribute a_home set to CSPP_GEOCAT/$PKG_NAME.

There will likely be a mismatch between the toolchain versions used to build libHimawari.so and the distributed python environment, resulting in an error like

geocat_l2.sh -h

Traceback (most recent call last): File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/geocat_l2.py", line 29, in from geocat_interface import execution_time, create_l2_products File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/geocat_interface.py", line 29, in import satellites File "/home/RH6B/GEOCAT/repos/git/cspp_geo_geocat/scripts/l2/satellites/init.py", line 24, in from himawari import Himawari File "/home/RH6B/GEOCAT/repos/git/cspp_geo_geocat/scripts/l2/satellites/himawari.py", line 28, in from libHimawari.HimawariScene import HimawariScene File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/libHimawari/HimawariScene.py", line 288, in _hsd = ffi.dlopen(libHimawariPath()) File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/ShellB3/lib/python2.7/site-packages/cffi-0.7.2-py2.7-linux-x86_64.egg/cffi/api.py", line 112, in dlopen lib, function_cache = _make_ffi_library(self, name, flags) File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/ShellB3/lib/python2.7/site-packages/cffi-0.7.2-py2.7-linux-x86_64.egg/cffi/api.py", line 366, in _make_ffi_library backendlib = backend.load_library(name, flags) OSError: cannot load library /home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/libHimawari/libHimawari.so: /home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/local/lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by /home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/libHimawari/libHimawari.so) In order to (temporarily) fix this, we copy the system library file `libstdc++.so.6` to the distribution package...bash cp /opt/gcc/4.9.2/lib64/libstdc++.so.6 /data/users/geoffc/CSPP_GEO_dev/GEOCAT/CSPP_GEOCAT//cspp-geo-geocat-1.0-rc1/common/local/lib64/ ```

* Test the new development environment...

```bash
PKG_NAME="cspp-geo-geocat-1.0-rc1"
export CSPP_GEO_GEOCAT_HOME=/data/users/geoffc/CSPP_GEO_dev/GEOCAT/CSPP_GEOCAT/$PKG_NAME ; . $CSPP_GEO_GEOCAT_HOME/geocat_env.sh
cd /mnt/WORK/work_dir/test_data/sample_data

geocat_l2.sh -h

for dirs in input/goes13_2015_143_1700; \
do \
    outdir="output_"$(basename $dirs); \
    rm -rf $outdir/*; \
    geocat_l2.sh -W $outdir --satellite goes $dirs --no_rtm --tmp_dir test_l1_dir --debug --line_segments 2 --element_segments 2 --num_cpu 4 ; \
done

for dirs in input/himawari08/2016_03_22_082_0220_R301; \
do \
    outdir="output_himawari08_"$(basename $dirs); \
    rm -rf $outdir/*; \
    geocat_l2.sh -W $outdir --satellite him8 $dirs --no_rtm --tmp_dir test_l1_dir --debug --line_segments 2 --element_segments 2 --num_cpu 4 ; \
done
```

```bash
for files in ../output_new_converter/geocatL1.GOES-13.2015143.1700*.nc; \
do \
    ql_geocat_level1.sh $files None --list_datasets ; \
done

for files in ../output_new_converter/geocatL1.GOES-13.2015143.1700*.nc; \
do \
    for dataset in $(ql_geocat_level1.sh $files None --list_datasets); \
    do \
        ql_geocat_level1.sh $files $dataset -O $(basename $files) -S 4 ; \
    done \
done

for files in ../output_new_converter/geocatL2.GOES-13.2015143.1700*.nc; \
do \
    ql_geocat_level2.sh $files None --list_datasets ; \
done

for files in ../output_new_converter/geocatL2.GOES-13.2015143.1700*.nc; \
do \
    for dataset in $(ql_geocat_level2.sh $files None --list_datasets); \
    do \
        ql_geocat_level2.sh $files $dataset -O $(basename $files) -S 4 ; \
    done \
done

```

* Construct the tarball file lists and scripts, which will be written to `$PKG_ROOT/prj_scripts` ...

```bash
cd $PKG_ROOT

$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -t $PKG_NAME
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json -t $PKG_NAME"_static"
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json -t $PKG_NAME"_cache"
```

* Run the scripts in `$PKG_ROOT/prj_scripts` to generate the tarballs in `$PKG_ROOT/packages`...
```bash
cd $PKG_ROOT/prj_scripts/
./$PKG_NAME"_tar.bash"
./$PKG_NAME"_static_tar.bash"
./$PKG_NAME"_cache_tar.bash"
```
which will result in the dir `$PKG_ROOT/packages/$pkg_dir` containing the tarballs. We generally then might move the tarball dir to
`/mnt/WORK/packages` as a backup.

* Untar the tarballs into a test dir and setup...

```bash
cd /mnt/WORK/work_dir
mkdir -p $PKG_NAME"_TEST/output"
ln -s $PKG_NAME"_TEST" cspp-geo-geocat_TEST
tar xzvf /mnt/WORK/packages/$PKG_NAME/$PKG_NAME".tar.gz"        -C $PKG_NAME"_TEST"
tar xzvf /mnt/WORK/packages/$PKG_NAME/$PKG_NAME"_static.tar.gz" -C $PKG_NAME"_TEST"
tar xzvf /mnt/WORK/packages/$PKG_NAME/$PKG_NAME"_cache.tar.gz"  -C $PKG_NAME"_TEST"

export CSPP_GEO_GEOCAT_HOME=/mnt/WORK/work_dir/$PKG_NAME"_TEST"/$PKG_NAME ; . $CSPP_GEO_GEOCAT_HOME/geocat_env.bash
```

* Run the example area files...

```bash
cd /mnt/WORK/work_dir/$PKG_NAME"_TEST"

geocat_l2.bash -W output --satellite goes /mnt/WORK/work_dir/input/goes13_2015_143_17* --no_rtm --tmp_dir test_l1_dir_goes --line_segments 2 --element_segments 2 --num_cpu 4
geocat_l2.bash -W output --satellite goes /mnt/WORK/work_dir/input/goes15_2015_143_21* --no_rtm --tmp_dir test_l1_dir_goes --line_segments 2 --element_segments 2 --num_cpu 4
geocat_l2.bash -W output --satellite him8 /mnt/WORK/work_dir/input/himawari08/2016_03_22_082_0220_* --no_rtm --tmp_dir test_l1_dir_him8 --line_segments 2 --element_segments 2 --num_cpu 4
```

* Create some quicklook images from the resulting Level1 NetCDF files...

Unnavigated...

```bash
mkdir -p png/unnavigated
navstr='_unnav --unnavigated'
```

```bash
l1_files=output/geocatL1.GOES-13.2015143.17*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done |  sort | uniq)

for files in $l1_files; \
do \
    for dataset in $l1_datasets; \
    do \
        ql_geocat_level1.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
    done \
done

l1_files=output/geocatL1.GOES-15.2015143.21*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done |  sort | uniq)

for files in $l1_files; \
do \
    for dataset in $l1_datasets; \
    do \
        ql_geocat_level1.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
    done \
done
```

Navigated...

```bash
mkdir -p png/navigated
navstr=' -m l -S 2'
```

```bash
l1_files=output/geocatL1.GOES-13.2015143.17*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done |  sort | uniq)

for files in $l1_files; \
do \
    for dataset in $l1_datasets; \
    do \
        ql_geocat_level1.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
    done \
done

l1_files=output/geocatL1.GOES-15.2015143.21*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done |  sort | uniq)

for files in $l1_files; \
do \
    for dataset in $l1_datasets; \
    do \
        ql_geocat_level1.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
    done \
done
```

* Create some quicklook images from the resulting Level2 NetCDF files...

```bash
l2_datasets="baseline_cmask_goes_nop_cloud_mask goesnp_ctype_cloud_phase goesnp_ctype_cloud_type ACHA_mode_7_goes_cloud_emissivity ACHA_mode_7_goes_cloud_optical_depth_vis ACHA_mode_7_goes_cloud_particle_effective_radius ACHA_mode_7_goes_cloud_top_height ACHA_mode_7_goes_cloud_top_pressure ACHA_mode_7_goes_cloud_top_temperature DCOMP_mode_3_cloud_albedo DCOMP_mode_3_cloud_spherical_albedo DCOMP_mode_3_cloud_ice_water_path DCOMP_mode_3_cloud_liquid_water_path DCOMP_mode_3_cloud_optical_depth_vis DCOMP_mode_3_cloud_particle_effective_radius goesr_fog_IFR_fog_probability  goesr_fog_LIFR_fog_probability goesr_fog_MVFR_fog_probability"
```

Unnavigated...

```bash
mkdir -p png/unnavigated
navstr='_unnav --unnavigated'
```

```bash
l2_files=output/geocatL2.GOES-13.2015143.17*.nc

for files in $l2_files; \
do \
    for dataset in $l2_datasets; \
    do \
        ql_geocat_level2.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
    done \
done

l2_files=output/geocatL2.GOES-15.2015143.21*.nc

for files in $l2_files; \
do \
    for dataset in $l2_datasets; \
    do \
        ql_geocat_level2.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
    done \
done
```

Navigated...

```bash
mkdir -p png/navigated
navstr=' -m l -S 2'
```

```bash
l2_files=output/geocatL2.GOES-13.2015143.17*.nc

for files in $l2_files; \
do \
    for dataset in $l2_datasets; \
    do \
        ql_geocat_level2.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
    done \
done

l2_files=output/geocatL2.GOES-15.2015143.21*.nc

for files in $l2_files; \
do \
    for dataset in $l2_datasets; \
    do \
        ql_geocat_level2.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
    done \
done

```

Command to get a list of links and their destination for a json file

for files in $(find common -maxdepth 1 -name "*" -print | grep -v \.pyc); \
do \
    pkg_loc=$(ls -la $files | grep git | gawk '{print $(NF-2)}'); \
    repo_loc=$(ls -la $files | gawk '{print $NF}') ; \
    echo "["$'\n'"    "\"$repo_loc\"","$'\n'"    "\"$pkg_loc\"","$'\n'"],"; \
done | sed 's/\/home\/RH6B\/GEOCAT\/repos\///'

tree -L 3 $PKG_NAME/ | grep -v \.pyc

cspp_geo_l2_packaging

Packaging CSPP-GEO