Packaging CSPP-GEO
The generic packaging scripts are in a git repo, and can be obtained by...
cd ~/GEOCAT/repos/git
git clone git@gitlab.ssec.wisc.edu:scottm/infra.git
creating the directory infra
.
There are three main packaging json files associated with CSPP-GEO, cspp_geocat_l2.json
, cspp_geocat_l2_cache.json
and cspp_geocat_l2_static.json
...
infra/configs/GEOCAT/cspp_geocat_l2.json
- This is the packaging setup file concerning the application component of the package, which includes the python scripting.
infra/configs/GEOCAT/cspp_geocat_l2_cache.json
- This is the packaging setup file concerning the cache component of the package.
infra/configs/GEOCAT/cspp_geocat_l2_static.json
- This is the packaging setup file concerning the vendor component of the package, primarily
geocat
and associated files.
These files all have the sections...
-
a_home
: The path of the dir (e.g.:~/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0b3
) that will be packaged. -
a_repo_home
: The root SVN repository (e.g.:repos
). -
copy_from_app
: -
dereferenced_links
: A collection of lists[file,link]
, which each contain a file or dir, and the location of a link to it. Might include links to third-party binary packages that want to be included (say, like, ShellB3). -
links_in_package
: A collection of lists[file,link]
, which each contain a file or dir, and the location of a link to it (both relative toa_home
). Links which help enforce a packaging scheme which differs from the physical file locations. -
links_from_repos
: Links to files in repositories which are being actively developed. -
dirs_in_package
: A list of dirs to create -
ignores
: A list of files and dirs which are ingnored by the tar command or rpath. -
trace
: -
repo_urls
: A list of version control urls which contribute files to the package.
Once we have updated the json files, in infra/tools
we have the following files...
-
prj-repos.bash
: generate a script to pull the repos. -
prj-create.bash
: This will create your project directory with links. -
prj-rpath.bash
: do the rpathes -
prj-trace.bash
: pull out binaries -
prj-tar.bash
: generates script to create tarball.
which take the command line args...
-H project home. ~/GEOCAT
-c "path to json config"
-
We first run
prj-repos.bash
to populate our work environment with the correct revisions from the desired repos. If we have been doing this by hand (most likely), then we can skip this step. -
Running
prj-create.bash
...verbosity=""; export PKG_ROOT=/data/users/geoffc/CSPP_GEO_dev/GEOCAT export INFRA_HOME=$PKG_ROOT/repos/git/infra export PKG_NAME=cspp-geo-geocat-1.0a1 cd $PKG_ROOT rm -rf $PKG_ROOT/CSPP_GEOCAT/$PKG_NAME ; $INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json $verbosity ; echo $'\n'; $INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json $verbosity; echo $'\n'; $INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json $verbosity
builds the software tree
cspp-geo-geocat-1.0-rc1
under/data/users/geoffc/CSPP_GEO_dev/GEOCAT
. Thecspp_geocat_l2*.json
files contain the entry"a_home": "CSPP_GEOCAT/cspp-geo-geocat-1.0-rc1"
, and so create a package dir tree under/data/users/geoffc/CSPP_GEO_dev/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc1
. The scriptprj-create.bash
can be run over and over again until the correct package dir tree is achieved, without the need to remove previous attempts (although this may be desirable). -
Running
prj-rpath.bash
...$INFRA_HOME/tools/prj-rpath.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -v
which inserts the correct
RPATH
into the listed executables. -
Running
prj-trace.bash
is not needed for this project. -
Running
prj-tar.bash
for the main software package configcspp_geocat_l2.json
...cd $PKG_ROOT $INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -t $PKG_NAME
generates these files in the
$PKG_ROOT/prj_scripts
directory...cspp-geo-geocat-1.0-rc1_links.bash cspp-geo-geocat-1.0-rc1_deref.bash cspp-geo-geocat-1.0-rc1_ignores.bash cspp-geo-geocat-1.0-rc1_tar.bash
-
Running
prj-tar.bash
for the static data configcspp_geocat_l2_static.json
...cd $PKG_ROOT $INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json -t $PKG_NAME"_static"
generates these files in the
$PKG_ROOT/prj_scripts
directory...cspp-geo-geocat-1.0-rc1_static_links.bash cspp-geo-geocat-1.0-rc1_static_deref.bash cspp-geo-geocat-1.0-rc1_static_ignores.bash cspp-geo-geocat-1.0-rc1_static_tar.bash
-
Running
prj-tar.bash
for the cache configcspp_geocat_l2_cache.json
...cd $PKG_ROOT $INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json -t $PKG_NAME"_cache"
generates these files in the
$PKG_ROOT/prj_scripts
directory...cspp-geo-geocat-1.0-rc1_cache.test_deref.bash cspp-geo-geocat-1.0-rc1_cache.test_ignores.bash cspp-geo-geocat-1.0-rc1_cache.test_links.bash cspp-geo-geocat-1.0-rc1_cache.test_tar.bash
-
Run the scripts generated in steps 5-7 in
$PKG_ROOT/prj_scripts
to generate the tarballs...cd $PKG_ROOT/prj_scripts/ ./$PKG_NAME"_tar.bash" ./$PKG_NAME"_static_tar.bash" ./$PKG_NAME"_cache_tar.bash"
-
Consolidated commands...
- Build the new development environment...
verbosity=""; export PKG_ROOT=$HOME/SSEC/CSPP-GEO/docker_test/GEOCAT export INFRA_HOME=$PKG_ROOT/repos/git/infra export PKG_NAME=cspp-geo-geocat-1.0a1 cd $PKG_ROOT rm -rf $PKG_ROOT/CSPP_GEOCAT/$PKG_NAME ; $INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json $verbosity ; echo $'\n'; $INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json $verbosity; echo $'\n'; $INFRA_HOME/tools/prj-create.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json $verbosity $INFRA_HOME/tools/prj-rpath.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -v cd $PKG_ROOT/CSPP_GEOCAT ln -s -f $PKG_NAME cspp-geo-geocat
ensuring that the JSON files have their attribute
a_home
set toCSPP_GEOCAT/$PKG_NAME
.- There will likely be a mismatch between the toolchain versions used to build
libHimawari.so
and the distributed python environment, resulting in an error like
geocat_l2.sh -h
Traceback (most recent call last):
File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/geocat_l2.py", line 29, in
from geocat_interface import execution_time, create_l2_products
File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/geocat_interface.py", line 29, in
import satellites
File "/home/RH6B/GEOCAT/repos/git/cspp_geo_geocat/scripts/l2/satellites/init.py", line 24, in
from himawari import Himawari
File "/home/RH6B/GEOCAT/repos/git/cspp_geo_geocat/scripts/l2/satellites/himawari.py", line 28, in
from libHimawari.HimawariScene import HimawariScene
File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/libHimawari/HimawariScene.py", line 288, in
_hsd = ffi.dlopen(libHimawariPath())
File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/ShellB3/lib/python2.7/site-packages/cffi-0.7.2-py2.7-linux-x86_64.egg/cffi/api.py", line 112, in dlopen
lib, function_cache = _make_ffi_library(self, name, flags)
File "/home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/ShellB3/lib/python2.7/site-packages/cffi-0.7.2-py2.7-linux-x86_64.egg/cffi/api.py", line 366, in _make_ffi_library
backendlib = backend.load_library(name, flags)
OSError: cannot load library /home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/libHimawari/libHimawari.so: /home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/common/local/lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by /home/RH6B/GEOCAT/CSPP_GEOCAT/cspp-geo-geocat-1.0-rc/l2/libHimawari/libHimawari.so)
In order to (temporarily) fix this, we copy the system library file `libstdc++.so.6` to the distribution package...
bash
cp /opt/gcc/4.9.2/lib64/libstdc++.so.6 /data/users/geoffc/CSPP_GEO_dev/GEOCAT/CSPP_GEOCAT//cspp-geo-geocat-1.0-rc1/common/local/lib64/
```
* Test the new development environment...
```bash
PKG_NAME="cspp-geo-geocat-1.0-rc1"
export CSPP_GEO_GEOCAT_HOME=/data/users/geoffc/CSPP_GEO_dev/GEOCAT/CSPP_GEOCAT/$PKG_NAME ; . $CSPP_GEO_GEOCAT_HOME/geocat_env.sh
cd /mnt/WORK/work_dir/test_data/sample_data
geocat_l2.sh -h
for dirs in input/goes13_2015_143_1700; \
do \
outdir="output_"$(basename $dirs); \
rm -rf $outdir/*; \
geocat_l2.sh -W $outdir --satellite goes $dirs --no_rtm --tmp_dir test_l1_dir --debug --line_segments 2 --element_segments 2 --num_cpu 4 ; \
done
for dirs in input/himawari08/2016_03_22_082_0220_R301; \
do \
outdir="output_himawari08_"$(basename $dirs); \
rm -rf $outdir/*; \
geocat_l2.sh -W $outdir --satellite him8 $dirs --no_rtm --tmp_dir test_l1_dir --debug --line_segments 2 --element_segments 2 --num_cpu 4 ; \
done
```
```bash
for files in ../output_new_converter/geocatL1.GOES-13.2015143.1700*.nc; \
do \
ql_geocat_level1.sh $files None --list_datasets ; \
done
for files in ../output_new_converter/geocatL1.GOES-13.2015143.1700*.nc; \
do \
for dataset in $(ql_geocat_level1.sh $files None --list_datasets); \
do \
ql_geocat_level1.sh $files $dataset -O $(basename $files) -S 4 ; \
done \
done
for files in ../output_new_converter/geocatL2.GOES-13.2015143.1700*.nc; \
do \
ql_geocat_level2.sh $files None --list_datasets ; \
done
for files in ../output_new_converter/geocatL2.GOES-13.2015143.1700*.nc; \
do \
for dataset in $(ql_geocat_level2.sh $files None --list_datasets); \
do \
ql_geocat_level2.sh $files $dataset -O $(basename $files) -S 4 ; \
done \
done
```
* Construct the tarball file lists and scripts, which will be written to `$PKG_ROOT/prj_scripts` ...
```bash
cd $PKG_ROOT
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2.json -t $PKG_NAME
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_static.json -t $PKG_NAME"_static"
$INFRA_HOME/tools/prj-tar.bash -H $PKG_ROOT -c $INFRA_HOME/configs/GEOCAT/cspp_geocat_l2_cache.json -t $PKG_NAME"_cache"
```
* Run the scripts in `$PKG_ROOT/prj_scripts` to generate the tarballs in `$PKG_ROOT/packages`...
```bash
cd $PKG_ROOT/prj_scripts/
./$PKG_NAME"_tar.bash"
./$PKG_NAME"_static_tar.bash"
./$PKG_NAME"_cache_tar.bash"
```
which will result in the dir `$PKG_ROOT/packages/$pkg_dir` containing the tarballs. We generally then might move the tarball dir to
`/mnt/WORK/packages` as a backup.
* Untar the tarballs into a test dir and setup...
```bash
cd /mnt/WORK/work_dir
mkdir -p $PKG_NAME"_TEST/output"
ln -s $PKG_NAME"_TEST" cspp-geo-geocat_TEST
tar xzvf /mnt/WORK/packages/$PKG_NAME/$PKG_NAME".tar.gz" -C $PKG_NAME"_TEST"
tar xzvf /mnt/WORK/packages/$PKG_NAME/$PKG_NAME"_static.tar.gz" -C $PKG_NAME"_TEST"
tar xzvf /mnt/WORK/packages/$PKG_NAME/$PKG_NAME"_cache.tar.gz" -C $PKG_NAME"_TEST"
export CSPP_GEO_GEOCAT_HOME=/mnt/WORK/work_dir/$PKG_NAME"_TEST"/$PKG_NAME ; . $CSPP_GEO_GEOCAT_HOME/geocat_env.bash
```
* Run the example area files...
```bash
cd /mnt/WORK/work_dir/$PKG_NAME"_TEST"
geocat_l2.bash -W output --satellite goes /mnt/WORK/work_dir/input/goes13_2015_143_17* --no_rtm --tmp_dir test_l1_dir_goes --line_segments 2 --element_segments 2 --num_cpu 4
geocat_l2.bash -W output --satellite goes /mnt/WORK/work_dir/input/goes15_2015_143_21* --no_rtm --tmp_dir test_l1_dir_goes --line_segments 2 --element_segments 2 --num_cpu 4
geocat_l2.bash -W output --satellite him8 /mnt/WORK/work_dir/input/himawari08/2016_03_22_082_0220_* --no_rtm --tmp_dir test_l1_dir_him8 --line_segments 2 --element_segments 2 --num_cpu 4
```
* Create some quicklook images from the resulting Level1 NetCDF files...
Unnavigated...
```bash
mkdir -p png/unnavigated
navstr='_unnav --unnavigated'
```
```bash
l1_files=output/geocatL1.GOES-13.2015143.17*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done | sort | uniq)
for files in $l1_files; \
do \
for dataset in $l1_datasets; \
do \
ql_geocat_level1.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
done \
done
l1_files=output/geocatL1.GOES-15.2015143.21*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done | sort | uniq)
for files in $l1_files; \
do \
for dataset in $l1_datasets; \
do \
ql_geocat_level1.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
done \
done
```
Navigated...
```bash
mkdir -p png/navigated
navstr=' -m l -S 2'
```
```bash
l1_files=output/geocatL1.GOES-13.2015143.17*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done | sort | uniq)
for files in $l1_files; \
do \
for dataset in $l1_datasets; \
do \
ql_geocat_level1.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
done \
done
l1_files=output/geocatL1.GOES-15.2015143.21*.nc
l1_datasets=$(for files in $l1_files; do ql_geocat_level1.sh $files None --list_datasets; done | sort | uniq)
for files in $l1_files; \
do \
for dataset in $l1_datasets; \
do \
ql_geocat_level1.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
done \
done
```
* Create some quicklook images from the resulting Level2 NetCDF files...
```bash
l2_datasets="baseline_cmask_goes_nop_cloud_mask goesnp_ctype_cloud_phase goesnp_ctype_cloud_type ACHA_mode_7_goes_cloud_emissivity ACHA_mode_7_goes_cloud_optical_depth_vis ACHA_mode_7_goes_cloud_particle_effective_radius ACHA_mode_7_goes_cloud_top_height ACHA_mode_7_goes_cloud_top_pressure ACHA_mode_7_goes_cloud_top_temperature DCOMP_mode_3_cloud_albedo DCOMP_mode_3_cloud_spherical_albedo DCOMP_mode_3_cloud_ice_water_path DCOMP_mode_3_cloud_liquid_water_path DCOMP_mode_3_cloud_optical_depth_vis DCOMP_mode_3_cloud_particle_effective_radius goesr_fog_IFR_fog_probability goesr_fog_LIFR_fog_probability goesr_fog_MVFR_fog_probability"
```
Unnavigated...
```bash
mkdir -p png/unnavigated
navstr='_unnav --unnavigated'
```
```bash
l2_files=output/geocatL2.GOES-13.2015143.17*.nc
for files in $l2_files; \
do \
for dataset in $l2_datasets; \
do \
ql_geocat_level2.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
done \
done
l2_files=output/geocatL2.GOES-15.2015143.21*.nc
for files in $l2_files; \
do \
for dataset in $l2_datasets; \
do \
ql_geocat_level2.sh $files $dataset -O png/unnavigated/$(basename $files)$navstr ; \
done \
done
```
Navigated...
```bash
mkdir -p png/navigated
navstr=' -m l -S 2'
```
```bash
l2_files=output/geocatL2.GOES-13.2015143.17*.nc
for files in $l2_files; \
do \
for dataset in $l2_datasets; \
do \
ql_geocat_level2.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
done \
done
l2_files=output/geocatL2.GOES-15.2015143.21*.nc
for files in $l2_files; \
do \
for dataset in $l2_datasets; \
do \
ql_geocat_level2.sh $files $dataset -O png/navigated/$(basename $files)$navstr ; \
done \
done
```
- Command to get a list of links and their destination for a json file
for files in $(find common -maxdepth 1 -name "*" -print | grep -v \.pyc); \ do \ pkg_loc=$(ls -la $files | grep git | gawk '{print $(NF-2)}'); \ repo_loc=$(ls -la $files | gawk '{print $NF}') ; \ echo "["$'\n'" "\"$repo_loc\"","$'\n'" "\"$pkg_loc\"","$'\n'"],"; \ done | sed 's/\/home\/RH6B\/GEOCAT\/repos\///' tree -L 3 $PKG_NAME/ | grep -v \.pyc