Skip to content

Changes

Page history
Update Fusion Matlab README authored by Geoff Cureton's avatar Geoff Cureton
Hide whitespace changes
Inline Side-by-side
Fusion-Matlab-README.rdoc 0 → 100644
View page @ 22b2239c
Contents
========
* Introduction
* Building
- Build requirements
- Build recommendations
* Installation
- Runtime requirements
- Runtime recommendations
- Environment setup
* Running
- Required inputs
- Exit status codes
- Expected outputs
- Example
* Troubleshooting
* FAQ
* Maintainers
Introduction
============
* Software Description
This software package provides codes and ancillary data necessary to produce synthetic infrared (IR)
bands by fusing imager and sounder radiance and geolocation data. While a sensor like MODIS has
multiple IR absorption bands, the current VIIRS imager has only IR window bands. The software
provides synthetic MODIS-like bands from VIIRS and CrIS data. With this approach applications that
require IR absorption bands (e.g. retrieval of cloud properties) can be continued over generations
of weather satellite sensors. The data fusion method consists of two main steps. The first step is
performing a nearest neighbor search using a k-d tree algorithm on both high-spatial and low-spatial
(i.e., geographic avaraged) resolution imager split-window 11 and 12 micron radiances. In the second
step, the new so-called fusion radiances (at original high spatial resolution) are obtained by
computing the mean of convolved sounder radiances for the filed-of-views (FOVs) selected in step 1.
Details of the data fusion process can be found in Weisz, E., B. A. Baum, and W. P. Menzel, 2017:
Fusion of Satellite-Based Imager and Sounder Data to Construct Supplementary High Spatial Resolution
Narrowband IR Radiances. In press, J. Appl. Remote Sens.
* Package Description
This package has the following directory structure:
fusion_matlab
├── dist
│ ├── bin
│ │ ├── imagersounderfusion_M
│ │ ├── imagersounderfusion_V
│ │ ├── run_imagersounderfusion_M.sh
│ │ └── run_imagersounderfusion_V.sh
│ ├── env
│ │ ├── bin
│ │ ├── conda-meta
│ │ ├── examples
│ │ ├── include
│ │ ├── lib
│ │ ├── share
│ │ └── ssl
│ └── luts
│ ├── NG_VIIRS_NPP_RSR_filtered_Oct2011_BA
│ ├── L2.chan_prop.2005.03.01.v9.5.1.txt
│ ├── modis_aqua.srf.nc
│ └── modis_conv_error_2005.mat
├── source
│ ├── fusion_glue
│ ├── modis-airs-fusion
│ ├── polar2grid
│ └── viirs-cris-fusion
├── test
│ ├── c_sh
│ │ └── run_fusion_1gran.sh
│ ├── inputs
│ │ ├── AIRS.2015.04.17.145.L1B.AIRS_Rad.v5.0.23.0.G15173170714.hdf
│ │ ├── AIRS.2015.04.17.146.L1B.AIRS_Rad.v5.0.23.0.G15173170305.hdf
│ │ ├── colloc.airs_aqua.modis_aqua.20150417T142930_143500.nc
│ │ ├── colloc.airs_aqua.modis_aqua.20150417T143530_143500.nc
│ │ ├── colloc.cris_snpp.viirs_m_snpp.20180202T183600_183600.nc
│ │ ├── MYD021KM.A2015107.1435.006.2015108162652.hdf
│ │ ├── MYD03.A2015107.1435.006.2015108161923.hdf
│ │ ├── SNDR.SNPP.CRIS.20180202T1836.m06.g187.L1B.std.v2_0_15.T.180203004403.nc
│ │ ├── VNP02MOD.A2018033.1836.001.2018033235822.uwssec.nc
│ │ └── VNP03MOD.A2018033.1836.001.2018033235737.uwssec.nc
│ ├── outputs
│ │ ├── MYD02FSN.A2015107.1435.006.2018053182304.nc
│ │ └── VNP02FSN.A2018033.1836.001.2018054223818.nc
│ ├── py
│ │ ├── log_common.py
│ │ ├── run_fusion_1gran.py
│ │ └── utils.py
│ └── run.sh
├── ANCILLARY.txt
├── HISTORY.txt
├── README.txt
└── VERSION.txt
The source/ directory contains the git repos modis-airs-fusion and viirs-cris-fusion, which are
matlab source codes for the MODIS/AIRS and VIIRS/CRIS fusion processes respectively. The fusion_glue
and polar2grid git repos contain the source code for the Matlab-to-HDF4 and Matlab-to-NetCDF4 conversions.
The dist/bin directory contains the compiled Matlab executables (and supporting scripts), while dist/luts contain
the static LUTS required by the Matlab executables:
modis_aqua.srf.nc : Aqua MODIS Spectral Response Functions (SRFs)
modis_conv_error_2005.mat : MAT-file of convolution corrections to be used when convolved AIRS
data to MODIS bands.
This correction accounts for the gaps within the AIRS spectrum.
Provided by Dave Tobin (SSEC).
L2.chan_prop.2005.03.01.v9.5.1.txt : AIRS channel property list (lists channel frequencies, bad_flags etc)
NG_VIIRS_NPP_RSR_filtered_Oct2011_BA: This is a directory containing the VIIRS Spectral Response Functions.
Here the NPP_VIIRS_NG_M*_filtered_Oct2011f_BA.dat-files are used.
The dist/env contains a complete python environment required to run the python elements of the fusion_matlab package.
The test/ directory contains a set of sample AIRS/MODIS and CrIS VIIRS inputs, the associated outputs generated by
the fusion_matlab package, and the script run.sh, which runs the package on the sample inputs.
To invoke the matlab executables (e.g., within a bash script) use :
./run_imagersounderfusion_M.sh input1 input2 input3 inpu4 input5 input6
where
input1 : The directory containing the Matlab runtime (i.e.: /opt/matlab/2015b/)
input2 : The MODIS or VIIRS geolocation file
input3 : The MODIS or VIIRS level-1b file
input4 : The AIRS or CrIS sounder level-1b files
input5 : The AIRS/MODIS or CrIS/VIIRS collocation files (must be same number of these files as sounder level-1b files)
input6 : The directory containing the luts required for the Matlab executables.
Building and Installation
=========================
- Matlab Code
The required Matlab codes are:
- MODIS/AIRS Fusion (https://gitlab.ssec.wisc.edu/eweisz/modis-airs-fusion.git)
* Commit: 869bbcd39660a1c31549691eac298746fad53ea7
- VIIRS/CrIS Fusion (https://gitlab.ssec.wisc.edu/eweisz/viirs-cris-fusion.git)
* Commit: 8e2c7887016d31fe453b4c945618014557f4c44b
To be able to run the Matlab executables without a Matlab license, the
source codes need to be compiled using the "mcc" command to create the matlab binaries
(executables) 'imagersounderfusion_M' and 'imagersounderfusion_V' as well as scripts
'run_imagersounderfusion_M.sh' and 'run_imagersounderfusion_M.sh'. A Matlab license is
required to build the Matlab binaries.
- Python Environment and CF-compliance converter
To create the python environment contained in dist/env, we create the python environment "python-matlab"
(assuming that a version of the Anaconda python distro is installed) by:
conda create --copy -n fusion-matlab h5py
source activate fusion-matlab
sips=http://sips.ssec.wisc.edu/conda
conda install --copy h5py netcdf4 scipy pyproj
conda install --copy -c $sips python-hdf4
conda install --copy -c $sips viirsmend
Remaining in the "python-matlab" python environment, we now install two Polar2Grid modules, polar2grid_core and polar2grid_modis:
* Polar2Grid (https://github.com/davidh-ssec/polar2grid.git),
- Commit: 6b7816ae62db43e74fcf823bd51ada4af73509ec
- polar2grid/py/polar2grid_core/setup.py
- requires: numpy
- polar2grid/py/polar2grid_modis/setup.py
- requires: numpy, python-hdf4, polar2grid.core
We begin by checking out the correct revision of Polar2Grid:
cd fusion_matlab/source/polar2grid
git checkout 6b7816ae62db43e74fcf823bd51ada4af73509ec
We next install the polar2grid_core module:
pip install py/polar2grid_core
The polar2grid_modis module requires modification of its setup.py script to use the correct version of pyhdf, so we apply the
following diff:
> git diff py/polar2grid_modis/setup.py
diff --git a/py/polar2grid_modis/setup.py b/py/polar2grid_modis/setup.py
index 67dcf56..c3eef00 100644
--- a/py/polar2grid_modis/setup.py
+++ b/py/polar2grid_modis/setup.py
@@ -82,7 +82,7 @@ setup(
namespace_packages=["polar2grid"],
include_package_data=True,
zip_safe=True,
- install_requires=['numpy', 'pyhdf', 'polar2grid.core'],
+ install_requires=['numpy', 'python-hdf4', 'polar2grid.core'],
entry_points={
'console_scripts': [],
'polar2grid.frontend_class': ['modis=polar2grid.modis:Frontend'],
and install the polar2grid_modis module:
pip install py/polar2grid_modis
Lastly, we install the fusion_glue module:
* Fusion Glue (https://gitlab.ssec.wisc.edu/sounder-imager-fusion/fusion_glue.git)
- Commit: c36966d00bc86c39c3744ed8b16971aa3ce775e3
- requires: netCDF4, python-hdf4, numpy, scipy, polar2grid.core, polar2grid.modis
- provides: l1b-fusion-modis-airs, l1b-fusion-viirs-cris
cd fusion_matlab/source/fusion_glue
git checkout develop
pip install py/fusion_l1b
The required python environment should now be present in your conda environment location
($HOME/.conda/envs/fusion-matlab for example). We finish by copying the python environment into
our package:
cp -R $HOME/.conda/envs/fusion-matlab fusion_matlab/dist/env/
Requirements
------------
Requirements to build the software:
- Continuum Analytics Anaconda for python 2.7 (https://www.continuum.io/downloads#linux)
- Matlab (2015b), and associated license
Recommendations
---------------
None
Environment Setup
-----------------
Describe the setup of the environment required to run the package, including:
- Environment Variables:
LD_LIBRARY_PATH=fusion_matlab/dist/env/lib
PATH=fusion_matlab/dist/env/bin:/usr/bin:/bin
- Configuration files: None
Troubleshooting
===============
- Describe any common issues and their potential solutions.
- Where to look for relevant output or logs
- etc ...
FAQ
===
Frequently asked questions
Maintainers
===========
Elisabeth Weisz <elisabeth.weisz@ssec.wisc.edu>
- Matlab code
Ray Keoni Garcia <rayg@ssec.wisc.edu>
- CF-Compliance converter
Geoff Cureton <geoff.cureton@ssec.wisc.edu>
- fusion_matlab packaging