process_isccp_l1g

This project contains tools for processing ISCCP-NG L1g into L2 data products.

Features

• produce a layered L2 netcdf file without saving intermediate files
• improve clavrx IO performance by decompressing the input netcdf files in parallel and placing the temporary files into /dev/shm
• determine which satellites we need to process from the (potentially subsetted) data and execute processing for all satellites in parallel
• run segments in parallel
• automatically download data from https://tyr.ssec.wisc.edu/isccp/l1g/ (see download_process_l1g.py)
• process subsets via lat-lon bounding box and strided sampling

Installation

This project requires Python 3.8 or greater.

It's pretty likely that these dependencies will already installed. The dependencies are listed in requirements.txt.

git clone https://gitlab.ssec.wisc.edu/cphillips/process_isccp_l1g.git
cd process_isccp_l1g

Create a conda environment

conda create -n isccp_l2 --file requirements.txt
conda activate isccp_l2

### Install dependencies with pip
pip install -r requirements.txt

Setup

Required

• clavrxorb - clavrx executable (source here)
• clavrx_options - template options file, last line gets replaced by script (specifies WMO id). I've included an example (clavrx_options_example)
• level2_list - list of L2 products to generate, sometimes different than clavrx names (see section on L2 products) (see level2_list_example)

L2 Products

Because of how tracer works, the available set of L2 products is different than normal clavrx. You can see all of the names in variable_database.json. Most noticably, the names of radiometric variables is based on the exact "clavrx" channel. For example temp_11_0um_nom -> temp_ch31.

download_process_l1g.py

download_process_l1g.py is a script that downloads ISCCP-NG L1g data from https://tyr.ssec.wisc.edu/isccp/l1g/ and processes it into L2 data products. It is basically a wrapper around process_l1g.py that handles downloading the data.

You can either specify the URL to the data you want to download or you can specify a version and date and have the script automatically determine the URL.

Examples

# print usage
python download_process_l1g.py --help

# download and process data from a specific URL
python download_process_l1g.py --url http://tyr.ssec.wisc.edu/isccp/l1g/demo_202307/2021/10/01/0000/ output_file.nc

# download and process data from a specific version and time
python download_process_l1g.py --version demo_202307 --dt 2021-10-01T00:00 output_file.nc

process_l1g.py

Examples

# print usage
python process_l1g.py --help

# process full L1g timestep
python process_l1g.py l1g_dir output_file.nc

# process a subset of the L1g timestep
python process_l1g.py --bbox 0 0 5 5 l1g_dir output_file.nc 

# process with parallel segments (at most have 4 segments per satellite running at once)
python process_l1g.py --pseg 4 l1g_dir output_file.nc