GRIB Processor

ETL pipeline for grib metadata.

Description

The grib processor extracts metadata from grib files and can optionally publish it to RabbitMQ servers.

The main usecase of this package is for the SSEC with the main purpose being to source grib files by watching a directory and publish all new metadata.

Limitations

Currently, this package can only process GRIB2 files, GRIB1 files will be ignored.

Installation

Requirements

• Python >= 3.8
• NCEPLIBS-g2c
  • This is the C library used to read GRIB files.

To install, simply clone the git repo:

git clone https://gitlab.ssec.wisc.edu/mdrexler/grib_rmq_stream.git --depth 1
cd grib_rmq_stream
./grib_processor --help

Setup

This will show you how to set up a virtual environment to run the processor with. It assumes that you've installed the grib processor and are in the grib_rmq_stream directory.

There are a couple of ways to get your environment working based on your environment manager. See anaconda if you want to use anaconda or virtualenv if you just want to use virutalenv.

Anaconda

Create the environment/install lib requirements.

conda create -n {env_name} python={py_version} nceplibs-g2c -c conda-forge

Replaec {env_name} with the name of your environment, and {py_version} with a version larger than 3.8.

Activate environment.

conda activate {env_name}

Install script requirements.

pip install -r requirements.txt

Virtualenv

Check the system python version.

python3 --version

If the version is less than 3.8, you're kind of out of luck. I recommend using anaconda to get a more up-to-date version of python.

Create an environment (this might ask you to install a package on linux).

python3 -m venv .venv

Activate the environment.

source .venv/bin/activate

Install the requirements.

python3 -m pip install -r requirements.txt

If you get an error during installation about not finding a library, set the environment variable G2C_DIR to be the path to the folder containing the 'lib' folder for NCEPLIBS-g2c. e.g.

If libg2c.so is under /home/user/.conda/envs/lib-env/lib/, you could install using:

G2C_DIR=/home/user/.conda/envs/lib-env/ python3 -m pip install -r requirements.txt

Usage

Once you've setup an environment, you can start the event processor simply by calling ./grib_processor {dir}. Where {dir} is the directory to watch for grib files. See configuration for more information about how to change the behavior of the processor.

Configuration

There are two ways to configure the grib_processer script: through a .env file and through the CLI. The .env file contains secret information that you might not want in the command name, and the CLI has all other configuration.

.env File

The .env file can have two keys: RMQ_USER and RMQ_PASS which specify the RabbitMQ username and password respectively.

CLI Options

Use ./grib_processor --help for a full list of command line arguments and what they do.

Author

Created by Max Drexler.

License

This software can be freely distributed and/or modified without consent of the original author. For more information, see the LICENSE file.