Skip to content
Snippets Groups Projects
Commit f3e280e1 authored by Max Drexler's avatar Max Drexler
Browse files

Update README.md

parent d4d83d83
No related branches found
No related tags found
No related merge requests found
......@@ -6,7 +6,7 @@ ETL pipeline for grib metadata.
The grib processor extracts [metadata](/docs/payload.md) from grib files and can optionally publish it to RabbitMQ servers.
The main use-case of this package is for the SSEC with the main purpose being to source grib files by watching a directory and publish all new metadata.
This package is made for use by the Satellite Data Service at the SSEC.
### Limitations
......@@ -17,96 +17,46 @@ Currently, this package can only process GRIB2 files, GRIB1 files will be ignore
### Requirements
* Python >= 3.8
* [NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c)
* [NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c) (`libg2c`)
* This is the C library used to read GRIB files.
To install, simply clone the git repo:
If `libg2c` is in a "common" installation path (`/usr/local`, `/sw`, `/opt`, `/opt/local`, `/opt/homebrew`, or `/usr`) then you can simply install the package from the GitLab package registry.
```bash
git clone https://gitlab.ssec.wisc.edu/mdrexler/grib_rmq_stream.git --depth 1
cd grib_rmq_stream
./grib_processor --help
python3 -m pip install grib-processor --upgrade --index-url https://gitlab.ssec.wisc.edu/api/v4/projects/2732/packages/pypi/simple
```
## Setup
If `libg2c` is on the system, but not in one of the previously listed paths, you can manually tell grib_processer where it is through the `G2C_DIR` environment variable. E.g. `libg2c` is in `/home/user/my_libs/lib/`, install grib_processer as so:
This will show you how to set up a virtual environment to run the processor with. It assumes that you've installed the grib processor and are in the `grib_rmq_stream` directory.
There are a couple of ways to get your environment working based on your environment manager. See [anaconda](#anaconda) if you want to use anaconda or [virtualenv](#virtualenv) if you just want to use virutalenv.
### Anaconda
Create the environment/install lib requirements.
```bash
conda create -n {env_name} python={py_version} nceplibs-g2c -c conda-forge
```
Replaec {env_name} with the name of your environment, and {py_version} with a version larger than 3.8.
Activate environment.
```bash
conda activate {env_name}
```
Install script requirements.
```bash
pip install -r requirements.txt
```
### Virtualenv
Check the system python version.
```bash
python3 --version
```
If the version is less than 3.8, you're kind of out of luck. I recommend using [anaconda](#anaconda) to get a more up-to-date version of python.
Create an environment (this might ask you to install a package on linux).
```bash
python3 -m venv .venv
```
Activate the environment.
```bash
source .venv/bin/activate
G2C_DIR=/home/user/my_libs python3 -m pip install grib-processor --upgrade --index-url https://gitlab.ssec.wisc.edu/api/v4/projects/2732/packages/pypi/simple
```
Install the requirements.
If the system doesn't have the `libg2c` binary at all, you can install it from [source](https://github.com/NOAA-EMC/NCEPLIBS-g2c), or, more easily, with conda.
```bash
python3 -m pip install -r requirements.txt
```
If you get an error during installation about not finding a library, set the environment variable `G2C_DIR` to be the path to the folder containing the 'lib' folder for [NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c). e.g.
If `libg2c.so` is under `/home/user/.conda/envs/lib-env/lib/`, you could install using:
```bash
G2C_DIR=/home/user/.conda/envs/lib-env/ python3 -m pip install -r requirements.txt
conda create -n my_env -y python=3.12 nceplibs-g2c -c conda-forge
conda run -n my_env python3 -m pip install grib-processor --upgrade --index-url https://gitlab.ssec.wisc.edu/api/v4/projects/2732/packages/pypi/simple
conda activate my_env
```
## Usage
Once you've [setup](#setup) an environment, you can start the event processor simply by calling `./grib_processor {dir}`. Where {dir} is the directory to watch for grib files. See [configuration](#configuration) for more information about how to change the behavior of the processor.
After installation, the command `grib_processer` will be added to your path. use `grib_processer --help` for more information about how to use the command.
### Configuration
The package also comes with a python interface to parse GRIB metadata.
There are two ways to configure the grib_processer script: through a .env file and through the CLI. The .env file contains secret information that you might not want in the command name, and the CLI has all other configuration.
```python3
import grib_processer.grib as grib
#### .env File
# If you would like to include XCD data.
grib.load_xcd_models()
The .env file can have two keys: `RMQ_USER` and `RMQ_PASS` which specify the RabbitMQ username and password respectively.
grib_path = '/path/to/my/grib_file.grib'
#### CLI Options
Use `./grib_processor --help` for a full list of command line arguments and what they do.
for data in grib.itergrib(grib_path):
print(data)
```
## Author
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment