|
|
|
Please see the [Building LightningCast wiki](https://gitlab.ssec.wisc.edu/jcintineo/lightningcast/-/wikis/Building-LightningCast) before running it in near-real time.
|
|
|
|
|
|
|
|
### Using `w2alg`
|
|
|
|
|
|
|
|
You can now run the program, `src/predict_ltg.py`. Do `python predict_ltg.py -h` to see the help options. Typically, realtime processing is started and stopped using the `w2alg` utility, which should be in your `$PATH`. You can modify `src/realtime/w2algrun/w2alg.conf` to make changes to processes started and stopped by `w2alg`.
|
|
|
|
|
|
|
|
In `w2alg.conf`, you will need to manually change the `MACHINE`, the "macros" (e.g., `M1`, `M2`), and the `LOG_DIR`. If you are using the default GOES-East and GOES-West models, `M3`\-`M6` don't need to be changed. You can also make changes to the commands themselves in `w2alg.conf`.
|
|
|
|
|
|
|
|
```
|
|
|
|
group GLOBAL
|
|
|
|
|
|
|
|
# Macros defined from M1 to M9.
|
|
|
|
# How to use a macro. Set name and then use
|
|
|
|
# Macros override like any other set. So you could have a different M1 per
|
|
|
|
# each group or even manager if you want
|
|
|
|
set MACHINE "skynet.ssec.wisc.edu"
|
|
|
|
|
|
|
|
set M1 "/home/jcintineo/lightningcast/src/"
|
|
|
|
set M2 "/data/PLTG/"
|
|
|
|
set M3 "static/"
|
|
|
|
set M4 "fit_conv_model.h5"
|
|
|
|
#for GOES-West
|
|
|
|
set M5 "static/GOES-West/"
|
|
|
|
set M6 "fit_conv_model.h5"
|
|
|
|
|
|
|
|
# Default W2Logger parameters per algorithm (can be overridden below)
|
|
|
|
set LOG_SIZE "10000000" # Size of log file in -s
|
|
|
|
set NUM_LOGS "2" # Max number of logs
|
|
|
|
set LOG_DIR "/data/PLTG/logs/"
|
|
|
|
```
|
|
|
|
|
|
|
|
`w2alg` has several arguments: "function", "group", and "process_name". Options for "function":
|
|
|
|
|
|
|
|
* `status` #tells you what command(s) is running
|
|
|
|
* `start` #starts the group and process
|
|
|
|
* `stop` #stops the group and process
|
|
|
|
|
|
|
|
`w2alg.conf` has several "groups": `MAIN`, `GE` (GOES-East), and `GW` (GOES-West). The `MAIN` group has a process for cleaning up data.
|
|
|
|
|
|
|
|
```
|
|
|
|
group MAIN
|
|
|
|
|
|
|
|
manager PROCESS-CLEAN
|
|
|
|
set PROCESS_EXE "python"
|
|
|
|
set PROCESS_PARAMS "$M1/cleanup.py -c 30"
|
|
|
|
```
|
|
|
|
|
|
|
|
Thus, `~]$ w2alg start main clean` will begin `cleanup.py`, and `~]$ w2alg stop main clean` will stop it.
|
|
|
|
|
|
|
|
### Using `amqpfind`
|
|
|
|
|
|
|
|
`amqpfind` uses [RabbitMQ](https://www.rabbitmq.com/) messaging to receive notifications. This is the best way to receive file notifications (i.e., a new GOES-R file was written) at SSEC. Outside of SSEC, some other notification mechanism may be necessary. The files in `src/amqpfind/` were obtained from Ray Garcia's `flaregun` repository.
|
|
|
|
|
|
|
|
- To start for GOES-East: `~]$ w2alg start ge amq`
|
|
|
|
- To stop for GOES-East: `~]$ w2alg stop ge amq`
|
|
|
|
- Only one instance of `amqpfind` is needed per satellite, assuming you receive notifications for each sector (RadC, RadF, RadM1, RadM2), which is the default.
|
|
|
|
|
|
|
|
### A LightningCast prediction stream
|
|
|
|
|
|
|
|
Use `w2alg` to start and stop a LightningCast prediction stream for a sector. We'll use GOES-East CONUS sector as an example.
|
|
|
|
- To start: `~]$ w2alg start ge conus`
|
|
|
|
- To stop: `~]$ w2alg stop ge conus`
|
|
|
|
|
|
|
|
I'll explain the arguments found here, in `w2alg.conf`:
|
|
|
|
|
|
|
|
```
|
|
|
|
manager PROCESS-CONUS
|
|
|
|
set PROCESS_EXE "python"
|
|
|
|
set PROCESS_PARAMS "$M1/predict_ltg.py $M2/logs/GE-AMQ-A1.log $M1/$M3/$M4 -o $M2/products/ -sec RadC -r -ru $M1/realtime/re_upload -gt -gr -ph 9000"
|
|
|
|
set NUM_LOGS "2"
|
|
|
|
```
|
|
|
|
|
|
|
|
* The `PROCESS_EXE` (process executable) is just `python`. You should make sure your `TF` environment is activated.
|
|
|
|
* `$M1/predict_ltg.py`: the program it is excecuting.
|
|
|
|
* `$M2/logs/GE-AMQ-A1.log`: the log file that gets generated when you do `w2alg start ge amq`. This file gets modified when new GOES data are finished writing to a disk. So this is the file that `predict_ltg.py` "listens" to.
|
|
|
|
* `$M1/$M3/$M4`: the CNN model file. Expanded out, it might look like this: `/home/jcintineo/lightningcast/src/static/fit_conv_model.h5`. Note that GOES-17 model is in `static/GOES-West/`. GOES-18 can use `static/fit_conv_model.h5`.
|
|
|
|
* `-o $M2/products` The output directory. In realtime, we can write netCDF and/or GeoJSON files.
|
|
|
|
* `-sec RadC`: The sector. Default is `RadC`.
|
|
|
|
* `-r`: a flag to run in realtime mode.
|
|
|
|
* `-ru $M1/realtime/re_upload`: RealEarth upload script, to upload GeoJSONs to RealEarth (UW-CIMSS only).
|
|
|
|
* `-mj`: a flag to create a GeoJSON.
|
|
|
|
* `-gt`: a flag to compute geotiffs to be uploaded to RealEarth. This enables the meteogram of probabilities in RealEarth (UW-CIMSS only).
|
|
|
|
* `-gr`: a flag to compute GRLevel placefile. It will also be uploaded to `cimss.ssec.wisc.edu/severe_conv` (UW-CIMSS only).
|
|
|
|
* `-ph 9000`: the static height (in meters) to do the parallax correction for the GRLevel placefile.
|
|
|
|
* `-x $M1/realtime/ldm_send.sh`: The extra script to run. The one and only input to this script is the created netCDF.
|
|
|
|
* `--savegrid 3`: Save a netCDF. 3 = save both plax-corrected and uncorrected grids.
|
|
|
|
|
|
|
|
The GOES-West and other GOES-East sectors are very analogous. |
|
|
|
\ No newline at end of file |
