RAIN Software Architecture

This repository (MetObsCommon) is the dumping ground for all functionality that may be shared between the software for the rooftop, SPARC, or any other instrument that is managed by the RAIN team. Below are some example executions of commands in this module that may be useful.

Note that some of the below commands may be calling code outside of this package like those in the AossTower or MendotaBuoy package. For more details and up to date examples and command line options see the documentation for those specific packages. In the future more examples and information will be provided in this gitlab project's Wiki.

Real world code can be found in the 'crontab' of the 'metobs' user on the rain01 and rain02 servers.

Backfill InfluxDB Database

Insert data from an old tower file:

python -m aosstower.level_00.influxdb -vvv --bulk 5000 /data1/raw/aoss/tower/2018/05/08/aoss_tower.2018-05-08.ascii

The above command sends data in blocks of 5000 records. This is to improve performance of sending data to the InfluxDB instead of sending one record at a time. A bulk value of 5000-10000 is preferred.

Compute the averages for 5 second tower and data:

python -m metobscommon.influxdb -vvv run_manual_cqs --symbol-list aosstower.level_00.influxdb.SYMBOLS --stations aoss.tower -s 2018-05-07T00:00:00 -e 2018-05-08T22:00:00 -d 1m 5m 1h

Note the above computes the 1m, 5m, and 1h averages. The time range (-s/-e) must be at whole intervals for the average intervals specified otherwise partial averages will be written. Any existing data points in the InfluxDB are overwritten by the new data points created during the execution of this script. Also note that long time ranges can take a while to process and may use a large amount of memory.

To insert a series of tower files:

find /data1/raw/mendota/buoy/2018/ -name "*.ascii" -print0 | sort -z | xargs -r0 -n1 python -m aosstower.level_00.influxdb -vvv --bulk 5000

The above command sorts the files by name which is important for the best performance.