Skip to content
Snippets Groups Projects
Verified Commit 11f1b39a authored by David Hoese's avatar David Hoese
Browse files

Update directories to match rain03

parent 39773aa0
Branches
No related tags found
No related merge requests found
......@@ -140,7 +140,7 @@ Backfill InfluxDB Database
Insert data from an old tower file:
python -m aosstower.level_00.influxdb --influxdb-token <READ_WRITE_TOKEN> -vvv --bulk 5000 /data1/raw/aoss/tower/2018/05/08/aoss_tower.2018-05-08.ascii
python -m aosstower.level_00.influxdb --influxdb-token <READ_WRITE_TOKEN> -vvv --bulk 5000 /data/raw/aoss/tower/2018/05/08/aoss_tower.2018-05-08.ascii
The above command sends data in blocks of 5000 records. This is to improve
performance of sending data to the InfluxDB instead of sending one record
......@@ -159,7 +159,7 @@ use a large amount of memory.
To insert a series of tower files:
find /data1/raw/mendota/buoy/2018/ -name "*.ascii" -print0 | sort -z | xargs -r0 -n1 python -m aosstower.level_00.influxdb -vvv --bulk 5000 --influxdb-token <READ_WRITE_TOKEN>
find /data/raw/mendota/buoy/2018/ -name "*.ascii" -print0 | sort -z | xargs -r0 -n1 python -m aosstower.level_00.influxdb -vvv --bulk 5000 --influxdb-token <READ_WRITE_TOKEN>
The above command sorts the files by name which is important for the best
performance.
......@@ -176,9 +176,9 @@ A bash one-liner for processing one year and running averages 1 month at a time
.. code-block:: bash
# AOSS Tower
for YEAR in `seq 2023 -1 2006`; do echo ${YEAR}; time find /data1/raw/aoss/tower/${YEAR} -name "*.ascii" -print0 | sort -z | xargs -r0 -n1 /opt/metobs/aoss_tower/bin/python -m aosstower.level_00.influxdb -v --sleep-interval 5.0 --bulk 10000 --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt); for month in `seq -f %02.0f 1 12`; do /opt/metobs/aoss_tower/bin/python -m metobscommon.influxdb -vvv --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt) run_manual_average --stations aoss.tower -s ${YEAR}-${month}-01T00:00:00 -e $(date -d "${YEAR}-${month}-01 + 1 month" +%Y-%m-%dT%H:%M:%S); sleep 5.0; done; echo "Done with $YEAR"; sleep 60; done
for YEAR in `seq 2023 -1 2006`; do echo ${YEAR}; time find /data/raw/aoss/tower/${YEAR} -name "*.ascii" -print0 | sort -z | xargs -r0 -n1 /opt/metobs/aoss_tower/bin/python -m aosstower.level_00.influxdb -v --sleep-interval 5.0 --bulk 10000 --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt); for month in `seq -f %02.0f 1 12`; do /opt/metobs/aoss_tower/bin/python -m metobscommon.influxdb -vvv --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt) run_manual_average --stations aoss.tower -s ${YEAR}-${month}-01T00:00:00 -e $(date -d "${YEAR}-${month}-01 + 1 month" +%Y-%m-%dT%H:%M:%S); sleep 5.0; done; echo "Done with $YEAR"; sleep 60; done
# Mendota Buoy
for YEAR in `seq 2023 -1 2013`; do echo ${YEAR}; time find /data1/raw/mendota/buoy/${YEAR} \( -name "*limnodata.*ascii" -o -name "*metdata.*.ascii" \) -print0 | sort -z | xargs -r0 -n1 /opt/metobs/mendota_buoy/bin/python -m mendotabuoy.level_00.influxdb -v --sleep-interval 5.0 --bulk 10000 --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt); for month in `seq -f %02.0f 2 11`; do /opt/metobs/mendota_buoy/bin/python -m metobscommon.influxdb -vvv --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt) run_manual_average --stations mendota.buoy -s ${YEAR}-${month}-01T00:00:00 -e $(date -d "${YEAR}-${month}-01 + 1 month" +%Y-%m-%dT%H:%M:%S); sleep 5.0; done; echo "Done with $YEAR"; sleep 60; done
for YEAR in `seq 2023 -1 2013`; do echo ${YEAR}; time find /data/raw/mendota/buoy/${YEAR} \( -name "*limnodata.*ascii" -o -name "*metdata.*.ascii" \) -print0 | sort -z | xargs -r0 -n1 /opt/metobs/mendota_buoy/bin/python -m mendotabuoy.level_00.influxdb -v --sleep-interval 5.0 --bulk 10000 --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt); for month in `seq -f %02.0f 2 11`; do /opt/metobs/mendota_buoy/bin/python -m metobscommon.influxdb -vvv --influxdb-token $(</home/metobs/influxdbv2_metobs_rw.txt) run_manual_average --stations mendota.buoy -s ${YEAR}-${month}-01T00:00:00 -e $(date -d "${YEAR}-${month}-01 + 1 month" +%Y-%m-%dT%H:%M:%S); sleep 5.0; done; echo "Done with $YEAR"; sleep 60; done
......@@ -32,9 +32,9 @@ import logging
from configparser import ConfigParser
INCOMING_ROOT = os.environ.get("METOBS_INCOMING", '/data1/incoming')
RAW_ROOT = os.environ.get("METOBS_RAW", '/data1/raw')
CACHE_ROOT = os.environ.get("METOBS_CACHE", '/data1/cache')
INCOMING_ROOT = os.environ.get("METOBS_INCOMING", '/data/incoming')
RAW_ROOT = os.environ.get("METOBS_RAW", '/data/raw')
CACHE_ROOT = os.environ.get("METOBS_CACHE", '/data/cache')
DATA_LEVELS = [
"00", "01",
......
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment