AWS summary un-concatenator
Some files of ours are concatenated from other files. This tool can reverse that.
Context
For example, the file
amrc:/data2/ftp/pub/aws/summary/monthly/2003/feb03e.sum
looks like
this:
STATION : 8981 MOUNT SIPLE LAT : 73.198S LONG : 127.052W ELEVATION : 230 M
FEB 03
HR PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD
DAY 1 DAY 2 DAY 3 DAY 4 DAY 5 DAY 6 DAY 7
00 -3 -3 -3 958 -3 -1 955 -3 953 -2
03 -3 955 -3 -3 956 -2 -2 954 -3 952 -3
06 957 -4 956 -3 957 -3 956 -3 961 -2 952 -3 953 -4
09 956 -4 955 -4 957 -4 957 -2 961 -3 952 -4 955 -4
12 956 -3 956 -4 957 -3 958 -2 960 -3 951 -4 955 -5
15 955 -2 957 -4 956 -2 958 -2 959 -2 950 -4 955 -5
18 -2 957 -3 958 -3 960 -2 958 -2 950 -3 955 -4
21 -2 -2 958 -2 -1 958 -2 952 -2 955 -3
DAY 8 DAY 9 DAY 10 DAY 11 DAY 12 DAY 13 DAY 14
00 955 -3 963 -3 965 -4 966 -4 -3 -4 -1
.
.
.
------------------------------------------------- MONTHLY SUMMARY ---------------------------------------------------
TEMPERATURE (C) * MEAN = -4.1 STD DEV = 1.7 MAX = 1.9 AT 1910 UT ON DAY 17 MIN = -10.4 AT 1150 UT ON DAY 24
THETA (K) * MEAN = 272.17
PRESSURE (MB) * MEAN = 958.8 STD DEV = 5.4 MAX = 968.2 AT 1740 UT ON DAY 16 MIN = 946.0 AT 1900 UT ON DAY 20
WINDS (M/S) MEAN = 0.0 STD DEV = 0.0 RESULTANT = 0.0 FROM 0 CONSTANCY =0.00 MAX =***** FROM **** AT 0 UT DAY 0
MISSING 3-HOURLY OBSERVATIONS * TEMPERATURE: 3.1 % PRESSURE: 16.1 % WINDS: 100.0 %
WIND SPEED (M/S): 0-2 2-4 4-6 6-8 8-10 10-12 12-14 14-16 16-18 18-20 20-25 25-30 30-35 35-40 40-45 45-50 50-55 55+
PERCENT: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
WIND DIRECTION: N NNE NE ENE E ESE SE SSE S SSW SW WSW W WNW NW NNW
PERCENT 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
STATION : 8984 POSSESSION ISLAND LAT : 71.891S LONG : 171.210E ELEVATION : 30 M
FEB 03
HR PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD PP TT VV DD
DAY 1 DAY 2 DAY 3 DAY 4 DAY 5 DAY 6 DAY 7
00 963 -2 967 -2 973 -2 978 -2 977 0 977 -1 980 -1
.
.
.
------------------------------------------------- MONTHLY SUMMARY ---------------------------------------------------
TEMPERATURE (C) * MEAN = -4.1 STD DEV = 2.5 MAX = 3.9 AT 20 UT ON DAY 9 MIN = -10.9 AT 920 UT ON DAY 21
.
.
.
PERCENT 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
STATION : 8985 HENRY LAT : 89.011S LONG : 1.025W ELEVATION : 2755 M
.
.
.
STATION : 8986 D-47 LAT : 67.397S LONG : 138.726E ELEVATION : 1550 M
.
.
.
STATION : 8988 CAPE DENISON LAT : 67.009S LONG : 142.664E ELEVATION : 31 M
.
.
.
STATION : 8989 DOME C II LAT : 75.121S LONG : 123.374E ELEVATION : 3250 M
.
.
.
The section from "STATION : 8981 MOUNT SIPLE" up to "STATION : 8984 POSSESSION ISLAND" was originally its own file. The same goes for the following sections POSSESSION ISLAND, HENRY, D-47, CAPE DENISON, and DOME C II.
Usage
To put the sections back in separate files, log into
amrc.ssec.wisc.edu
and run:
~ograham/uncat-sum/uncat-sum -v /data2/ftp/pub/aws/summary/monthly/2003/feb03e.sum
This will create the following files:
-
089810203.sum
from 8981 MOUNT SIPLE -
089840203.sum
from 8984 POSSESSION ISLAND -
089850203.sum
from 8985 HENRY -
089860203.sum
from 8986 D-47 -
089880203.sum
from 8988 CAPE DENISON -
089890203.sum
from 8989 DOME C II
The uncatted files will be created in the current directory.
To create them in some other directory DIR, add the option --dir DIR
to the command above.
The filenames are generated from the station ID, month, and 2-digit year from the section. This is hard-coded and can be changed by editing the Python script.
Advanced example usage
To uncat all the summaries in /data2/ftp/pub/aws/summary/monthly/
into the dir summaries
, you would run:
for i in /data2/ftp/pub/aws/summary/monthly/*/*[0-9][0-9][ew].sum; do
~ograham/uncat-sum/uncat-sum "$i" --dir summaries
echo "uncatted $(basename "$i")"
done