Skip to content
Snippets Groups Projects
Name Last commit Last update
README.md
uncat-sum

AWS summary un-concatenator

Some files of ours are concatenated from other files. This tool can reverse that.

Context

For example, the file amrc:/data2/ftp/pub/aws/summary/monthly/2003/feb03e.sum looks like this:

                    STATION : 8981  MOUNT SIPLE       LAT : 73.198S   LONG : 127.052W   ELEVATION :  230 M
 FEB  03
 HR   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD
         DAY  1          DAY  2          DAY  3          DAY  4          DAY  5          DAY  6          DAY  7
 00       -3              -3              -3         958  -3              -1         955  -3         953  -2
 03       -3         955  -3              -3         956  -2              -2         954  -3         952  -3
 06  957  -4         956  -3         957  -3         956  -3         961  -2         952  -3         953  -4
 09  956  -4         955  -4         957  -4         957  -2         961  -3         952  -4         955  -4
 12  956  -3         956  -4         957  -3         958  -2         960  -3         951  -4         955  -5
 15  955  -2         957  -4         956  -2         958  -2         959  -2         950  -4         955  -5
 18       -2         957  -3         958  -3         960  -2         958  -2         950  -3         955  -4
 21       -2              -2         958  -2              -1         958  -2         952  -2         955  -3
         DAY  8          DAY  9          DAY 10          DAY 11          DAY 12          DAY 13          DAY 14
 00  955  -3         963  -3         965  -4         966  -4              -3              -4              -1
.
.
.
-------------------------------------------------  MONTHLY SUMMARY  ---------------------------------------------------
TEMPERATURE (C) * MEAN =  -4.1  STD DEV =   1.7  MAX =   1.9 AT 1910 UT ON DAY 17  MIN = -10.4 AT 1150 UT ON DAY 24
THETA (K) *  MEAN =  272.17
PRESSURE (MB) *   MEAN = 958.8  STD DEV =   5.4  MAX = 968.2 AT 1740 UT ON DAY 16  MIN = 946.0 AT 1900 UT ON DAY 20
WINDS (M/S) MEAN =  0.0 STD DEV = 0.0 RESULTANT =  0.0 FROM   0 CONSTANCY =0.00 MAX =***** FROM  **** AT    0 UT DAY  0
MISSING 3-HOURLY OBSERVATIONS *   TEMPERATURE:   3.1 %   PRESSURE:  16.1 %   WINDS: 100.0 %
WIND SPEED (M/S): 0-2 2-4 4-6 6-8 8-10 10-12 12-14 14-16 16-18 18-20 20-25 25-30 30-35 35-40 40-45 45-50 50-55 55+
PERCENT:            0   0   0   0    0     0     0     0     0     0     0     0     0     0     0     0     0   0
WIND DIRECTION:    N   NNE   NE   ENE   E   ESE   SE   SSE   S   SSW   SW   WSW   W   WNW   NW   NNW
PERCENT            0     0    0     0   0     0    0     0   0     0    0     0   0     0    0     0


                    STATION : 8984  POSSESSION ISLAND LAT : 71.891S   LONG : 171.210E   ELEVATION :   30 M
 FEB  03
 HR   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD   PP  TT VV  DD
         DAY  1          DAY  2          DAY  3          DAY  4          DAY  5          DAY  6          DAY  7
 00  963  -2         967  -2         973  -2         978  -2         977   0         977  -1         980  -1
.
.
.
-------------------------------------------------  MONTHLY SUMMARY  ---------------------------------------------------
TEMPERATURE (C) * MEAN =  -4.1  STD DEV =   2.5  MAX =   3.9 AT   20 UT ON DAY  9  MIN = -10.9 AT  920 UT ON DAY 21
.
.
.
PERCENT            0     0    0     0   0     0    0     0   0     0    0     0   0     0    0     0


                    STATION : 8985  HENRY             LAT : 89.011S   LONG :   1.025W   ELEVATION : 2755 M
.
.
.
                    STATION : 8986  D-47              LAT : 67.397S   LONG : 138.726E   ELEVATION : 1550 M
.
.
.
                    STATION : 8988  CAPE DENISON      LAT : 67.009S   LONG : 142.664E   ELEVATION :   31 M
.
.
.
                    STATION : 8989  DOME C II         LAT : 75.121S   LONG : 123.374E   ELEVATION : 3250 M
.
.
.

The section from "STATION : 8981 MOUNT SIPLE" up to "STATION : 8984 POSSESSION ISLAND" was originally its own file. The same goes for the following sections POSSESSION ISLAND, HENRY, D-47, CAPE DENISON, and DOME C II.

Usage

To put the sections back in separate files, log into amrc.ssec.wisc.edu and run:

~ograham/uncat-sum/uncat-sum -v /data2/ftp/pub/aws/summary/monthly/2003/feb03e.sum

This will create the following files:

  • 089810203.sum from 8981 MOUNT SIPLE
  • 089840203.sum from 8984 POSSESSION ISLAND
  • 089850203.sum from 8985 HENRY
  • 089860203.sum from 8986 D-47
  • 089880203.sum from 8988 CAPE DENISON
  • 089890203.sum from 8989 DOME C II

The uncatted files will be created in the current directory. To create them in some other directory DIR, add the option --dir DIR to the command above.

The filenames are generated from the station ID, month, and 2-digit year from the section. This is hard-coded and can be changed by editing the Python script.

Advanced example usage

To uncat all the summaries in /data2/ftp/pub/aws/summary/monthly/ into the dir summaries, you would run:

for i in /data2/ftp/pub/aws/summary/monthly/*/*[0-9][0-9][ew].sum; do
    ~ograham/uncat-sum/uncat-sum "$i" --dir summaries
    echo "uncatted $(basename "$i")"
done