EdosL0Util
Tools for mangling EDOS L0 PDS files. This package was primarily written to support work with EDOS L0 PDS files but can more generically be considered a CCSDS packet file manipulation toolkit.
Features:
- API for streaming packets
- CLI tools for:
- merging multiple PDS files
- splitting PDS into X minute files
- truncating files to specified time range
- inspecting PDS info
- Dumping JPSS H5 RDRs to L0 PDS files for VIIRS, CrIS, and ATMS
- Supports both Python 2 & 3
- Etc ...
There is also support for creating a packet stream from Aqua L0 PDS files, but it has not been tested much. Most functionality assumes EDOS L0 PDS files.
Packet streaming
To obtain a simple stream of packets, with support for secondary header
parsing use the PacketStream class as an iterator.
>>> from edosl0util.stream import PacketStream
>>> stream = PacketStream(open('P1570826VIIRSSCIENCEAS15286023813501.PDS', 'rb'))
>>> stream.next()
<Packet apid=826 seqid=13072 stamp=2015-10-13 00:22:04.042798>
>>> packets = list(stream)
>>> first, last, info = stream.info()
>>> print("First:{}\nLast:{}".format(first, last))
First:2015-10-13 00:22:04.042798
Last:2015-10-13 02:04:58.107582
>>> print info
defaultdict(<function edosl0util.stream.<lambda>>,
{800: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
801: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
802: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
803: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
804: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
805: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
806: {'count': 59005, 'last_seqid': 2684, 'num_missing': 0},
807: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
808: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
809: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
810: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
811: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
812: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
813: {'count': 114543, 'last_seqid': 5214, 'num_missing': 0},
814: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
815: {'count': 58994, 'last_seqid': 2673, 'num_missing': 0},
816: {'count': 58990, 'last_seqid': 2669, 'num_missing': 0},
817: {'count': 114510, 'last_seqid': 5181, 'num_missing': 0},
818: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
819: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
820: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
821: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
825: {'count': 83280, 'last_seqid': 3768, 'num_missing': 0},
826: {'count': 3471, 'last_seqid': 158, 'num_missing': 0}})To handle PacketTooShort and NonSequentialSeqId errors you have to
iterate manually:
>>> from edosl0util.stream import PacketStream
>>> stream = PacketStream(open('P1570826VIIRSSCIENCEAS15286023813501.PDS'))
>>> while True:
...: try:
...: packet = stream.next()
...: print(packet)
...: except PacketTooShort as err:
...: print("Corrupt packet stream: {:s}".format(packet))
...: break
...: except NonConsecutiveSeqId as err:
...: print("Missing packet: {:s}".format(err))
...: except StopIteration:
...: breakCommand Line Tools
There are command line tools for merging, splitting, truncating and inspecting L0 PDS files.
* edosl0merge
* edosl0split
* edosl0trunc
* edosl0info
* rdr2l0Creating L0 PDS Files from RDRs
The rdr2l0 tool will dump out an RDR in an order not suitable for use as a
NASA L0 PDS file. To generate a L0 PDS file actually requires 2 steps:
$> rdr2l0 --diary --skipfill RNSCA-RVIRS_npp_d20151018_t2359214_e0000467_b20595_c20151019015559980487_noaa_ops.h5
$> edosl0merge -o out.PDS RNSCA-RVIRS_npp_d20151018_t2359214_e0000467_b20595_c20151019015559980487_noaa_ops.h5.science.PDSThe resulting file out.PDS will have packets sorted by time and APID with
any duplicates removed.
The typical case would be to dump multiple RDR files to PDS then merge the resulting PDS files into a 6 minute granule like so:
$> rdr2l0 --diary --skipfill RNSCA-RVIRS_npp_d20151018_t000*.h5
$> edosl0merge -o out.PDS --trunc_to "2015-10-18 00:00:00,2015-10-18 00:06:00" *.science.PDS Which will create a proper 6 minute NASA L0 PDS file with any fill packets filtered out.
If you have quesions or comments related to this software contact brucef@ssec.wisc.edu.