Skip to content
Snippets Groups Projects
Select Git revision
  • jpss3-rdr
  • master default protected
  • 4-support-omps-for-rdr-meging-2
  • 4-support-omps-for-rdr-meging
  • scanbytes
  • develop protected
  • 0.17.2
  • 0.17.1
  • 0.17.0
  • 0.16.0
  • 0.15.7
  • 0.15.6
  • 0.15.5
  • 0.15.4
  • 0.15.3
  • 0.15.2
  • 0.15.1
  • 0.15.0
  • 0.14.4
  • 0.14.3
  • 0.14.2
  • 0.14.1
  • 0.14.0
  • 0.13.5
  • 0.13.4
  • 0.13.3
26 results
Compare
Name Last commit Last update
.idea
edosl0util
scripts
tests
.dockerignore
.gitattributes
.gitignore
.gitlab-ci.yml
MANIFEST.in
README.md
setup.py
README.md

EdosL0Util

Tools for mangling EDOS L0 PDS files. This package was primarily written to support work with EDOS L0 PDS files but can more generically be considered a CCSDS packet file manipulation toolkit.

Features:

  • API for streaming packets
  • CLI tools for:
    • merging multiple PDS files
    • splitting PDS into X minute files
    • truncating files to specified time range
    • inspecting PDS info
  • Dumping JPSS H5 RDRs to L0 PDS files for VIIRS, CrIS, and ATMS
  • Supports both Python 2 & 3
  • Etc ...

There is also support for creating a packet stream from Aqua L0 PDS files, but it has not been tested much. Most functionality assumes EDOS L0 PDS files.

Packet streaming

To obtain a simple stream of packets, with support for secondary header parsing use the PacketStream class as an iterator.

>>> from edosl0util.stream import PacketStream
>>> stream = PacketStream(open('P1570826VIIRSSCIENCEAS15286023813501.PDS', 'rb'))
>>> stream.next()
    <Packet apid=826 seqid=13072 stamp=2015-10-13 00:22:04.042798>
>>> packets = list(stream)
>>> first, last, info = stream.info()
>>> print("First:{}\nLast:{}".format(first, last))
    First:2015-10-13 00:22:04.042798
    Last:2015-10-13 02:04:58.107582
>>> print info
    defaultdict(<function edosl0util.stream.<lambda>>,
            {800: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             801: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             802: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             803: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             804: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             805: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             806: {'count': 59005, 'last_seqid': 2684, 'num_missing': 0},
             807: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             808: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
             809: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
             810: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
             811: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
             812: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
             813: {'count': 114543, 'last_seqid': 5214, 'num_missing': 0},
             814: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
             815: {'count': 58994, 'last_seqid': 2673, 'num_missing': 0},
             816: {'count': 58990, 'last_seqid': 2669, 'num_missing': 0},
             817: {'count': 114510, 'last_seqid': 5181, 'num_missing': 0},
             818: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
             819: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
             820: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
             821: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
             825: {'count': 83280, 'last_seqid': 3768, 'num_missing': 0},
             826: {'count': 3471, 'last_seqid': 158, 'num_missing': 0}})

To handle PacketTooShort and NonSequentialSeqId errors you have to iterate manually:

>>> from edosl0util.stream import PacketStream
>>> stream = PacketStream(open('P1570826VIIRSSCIENCEAS15286023813501.PDS'))
>>> while True:
   ...:     try:
   ...:         packet = stream.next()
   ...:         print(packet)
   ...:     except PacketTooShort as err:
   ...:         print("Corrupt packet stream: {:s}".format(packet))
   ...:         break
   ...:     except NonConsecutiveSeqId as err:
   ...:         print("Missing packet: {:s}".format(err))
   ...:     except StopIteration:
   ...:         break

Command Line Tools

There are command line tools for merging, splitting, truncating and inspecting L0 PDS files.

* edosl0merge
* edosl0split
* edosl0trunc
* edosl0info
* rdr2l0

Creating L0 PDS Files from RDRs

The rdr2l0 tool will dump out an RDR in an order not suitable for use as a NASA L0 PDS file. To generate a L0 PDS file actually requires 2 steps:

$> rdr2l0 --diary --skipfill RNSCA-RVIRS_npp_d20151018_t2359214_e0000467_b20595_c20151019015559980487_noaa_ops.h5
$> edosl0merge -o out.PDS RNSCA-RVIRS_npp_d20151018_t2359214_e0000467_b20595_c20151019015559980487_noaa_ops.h5.science.PDS

The resulting file out.PDS will have packets sorted by time and APID with any duplicates removed.

The typical case would be to dump multiple RDR files to PDS then merge the resulting PDS files into a 6 minute granule like so:

$> rdr2l0 --diary --skipfill RNSCA-RVIRS_npp_d20151018_t000*.h5
$> edosl0merge -o out.PDS --trunc_to "2015-10-18 00:00:00,2015-10-18 00:06:00" *.science.PDS

Which will create a proper 6 minute NASA L0 PDS file with any fill packets filtered out.

If you have quesions or comments related to this software contact brucef@ssec.wisc.edu.