# EdosL0Util
Tools for mangling EDOS L0 PDS files. This package was primarily written to
support work with EDOS L0 PDS files but can more generically be considered a
CCSDS packet file manipulation toolkit.

### Features:

* API for streaming packets
* CLI tools for:
    - merging multiple PDS files
    - splitting PDS into X minute files
    - truncating files to specified time range
    - inspecting PDS info
* Dumping JPSS H5 RDRs to L0 PDS files for VIIRS, CrIS, and ATMS
* Supports both Python 2 & 3
* Etc ...

There is also support for creating a packet stream from Aqua L0 PDS files, but
it has not been tested much. Most functionality assumes EDOS L0 PDS files.


### Packet streaming
To obtain a simple stream of packets, with support for secondary header
parsing use the ``PacketStream`` class as an iterator.

    >>> from edosl0util.stream import PacketStream
    >>> stream = PacketStream(open('P1570826VIIRSSCIENCEAS15286023813501.PDS', 'rb'))
    >>> stream.next()
        <Packet apid=826 seqid=13072 stamp=2015-10-13 00:22:04.042798>
    >>> packets = list(stream)
    >>> first, last, info = stream.info()
    >>> print("First:{}\nLast:{}".format(first, last))
        First:2015-10-13 00:22:04.042798
        Last:2015-10-13 02:04:58.107582
    >>> print info
        defaultdict(<function edosl0util.stream.<lambda>>,
                {800: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 801: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 802: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 803: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 804: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 805: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 806: {'count': 59005, 'last_seqid': 2684, 'num_missing': 0},
                 807: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 808: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
                 809: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
                 810: {'count': 32351, 'last_seqid': 16043, 'num_missing': 0},
                 811: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
                 812: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
                 813: {'count': 114543, 'last_seqid': 5214, 'num_missing': 0},
                 814: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
                 815: {'count': 58994, 'last_seqid': 2673, 'num_missing': 0},
                 816: {'count': 58990, 'last_seqid': 2669, 'num_missing': 0},
                 817: {'count': 114510, 'last_seqid': 5181, 'num_missing': 0},
                 818: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
                 819: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
                 820: {'count': 62766, 'last_seqid': 11834, 'num_missing': 0},
                 821: {'count': 59007, 'last_seqid': 2686, 'num_missing': 0},
                 825: {'count': 83280, 'last_seqid': 3768, 'num_missing': 0},
                 826: {'count': 3471, 'last_seqid': 158, 'num_missing': 0}})

To handle ``PacketTooShort`` and ``NonSequentialSeqId`` errors you have to 
iterate manually:

    >>> from edosl0util.stream import PacketStream
    >>> stream = PacketStream(open('P1570826VIIRSSCIENCEAS15286023813501.PDS'))
    >>> while True:
       ...:     try:
       ...:         packet = stream.next()
       ...:         print(packet)
       ...:     except PacketTooShort as err:
       ...:         print("Corrupt packet stream: {:s}".format(packet))
       ...:         break
       ...:     except NonConsecutiveSeqId as err:
       ...:         print("Missing packet: {:s}".format(err))
       ...:     except StopIteration:
       ...:         break


### Command Line Tools
There are command line tools for merging, splitting, truncating and inspecting
L0 PDS files.

    * edosl0merge
    * edosl0split
    * edosl0trunc
    * edosl0info
    * rdr2l0


### Creating L0 PDS Files from RDRs
The ``rdr2l0`` tool will dump out an RDR in an order not suitable for use as a
NASA L0 PDS file. To generate a L0 PDS file actually requires 2 steps:
```
$> rdr2l0 --diary --skipfill RNSCA-RVIRS_npp_d20151018_t2359214_e0000467_b20595_c20151019015559980487_noaa_ops.h5
$> edosl0merge -o out.PDS RNSCA-RVIRS_npp_d20151018_t2359214_e0000467_b20595_c20151019015559980487_noaa_ops.h5.science.PDS
```
The resulting file ``out.PDS`` will have packets sorted by time and APID with
any duplicates removed.

The typical case would be to dump multiple RDR files to PDS then merge the 
resulting PDS files into a 6 minute granule like so:
```
$> rdr2l0 --diary --skipfill RNSCA-RVIRS_npp_d20151018_t000*.h5
$> edosl0merge -o out.PDS --trunc_to "2015-10-18 00:00:00,2015-10-18 00:06:00" *.science.PDS 
```
Which will create a proper 6 minute NASA L0 PDS file with any fill packets
filtered out.


If you have quesions or comments related to this software contact
brucef@ssec.wisc.edu.