|
|
|
Glance comes with a set of built in data filters for use in creating Virtual Data Sets. These filters include: trimming functions to remove any number of elements from any edge of a 2D data set, a slicing function to slice out a 2D data from a 3D data set (slicing across the third index dimension), a function to classify data into consecutive "bin" ranges and relabel the data with a new set of values (by bin), a function to extract a single bit from an integer that represents a packed bit mask, a function for rotating the order of indexes in a 3D data set, a function for filtering a data set based on the bounds in another data set, and a filter to arrange IPOPP cris data into a pseudo image for inspection.
|
|
|
|
|
|
|
|
In order to use the built in filters you need to import them in your configuration file.
|
|
|
|
|
|
|
|
`import glance.filters as filters`
|
|
|
|
|
|
|
|
###Reshaping and Reordering
|
|
|
|
|
|
|
|
|
|
|
|
The simplest of the filter functions are the trimming functions. These remove a specified number of elements from an edge of a 2D data set. Besides the obvious use of narrowing the window of examination, they may also be used to shift data very easily in the case where the edges of your data are mostly non-finite/missing.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
trim_off_of_top (data, num_elements_to_trim)
|
|
|
|
"""
|
|
|
|
Remove num_elements_to_trim rows from the top of the data array
|
|
|
|
and return a copy of the remaining data
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
trim_off_of_bottom (data, num_elements_to_trim)
|
|
|
|
"""
|
|
|
|
Remove num_elements_to_trim rows from the bottom of the data array
|
|
|
|
and return a copy of the remaining data
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
trim_off_of_right (data, num_elements_to_trim)
|
|
|
|
"""
|
|
|
|
Remove num_elements_to_trim columns from the right side of the data array
|
|
|
|
and return a copy of the remaining data
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
trim_off_of_left (data, num_elements_to_trim)
|
|
|
|
"""
|
|
|
|
Remove num_elements_to_trim columns from the left side of the data array
|
|
|
|
and return a copy of the remaining data
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
The slicing function is a good way to reduce 3D data sets to 2D ones that can be more easily visualized. At the moment the slicing function assumes that you want to slice across the third index of the array.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
select_slice_from_3D_last (data, slice_index)
|
|
|
|
"""
|
|
|
|
Select a slice from a 3 dimensional data set.
|
|
|
|
slice_index indicates the index of the slice that you would like returned
|
|
|
|
by this function
|
|
|
|
|
|
|
|
note: this assumes you wish to slice across the dimension that you would index
|
|
|
|
into last
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
The rotate_indexes_right function allows you to reorder the indexes of a three dimensional array. It takes an array that is indexed [a, b, c] and changes the order to [c, a, b]. You can apply the filter multiple times to achieve other orders.
|
|
|
|
|
|
|
|
Note: Not all orders can be achieved with this method. If more complex reordering filters are needed in the future please feel free to create them in a similar style.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def rotate_indexes_right (data)
|
|
|
|
"""
|
|
|
|
move the order of the indexes in the array to the right in order, taking the last one
|
|
|
|
and putting it in the first index spot
|
|
|
|
note: at the moment this filter only works with 3 dimentional data sets
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
The reverse_2D_data_vertically and reverse_2D_data_horizontally functions allow you to reverse the order of two dimensional data arrays along the dimensions that Glance displays vertically or horizontally respectively.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def reverse_2D_data_vertically (data)
|
|
|
|
"""
|
|
|
|
Reverse two dimensional data along it's first dimension.
|
|
|
|
For most satellite data this will result in it flipping vertically
|
|
|
|
when glance displays it.
|
|
|
|
"""
|
|
|
|
|
|
|
|
def reverse_2D_data_horizontally (data)
|
|
|
|
"""
|
|
|
|
Reverse two dimensional data along it's second dimension.
|
|
|
|
For most satellite data this will result in it flipping horizontally
|
|
|
|
when glance displays it.
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
Data Value Based Filters
|
|
|
|
|
|
|
|
|
|
|
|
The sorting/flattening function allows you to specify a set of bins and new values that you would like assigned to the values that fall into those bins. This, in essence, will allow you to reduce continuous data to a more discrete form or reduce discrete data to a smaller number of categories. For example, you could reduce cloud mask data to simply "clear" and "cloudy" rather than the various possible intermediate values.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
flatten_data_into_bins (data, ranges, new_values, missing_value, return_data_type)
|
|
|
|
"""
|
|
|
|
Sort the data into the given ranges. Each range should correspond to a value
|
|
|
|
given in the list of new_values; this value will be used for all data points
|
|
|
|
that fall within that range. Ranges will be treated as a continuous spectrum
|
|
|
|
So please list them in increasing order.
|
|
|
|
|
|
|
|
Note: Data values that fall directly between two ranges will be grouped with
|
|
|
|
the larger range.
|
|
|
|
|
|
|
|
Also Note: If a data point is not found to fall within any of the ranges,
|
|
|
|
the missing_value will be filled into that spot instead.
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
Glance also provides a function that will extract a single bit of "mask" data from an integer data set. This method allows you to unpack standard packed bit masks and specify the "true" and "false" values that they should be interpreted into.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
extract_bit_from_packed_mask (data, index_of_bit_to_extract,
|
|
|
|
truth_value, false_value,
|
|
|
|
return_data_type)
|
|
|
|
"""
|
|
|
|
Extract a one bit boolean mask from a larger packed data set. The bit that
|
|
|
|
you wish to extract must be identified by it's index from the lower end of
|
|
|
|
the array of bits (with 0 being the bit that would represent a value of 1
|
|
|
|
if the mask was an integer; so the 1 indexed bit would represent the integer
|
|
|
|
value of 2, the 2 indexed bit a value of 4, and so on).
|
|
|
|
Note: It is assumed that the endian-ness of your data was handled correctly
|
|
|
|
by whatever code loaded it from the file.
|
|
|
|
|
|
|
|
The return data will be filled with the truth_value and false_value based
|
|
|
|
on whether the bit was 1 (true) or 0 (false) and will be of the requested
|
|
|
|
return_data_type.
|
|
|
|
|
|
|
|
Note: If you wish to examine or compare more than one packed bit, you may
|
|
|
|
use this filter multiple times to extract each separately and then create
|
|
|
|
your own filter function to perform any comparisons.
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
The set_to_value_between_bounds function allows you to set all data in an array that is non-finite or outside a given set of bounds to some inputted value. This is good if you need to analyze a data set while ignoring certain types of invalid (ex. out of range) data by setting it to some known value, possibly the missing value.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def set_to_value_between_bounds(data, value_to_set_to, bottom_bound_exclusive, top_bound_exclusive) :
|
|
|
|
"""
|
|
|
|
Wherever the data is non-finite or outside the given bounds, set it to the given value.
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
Complex Filters
|
|
|
|
|
|
|
|
|
|
|
|
The filter_based_on_additional_data_set_min_max_bounds filter can be used to remove data from your data set (by setting it to the missing value) based on where data of certain values is in a secondary "filter" data set. For example, this filter might be useful in selecting only the winds vectors that correspond to a certain flag value or range of quality index values.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Note: This filter was originally created to be used with secondary variable dependent filters.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def filter_based_on_additional_data_set_min_max_bounds(data, filterData, missingValue=None,
|
|
|
|
minOkFilterValue=None, maxOkFilterValue=None) :
|
|
|
|
"""
|
|
|
|
filter a data set based on values in another data set
|
|
|
|
|
|
|
|
if some of the filter data is above/below the optional min/max values the corresponding values in the
|
|
|
|
primary data will be set to the given missingValue
|
|
|
|
|
|
|
|
ex. this filter might be used to remove winds data that has a quality index below a certain threshold
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
The organize_ipopp_data_into_image filter reorders IPOP Cris data into a pseudo-image for display. The image is composed of either one selected wave number or the mean of all wave numbers at a given point. Each pixel will represent the data at one detector, in one field of regard, on one scan line. The filter description below gives more explicit details on the physical ordering of the data in the image. The purpose of the image created by this filter is primarily to give you an idea of general spatial problems and groupings of missing data in the image taken by the hardware.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def organize_ipopp_data_into_image(original_ipopp_data, wave_number=None, missing_value=None,
|
|
|
|
propagate_partial_missing_values=False) :
|
|
|
|
"""
|
|
|
|
organize the ipopp data spatially into an 'image' of sorts
|
|
|
|
this basically consists of:
|
|
|
|
|
|
|
|
-> the 30 fields of regard
|
|
|
|
---------------
|
|
|
|
| | |
|
|
|
|
| | V
|
|
|
|
| | the 4 scan lines
|
|
|
|
| |
|
|
|
|
---------------
|
|
|
|
|
|
|
|
for each field of regard/scan line
|
|
|
|
_______
|
|
|
|
|_|_|_|
|
|
|
|
|_|_|_|
|
|
|
|
|_|_|_|
|
|
|
|
|
|
|
|
block of the 9 detectors
|
|
|
|
|
|
|
|
with the index to physical mapping:
|
|
|
|
|
|
|
|
0 1 2
|
|
|
|
3 4 5
|
|
|
|
6 7 8
|
|
|
|
|
|
|
|
for each detector point, if the wave_number was given
|
|
|
|
in the parameters, that specific interferogram data pt will be used
|
|
|
|
if no wave_number was given, the mean of the 717 pts will be used
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
Chaining Filters
|
|
|
|
|
|
|
|
|
|
|
|
Multiple filters can be chained to produce more complex behavior.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# unpack a bit mask and then clip some data off of the edges
|
|
|
|
filterFunction = (lambda data: filters.trim_off_of_top(
|
|
|
|
filters.trim_off_of_right(
|
|
|
|
filters.extract_bit_from_packed_mask(data, 3, (1,0), int), 2), 2))
|
|
|
|
|
|
|
|
\ No newline at end of file |
