Minimum Valid Pixel/Days in D3/M3 for L3 Aerosol Products
Request
A two-layer filtering is requested by the Aerosol Team to produce M3 products with Yori:
- A minimum number of
Pixel_Counts
in a given 1x1 grid cell is required to populate any Aerosol-related group in the D3 product - A minimum number of "valid days" in a given 1x1 grid cell is required to populate any Aerosol-related group in the M3 product
Note:
the minimum value for both the Pixel_Counts
and the "valid days" is subject to change and the user should be allowed to set it
Implementation
The implementation of this change is split in two parts, in line with the two bullets listed above. All the changes will require the user to specify they want to use these features; this will allow to add functionalities to Yori while maintaining the same workflow for current users.
Pixel_Counts
Minimum The minimum Pixel_Counts
will be implemented in the gridding phase. As such, an additional, optional line will be added to the configuration file that will specify the minimum threshold.
The following example shows how this option can be added to the configuration file (names are not final):
variable_settings:
- name_in: Aerosol_Variable
name_out: Aerosol_Variable_Out
attributes:
...
min_pixel_counts: 4
On the code side this should be a relatively easy feature to add. Yori at the moment does this filtering implicitly by populating grid cells with fill values whenever Pixel_Counts = 0
. We can implement this functionality by passing the min_pixel_counts
to the ComputeVariables
class in the gridtools
module. Again, this parameter will default to zero so yori-grid
won't change unless explicitly requested by the user.
Minimum "Valid Days"
Valid Days is defined as the number of D3 files with more than zero Pixel_Counts
values over a month period for a given grid cell.
The hard requirement for making the M3 products filtered by minimum valid days is having a month worth of D3 products ready to be passed to yori-aggr
.
During the aggregation from D3 to M3 a temporary 360x180 valid_days
array is initialized with zeros for each variable that requires the "minimum valid days" filtering. Iterating over the D3 files, a one is added to every grid cell that is not empty. At the end of the 30-day aggregation each grid cell in the valid_days
array will have a value between 0 and 30. This array is used to determine whether a given grid cell meets the criteria and deleted from the final M3 file. This idea is basically what Paul also suggested, but I'm still thinking of a better alternative.