Minimum Valid Pixel/Days in D3/M3 for L3 Aerosol Products
A two-layer filtering is requested by the Aerosol Team to produce M3 products with Yori:
- A minimum number of
Pixel_Countsin a given 1x1 grid cell is required to populate any Aerosol-related group in the D3 product
- A minimum number of "valid days" in a given 1x1 grid cell is required to populate any Aerosol-related group in the M3 product
the minimum value for both the
Pixel_Counts and the "valid days" is subject to change and the user should be allowed to set it
The implementation of this change is split in two parts, in line with the two bullets listed above. All the changes will require the user to specify they want to use these features; this will allow to add functionalities to Yori while maintaining the same workflow for current users.
Pixel_Counts will be implemented in the gridding phase. As such, an additional, optional line will be added to the configuration file that will specify the minimum threshold.
The following example shows how this option can be added to the configuration file (names are not final):
variable_settings: - name_in: Aerosol_Variable name_out: Aerosol_Variable_Out attributes: ... min_pixel_counts: 4
On the code side this should be a relatively easy feature to add. Yori at the moment does this filtering implicitly by populating grid cells with fill values whenever
Pixel_Counts = 0. We can implement this functionality by passing the
min_pixel_counts to the
ComputeVariables class in the
gridtools module. Again, this parameter will default to zero so
yori-grid won't change unless explicitly requested by the user.
Minimum "Valid Days"
Valid Days is defined as the number of D3 files with more than zero
Pixel_Counts values over a month period for a given grid cell.
The hard requirement for making the M3 products filtered by minimum valid days is having a month worth of D3 products ready to be passed to
During the aggregation from D3 to M3 a temporary 360x180
valid_days array is initialized with zeros for each variable that requires the "minimum valid days" filtering. Iterating over the D3 files, a one is added to every grid cell that is not empty. At the end of the 30-day aggregation each grid cell in the
valid_days array will have a value between 0 and 30. This array is used to determine whether a given grid cell meets the criteria and deleted from the final M3 file. This idea is basically what Paul also suggested, but I'm still thinking of a better alternative.