himawari issueshttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues2022-01-27T16:23:19Zhttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/34Make python code py3 where useful, remove where not2022-01-27T16:23:19ZRay GarciaMake python code py3 where useful, remove where notWhile the module has py3 binding capability, included sample and test code should retire py2.While the module has py3 binding capability, included sample and test code should retire py2.Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/33Add support for Meteosat HRIT format2023-08-18T16:36:29ZJustin SieglaffAdd support for Meteosat HRIT formatAs part of the VOLCAT in NOAA Cloud Project, @scottm has agreed to add support to the Himawari HRIT reader to also support Meteosat HRIT reader (Meteosat HRIT data is the format available in NOAA cloud, not AREA).
The datacenter has pro...As part of the VOLCAT in NOAA Cloud Project, @scottm has agreed to add support to the Himawari HRIT reader to also support Meteosat HRIT reader (Meteosat HRIT data is the format available in NOAA cloud, not AREA).
The datacenter has provided some sample Meteosat HRIT data:
/ships19/grain/msg-example-files.tar.gz
They also provided link to format documentation:
https://www.eumetsat.int/media/45126Scott MindockScott Mindockhttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/32implement and test AMI C++ module for F9x/C user applications2019-11-12T21:25:17ZRay Garciaimplement and test AMI C++ module for F9x/C user applicationsNetCDF format for AMI L1b is approaching finalization, implement a compatibility reader.NetCDF format for AMI L1b is approaching finalization, implement a compatibility reader.Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/31out-of-bounds reads may be best to return sentinels instead of errors2019-05-10T19:57:47ZRay Garciaout-of-bounds reads may be best to return sentinels instead of errorsIn integrating libHimawari to CLAVR-x, one of the gotchas involved querying for segments of data that extended beyond the extent of the scene. In that case, e.g. a last segment that doesn't conveniently end on scene boundary, an error wa...In integrating libHimawari to CLAVR-x, one of the gotchas involved querying for segments of data that extended beyond the extent of the scene. In that case, e.g. a last segment that doesn't conveniently end on scene boundary, an error was returned by libHimawari.
Investigate whether it is still correct if data requests extending beyond scene boundaries to simply fill out-of-scene pixels with sentinel values, or whether this should be an extended boolean toggle that defaults to the existing behavior, e.g. sentinel_outside_scene_extent=true/false
This is a feature request and not a bugfix, since the issue is currently solvable by just insuring that no queries extend outside the available scene data.https://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/29HSD reading is not expected to work on big endian systems2019-04-02T17:12:59ZRay GarciaHSD reading is not expected to work on big endian systems* [ ] the mapping of basic_info needs to determine whether byte-flipping is needed and then flip the struct
* [ ] the mmap_struct code needs to byte-flip before confirming block number* [ ] the mapping of basic_info needs to determine whether byte-flipping is needed and then flip the struct
* [ ] the mmap_struct code needs to byte-flip before confirming block numberRay GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/28warnings during hsd2nc runs, presumably due to mask/nan content2018-10-16T19:37:16ZRay Garciawarnings during hsd2nc runs, presumably due to mask/nan content```
axi-tools-v20171002/ShellB3/lib/python3.6/site-packages/himawari-20171002T193124-py3.6-linux-x86_64.egg/himawari/hsd2nc.py:297: RuntimeWarning: invalid value encountered in less
assert(not np.any(counts<0.0))
axi-tools-v20171002/She...```
axi-tools-v20171002/ShellB3/lib/python3.6/site-packages/himawari-20171002T193124-py3.6-linux-x86_64.egg/himawari/hsd2nc.py:297: RuntimeWarning: invalid value encountered in less
assert(not np.any(counts<0.0))
axi-tools-v20171002/ShellB3/lib/python3.6/site-packages/himawari-20171002T193124-py3.6-linux-x86_64.egg/himawari/hsd2nc.py:298: RuntimeWarning: invalid value encountered in greater_equal
assert(not np.any(counts>=self.imissing))
```
If we can get rid of these warnings in the future, that would be great. Opening hypothesis is that NaNs are present in the data and we need to ignore them in the `>=` and `<` .Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/27add downsampling and radiance output options to cmi_changer2018-04-23T16:55:17ZRay Garciaadd downsampling and radiance output options to cmi_changerIn order to fully sunset hsd2nc, we need options for writing L1B-ish radiance files at 2km resolution for AHI.
* [ ] implement interpolated downsampling and striding options
* [ ] implement radiance output option
* [ ] implement geo (l...In order to fully sunset hsd2nc, we need options for writing L1B-ish radiance files at 2km resolution for AHI.
* [ ] implement interpolated downsampling and striding options
* [ ] implement radiance output option
* [ ] implement geo (lat/lon) export optionhttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/26hsd2nc downsample does not work in py32018-04-23T16:53:20ZRay Garciahsd2nc downsample does not work in py3GregQ notes that hsd2nc with downsampling option does not work in axi-tools package due to py3 division semantics.GregQ notes that hsd2nc with downsampling option does not work in axi-tools package due to py3 division semantics.https://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/25Resolve off-disk HCAST sentinel value concerns2017-11-15T19:50:16ZRay GarciaResolve off-disk HCAST sentinel value concernsWe still have an issue where some bands in HCAST show off-disk pixels as being very low radiance instead of being missing; this is not the case for all HCAST.
* [ ] review HRIT docs http://www.data.jma.go.jp/mscweb/en/operation/sample/...We still have an issue where some bands in HCAST show off-disk pixels as being very low radiance instead of being missing; this is not the case for all HCAST.
* [ ] review HRIT docs http://www.data.jma.go.jp/mscweb/en/operation/sample/JMA_HRIT_Issue1.2.pdf
* [ ] review HRIT reader code regarding off-disk sentinel values
* [ ] fix or confirm "works as required"https://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/24Produce doxygen documentation for fortran/C API set2017-11-15T19:48:28ZRay GarciaProduce doxygen documentation for fortran/C API set* [ ] doxygen markup existing interface files
* [ ] generate doxygen output and place it on SSEC web
* [ ] review it with Mike Pavolonis or another API end-user* [ ] doxygen markup existing interface files
* [ ] generate doxygen output and place it on SSEC web
* [ ] review it with Mike Pavolonis or another API end-userhttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/23resolve competition between hsd2nc.py, ahi2cmi.py, and cmi_changer2017-11-15T19:47:20ZRay Garciaresolve competition between hsd2nc.py, ahi2cmi.py, and cmi_changerRight now we have several variants of CMI and L1B conversion for AHI.
Look at reducing this to two, the "retro" hsd2nc and the "neo" cmi_changer; this requires cmi_changer have a radiance option like ahi2cmi (which was originally a test...Right now we have several variants of CMI and L1B conversion for AHI.
Look at reducing this to two, the "retro" hsd2nc and the "neo" cmi_changer; this requires cmi_changer have a radiance option like ahi2cmi (which was originally a test pattern going into cmi_changer development)https://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/22Warnings during HSD conversion to SCMI using cmi_changer2017-11-15T19:11:19ZRay GarciaWarnings during HSD conversion to SCMI using cmi_changerKaba writes:
```
I was able to run and the AHI HCAST sample files and generate SCMI files just fine but had
some warnings with the AHI HSD test case.
Here are the details:
[kbah@imaginator hsd]$ /home/kbah/code/axi_tools/axi-tools-v...Kaba writes:
```
I was able to run and the AHI HCAST sample files and generate SCMI files just fine but had
some warnings with the AHI HSD test case.
Here are the details:
[kbah@imaginator hsd]$ /home/kbah/code/axi_tools/axi-tools-v20171020/bin/cmi_changer.sh -E OT -R HDF -S Full\ Disk $(find /fjord/awg/ait/users/rayg/Data/axi-tools-test-data/input/hsd -name 'HS_H08_*DAT' |sed 's/_s.*//g' |sort |uniq)
> OT_HDF-010-B11-M1C01-T000_H8_s2017089021000_c2017318181351.nc
> OT_HDF-010-B11-M1C01-T001_H8_s2017089021000_c2017318181351.nc
> OT_HDF-010-B11-M1C01-T002_H8_s2017089021000_c2017318181351.nc
> OT_HDF-010-B11-M1C01-T003_H8_s2017089021000_c2017318181351.nc
> OT_HDF-010-B11-M1C01-T004_H8_s2017089021000_c2017318181351.nc
> OT_HDF-010-B11-M1C01-T005_H8_s2017089021000_c2017318181351.nc
> OT_HDF-010-B11-M1C01-T006_H8_s2017089021000_c2017318181351.nc
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
/home/kbah/code/axi_tools/axi-tools-v20171020/ShellB3/lib/python3.6/site-packages/goesr-0.5.1-py3.6.egg/goesr/rockfall.py:499: RuntimeWarning: invalid value encountered in less
X[:] = V[:]
/home/kbah/code/axi_tools/axi-tools-v20171020/ShellB3/lib/python3.6/site-packages/goesr-0.5.1-py3.6.egg/goesr/rockfall.py:499: RuntimeWarning: divide by zero encountered in true_divide
X[:] = V[:]
/home/kbah/code/axi_tools/axi-tools-v20171020/ShellB3/lib/python3.6/site-packages/himawari-20171020T203829-py3.6-linux-x86_64.egg/himawari/ahi2cmi.py:414: RuntimeWarning: Mean of empty slice
v_image = np.nanmean(v)
/home/kbah/code/axi_tools/axi-tools-v20171020/ShellB3/lib/python3.6/site-packages/himawari-20171020T203829-py3.6-linux-x86_64.egg/himawari/ahi2cmi.py:419: RuntimeWarning: All-NaN axis encountered
vrange = np.array([np.nanmin(v), np.nanmax(v)])
/home/kbah/code/axi_tools/axi-tools-v20171020/ShellB3/lib/python3.6/site-packages/himawari-20171020T203829-py3.6-linux-x86_64.egg/himawari/ahi2cmi.py:419: RuntimeWarning: All-NaN slice encountered
vrange = np.array([np.nanmin(v), np.nanmax(v)])
WARNING: not all data for requested region available - fill data will be included
/home/kbah/code/axi_tools/axi-tools-v20171020/ShellB3/lib/python3.6/site-packages/himawari-20171020T203829-py3.6-linux-x86_64.egg/himawari/ahi2cmi.py:668: RuntimeWarning: Mean of empty slice
return np.nanmean(y), np.nanmean(x)
> OT_HDF-010-B11-M1C01-T000_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T001_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T002_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T003_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T004_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T005_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T006_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T007_H8_s2017089021000_c2017318181453.nc
> OT_HDF-010-B11-M1C01-T008_H8_s2017089021000_c2017318181453.nc
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
WARNING: not all data for requested region available - fill data will be included
```Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/20HSD format 1.3 as of 20170725, support additional calibration block fields2018-02-14T22:58:15ZRay GarciaHSD format 1.3 as of 20170725, support additional calibration block fieldsImpact of the below change is that the additional cal fields will not be available, but the existing code is sufficiently forward compatible with the new block size.
* [x] allow different versions of calibration block to be made availa...Impact of the below change is that the additional cal fields will not be available, but the existing code is sufficiently forward compatible with the new block size.
* [x] allow different versions of calibration block to be made available based on length of block provided
* [x] examine using file format version in the main header as well
* [x] verify with pre-20170725 and post-20170725 HSD data files
"""
Dear HimawariCast/HimawariCloud users,
The Japan Meteorological Agency (JMA) plans to modify its Himawari-8
ground processing system at 04:00 UTC on 25 July 2017 in order to improve the
quality of Himawari-8 imagery and to update the Himawari Standard Data (HSD)
format with the latest calibration coefficients in consideration of sensor sensitivity.
The modification will include:
1) Reduction of banding and stripe noise
2) Improvement of quantization noise
3) Updating of the HSD format
Please see the the details below.
http://www.data.jma.go.jp/mscweb/en/operation8/eventlog/Improvement_of_Himawari-8_data_quality.pdf
http://www.data.jma.go.jp/mscweb/en/operation/calibration/hm8/CorrectionOfSensitivityTrend.pdf
http://www.data.jma.go.jp/mscweb/en/himawari89/space_segment/hsd_sample/HS_D_users_guide_en_v13.pdf
Best regards,
Satellite Program Division
Japan Meteorological Agency
"""Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/19scaled integer radiances desirable for HCAST2018-09-08T20:44:28ZWilliam Strakascaled integer radiances desirable for HCASTIn the latest version of hsd2nc.py, it no longer produces **scaled** radiances. Instead, the radiance (Rad) variable is *un*scaled. There should be a command-line option to output the files in a similar format to the ABI L1b files (i.e. ...In the latest version of hsd2nc.py, it no longer produces **scaled** radiances. Instead, the radiance (Rad) variable is *un*scaled. There should be a command-line option to output the files in a similar format to the ABI L1b files (i.e. **scaled** radiances, scale_factor/add_offset attributes, etc.).
The line in question is line 275
if not self.rad_as_float: # then we're doing scaled integer counts, which works for HSD but not HCASThttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/18reading a single section of HCAST does not get valid time metadata2018-09-08T20:44:28ZRay Garciareading a single section of HCAST does not get valid time metadataGeoffC is having trouble getting time information from the libHimawari metadata structure for the segmented HCAST files.
The first segment file (`IMG_DK01B05_201508290810_001.bz2`, say) has the correct time information in the metadata ...GeoffC is having trouble getting time information from the libHimawari metadata structure for the segmented HCAST files.
The first segment file (`IMG_DK01B05_201508290810_001.bz2`, say) has the correct time information in the metadata attribute...
```
2016-10-13 21:50:49.206 (INFO) : IMG_DK01B05_201508290810_001.bz2 : year = 2015
2016-10-13 21:50:49.207 (INFO) : IMG_DK01B05_201508290810_001.bz2 : month = 8
2016-10-13 21:50:49.207 (INFO) : IMG_DK01B05_201508290810_001.bz2 : day = 29
2016-10-13 21:50:49.207 (INFO) : IMG_DK01B05_201508290810_001.bz2 : hour = 8
2016-10-13 21:50:49.207 (INFO) : IMG_DK01B05_201508290810_001.bz2 : minute = 10
2016-10-13 21:50:49.207 (INFO) : IMG_DK01B05_201508290810_001.bz2 : second = 21
2016-10-13 21:50:49.207 (INFO) : IMG_DK01B05_201508290810_001.bz2 : microsecond = 360000
```
But looking at any other segment file (`IMG_DK01B05_201508290810_003.bz2`, say)...
```
2016-10-13 21:51:54.207 (INFO) : IMG_DK01B05_201508290810_003.bz2 : year = 0
2016-10-13 21:51:54.208 (INFO) : IMG_DK01B05_201508290810_003.bz2 : month = 0
2016-10-13 21:51:54.208 (INFO) : IMG_DK01B05_201508290810_003.bz2 : day = 0
2016-10-13 21:51:54.208 (INFO) : IMG_DK01B05_201508290810_003.bz2 : hour = 0
2016-10-13 21:51:54.208 (INFO) : IMG_DK01B05_201508290810_003.bz2 : minute = 0
2016-10-13 21:51:54.208 (INFO) : IMG_DK01B05_201508290810_003.bz2 : second = 0
2016-10-13 21:51:54.208 (INFO) : IMG_DK01B05_201508290810_003.bz2 : microsecond = 0
```
Test sequence:
```python
import HimawariScene as hsd
fn='espresso/fjord/awg/ait/dc/jma/himawari08/ahi/2016/2016_10_12_286/0040/HS_H08_20161012_0040_B01_FLDK_R10_S0610.DAT'
hc = hsd.HimawariScene(fn)
[getattr(hc.metadata.start_time,x) for x in ('year', 'month', 'day', 'hour', 'minute', 'second')]
Out[6]: [2016, 10, 12, 0, 45, 4]
```
Goal is that partial scenes (sections missing or deliberately omitted) should still be valid scenes.https://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/17cgms_x not produced if first segment not available2018-09-08T20:44:28ZRay Garciacgms_x not produced if first segment not availableusing S0610.DAT results in NaN-filled cgms_x variable while it works OK for S0110. Reported by RKuehnusing S0610.DAT results in NaN-filled cgms_x variable while it works OK for S0110. Reported by RKuehnRay GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/16Add a loglevel option when attaching a scene2018-09-08T20:44:28ZRay GarciaAdd a loglevel option when attaching a sceneAPI (especially in Python) should have a loglevel setting which lets the client set the default log level for the scene, if not the whole library. This should be in addition to the LogFunction optional.
API (especially in Python) should have a loglevel setting which lets the client set the default log level for the scene, if not the whole library. This should be in addition to the LogFunction optional.
Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/14assess whether viewing, solar, lunar angles and geometry calculations can/sho...2018-09-08T20:44:28ZRay Garciaassess whether viewing, solar, lunar angles and geometry calculations can/should be includedA user has requested solar and viewing geometry queries be available on a pixel basis. Look at existing implementations of these calculations (geocat etc) and see if it's appropriate to include one or more geometry APIs. A user has requested solar and viewing geometry queries be available on a pixel basis. Look at existing implementations of these calculations (geocat etc) and see if it's appropriate to include one or more geometry APIs. Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/13reduce memory footprint on hsd2nc.py2018-09-08T20:44:28ZRay Garciareduce memory footprint on hsd2nc.pyhsd2nc uses too much RAM when working with 0.5km resolution files. Conversion of data can and should be done in smaller chunks to preserve memory.
hsd2nc uses too much RAM when working with 0.5km resolution files. Conversion of data can and should be done in smaller chunks to preserve memory.
Ray GarciaRay Garciahttps://gitlab.ssec.wisc.edu/rayg/himawari/-/issues/11add ssp_latitude and _longitude to hsd2nc output2018-09-08T20:44:28ZRay Garciaadd ssp_latitude and _longitude to hsd2nc outputWCS3 requests addition of Sub-satellite point coordinates to the hsd2nc generated files.
WCS3 requests addition of Sub-satellite point coordinates to the hsd2nc generated files.
Ray GarciaRay Garcia