Skip to content

grb_latency.py nickb recommendations

Nick Bearson requested to merge grb-latency-extended into develop

Tried to order the commits from least -> most controversial, so that easier cherry-picking can be done if necessary.

I sent this on slack, but for posterity, when examining a stack of log files I started to see larger outliers:

Log start time:  2020-04-27 16:06:24.203505
Log end time:    2020-04-27 17:57:07.672923
Per-product latencies in seconds (observation end time to file creation time on GRB receiving system):
  ABI Continental United States                                          mean:  22.1   median:  21.1   max:  28.2   min:  15.3   count:    352
  ABI Full Disk                                                          mean:  19.4   median:  19.5   max:  25.3   min:  13.6   count:    176
  ABI Mesoscale                                                          mean:  17.4   median:  16.9   max:  22.9   min:  14.2   count:   3536
  EXIS Solar Flux: EUV                                                   mean:  18.2   median:  16.7   max:  21.7   min:  16.3   count:    221
  EXIS Solar Flux: X-Ray                                                 mean:  13.4   median:  13.4   max:  15.0   min:  13.1   count:    221
  GLM                                                                    mean:  15.4   median:  15.0   max:  30.5   min:  13.6   count:    332
  MAG                                                                    mean:  13.2   median:  13.1   max:  14.4   min:  13.0   count:    110
  SEISS Magnetospheric Electrons and Protons: Low Energy                 mean:  13.5   median:  13.5   max:  14.4   min:  13.3   count:    221
  SEISS Magnetospheric Electrons and Protons: Medium and High Energy     mean:  21.6   median:  13.7   max: 161.5   min:  13.4   count:    221
  SEISS Solar and Galactic Protons                                       mean:  22.9   median:  13.8   max: 191.4   min:  13.6   count:    110
  SEISS Energetic Heavy Ions                                             mean:  23.5   median:  12.5   max: 132.4   min:  12.4   count:     22
  SUVI                                                                   mean:  33.1   median:  33.1   max:  38.5   min:  29.6   count:    662

Wow look at those outliers! Just looking at one of their filenames you can see there's going to be ~2 minutes of delay, between the "end time" and "created time":

OR_SEIS-L1b-EHIS_G16_s20201181735590_e20201181740580_c20201181742592.nc

"observation end time" is probably appropriate for what Graeme was reporting to Jim McNitt for the program's use, but for an individual reception site I think they're probably more interested in comparing against the ground system file creation time. This is what the numbers look like after making that edit:

Log start time:  2020-04-27 16:06:24.203505
Log end time:    2020-04-27 17:57:21.016370
Per-product latencies in seconds (ground system creation time to file creation time on GRB receiving system):
  ABI Continental United States                                          mean:  14.1   median:  14.3   max:  17.3   min:  11.2   count:    352
  ABI Full Disk                                                          mean:  13.5   median:  13.5   max:  16.3   min:  11.0   count:    176
  ABI Mesoscale                                                          mean:  13.1   median:  12.7   max:  17.6   min:  10.7   count:   3552
  EXIS Solar Flux: EUV                                                   mean:  12.7   median:  11.1   max:  16.2   min:  10.6   count:    222
  EXIS Solar Flux: X-Ray                                                 mean:  13.1   median:  13.0   max:  14.6   min:  12.6   count:    222
  GLM                                                                    mean:  12.9   median:  12.6   max:  17.4   min:  11.2   count:    333
  MAG                                                                    mean:  12.7   median:  12.6   max:  13.9   min:  12.4   count:    111
  SEISS Magnetospheric Electrons and Protons: Low Energy                 mean:  12.2   median:  12.2   max:  13.2   min:  11.8   count:    222
  SEISS Magnetospheric Electrons and Protons: Medium and High Energy     mean:  12.3   median:  12.3   max:  13.7   min:  11.1   count:    222
  SEISS Solar and Galactic Protons                                       mean:  12.4   median:  12.4   max:  14.0   min:  11.1   count:    111
  SEISS Energetic Heavy Ions                                             mean:  11.2   median:  11.1   max:  11.4   min:  11.1   count:     22
  SUVI                                                                   mean:  13.3   median:  12.3   max:  17.4   min:  11.0   count:    663

Merge request reports