- when updating the cache, inside a given retry (assumption: that's the outer
loop!), track downloads:
- Successful - permanently drop from other FileSet lists, as they'll
generate misleading cache-hit statistics. Or somehow suppress? Should
statistics take a filename and don't include a given filename more than once?
- Failed - Don't retry?
- What other usage patterns are worth handling?
- What other usage patterns are worth handling?
- "I need a pair of times around my target time" - Does anyone need this at
- "I need a pair of times around my target time" - Does anyone need this at
all?
all?
...
@@ -22,19 +8,3 @@ This should move into GitLab issues at some point.
...
@@ -22,19 +8,3 @@ This should move into GitLab issues at some point.
- Target time: 15:15Z
- Target time: 15:15Z
- Result: 15:00Z AND 16:00Z
- Result: 15:00Z AND 16:00Z
- "I need a pair of forecasts around my target time" - cspp-geo-aitf needs
GFS data exactly like this!
- Example:
- A set of forecasts is generated every 6 hours starting at 0:00Z.
The set of forecasts make predictions for every 3 hours out
- Target time: 15:15Z
- Result:
- Ideal: 12:00Z's +3 hour and +6 hour forecasts
- Fallback: 6:00Z's +9 hour and +12 hour forecasts
- Fallback: 6 hours earlier using 6 later later forecasts,
repeating until hitting 72 hour forecasts.
- Should csppfetch.Downloader.download_for_time and csppfetch.Downloader.update_cache automatically construct and return a DownloadStatistics object? Simplifies API a little bit. DownloadStatistics would need to store a list of downloaded files for download_for_time's caller to use, but maybe that's a good idea anyway? If so, becomes a logical place for progress information (see "add a progress callback" below").
- Add a progress callback
- It's silly for csppfetch to log progress to logging.progress if available and logging.info if not. csppfetch.Downloader.download_for_time and csppfetch.Downloader.update_cache should take a callback which will be called occasionally with useful information and a copy of the DownloadStatistics if not-null.