Skip to content
Snippets Groups Projects

Add initial attempt at S3 geotiff reading

Merged Add initial attempt at S3 geotiff reading
feature-s3-geotiffs into master
Merged David Hoese requested to merge feature-s3-geotiffs into master

This adds the initial configuration changes and testing for loading geotiff data from an S3 bucket. It currently uses MinIO as a temporary S3 server. It includes a lot of new configuration/environment variable options that maybe should have been added earlier for performance. Some are specific to curl/s3 loading and authentication.

TODO:

  • Test performance differences between local geotiff and local S3 server. This won't be a fair comparison as S3 will be remote, but it should reveal any major problems (I hope).
  • Allow more of the configuration values to be specified from environment variables.
  • Update the helm chart to load these environment variables from the values.yaml
  • Think of a transition plan for geosphere-test (and geosphere) for switching to S3. Either transition all geotiffs (manual copy/upload) or all future geotiffs or wait to enable it at a later date.

This is step one in getting this to work. The other steps include:

  1. tile-gen needs to glob/search the S3 bucket for all geotiffs instead of expecting them locally
  2. geo2grid needs to push geotiffs to S3. It probably shouldn't do local and S3 as it will slow processing...but maybe if tested we can check how long this really would be and do it on geosphere-test to verify operations.
Edited by David Hoese

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply