Skip to content

Add support for S3 download and upload

David Hoese requested to merge feature-s3-upload into master

This merge request, as of this writing, is what I have so far for getting S3 upload support added to this geo2grid component. It includes adding awscli to the docker container and laying out a separate bash function for doing the S3 upload instead of doing the local copy to /dst. The main reason this code is considered not done yet is that it really needs a testing script added to ci/. You can see there is one there already but it is extremely basic. I've made some pretty decent complex scripts in other components like geosphere-tile-gen here:

https://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-tile-gen/-/blob/master/ci/test_tile_gen_exec.sh

So things that are left to do:

  1. Add test script that tests basic local copying
  2. Update script with additional test case that tests S3 upload instead of local copying (or maybe in addition to since we may want both initially)
  3. Add/update helm chart values.yaml and load necessary S3-related environment variables in deployment.yaml. I think it might be best to assume the model that Ceph produces: one Secret for the AWS access key and secret and one ConfigMap for all the other parameters (BUCKET_HOST, BUCKET_PORT, BUCKET_NAME).

CC @wroberts

Edited by David Hoese

Merge request reports