Add support for S3 download and upload
This merge request, as of this writing, is what I have so far for getting S3 upload support added to this geo2grid component. It includes adding awscli to the docker container and laying out a separate bash function for doing the S3 upload instead of doing the local copy to /dst
. The main reason this code is considered not done yet is that it really needs a testing script added to ci/
. You can see there is one there already but it is extremely basic. I've made some pretty decent complex scripts in other components like geosphere-tile-gen
here:
So things that are left to do:
- Add test script that tests basic local copying
- Update script with additional test case that tests S3 upload instead of local copying (or maybe in addition to since we may want both initially)
- Add/update helm chart
values.yaml
and load necessary S3-related environment variables indeployment.yaml
. I think it might be best to assume the model that Ceph produces: one Secret for the AWS access key and secret and one ConfigMap for all the other parameters (BUCKET_HOST, BUCKET_PORT, BUCKET_NAME).
CC @wroberts
Edited by David Hoese