Add support for S3 download and upload
This merge request, as of this writing, is what I have so far for getting S3 upload support added to this geo2grid component. It includes adding awscli to the docker container and laying out a separate bash function for doing the S3 upload instead of doing the local copy to /dst. The main reason this code is considered not done yet is that it really needs a testing script added to ci/. You can see there is one there already but it is extremely basic. I've made some pretty decent complex scripts in other components like geosphere-tile-gen here:
So things that are left to do:
- Add test script that tests basic local copying
- Update script with additional test case that tests S3 upload instead of local copying (or maybe in addition to since we may want both initially)
- Add/update helm chart
values.yamland load necessary S3-related environment variables indeployment.yaml. I think it might be best to assume the model that Ceph produces: one Secret for the AWS access key and secret and one ConfigMap for all the other parameters (BUCKET_HOST, BUCKET_PORT, BUCKET_NAME).
CC @wroberts
Edited by David Hoese