geosphere-mapserver merge requestshttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests2023-01-09T21:38:05Zhttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests/6Add horizontal pod autoscaling2023-01-09T21:38:05ZDavid HoeseAdd horizontal pod autoscalingDavid HoeseDavid Hoesehttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests/5Add initial attempt at S3 geotiff reading2022-11-02T18:48:20ZDavid HoeseAdd initial attempt at S3 geotiff readingThis adds the initial configuration changes and testing for loading geotiff data from an S3 bucket. It currently uses MinIO as a temporary S3 server. It includes a lot of new configuration/environment variable options that maybe should h...This adds the initial configuration changes and testing for loading geotiff data from an S3 bucket. It currently uses MinIO as a temporary S3 server. It includes a lot of new configuration/environment variable options that maybe should have been added earlier for performance. Some are specific to curl/s3 loading and authentication.
TODO:
- [ ] Test performance differences between local geotiff and local S3 server. This won't be a fair comparison as S3 will be remote, but it should reveal any major problems (I hope).
- [ ] Allow more of the configuration values to be specified from environment variables.
- [ ] Update the helm chart to load these environment variables from the values.yaml
- [ ] Think of a transition plan for geosphere-test (and geosphere) for switching to S3. Either transition all geotiffs (manual copy/upload) or all future geotiffs or wait to enable it at a later date.
This is step one in getting this to work. The other steps include:
1. tile-gen needs to glob/search the S3 bucket for all geotiffs instead of expecting them locally
2. geo2grid needs to push geotiffs to S3. It probably shouldn't do local and S3 as it will slow processing...but maybe if tested we can check how long this really would be and do it on geosphere-test to verify operations.Phase 2 Task 1 - Processing chain improvementsDavid HoeseDavid Hoesehttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests/4Fix CI bind mount issues2022-04-26T19:04:45ZDavid HoeseFix CI bind mount issueshttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests/3Switch to pulling tile index information from postgres server2022-04-29T12:09:01ZDavid HoeseSwitch to pulling tile index information from postgres serverThis updates the low-level mapserver logic to pull layer information from the postgres server rather than an on-disk shapefile. This means that the "tile gen" component can completely stop creating/using shapefiles. However, these change...This updates the low-level mapserver logic to pull layer information from the postgres server rather than an on-disk shapefile. This means that the "tile gen" component can completely stop creating/using shapefiles. However, these changes require a change to the database itself. The `start_time` column in the PG database needs to be of the `timestamp` type but is currently just a string. This requires updates to tile gen and to mapcache which write and read this time field. I have locally updated my tile gen repository to run `ALTER TABLE` on the database and make this change "in place". As long as the mapserver and mapcache deployments are updated at the same time we should see near-zero down time...I think.Phase 2 Task 1 - Processing chain improvementsDavid HoeseDavid Hoesehttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests/2Fix mapserver not being built with FCGI support2022-04-20T18:29:53ZDavid HoeseFix mapserver not being built with FCGI supportDavid HoeseDavid Hoesehttps://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-mapserver/-/merge_requests/1Upgrade mapserver image to ubuntu 20.04 and update all dependencies2022-04-16T01:47:00ZDavid HoeseUpgrade mapserver image to ubuntu 20.04 and update all dependenciesPhase 2 Task 1 - Processing chain improvementsDavid HoeseDavid Hoese