CSPP Geo GeoSphere

GeoSphere is a website and processing backend for viewing low-latency high-resolution GOES-16 ABI imagery. The frontend website consists of a static HTML and Javascript website. The backend consists of a series of cloud-friendly Kubernetes deployments released as Helm Charts. Each component of GeoSphere has its own GitLab repository (see below), but the main automated testing and deployment is done by the geosphere-deploy repository. See the below sections for more details.

GeoSphere Production Website: https://geosphere.ssec.wisc.edu/

GeoSphere Test Website: https://geosphere-test.ssec.wisc.edu/

As part of the GeoSphere project guides have been created to provide an overview of common "gotchas" that are encountered when working with containers and cloud technologies. You can find the containerization guide here and the cloud technologies guide here.

GeoSphere Repositories

All GeoSphere related repositories are stored in the "csp_geo/geosphere" gitlab group here. The main component repositories are summarized below. Each repository contains one or more container image definitions, a helm chart definition that uses these containers, and a container registry where the built images are stored. The container registry also contains a bundled OCI version of the helm chart using a name with charts/. This OCI form of the helm chart is currently experimental. See the helm documentation for more information.

Note that the images in the container registry are tagged with the git commit SHA instead of a more human-readable version number. This was done to simplify the development process, but has the downside of the tags not being sortable.

Any other repositories in the group that are not listed below are either legacy efforts or utilities for working with deployment.

• geosphere-grb: CSPP Geo GRB processing. Includes functionality to produce "fake" GRB stream from CADU files for testing and a notifier container for copying produced L1b files to a long term storage. (Container Registry)

• geosphere-geo2grid: CSPP Geo Geo2Grid processing. (Container Registry)

• geosphere-tile-gen: Tile generation to update a database with information about new geotiffs created by Geo2Grid. (Container Registry)

• geosphere-mapserver: MapServer - open source WMS server. Includes non-standard CGI scripts for providing information about what times and products are available from the WMS server. The container uses Apache to serve MapServer. (Container Registry)

• geosphere-mapcache: MapCache - open source WMTS and caching solution. Requests small "tile" regions of images from MapServer, caches them on disk, then provides them to the user. This is served using Apache. (Container Registry)

• geosphere-client: GeoSphere web client. Serves static HTML/Javascript using an Nginx server. (Container Registry)

• geosphere-deploy: GeoSphere automated testing and deployment configurations, utilities, and tools.

In addition to these custom container and helm chart implementations, GeoSphere also uses third-party helm charts for a few components including:

• RabbitMQ
• PostGIS Database

GeoSphere Deploy

This repository holds the various configuration files needed to deploy GeoSphere data processing jobs on the GeoSphere Kubernetes cluster(s). To accomplish this deployment a series of GitLab "triggers" are configured so when sub-components in the https://gitlab.ssec.wisc.edu/cspp_geo/geosphere GitLab group are updated, deployment is executed and operational processing is upgraded.

This repository also contains "helper" CI configs for sub-component repositories to use for building and testing their pieces. See below for more detailed information about the structure of this repository and the locations of the other repositories related to GeoSphere.

Repository Structure

• admin/: Kubernetes resources that should be manually installed to define cluster resources available to the main geosphere application(s). These are not run as part of continuous integration.
• ci_aws/: Helm chart "values" YAML files to configure GeoSphere for deployment on Amazon Web Services (AWS). This directory also contains the GitLab CI configurations specific to this deployment.
• ci_gcp/: Helm chart "values" YAML files to configure GeoSphere for deployment on Google Cloud Platform (GCP). This directory also contains the GitLab CI configurations specific to this deployment.
• ci_geosphere-test/: Helm chart "values" YAML files to configure GeoSphere for the development (geosphere-test) environment. This directory also contains the GitLab CI configurations specific to this deployment.
• ci_geosphere/: Helm chart "values" YAML files to configure GeoSphere for the production environment. This directory also contains the GitLab CI configurations specific to this deployment.
• ci_tests/: Configuration files and scripts for performing basic tests as part of the "test" stage of CI. Each sub-directory contains the helm "values" for a particular test being performed.
• cichart/: Docker image configuration used by certain CI jobs to build or download helm charts.
• helpers/: Scripts and YAML files to help with CI jobs. Soem of these are used by sub-component repositories to reduce the amount of duplicated code between GeoSphere CI environments.
• sidecars/: Docker image configurations for various "sidecar" containers that don't fit in one subcomponent's project.
• .gitlab-ci.yml: Continuous Integration (CI) configuration. This file uses various scripts and sub-configuration files throughout this repository.
• tag_release.sh: Helper script for assigning a git tag to the repository for the various types of deployments (production, test, and GCP environments).