Add initial proof of concept DB sync script

README.md

@@ -110,9 +110,9 @@ and the locations of the other repositories related to GeoSphere.
   production website running on the SSEC gamma cluster.
 * **gamma-geosphere-test/**: Helm chart and ArgoCD application for the
   dev site running on the SSEC gamma cluster.
-* **helpers/**: Scripts and YAML files to help with CI jobs. Soem of these are
+* **helpers/**: Scripts and YAML files to help with CI jobs. Some of these are
   used by sub-component repositories to reduce the amount of duplicated code
-  between GeoSphere CI environments.
+  between GeoSphere CI environments. Other helpful scripts are also stored here.
 * **sidecars/**: Docker image configurations for various "sidecar" containers that
   don't fit in one subcomponent's project.
 * **.gitlab-ci.yml**: Continuous Integration (CI) configuration. This file

helpers/sync_s3_geotiffs_to_db.sh

+#!/usr/bin/env bash
+# Purpose: Scan through every geotiff in an S3 bucket and add an associated
+#    to the GeoSphere database.
+
+debug() {
+    >&2 echo "DEBUG: $@"
+}
+
+error() {
+    >&2 echo "ERROR: $@"
+    exit 1
+}
+
+if [[ $# -ne 3 ]]; then
+    error "Usage: ./sync_s3_geotiffs_to_db.sh <namespace> <objectBucketClaim name> <postgres hostname>"
+fi
+
+export NAMESPACE=$1
+export OBC_NAME=$2
+export POSTGRES_HOST=$3
+export POSTGRES_SECRET="geosphere-postgresql"
+SCRIPT_DIR="$( cd -P "$( dirname "$(readlink -f "${BASH_SOURCE[0]}")" )" && pwd )"
+
+tty=$(tty)
+envsubst < "${SCRIPT_DIR}/sync_s3_geotiffs_to_db_job_tmpl.yaml" | \
+    tee ${tty} | kubectl apply -n ${NAMESPACE} -f -
\ No newline at end of file

helpers/sync_s3_geotiffs_to_db_job_tmpl.yaml

+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: geosphere-db-sync
+  namespace: "${NAMESPACE}"
+  labels: {}
+  annotations: {}
+spec:
+  ttlSecondsAfterFinished: 300
+  template:
+    spec:
+      securityContext:
+        runAsUser: 1000
+      restartPolicy: Never
+      volumes:
+        - name: postgres-secret
+          secret:
+            secretName: "${POSTGRES_SECRET}"
+      containers:
+        - name: refresh-db
+          image: "gitlab.ssec.wisc.edu:5555/cspp_geo/geosphere/geosphere-tile-gen/geosphere-tile-gen:0b8e9299"
+          imagePullPolicy: IfNotPresent
+          args:
+            - /work/refresh_missing_db_records.py
+            - "geotiffs/{satellite_id}/{instrument_id}/{sector_id}"
+            - --postgres-host
+            - "${POSTGRES_HOST}"
+            - --postgres-password-file
+            - "/secrets/postgres-password"
+          resources:
+            limits:
+              cpu: "2.0"
+              memory: "4Gi"
+            requests:
+              cpu: "1.0"
+              memory: "2Gi"
+          securityContext:
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            - name: postgres-secret
+              mountPath: "/secrets"
+              readOnly: true
+          env:
+            - name: G2G_PRODUCTS
+              value: ""
+            - name: PRODUCT_SECTORS
+              value: ""
+            - name: PRODUCT_SATELLITES
+              value: ""
+            - name: "AWS_S3_ENDPOINT"
+              valueFrom:
+                configMapKeyRef:
+                  name: "${OBC_NAME}"
+                  key: "BUCKET_HOST"
+            - name: "AWS_S3_ENDPOINT_PORT"
+              valueFrom:
+                configMapKeyRef:
+                  name: "${OBC_NAME}"
+                  key: "BUCKET_PORT"
+            - name: SRC_BUCKET
+              valueFrom:
+                configMapKeyRef:
+                  name: "${OBC_NAME}"
+                  key: "BUCKET_NAME"
+            - name: AWS_ACCESS_KEY_ID
+              valueFrom:
+                secretKeyRef:
+                  name: "${OBC_NAME}"
+                  key: "AWS_ACCESS_KEY_ID"
+            - name: AWS_SECRET_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: "${OBC_NAME}"
+                  key: "AWS_SECRET_ACCESS_KEY"
+            - name: RABBITMQ_OUT_HOST
+              value: ""
+            - name: RABBITMQ_OUT_EXCHANGE_NAME
+              value: "satellite"
+            - name: RABBITMQ_OUT_USERNAME
+              value: "user"
+#            - name: RABBITMQ_OUT_PASSWORD
+#              valueFrom:
+#                secretKeyRef:
+#                  name: "geosphere-rabbitmq"
+#                  key: rabbitmq-password