-
David Hoese authored
For some reason it was at 8MB by default when it was supposed to be 128MB.
- Cluster Resource Administration
- K3s - Kubekorner
- Nginx Ingress Controller
- Local Path Configuration
- MinIO - Local S3 storage
- Upgrading existing MinIO installation
- Longhorn - Shared Block Storage
- Storage - Local Large Cache
- Storage - Local Medium Archive
- Configure HTTPS on Ingress
- Monitoring a cluster with Prometheus
- Customizing Prometheus rules
- Customizing Prometheus Alerts
Cluster Resource Administration
This directory includes Kubernetes resources that should be installed on
Kubernetes clusters that will have GeoSphere deployed on them. While there
may be local cluster builtin equivalents to the resources defined here, these
builtin names are not used in the current configuration in this deploy
repository. The builtin resources could be used instead of installing the
resources defined in this directory by updating the values-X.yaml
files in
the various directories and in the .gitlab-ci.yml
configuration file.
K3s - Kubekorner
Nginx Ingress Controller
At the time of writing, K3s comes with the traefik ingress controller with a version less than 2.0. It is our (geosphere project) that this controller is buggy and doesn't handle HTTPS certificates in an expected way. We've chosen to uninstalled the traefik controller and instead install the nginx ingress controller. It is possible in the future that newer versions of traefik (2.3+ is availabe but not supported by k3s) will not have the issues we've run into. It is also possible nginx will be used by K3s as an alternative ingress option.
The k3s FAQ includes the following:
How can I use my own Ingress instead of Traefik?
Simply start K3s server with --disable traefik and deploy your ingress.
After further research we discovered that additional steps may be required:
See https://github.com/rancher/k3s/issues/1160#issuecomment-561572618
For the record and future me, this is what needs to be done to disable Traefik during initial setup:
Remove traefik helm chart resource: kubectl -n kube-system delete helmcharts.helm.cattle.io traefik
Stop the k3s service: sudo service k3s stop
Edit service file sudo nano /etc/systemd/system/k3s.service and add this line to ExecStart:
--no-deploy traefik \
Reload the service file: sudo systemctl daemon-reload
Remove the manifest file from auto-deploy folder: sudo rm /var/lib/rancher/k3s/server/manifests/traefik.yaml
Start the k3s service: sudo service k3s start
Note the above --no-deploy
flag is deprecated and --disable
should be used.
Alternatively, k3s could be updated completely with the --disable traefik
flag added:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --no-deploy traefik --write-kubeconfig-mode 644" sh
Then nginx can be installed by following the instructions and settings described here: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install -n kube-system ingress-nginx ingress-nginx/ingress-nginx --set controller.metrics.enabled=true --set controller.metrics.serviceMonitor.enabled=true --set controller.metrics.serviceMonitor.namespace="monitoring" --set controller.metrics.serviceMonitor.additionalLabels.release="prometheus-operator"
Note the above includes enabling metric gathering for a Prometheus server. We enable the metrics endpoint on the controller, then enable the ServiceMonitor which is Prometheus resource that tells Prometheus about the metrics. We also add an extra label for kubekorner's particular installation of Prometheus so our ServiceMonitor can be found automatically.
Local Path Configuration
When running on a K3S-based (rancher) cluster like the one currently running
on kubekorner.ssec.wisc.edu, the local path provisioner should be updated to
point to larger storage paths. The K3S cluster software comes with a local
path provisioner as the default storage provisioner. This means that when an
application asks for generic storage (PersistentVolumeClaim), this provisioner
will be used to find and provide the storage. However, by default this
provisioner is configured to give access to
/var/lib/rancher/k3s/storage
which is typically space limited.
By modifying the config.json
stored in the local-path-config
ConfigMap,
we can tell the provisioner where storage should be provided from for each
node. See
https://github.com/rancher/local-path-provisioner/blob/master/README.md#configuration
for more information.
To apply:
echo -e "data:\n config.json: |-" > tmp.yaml
cat k3s-local-path-config.json | awk '{ print " " $0 }' >> tmp.yaml
# dry run
kubectl patch -n kube-system cm/local-path-config --type merge --patch "$(cat tmp.yaml)" --dry-run=client
# not dry run
kubectl patch -n kube-system cm/local-path-config --type merge --patch "$(cat tmp.yaml)"
MinIO - Local S3 storage
For easy data storage using an S3 interface we install MinIO on our K3s cluster. This will take advantage of the local path provisioner we configured above so that the storage has more than the couple hundred gigabytes of storage in the default location.
To do the initial MinIO installation run the following in the bash terminal on the cluster:
namespace="geosphere-test"
helm upgrade -v 2 --install -f admin/values-geosphere-minio.yaml --set accessKey=false --set secretKey=false -n $namespace geosphere-minio stable/minio
The values YAML file provides configuration information specific to this MinIO
installation. The accessKey and secretKey set to false
cause the helm chart
to generate random values for these. These values are then used to authenticate
to the S3 storage in the application. Because of this, it is important that the
"release" be called "geosphere-minio" as above so the various parts of this
installation can be found by the geosphere application.
Note, if your helm installation doesn't already have the stable chart repository added you may need to do:
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update
Next, we need to configure life cycle policies for the MinIO buckets so that they are automatically cleared of old data. On the cluster run:
namespace="geosphere-test"
ak=$(kubectl get secret -n "$namespace" geosphere-minio -o jsonpath="{.data.accesskey}" | base64 -d)
sk=$(kubectl get secret -n "$namespace" geosphere-minio -o jsonpath="{.data.secretkey}" | base64 -d)
curl -O "https://gitlab.ssec.wisc.edu/cspp_geo/geosphere/geosphere-deploy/-/blob/master/admin/abi-netcdf-bucket-lifecycle.json"
for bucket in g16-abi-l1b-netcdf g17-abi-l1b-netcdf; do
kubectl run -n "$namespace" --env=AWS_ACCESS_KEY_ID="$ak" --env=AWS_SECRET_ACCESS_KEY="$sk" --restart=Never --rm -it --image=amazon/aws-cli set-bucket-lifecycle -- --endpoint-url "http://geosphere-minio:9000" s3api put-bucket-lifecycle-configuration --bucket "$bucket" --lifecycle-configuration="$(cat abi-netcdf-bucket-lifecycle.json)"
done
Upgrading existing MinIO installation
If upgrading an existing installation of MinIO then we must make sure that we tell the helm chart what the existing accessKey and secretKey are or it will generate new random values for these and clients may become out of sync.
To do this, run the following in bash on the cluster:
ak=$(kubectl get secret -n "$namespace" geosphere-minio -o jsonpath="{.data.accesskey}" | base64 -d)
sk=$(kubectl get secret -n "$namespace" geosphere-minio -o jsonpath="{.data.secretkey}" | base64 -d)
EXTRA_ARGS="--set accessKey=$ak --set secretKey=$sk"
helm upgrade -v 2 --install -f admin/values-geosphere-minio.yaml $EXTRA_ARGS -n $namespace geosphere-minio stable/minio
Note, geosphere-minio
in the above commands must match the name of the release
from the original installation.
Longhorn - Shared Block Storage
Most cloud platforms have some concept of a shared block storage (AWS EBS, GCP Persistent Storage, etc). These can be mounted as normal volumes in our containers. Although our K3S installation has a local path provisioner these volumes are limited to one single node. We need another solution that shares the volumes between nodes. That's where longhorn comes in.
Follow the official longhorn installation instructions:
https://longhorn.io/docs/1.0.0/deploy/install/install-with-helm/
Unless newer versions no longer require it, on kubekorner we needed to install and enable a iscsi daemon:
yum install iscsi-initiator-utils
systemctl enable iscsid
systemctl start iscsid
If you have a particular mount on the cluster nodes that has more space than
the default /var
path, you may want to customize this setting. For longhorn
1.0 you can do this by adding --set defaultSettings.defaultDataPath=/data
to
your helm install command.
Additionally, if your cluster only has 1 or 2 nodes you may want to change the default number of replica volumes longhorn attempts to create. Otherwise, by default, longhorn's "hard affinity" will stop volumes from being created since it can't make all of the replicas (only one replica per node).
At the time of writing, kubekorner has had its longhorn instance installed with:
helm install longhorn ./chart/ --namespace longhorn-system --set persistence.defaultClass=false --set defaultSettings.defaultReplicaCount=1 --set persistence.defaultClassReplicaCount=1 --set ingress.enabled=true --set ingress.host="kubekorner.ssec.wisc.edu" --set defaultSettings.defaultDataPath="/data"
From the webUI or following longhorn's current instructions we can change most if not all of these settings. If a cluster with one node has more nodes added on in the future you may want to consider increasing the replicate count.
NOTE: The web UI does NOT have authentication by default. It must be configured manually. See https://longhorn.io/docs/1.0.0/deploy/accessing-the-ui/longhorn-ingress/ for more information.
Storage - Local Large Cache
DEPRECATED: See local path provisioner above.
This storage class and persistent volume can be used for cases where a
GeoSphere component needs relatively high performance and
large capacity storage. Both the StorageClass and the PersistentVolume
are defined in local-large-cache.yaml
. This storage is primarily used
for GeoSphere's tile cache (used by MapCache). It defines large storage
that is physically located/connected to the node where the pod is being
run or at least performs like it is. The term "large" here refers to
multiple terabytes (3-10TB). While this isn't large in generic storage
terms, it is considered large for a "cache" which is not guaranteed to
persist.
To apply:
kubectl apply -f local-large-cache.yaml
To delete (make unavailable):
kubectl delete pv/local-large-cache
kubectl delete sc/local-large-cache
Storage - Local Medium Archive
DEPRECATED: See local path provisioner above.
Similar to Local Large Cache above, but larger available space. Note this should only be used for testing as data will be deleted when the claim is removed.
Configure HTTPS on Ingress
Web services being served on the cluster via HTTP can be made available via HTTPS by enabling TLS on the Ingress controller of the cluster. The below instructions will walk through how to enable this.
First, we must create a Secret to store the certificates. For SSEC-based services, certificates should be requested from Technical Computing (TC). To create the secret, have the certificate file and key file available in your current directory and run:
kubectl create secret tls mysite-tls-certs --cert=mycert.crt --key=mycert.key
Where mysite-tls-certs
is the name of the secret, tls
is the type of the
secret, and mycert.crt
and mycert.key
are the actual certificate files.
Make sure if this certificate is for a specific namespace that you add
-n mynamespace
.
Then we need to make sure our Service definition includes something like:
tls:
- hosts:
- mysite.ssec.wisc.edu
secretName: mysite-tls-certs
Once this is deployed the certificate should now be used when requesting
the HTTPS version of your service. You may also want to add the following
to force users to be redirected to HTTPS from HTTP requests. This is what
it looks like in the values.yaml
file, but shows up in the metadata
section of the Ingress
definition.
ingress:
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
Note: this annotation applies to the traefik ingress controller and may not be the same for nginx or other ingress controllers installed on a cluster.
Monitoring a cluster with Prometheus
One of the best ways to fully monitor your cluster is to install Prometheus. Prometheus is itself a separate service for collecting metrics from various sources and presenting them to the user. One of the best ways to get this functionality on a Kubernetes cluster is by installing Prometheus Operator. Prometheus Operator will install its own custom resources definitions (CRDs) to allow other applications to create their own ways of interacting with Prometheus.
To install this on the Kubekorner K3s cluster we will use the prometheus-community prometheus stack helm chart maintained by the helm community:
https://github.com/prometheus-community/helm-charts
First we will create a namespace specifically for prometheus:
kubectl create namespace monitoring
If your helm installation doesn't already have the necessary chart repositories, they can be added by doing:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo update
Then we will install the helm chart in that namespace with the release name "prometheus-operator".
helm install -n monitoring prometheus-operator prometheus-community/kube-prometheus-stack
Also note at the time of writing this installation results in some warnings:
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
This is described in a GitHub issue here: https://github.com/helm/charts/issues/17511
Customizing Prometheus rules
In order to get the most out of Prometheus, it is a good idea to set up rules for alerts to send to the AlertManager servers created by Prometheus. We can then configure AlertManager to notify our development team of different conditions if needed.
First, we need to create a set of rules that we want to be notified about. To
configure these we create one or more PrometheusRule
objects. Here is an
example:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
creationTimestamp: null
labels:
app: kube-prometheus-stack
release: prometheus-operator
name: prometheus-example-rules
spec:
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
This creates an alert called "ExampleAlert" that is fired when expr
is true.
In this case vector(1)
is the equivalent of always true. The expr
is
a PromQL query that has access to any field recorded by Prometheus.
Normally these rules should be automatically picked up by the Prometheus
server(s) by matching labels
. By default, the Prometheus Operator installed
above will use the name of the helm chart for app
and the name of the helm
release for release
to match against.
To check, run:
$ kubectl -n monitoring get prometheus/prometheus-operator-kube-p-prometheus -o go-template="{{ .spec.ruleSelector }}"
map[matchLabels:map[app:kube-prometheus-stack release:prometheus-operator]]
Although a little cryptic, this is showing:
matchLabels:
app: kube-prometheus-stack
release: prometheus-operator
If the above yaml PrometheusRule configuration was stored in a example_rule.yaml
we could
deploy it by running:
kubectl create -n monitoring -f example_rule.yaml
If you've installed these rules in the past and would like to update them, use
the replace
command instead:
kubectl replace -n monitoring -f example_rule.yaml
To investigate if our rules are showing up in Prometheus we can forward the service to the cluster node and then forward that to our local machine with SSH. Note you'll need to use the name of your service in your installation.
kubectl -n monitoring port-forward service/prometheus-operated 9995:9090
If we go to http://localhost:9995/alerts
we will see the current alerts
Prometheus is aware of. We can click on "Graph" at the top and query the
Prometheus PromQL that we might want to use in our other rules.
We can do a similar check for firing alerts in the alertmanager by forwarding another port:
kubectl -n monitoring port-forward service/prometheus-operator-kube-p-alertmanager 9993:9093
And going to http://localhost:9993
.
Customizing Prometheus Alerts
Now that the rules should have been picked up, we need to configure the alertmanager to do something when these alerts are fired. The below instructions are one approach to configuring the alertmanager. The available methods are changing over time as the prometheus community grows the helm chart used above. Other solutions may involve ConfigMap resources or mounting additional volumes for alertmanager. The below approach is the simplest but does require "upgrading" the Prometheus Operator installation whenever it changes.
To configure how alerts are handled by alertmanager we need to modify the alertmanager configuration. Below we've embedded our alertmanager configuration in a YAML file that we will provide to our helm chart upgrade as the new "values" file.
alertmanager:
## Alertmanager configuration directives
## ref: https://prometheus.io/docs/alerting/configuration/#configuration-file
## https://prometheus.io/webtools/alerting/routing-tree-editor/
##
config:
global:
resolve_timeout: 5m
slack_api_url: "https://hooks.slack.com/services/blah/blah/blah"
route:
group_by: ["instance", "severity"]
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: "null"
routes:
- match:
alertname: ExampleAlert
receiver: "geosphere-dev-team"
receivers:
- name: "null"
- name: "geosphere-dev-team"
slack_configs:
- channel: "#geo2grid"
text: "summary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"
To upgrade the prometheus operator installation and assuming the above is in a
file called custom_prom_values.yaml
:
helm upgrade --reuse-values -n monitoring -f custom_prom_values.yaml prometheus-operator prometheus-community/kube-prometheus-stack
You can verify that the upgrade updated the related secret with:
kubectl -n monitoring get secrets alertmanager-prometheus-operator-kube-p-alertmanager -o jsonpath="{.data.alertmanager\.yaml}" | base64 -d
You should also see the config-reloader for alertmanager eventually pickup on the new config:
kubectl -n monitoring logs pod/alertmanager-prometheus-operator-kube-p-alertmanager-0 -c config-reloader --tail 50 -f