Investigate mapserv.fcgi OOM
Prometheus alerted us to a process being killed because it was out of memory. @jstroik check the logs for gamma-ait2 where this OOM occurred and found:
Oct 12 17:25:33 gamma-ait2 kernel: Memory cgroup out of memory: Killed process 100231 (mapserv.fcgi) total-vm:550788kB, anon-rss:455688kB, file-rss:26520kB, shmem-rss:0kB, UID:33 pgtables:1060kB oom_score_adj:997
Oct 12 17:25:33 gamma-ait2 kernel: oom_reaper: reaped process 100231 (mapserv.fcgi), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
If we check the grafana logs for memory usage of the mapserver pod:
This memory usage is periodic and the actual OOM killed process (mapserv.fci) at 17:25Z only resulted in a small reduction in memory usage.