Add ability to run Geo2Grid profiling and observe diagnostic plots
I had a late night idea a couple days ago and even though I'm not sure if it is worth implementing in the long term I think I should at least write it down here.
Geo2Grid has the ability to create dask performance diagnostics by specifying the --create-profile
flag and an optional HTML filename to save to (default filename includes reader/writer names and execution date/time). This should NOT run on every execution of Geo2Grid as it would have major performance penalties. It also shouldn't change much if at all between executions once a single set of settings is chosen. These settings (like number of dask workers and which products to produce) don't change once Geo2Grid on GeoSphere is deployed.
So this flag produces an HTML file that embeds all the profiling data (CPU usage, memory usage, number of tasks, etc) into it and uses bokeh JS to visualize the plots. So we need to get access to this HTML file in some way and given the above limitations on when the profiling should be run this puts us in a weird spot. Even still, this could be important to reveal a misconfiguration or poor configuration that could lead to a better configuration and faster execution or better resource usage.
Possible solution(s)/parts for when to run the profiling:
- A cronjob that runs every N days (once a week at midnight?) with the same configuration as the regular Geo2Grid but in a separate Pod.
- Include the flag 1 time at midnight on a specific day of the week.
- Run once or twice on initial deployment from a separate Pod (Job as a helm hook)
Possible solutions for serving the result:
- Upload HTML to S3 and serve from special nginx Pod
- Upload to PersistentVolume for ephemeral (temporary) storage to be served by special nginx
- Write a special grafana exporter/converter for dask's diagnostics to upload to the grafana server. This is the best longest term solution probably, but by far the most work. It would also require converting bokeh plots to a grafana dashboard which may not be easy or even possible if the exact same plot elements aren't available.
- Don't serve them. Just download them. They are just static HTML files so even doing a copy from the container to my local machine would work.
- Email? I see some of just some true color generation with nearest neighbor resampling done on my machine that are ~44MB. Not great as an attachment.
- Upload to SSEC FTP to be viewed on bin.ssec.wisc.edu?