GOES16 retraining
Creating distributions and scoring their impact on ProbSevere has been delayed, mostly due to uncovering a bug in training.
After getting distributions for maxrc_emiss and maxrc_icecf, scoring was performed on ProbSevere with GOES16 and GOES-NOP predictors. I found a very large increase in FAR, which seemed very wrong to me. Upon much digging, I found a bug in creating the distributions, whereby the severe distributions used the maximum rcs, and the non-severe used every rc (note that this doesn't affect the GOES-NOP LUTs, as this was entirely new code created for GOES16).
So, the bug was fixed, and new GOES16 distributions created:
Edit, 12/15/2017: New maxrc_emiss_ABI distribution, adding in about 20 more days
Here are the GOES-NOP distributions, for comparison:
As you can imagine from the above distributions, the GOES16 maxrc_icecf doesn't really add anything. I scored ProbSevere(GOES-NOP), ProbSevere(GOES16), and ProbSevere(GOES16-maxrc_emiss only), for a large sample of independent data. In scoring these three, I found the ProbSevere(GOES16-maxrc_emiss only) to be the best. When compared to ProbSevere(GOES-NOP) (i.e., control), there is a very small increase in max CSI, of about 0.004 at the 70% threshold. The 10-40% thresholds had 0.01 increase in CSI. These scorings are for storms with maxrc_emiss > 0, only. The leadtime appears about the same for ProbSevere(GOES16-maxrc_emiss only) and ProbSevere(GOES-NOP), with 0-2 min decrease in leadtime for ProbSevere(GOES16-maxrc_emiss only), depending on threshold.
We are still looking into why maxrc_icecf does not appear to be skillful, and I think a lot of questions still exist into how much utility we have with these new data.
Edit, 12/6/2017: We looked at a few distributions for our training data set, with respect to background max emissivity, ice cloud fraction (icecf), for ABI and NOP severe storms (all during the checkout period in 2017 for GOES16).
This first set of images is the background max emiss11_tot, when the maxrc_emiss occurred (i.e., the time prior). We can see that this distribution shifts to the right a little bit and tightens for ABI, wrt to NOP. This is about what I expected, since our identification/tracking has a higher floor (40%).
This next set of images is shows the background max emiss11_tot prior to the storms' maxrc_icecf, for ABI and NOP. We see a very large shift of the distribution rightward, whereby the median is over 80% for ABI. This seems much higher than we expected.
This third set of images are just histograms of the background icecf prior to maxrc_icecf. We see that a much greater proportion of ABI storms start out with at least partial glaciation compared to NOP storms.
Justin and I looked at some storms on 20170515 and 16. It seems like the enterprise cloud phase is simply seeing a lot more ice than the goesim_cloud_phase algorithm. In particular, the enterprise alg. sees ice sooner than NOP alg. for growing storms, in general. Looking at some storms in the WDSS2 GUI bears this out, as well (see images below). Justin started digging into the enterprise algorithm, and found a number of tests involving the 8.5-11 beta ratio and the 6.7-11 beta ratio (I think). If any one of these returned true, the cloud is called 'ice'. So the presence of the 8.5µm channel on GOES-16 is principally what is resulting in more pixels being called ice (probably rightfully so); the higher spatial resolution might also be a contributing factor.
For delineating between thick ice and cirrus, there in an 'OR' test, with the first part being if the emiss11_tot was ≥ 0.40, it is thick ice. So, if we changed the glaciation rate to be conversion of thick ice, we don't think that would help us out much, since the threshold seems rather low. However, we surmise that perhaps if that threshold is higher, say 0.6 or 0.8, that might eek out a bit more value in this predictor. I guess this predictor would kind of become conversion rate to "really thick ice", or "confidently thick ice"? We wouldn't need to run the algorithm/GEOCAT again, since we have the full emissivity distributions. Justin also mused that this might simply be the conversion rate of emissivity to some "high" threshold (e.g., 0.4, 0.6, 0.8). Pixels would of course need to be called 'ice' first, but how many pixels at 0.4, 0.6, 0.8 are NOT called ice? Perhaps some at 0.4, but probably not many at 0.6.