Reflection on 26 April 2016

Some reflections on "busted" forecasts and some quotes from the Bosart Symposium.


From Bosart (2017): 

1. "You have to be honest about the forecast."

2. "You aren't going to get better if you aren't honest about where the problems are."

3. "The atmosphere has a way of making fools of us when we think we have it figured out."

From the latest Freakonomics podcast:

4. "... [this is a] reminder of how humble we should all be 

when we make a prediction and when we hear one."

The challenge from Rick Smith (2017):

5. "How can weather messages provide the proper balanced information to promote rational fear/preparedness and reduce the degree to which we scare people?"

I used these quotes because they all tie into one event on 26 April 2016.

Facts:

1. The word tornadoes and "outbreak" appeared early in the forecast cycle of SPC day 4-8 product suite. To achieve consistency, many in our community began to message the same.

2. The event was ramped down for a period of time as concern grew it may not end up being a big tornado day but still a big severe weather event. The day of the event, a PDS tornado watch emerged amidst the impression of a ramp down.

Here is the outlook sequence:

Since forecasts out to day 7 and even day 4 are often difficult, depending on details that are in no way shape or form present (or perhaps predictable) in the forecast, we have implicitly accepted pattern recognition to steady the forecasts from flip flopping every model cycle.

One aspect I find interesting is the relative small-ness of the day 4-8 risk areas. This usually stands in contrast to the larger areas of the Day 1 and 2 forecast. There is good reason for this - confidence or certainty or willingness to go out on a limb. This is arguably a conservative approach. In much the same conservative way, the forecast areas get larger as the event approaches. This might seem counterintuitive. It took me a while to understand this tendency. If you draw the risk areas too fine, you will miss. There are just too many details, too many ways to be humbled by the atmosphere.

In many ways these conservative approaches allow for consistent forecasts, fine tuning but not fine tuning too much. This is evident on the maps, for which we have verification information.

While conservatism works for spatial forecast areas, how can it be used conceptually in communication, to foster the same style of consistency?

The use of the terms outbreak or tornado should be rare at long lead time. But they should be used when they need to be. We have to balance the intuitive nature of the forecast with the available evidence. Especially because we use pattern recognition methods to counter the unpredictable, specific fine details that ALWAYS complicate mesoscale convective weather forecasts.

Forecasters should have the space to express their forecast judgments, as in Murphy (1993).  As such we should hold forecasts accountable to verification (quality) and value. A forecast can only be valuable if it is issued at times where it can be used and be useful. To be useful requires consistency, because a good forecast surrounded by two equally bad forecasts .... is a signal buried in noise.

In many ways, when I look at SPC forecasts I can see a few different kinds of days. 

Some days are tornado deadly (27 April 2011).  The environment is primed for supercells for a prolonged period and the storms produce many tornadoes for a long time. Probabilistically, this is much more dangerous and perhaps more obvious.  Many days are deadly because of a brief window of time for locally one tornado (20 May 2013). You have all the ingredients, you have the storms, but its just one or two storms that do most of the damage.

I find these days to be justifiably intense, where forecasters intuitively are willing to be on the higher risk side than not*. "It could be as high as this" in the snow experiment parlance. A worst case scenario.

Then there are types of days where "at least this can happen" to "this is the most likely outcome".  Sometimes these are the hardest days because they contain just enough signal to emerge from the noise, but no clear indication where or when to expect the worst...if it even happens!

In this way I find the communication to be THE challenge. Because it is not a simple endeavor to categorize these events prior to their occurrence. Yet intuitively, SPC has developed an expertise, developed heuristics, and in a sense, has the capital to take risks ... to send communication cues, not just in their discussions and graphics, but in the way they think about making these forecasts. And this is hard to communicate. In other forecast venues, confidence has been suggested as a piece of information.

The last thing we want is to stifle that creative, intuitive culture. Instead I think we want to extract more information from forecasters to understand what they know intuitively.

So, as Rick Smith asked and challenged us, how do we execute these forecasts without stirring anxiety? Good question and at the same time I don't think we know.  Tornadoes should cause some anxiety, after all they are dangerous. And they are rare**.  Even rarer are outbreak days. And on those days, we have maximum loss of life. So the stakes are higher, and the motivations to communicate these days as outbreaks has the maximum possible reward -- because outbreaks are THE DAYS to save lives!

Lastly, social science work on warnings has been stressing to us that in a crisis people need to personalize the risks, that this event could effect them. While outlooks mostly fall outside the range of a crisis, they are in an unenviable position of creating one -- for days. We know that people need TIME to PERSONALIZE the risk. How much time is enough? Is there too much time?

The goal is to give you time to prepare, time to act, and ultimately time to save your life.  While no one intends to scare, we have to be honest about the forecast (mentioning tornados) and honest that it has the power to scare. As careful as we are to use language commensurate with the threats, there will always be just enough uncertainty to humble us. But we don't want that uncertainty to prevent us from identifying those days where lives can be saved. Erring on the side of caution as the stakes get higher has some trade offs, some serious ones. We have to keep working together to find a way, a method of communication that can get us through that anxiety ... alive.

- - - - - - - - - -

*Also relevant, this is what makes it hard to verify forecasts using only objective methods. The subjective importance of some deadly or impressive events, can and should take precedence in our verification. Risk is a social construct. The penalties are asymmetric ... its more important to get a dangerous forecast correct than it is a benign forecast. As such our evaluation of value naturally hinges on getting the right forecasts correct.

**About 20 killer tornadoes occur per year compared to 500 tornadoes rated EF1+ per year, while on average 14 days per year with a killer tornado occur out of 110 days per year with an (E)F1+ tornado.