charles-chandler.org

This topic is for discussion of the graphical, online Tornado Outlook Contest at charles-chandler.org.

Here is an example of an outlook:

First of all, I'd like to say thank you for running this contest, I'm finding it enjoyable and a good way to test my own forecasting skills (and learn from what I'm doing wrong). I'm also participating in the WxChat.com Virtual Chasing Contest for the first time.

I'm somewhat impartial to the scoring idea you mentioned today; I'm really doing this, as I said above, for the enjoyment and evaluation of my forecasting. I do understand what you are saying about "all or nothing." At one point yesterday, the first tornado report in northeast central Kansas missed one of my polygons by 14 (miles?), while it was just inside another one (I think it was yours) by I believe even less than that distance. As a result, at the time (and if that had been the only tornado report between 18z and 6z), that gave my outlook 0 extra points (0% inside the polygon) and the other outlook 100 extra points (100% inside the polygon), even though the two polygons couldn't have been more than 20-25 miles or so apart in that location. This is probably a similar but different situation than what you mentioned with the small but accurate outlooks, but maybe it's actually an example of that.

I think what you mentioned would be good, with full credit for reports inside the outlook polygon and diminishing scores for reports outside the polygon. In the above example, it was 100 vs. 0 points with a less than 25 mile difference in outlooks. Like you said, all or nothing.

I had a possible suggestion for the scoring table on the main page - what do you think about putting the number of outlooks associated with each person's average?

Thanks for the recognition on my outlook for last Saturday. I would have done much better if I had gone with 18z instead of 0z for the valid time, so I was not happy with myself about that. I actually was watching the potential all week, and probably would have drawn an outlook for the same general area more than 78 hours in advance if I had the opportunity.

Nick

Here is a thought on a possible alternative to the scoring. Instead of using percent of reports inside the polygon and distance the reports are away from the polygon, and using your suggestion about giving full credit for reports inside the polygon, with points decreasing with distance from the polygon:

Give each tornado report (event) a maximum score of 100 divided by the number of reports (so they total to 100). For example, if there are 10 reports, each one would have a maximum score of 10. If there are 20 reports, each would have a maximum of 5. If there are 5, each one would have a maximum of 20 (tenths, hundredths, or thousandths of a point can be used for calculation purposes for numbers not divisible into 100, then the total score rounded to the nearest integer at the end). If a report is inside the outlook polygon, it gets the maximum number of points for that report. Otherwise, it gets a decreasing value defined by distance (i.e. it can be set that a report 100 miles or 200 miles or 500 miles, or whatever you decide, gets 0 points) - so if the 0-point threshold is 200 miles, and a report comes in 100 miles away from the polygon, it would get half the maximum points. Add up the points for each report and then get a score with a maximum value of 100.

Alternately, and this is really the same exact thing as above but stated in a different way, each report could have 100 maximum points. As above, a report inside the polygon would receive 100 points, and one outside would decrease by distance (in the example above, it would get 50 points). Then to get a score out of 100, an average of all the reports' scores would be taken. Again, this is the same as above, just the dividing by the number of reports occurs at the end instead of at the beginning, which may make calculations easier to understand.

The area and perimeter of the polygon, as well as points for lead time, could be retained. Just some thoughts.

OK, I got this done. To make it as explicit as possible (and easier to sanity check), I implemented the new "% nearby" factor, which scales from full credit within 1 mile of the poly, down to zero credit 100 miles away. Maybe this is a little too tight, but I can tweak that easily, now that the code is in place. Perhaps it should fare down to zero at 200 miles?

I also found & fixed a problem with duplicate event reports in the database, which was throwing off the scores (good and bad!). So I re-scored all of the existing outlooks, with the new algorithm, and with no dupes in the database.

Thanks for your interest! I'm a big believer in objective metrics, even when it comes to simple contests. So when I saw an absence of this kind of thing with respect to tornadoes, I got the idea to code it up. I'm truly a novice at forecasting, but I figured I could still make a contribution as a programmer!

Charles

The new scoring factor looks good, so that if an event misses your polygon by a few miles, you don't lose full credit for it. Thanks for implementing it! I realized that the proposal I made had a few issues to it. I'm not sure what the best distance would be, but I wonder how the scores would change if it was 200 miles. Any way you could determine how they would change without actually changing them, to see what differences it would make?

I was also wondering if something was wrong with the reports, because one score I had from last week had suddenly gone down to 20-some, and I couldn't figure out why. That explains it. It was a bad forecast with little time to investigate things - I had actually done much better on my 78- and 66-hour forecasts for the same time than the closer forecasts. Maybe I should stick to the longer-range ones? I'm glad to see that my attempt at a nighttime, 12z outlook three days in advance, instead of a 0z forecast, paid off extremely well. There were just so many ingredients coming together for the second half of the week, just as there were for the week before.

OK, I implemented a blue polygon that is offset 100 miles from the outlook poly. This way, you can just look at it and get a general sense for how often near misses are inside/outside it. The offset routine isn't firing on all cylinders yet (it's choking on concave vertexes), but enough of it is OK that you can get the idea. Remember that the credit fares from 100% to 0% from the green to the blue poly. I'll turn this blue poly back off once we've made up our minds, since it's a bit distracting. But I figured that this would be the best way to evaluate the distances.

I said earlier that I think that 100 miles is too tight, but at 200 miles, the faring poly would be larger than a number of the outlook polys. Hmmm... Anyway, let me know your thoughts.

BTW, what model maps do you use? I'm such a newbie (and I never took any classes for this), that I'm not sure I'm even using the correct info. I just look at the Unisys 850 mb, 300 mb, lift/RH, and CAPE maps. Where can I find future Total Totals and K Index maps? Unisys just has the current WXP output. I'd also like to find a helicity map. Right now I just kinda guess at it by flipping through the 850/500/300 mb maps.

Speaking of helicity, what the heck happened 2010-05-03? I thought for sure that front was going to produce another outbreak. Was it just that there was no helicity? Or not enough of a collision? The cold air side was weak, but I still thought that there would be enough convergence to trigger storms all along the front. (I'm a newbie -- what can I say? )

I'm liking the blue polygon, except in those concave places. So, if there is a report of a brief tornado in your polygon, you get 100 more points than you would if there was a funnel cloud that just barely missed touching down, or a tornado that no one saw or reported?

I'm not sure about the distance yet. I know what you mean about 200 miles... I think 100 is good, but more test cases would be good.

A good place to look is the Short Range Ensemble Forecast Models. Not only is it designed for the short term, but it is comprised of ensembles, so it can show you the range of the same model ran multiple times with slightly different initial conditions. Best place to view it is at http://w1.spc.woc.noaa.gov/exper/sref/ (on the Storm Prediction Center website). Helicity is included there, but the Total Totals and K index are not. I'm not really sure a good location for those, I don't look at those, honestly. For tornadoes, I do like the Significant Tornado Parameter, which is found on the SREF. It's a combination index using helicity, CAPE, CIN, LCL height, and 0-6km wind shear (http://w1.spc.woc.noaa.gov/exper/mesoan ... _stor.html).

I honestly don't remember May 3... it seems I didn't place an outlook for that day, but I can't tell you why because I don't remember what it looked like.

Can you add this tornado to the scoring?

http://www.erh.noaa.gov/cle/wx_events/2 ... terisk.php
http://www.spc.noaa.gov/climo/reports/100507_rpts.html

OK, I got that fixed. The routine was set to import any tornado reports added in the last 3 days. I figured that after 3 days, anything that is going to be reported has already been reported. But not so, as that report came in even later than that after the fact. So I set it to import any new reports from the last 10 days.

Thanks for the links to the other forecast models -- I'll start trying to figure out what that stuff means, as time permits.

This topic is getting locked. Further discussions should be on one of my other boards:

http://scs-inc.us/Other/QuickDisclosure/?top=5660