Rantin' Bout Ratings

[MTL21676]

My latest blog entry.

Enjoy

http://mtlscienfiticdiscgolf.blogspo...t-ratings.html

[gotcha]

From MTL's latest blog entry on player ratings:

"As the creator of the now standardized rating system the PDGA uses, Kennedy was able to seemingly fix one of the main problems within disc golf that ball golf never had an issue with; standardization of par."

* * * * * * * * * * * *

Standardization of par?

[MTL21676]

What about that doesn't make sense Jerry?

[gotcha]

I have never considered the Player Rating system a "standardization of par".......two separate things (imo).

[MTL21676]

It's not. This is why I used the word "seemingly" and went to explain how it to some degree standardizes par b/c you can, to some degree, compare an SSA 49 course to an SSA 54 course based on the rating.

Thanks for reading, btw

[gotcha]

Is too!

I now understand what you were trying to say in your blog, however, I disagree that the player rating system is a "standardization of par". Player ratings won't tell me who's in the lead after nine holes of play during a final round......a player's score in relation to par does. Player ratings can change from day to day, year to year.....course par does not.

[MTL21676]

I agree.

But as I said, I really think of the primary benefits of ratings is like what I talked about with the golf course / disc golf course example and was going along the lines of a lot more people understand I shoot 990 at Cedar Hills than they do I shoot 50.

[gotcha]

Ideally, I would like to see the implementation of a course rating system in addition to the current player rating system. It would be difficult and time consuming to create a course rating system (particulary on courses where multiple pin placements result in par/SSA differentiation), however, a static course rating would nicely compliment the player rating system which is currently exclusive to PDGA membership.

[jebbeer]

Like the name drop MTL, good article.

[cgkdisc]

It's apparent he still doesn't understand the ratings or he wouldn't have written the second half of the blog that way. Those who understand ratings do understand why the way we do it is better and in fact the only functional way to do it. But he's young and has many more years to eventually understand.

[MTL21676]

Perhaps my favorite example this came on at Glenburnie Park in New Bern, NC. This past year on the short tees, I put together my best personal round of 46. Mike Hofmann, now rated 1005, matched this round as well. However, North Carolina's newest 1000 rated player, Jeb Bryant, smoked the course and fired a course record 41. Since I had shot a 49 at a previous tournament on this course in similar conditions and received a round rating 1008, I was anxious to see a round rating of possibly 1040, which would be my third highest ever. However, since Jeb and others played really well that round, my 46 was rated LOWER than my 49. Exact same course. Same conditions. Three strokes better. Two points lower.

But he's young and has many more years to eventually understand.

I really don't wanna understand something like that....

[cgkdisc]

The occasional scenario like that is miniscule in comparison to the error induced from weather variations every weekend using fixed SSA values like golf, let alone the impossible data management tasks involved in trying to have TDs figure out what SSA should be used in the report when more than 1/3 of the events have new courses or temp configurations with no SSA value. TDs have hard enough time doing the work to make sure players are in the proper division. Fixed SSAs would also mean no unofficial ratings during or right after the event. Ratings would only be provided during official updates.

Since you are forced to determine SSAs from propagators regularly anyway in order to figure out the official fixed SSA value, it's better to be consistent and do it that way for every round. Fixed SSAs would have brought the data management system in 1999 to its knees before it got off the ground and more divisions and players would not have gotten ratings. In fact, the way we did it was the only way ratings would have gotten underway since Roger and I did it free for five years with little PDGA backing. And we wouldn't have done it with fixed SSAs because the process would have been unmanageable.

[pgyori]

But he's young and has many more years to eventually understand.

I really don't wanna understand something like that....

[/QUOTE]

Yet it makes sense to you that a 54 on a particular course, on a sunny, calm day should be rated the same as a 54 on the same course in pouring rain and high winds?

Do you think it remotely feasible to accurately identify the SSA for each course, with varying pin positions, tee pads, and an ever changing fairway? After all, most disc golf courses evolve with time, as trees grow and die. Holes change and the course rating would be outdated in no time.

Comparing a player's average to the field provides the best meaningful rating for that day. It is certainly not indicative of absolute performance, and you certainly risk anomolies that you describe above, but other systems would be more flawed and less meaningful.

I do disagree that outliers shouldn't be included in your ratings (they should be), and that bailing on a bad round should help your rating (it does, but it shouldn't), but the overall premise is sound.

[MTL21676]

The occasional scenario

That made me laugh.

[cgkdisc]

There will absolutely be some variances like the one you point out. The good news is that because it's a statistical process, you get just as much 'good luck' as 'bad luck' so it won't matter overall. In fact, the odds of getting the exact same SSA within one decimal place on the same layout with the same players in the second round is less than 10%. That's why we now average the two rounds exactly for the reason having fixed SSAs is good - so players get the same rating for the same score. It doesn't take long to go thru the sequence of what would be required to try a fixed SSA process to see why you should thank your lucky stars the way we do it is possible. But even with your TD experience, you won't acknowledge that impossibility.

[MTL21676]

I actually have said many times that doing the ratings for all rounds played one layout was a good thing and a step towards doing them the right way.

However, as soon as I said that I played in perfect conditions on Saturday and 50 MPH winds on Sunday. This lead to a very high rating Saturday and a very low rating Sunday, so I guess it averages out.

[cgkdisc]

It's up to the TD to tell us not to combine the rounds.

And you think they should review a database of fixed SSAs and figure out whether an exisiting value is the same or different and how much to adjust it for the temp hole and the big tree that came down? And remember that a pin placement was hit by the mower and moved over. Then figure it out for the divisions playing different layouts and then adjust for wind factor we'd provide them to determine on a graph?

[Jeff_LaG]

It's apparent he still doesn't understand the ratings or he wouldn't have written the second half of the blog that way.

That's what bothers me the most. Ignoring the whole dynamic SSA vs. fixed SSA argument, I look at the two paragraphs that begin with "Here is another level of stupidity about the ratings" and "It gets worse" and it's obvious Robert doesn't even understand how the PDGA ratings system works.

A responsible journalist would have done more thorough research before putting out such an article, and similar to his "PDGA broke its own rules" blog post, I think what we have here is another article which relies on sensationalism to attract readers.

[MTL21676]

Three shots better. 2 Points worse.

I can not repeat this phrase enough.

And the crazy part is Chuck will be like "yeah that happens but its rare" but there are examples ALL over the place.

Here are the stats on the four courses played in my last tournament. This four things SCREAM consistency don't they....

http://www.pdga.com/tournament/cours...ngCourseID=343

http://www.pdga.com/tournament/cours...ngCourseID=102

http://www.pdga.com/tournament/cours...gCourseID=1587

http://www.pdga.com/tournament/cours...ngCourseID=134

[gotcha]

Three shots better. 2 Points worse.

I can not repeat this phrase enough.

I guess it's really not a "standardization of par" then... [img]/msgboard/images/graemlins/smirk.gif[/img]

[cgkdisc]

3 shots variance on a 50 SSA is still only 6% variance. That's about the difference in sales tax. First, it's statistically uncommon and you can just as easily get a benefit the other direction some other time. Many player's ratings precision expectations are out of line because each point seems so precise. But one rating point is just 0.1% variance and a cut thru can be 10 points or 2% variance in your score and one centimeter variance in a single throw at USDGC can produce a 3% variance. The SSA variance is less than those common occurrences per round and insignificant overall due to averaging. However, fixed SSAs if actually known are ALWAYS wrong in the same direction due to no weather adjustment which would occur in more than 1/3 of the events each weekend but impact certain regional areas more than others.

[MTL21676]

I don't know why people think I am in favor of a fixed SSA across the board.

Of course if I play a course in rainy conditions that course will play more difficult than the exact course in perfect confiditions - that I am not arguing one bit.

However, there needs to be a way to alter from the fixed SSA of the layout when weather is bad and I don't think basing it on peoples ratings is the way. I think if the course info had something along the lines of - Joe Doe Memorial Park - 30 MPH Winds from south instead of Joe Doe Memorial Park Joe Doe Memorial Tournament Round 2 - that could be a step in the right direction. The first example provides an explination to an outside eye why the course was that difficult that day. The second example and current use only is usefull to those who played the tournament or have access to the TD report (And since you love stats so much, Chuck) or only roughly .005% of the current membership. That is a pretty usefull stat - only .005% of PDGA members can explain why a course played 2 strokes harder that day.

The problem with the basing it off of what everyone shoots, I feel, is that when the conditions are consistent from tournament to tournament, much like the example I have listed in this thread, then players are penalyzed when the field plays good.

If ratings were in essence nothing more than just something we did as a sport for fun, I wouldn't care much. However, due to the many things I listed in in the blog, that is not the case.

I feel if you polled all PDGA members you would find that at least 50% (probably closer to 60%) of them think the ratings system is flawed. The main problem with that I think is how Chuck presents his arguments.

He is never wrong and never will be wrong.

[cgkdisc]

I feel if you polled all PDGA members you would find that at least 50% (probably closer to 60%) of them think the ratings system is flawed.

I would be in that 60% and have identified the challenges. But the alternative is worse. Likely 90% think that the tax system is unfair in some areas.

The key problem with fixed SSAs is there's no basis upon which to determine their values. And we know how much agreement there is on determining par which is much simpler in theory. This is the key problem with ball golf course ratings - no objective way to do it. Second, there's no proof that a course plays with the same challenge in the morning and afternoon under the same conditions. There's no way to verify that it does play the same or it doesn't.

[scottcwhite]

MTL - you presented the problem with the current ratings system, which has some valid concerns IMO. However you fail to present a feasible solution. CK seems to argue that yes the ratings are not perfect but they work under most situations and it's easy for ALL tournaments to have ratings no matter how dumb the TD is.

Unless you are offering a solution then you are just complaining. What do you want to change how do we do it?

Dave Feldberg is the highest rated player and also the best at disc golf. I think that indicates that the ratings work at some level.

[MTL21676]

did you not read 2 posts up?

[scottcwhite]

Your proposed solution is to have TDs list weather conditions as part of the course info?

How many TDs will actually list weather? How do they know what to list? 5mph wind with 20 mph gusts for 20 seconds at a time, followed by a 20 min period of rain and then sunshine for the remainder of the round... you can see where that goes.

"step in the right direction" - maybe. What direction?

[janttila]

I think Mr. EsDubYa is presenting some skillfull moderating work here or in other words MB facilitation. You facilitate meetings and productive conversations for a living or something?

[bobsted]

You complain a lot about ratings, but how close do you think a players overall rating is to there actual level? The good players around here have high ratings and the bad players have low ratings. Yes, every round rating is not an exact measurement but nothing ever will be. It sounds like your complaint is that your rating is based on your play against other players and not based on how well you played. This is true of any ladder system which is basically what ratings are. If you understand the whole process, then you mus be pretty smart guy because it is some pretty complex math. So as a smart guy, why don't you come up with another option.

[MTL21676]

I def. do not understand all the numbers and the calculations but I do understand the overwhelming complaining I hear just about every weekend about the ratings.

There is no possible way for me, with no background in statistics, to develop such a system.

The only thing I can do to help is point out the things I see as problems and hope those with knowledge of how to fix them can. If I am seen as a complainer, that is fine by me. I know what I am doing is standing up and voicing my opinion about something I don't agree with and I bet more than likely most people are with me on that.

[bobsted]

Your first complaint is that your rating is based on how the players you played against that day did and you think it should be based on course history. The problem with comparing scores against previous tournaments is the conditions are different. You even use an example saying exact same conditions. Personally I doubt they were exact. Could you at least give us the dates of the tournaments, the temp., the average wind speed, and the humidity for the 2 tournaments. Or were they all the exact same? To start using these pieces of date in the rating equation a lot of data would be needed to see how each effects the scores during rounds. If enough data was gathered trends would probably exist that could be put into an equation to alter the SSA for a course. Courses also change when trees fall and trees grow. These I don't think could be measured so that would always be a problem. Also, as better discs come out courses get easier. I know players at my local course are shooting a lot better scores then they did 10 years ago. I am not sure how you would put that into the equation.

Your second complaint is about 20 players a 1000 player, a 990 player, a 980 player, ..., down to 800 (which fyi is 21 players). I am not sure I understand your argument, but I will try and address it anyways. You are saying complaining that it bases the SSA off of the idea that the 1000 rated players ability is 1000 and the 990 rated players ability is 990, etc. Yes, that is a problem if a player is a lot better than your rating or a player s injured and a lot worse than their rating. I think any system that bases the SSA on how players shoot and what their ability is based on previous rounds is going to have this problem. Even looking at previous rounds at a course would have this same problem. The overall percentage of these players would be the same, but yes it would not affect one tournament as much as does now. If you play enough tournaments it will average out and your overall rating will be the same.

Your third complaint is high ratings on courses with SSA 50 -54 and low ratings on courses with SSA over 60. I had to think about this quite a bit and I still do. I am not sure how you would fix this. Without varying the amount that each stroke is worth. The problem is the number of strokes between a 1100 rated round and a 1000 rated round is the same number of strokes between a 1000 rated round and a 900 rated round. You take a bunch of 900 rated players see what they shoot. Take a bunch of 1000 rated players see what they shoot. Add the difference to 1000 rated players average and that is 1100 rated round. Why players don't shoot that score on hard courses I am not sure. I would guess is the more throws that are involved the bigger the difference between these 2 groups because the 900 players are just so inconsistent. The bigger the difference that just means the more strokes a player has to shoot better than a 1000 rated round to get to 1100. On these hard courses it is just impossible to get there. Now that I have explained why this is I guess the question comes, why does this matter? I would guess that if a player played all easy courses their overall rating would be the same as their rating would be if they played all hard courses. They would have a bigger range of ratings on the easy course, but the average will be the same. So your complaint is just that the best rated round of all time will be on easy course? You know the best round in PGA history is on an easy course also, so guess that is just the way it is. If you wanted to figure out what was the best round in PDGA history, it would not necessarily be the highest rated round. I would have to think about it, but an equation could be written to compare rounds based on the SSA and the rating.

Your last complaint is that no one outside the sport knows what a rating is. Personally, I take my rating take the difference from 1000 divide by 10 and tell people that is my handicap. Anyone who is in a golf league totally understands that. Your comparing of rounds of 1010 and 1020 to an non player and trying to explain why one is better than another. Wow you must bore people with this story. I already addressed the issue of the round be rated without looking at other rounds. Also, in ball golf you can't totally compare scores from different courses either. Otherwise after the US Open I would be telling everyone I should of played, because I shot even at my local course that day. Or should we start explaining to non golfers that there are different tees, some greens are fast and some are slow, and some rough is rough and some rough is really rough.

I just thought I would add a few of my thoughts to your complaints. Note, I do not think that the rating system is perfect. I think it does a good enough job that the PDGA should stick with it. Actually, I was disappointed to here that each round doesn't stand by itself anymore. I understand that the more rounds the more accurate the rating, but I just think there are too many other factors involved as listed above to make this fair.

You do come off as a complainer, but it is good that it is fine for you. And yes most people are like you in that they are not good at statistics, would not know how to specifically improve the ratings system or make something better, and yet they complain about it anyways. [img]/msgboard/images/graemlins/smirk.gif[/img]