Forum Search:
RSD No Spam
rec.sport.disc without the spam


Home » RSD » RSD Posts » Nationals Bids Chaos (Serious flaws in the USAU bid distribution system and a possible solution)
Nationals Bids Chaos [message #111789] Wed, 07 March 2012 16:58 Go to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
The Bid Distribution System that USAU plans to use this year is very flawed. It was flawed last year, but by dropping the 4 "region strength" bids and using only "team strength" bids, they have made it worse.

Right now it is only the cutoff between the #20 and #21 team that determines how many bids a region will get (assuming all 10 regions have at least 1 team in the top 20, but that doesn't change the main point). Let's say a region has 4 teams ranked #4, #13, #22, #28. Teams #4 and #13 are encouraged to throw games to #22 and #28 so that as many of the 4 teams can rise into the top 20. The loss of ranking for #4 and #13 will have no effect on bids, until one of them drops out of the top 20.

Now, I think that dropping the region strength bids is good, because most regions are not 8 teams deep (at least not when you consider the 8th team's ability to compete with nationals level teams), but they need to replace the way in which the 10 strength bids are allocated. As it stands now, the system is very vulnerable to being gamed and can strongly incentivize teams to lose/throw games to their opponents. Additionally, I don't think that the bid distribution that results will be the most fair, but it is the lack of tolerance to gaming that is most important right now.

I have worked with a few friends to develop an alternative system named Top30, which is described below. My current plan is to submit an official proposal to the USAU, but I would like to submit it to the general community for comment and feedback first.


Goals/Assumptions:

• Teams should not be able to game the system by intentionally losing games.
• Each region deserves at least 1 bid (autobid).
• Any ranking algorithm will be imperfect, but we are assigning bids, not spots to nationals. The teams still have to win on the field.
• Teams 20-30 are very much in play in the competition for nationals spots. This should be reflected in the bid distribution system.
• The current system has a hard cutoff between the last team (team ~20, depends on if all regions have a top 20 team) and the next team (~21). A gradient is strongly preferred to a hard cutoff.


How it works:

• The top 30 teams are assigned points by their ranking. Team 1 = 30 pts, Team 2 = 29 pts, Team 30 = 1pt, Team 31 = 0 pts...
• A region's point total is calculated by summing the points of their teams.
• The region with the most points gets the 1st strength bid. 30 points are subtracted from that region's total and the point totals are recalculated.
• The 2nd strength bid is awarded. 29 points are subtracted from that region and the process is repeated until all of the strength bids are allocated.
• Points are subtracted from a region's total according to the round of strength bid that they acquire; 30 pts for the 1st bid, 29 for the 2nd, ... , 22 pts for the 9th. *(see note in "comments and notes")
• In the event of a tie, the bid is awarded to the region with the highest "remaining" team. If the region already has 2 strength bids, you would look at their 3rd highest ranked team.
• A region cannot have more total bids than teams in the top 30 rankings. If a region has no teams in the top 30, they will receive exactly 1 bid (their autobid).** (see note)


Comments and Notes:

• Teams cannot intentionally lose games since the rise is ranking for the lower team is balanced by the loss in ranking (and points) for the better team.
• The strength of teams 21-30 is factored into the bid distribution.
• *Originally I intended to remove the top team from the winning region and their associated points with each iteration. However, this caused the problem that top teams are again encouraged to lose to 2nd and 3rd teams in their region, since once the top team's points are removed, the lower ranked teams' point total is now higher. By subtracting a fixed point total with each iteration, the value of the top team's points is always counted.
• ** A region that has one great team (i.e. #5) and no other good teams, could theoretically obtain a strength bid on the #5 team's points alone. This clause is added to prevent a region with only one good team from getting 2 bids to nationals.


I ran last year's regular season rankings (4/8/11) through the Top30 system, and this is what I got (counting strength bids, not total bids):
GL:1 NC:2 NE:1 NW:2 SC:2 SW:2

Last year's actual strength bids:
NC:3 NE:1 NW:3 SC:2 SW:1

If they had used this year's algorithm with last year's rankings:
NC:3 NE:1 NW:3 SC:2 SW:1 Turns out to be the same anyway. (Glad I spent 5-10min working that out).


-Ariel Jackson
Re: Nationals Bids Chaos [message #111795 is a reply to message #111789] Wed, 07 March 2012 17:59 Go to previous messageGo to next message
Ryan Thompson
Messages: 364
Registered: September 2008
Senior Member
There are two main issues that this plan tries to address (I'm one of
the aforementioned "friends" who helped Ariel develop this):

1) The current ranking system is imprecise and cannot accurately rank
the teams in a way that justifies granting a bid to the 20th and not
the 21st based solely on ranking. The system does a good job of saying
"Team A ranked X is better than Team B ranked X+10", but not "Team C
ranked Y is better than Team D ranked Y+1".

2) The current system incentivizes intentionally losing to a worse
team in order to secure an additional bid for the region. See Harvard
vs Tufts last year (NE probably would have secured a 2nd bid even if
Harvard beats Tufts 15-9 instead of vice versa, but the incentive is
still there). This is a real problem, not an imagined one.
Re: Nationals Bids Chaos [message #111797 is a reply to message #111795] Wed, 07 March 2012 18:54 Go to previous messageGo to next message
Krishna
Messages: 98
Registered: October 2008
Member
Nice work.

Also, beyond the issue of how bids are allocated. There are serious problems with the ranking system itself.

For example, not counting forfeits in the rankings makes no sense. It incentivizes teams to run tight lines early in a tournament to pick up wins, and then forfeit late games with no penalty.

It also dramatically punishes any team who plays far inferior competition. The best a team can do is perform 600 points above an opponent. That means for example, one of the worst games (according to the rankings) UNCW has played this year, is when they beat their B team 15-1.





Re: Nationals Bids Chaos [message #111799 is a reply to message #111797] Wed, 07 March 2012 19:37 Go to previous messageGo to next message
sswang
Messages: 29
Registered: June 2010
Junior Member
I totally agree that the rankings are not accurate enough to justify giving the last bid to nationals based on the rating of one single team compared to another team. I brought up similar concerns in the fall: http://boards.usaultimate.org/showthread.php?994-Team-Streng th-Bids

Not much response from USAU, unfortunately.

Krishna, I think your concern about what happens when a team plays vastly inferior competition is actually taken care of. It doesn't say so in the algorithm page, but I remember reading RSD a few years back when someone said that if there is a game in which the maximum differential is reached, it doesn't count in the averaging if it hurts the winning team (and, conversely, benefits the losing team). I sincerely hope my memory is correct, because you're right, it would be awful if this fix was not implemented.

There are several other unaddressed issues with the algorithm, though. The two biggest are the non-convexity of the differential, which means there is a greater difference in ranking between winning 13-7 and 13-6 than between 13-12 and 13-11, and a dependence on the date of the running of the algorithm - meaning that if you run the algorithm on one day, you'll get a different set of rankings than if you run it on another day, even if there are no new scores reported. I wrote this up last summer and submitted a proposal for ranking system amendments to USAU but whatever committee they have to deal with it was unable to get around to field-testing new ranking algorithms.
Re: Nationals Bids Chaos [message #111801 is a reply to message #111789] Wed, 07 March 2012 20:21 Go to previous messageGo to next message
mgd.mitch
Messages: 1207
Registered: January 2009
Senior Member
Jackson wrote on Wed, 07 March 2012 19:58
Now, I think that dropping the region strength bids is good
I'm a little curious why you would say this given you are suggesting moving almost completely to a regional strength system, you just changed the averaging process. I'm not saying your suggestion is bad or flawed, but it is absoluely a regional strength system, not a team strength system.
Re: Nationals Bids Chaos [message #111802 is a reply to message #111799] Wed, 07 March 2012 20:16 Go to previous messageGo to next message
Ryan Thompson
Messages: 364
Registered: September 2008
Senior Member
It doesn't appear that the "fix" was implemented - Adam Lerman
calculated Whitman's rating last year and it definitely incorporated
15-3 wins over UPS-B by lowering Whitman's score. Matt Bourland from
USAU said that the intended fix created other issues with the ranking
system, so they didn't do it.

But if they are going to keep a flawed ranking system, they shouldn't
base bids around it as if it's perfect.
Re: Nationals Bids Chaos [message #111803 is a reply to message #111799] Wed, 07 March 2012 21:27 Go to previous messageGo to next message
Krishna
Messages: 98
Registered: October 2008
Member
sswang wrote on Wed, 07 March 2012 22:37
and a dependence on the date of the running of the algorithm - meaning that if you run the algorithm on one day, you'll get a different set of rankings than if you run it on another day, even if there are no new scores reported.

...

I wrote this up last summer and submitted a proposal for ranking system amendments to USAU but whatever committee they have to deal with it was unable to get around to field-testing new ranking algorithms.


Yeah I noticed the timing issue, but didn't have a quick way to see how much it mattered.

What was your proposal for amending the ranking system?
Re: Nationals Bids Chaos [message #111804 is a reply to message #111803] Wed, 07 March 2012 21:56 Go to previous messageGo to next message
Franzia
Messages: 79
Registered: October 2010
Location: Cleveland, OH
Member
I don't see why we should be concerned with what day they run the algorithm, as long as they specify beforehand and remain consistent in their application. Also, if the algorithm is set to run on Monday and they have "computer problems" and can't run it until Tuesday or Wednesday, they can just alter the time-delay expression.

I do agree with the point differential issue - winning 9-7 is "more impressive" than winning 17-15 according to the algorithm, even though a game to 15 in overtime is a more rigorous test.
Re: Nationals Bids Chaos [message #111806 is a reply to message #111801] Wed, 07 March 2012 22:28 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
mgd.mitch wrote on Wed, 07 March 2012 23:21
Jackson wrote on Wed, 07 March 2012 19:58
Now, I think that dropping the region strength bids is good
I'm a little curious why you would say this given you are suggesting moving almost completely to a regional strength system, you just changed the averaging process. I'm not saying your suggestion is bad or flawed, but it is absoluely a regional strength system, not a team strength system.


I was unclear. I meant the specific "region strength" bids used in 2011 where the top 8 teams' ranking per region were averaged. Dropping this particular bid allocation criteria was good, because most--if not all--regions are not 8 teams deep.

I agree that my Top30 system is heavily based on regional strength.
Re: Nationals Bids Chaos [message #111807 is a reply to message #111789] Wed, 07 March 2012 23:34 Go to previous messageGo to next message
Krishna
Messages: 98
Registered: October 2008
Member
Jackson wrote on Wed, 07 March 2012 19:58

• Teams cannot intentionally lose games since the rise is ranking for the lower team is balanced by the loss in ranking (and points) for the better team.


Teams can still intentional lose games and game your system though. There are two ways to game the current system, with its binary cutoff between 20 and 21:

1) Teams well over that cut off have an incentive to lose to teams just below the cut off to pull them over.

2) Teams well below the cut off have an incentive to lose to teams just below the cut off to pull them over.

You and your "friend's" (I love how Pulse felt the need to add quotes) system eliminates the first, but just moves the cutoff from 20 to 30 for the second.

Re: Nationals Bids Chaos [message #111808 is a reply to message #111789] Thu, 08 March 2012 03:57 Go to previous messageGo to next message
Reggie Fanelli
Messages: 1958
Registered: September 2011
Senior Member
the UOA Top 25 rankings could be used.
we have a nation wide panel of experts casting ballots every other
week to determine the rankings.....
Re: Nationals Bids Chaos [message #111809 is a reply to message #111799] Thu, 08 March 2012 04:00 Go to previous messageGo to next message
Reggie Fanelli
Messages: 1958
Registered: September 2011
Senior Member
> Not much response from USAU, unfortunately.
~~~~~

--they don't care....they've got most of your teams trapped already.
they've got all your membership dues...and entry fees...


escape.....to the UOA.
Re: Nationals Bids Chaos [message #111811 is a reply to message #111795] Thu, 08 March 2012 04:03 Go to previous messageGo to next message
Reggie Fanelli
Messages: 1958
Registered: September 2011
Senior Member
> 2) The current system incentivizes intentionally losing to a worse
> team in order to secure an additional bid for the region. See Harvard
> vs Tufts last year (NE probably would have secured a 2nd bid even if
> Harvard beats Tufts 15-9 instead of vice versa, but the incentive is
> still there).
~~~~~~~~

--whoa....a system where two genius schools figure out a loop hole
where losing is better than winning....
too complicated.
keep it simple.

ditch the 'sanctioned games' and all that....
Re: Nationals Bids Chaos [message #111813 is a reply to message #111811] Thu, 08 March 2012 05:05 Go to previous messageGo to next message
jimmm
Messages: 122
Registered: May 2010
Senior Member
What does the UPA mean by "sanctioned"? What is the difference between sanctioned and non-sanctioned events?(outside of just simply using the term)
Re: Nationals Bids Chaos [message #111814 is a reply to message #111789] Thu, 08 March 2012 05:14 Go to previous messageGo to next message
sswang
Messages: 29
Registered: June 2010
Junior Member
@Franzia: If every tournament got rained out this coming weekend (except maybe one that had only teams that hadn't yet played a game this season), and USAU reruns the rankings next week, you'd want them to remain the same, right (except for inserting the new teams in various spots)? They won't.

Now that I re-read my proposal, this probably isn't among the biggest issues however... especially compared to the "winning a game 15-0 might still cause you to drop in ranking" problem! Did Matthew say exactly what issues the fix caused? I can imagine wild scenarios in which implementing it in a certain way might cause non-convergence but I'm sure there are easy workarounds for that.

Another issue I identify in the proposal is the one that Franzen noted, where winning 5-3 is as impressive to the algorithm as 15-9 - games that last to a higher point total should be weighted more - but that isn't quite what I meant by "non-convexity"*. It's hard to explain on RSD, but if anyone wants to see the proposal (it's a 5-page PDF with graphs and formulae), just send me a private message with your email address.

*: and I should have said "non-concavity". Oh well.
Re: Nationals Bids Chaos [message #111815 is a reply to message #111813] Thu, 08 March 2012 05:27 Go to previous messageGo to next message
Reggie Fanelli
Messages: 1958
Registered: September 2011
Senior Member
> What does the UPA mean by "sanctioned"?  What is the
> difference between sanctioned and non-sanctioned
> events?(outside of just simply using the term)
~~~~~~

---sanctioned means.....what?.....they get a cut because everyone has
to have paid their upa dues to compete?
Re: Nationals Bids Chaos [message #111817 is a reply to message #111815] Thu, 08 March 2012 06:49 Go to previous messageGo to next message
Ryan Thompson
Messages: 364
Registered: September 2008
Senior Member
On Mar 8, 8:27 am, Reggie Fanelli <ageric...@gmail.com> wrote:
> > What does the UPA mean by "sanctioned"?  What is the
> > difference between sanctioned and non-sanctioned
> > events?(outside of just simply using the term)
>
> ~~~~~~
>
> ---sanctioned means.....what?.....they get a cut because everyone has
> to have paid their upa dues to compete?

It means that all of the players have their eligibility verified and
TD's publish a list of every team's roster so that other teams can
check for roster violations.

It's also supposed to mean a minimum quality standard that tournaments
must uphold, but that seems to be rarely if ever enforced.

But back to the bid system and addressing Krishna's points:

A team ranked #8 losing intentionally to a team ranked #31 in the
proposed system does not have the same benefit as a team ranked #8
losing to a team ranked #21 in the current system. In the current
system if team #8 drops to team #10 and team #21 jumps to team #19,
the region gets an extra bid. In our proposed system, the region would
have had 23 points from the team ranked #8 and 0 from the team ranked
#31. After the team ranked #31 jumps to #29 and the team ranked #8
falls to #10, the region now just has 21+2 = 23 points, or the same as
they did before.

There is this case for gaming the system, but it's much more difficult/
less likely:

Let's say Pittsburgh has not submitted a correct roster all season and
all of their results get wiped from the rankings. At the last
tournament of the season, Ohio is ranked #5 and the region would have
2 bids for sure if Pittsburgh were still at #3. Penn State is ranked
#34. Right now the Ohio Valley is only getting 1 bid from Ohio being
#5, but they have enough points that they could get a second bid, but
they only have 1 team in the top 30. Ohio loses a game to Penn State
on purpose to bring them into the rankings at #30, dropped down to #8
but still good enough to get a 2nd bid.

This scenario could arise any time a team is capped by the "can only
get the # of bids equal to # of teams in the top 30" rule, but it's
much less likely that a team ranked in the top 6 (generally the cutoff
for this scenario) would have the next closest team outside of the top
30 and be worried about the number of bids.
Re: Nationals Bids Chaos [message #111818 is a reply to message #111814] Thu, 08 March 2012 07:03 Go to previous messageGo to next message
Franzia
Messages: 79
Registered: October 2010
Location: Cleveland, OH
Member
sswang wrote on Thu, 08 March 2012 08:14
@Franzia: If every tournament got rained out this coming weekend (except maybe one that had only teams that hadn't yet played a game this season), and USAU reruns the rankings next week, you'd want them to remain the same, right (except for inserting the new teams in various spots)? They won't.


oh ok I see what you mean - you're saying that if no one plays a game during a given week, the rankings should not change.

I was under the impression that the change in ratings points would be negligible, but I crunched some numbers and found that ratings points can change by 1-8 points in one week if no one plays a game. I guess if one team sitting at #20 with like 1600 ratings points loses 8 and another team sitting at #21 with like 1590 ratings points gains 8 in this situation, you could have a sticky situation on your hands.

My overall feelings is that the decay function is meant to weight games in late March heavier than games in January. The loophole we're discussing is unfortunate but I think teams go into the season knowing that a computer is rating them and any sorts of BS including rainouts could happen. If you can keep the weighting function and find a way to correct for this rare occurrence, that would be amazing.

But, for the time being, all I can see is that the top 20 teams hold the 20 bids to nationals. Isn't that what we've always wanted???
Re: Nationals Bids Chaos [message #111819 is a reply to message #111818] Thu, 08 March 2012 07:14 Go to previous messageGo to next message
Ryan Thompson
Messages: 364
Registered: September 2008
Senior Member
"But, for the time being, all I can see is that the top 20
teams hold the 20 bids to nationals. Isn't that what we've
always wanted??? "

As I stated upthread, there are two problems with this:

-- The top 20 ranked teams are not necessarily the top 20 teams due to
a flawed and imprecise ranking system. The top 10 teams are definitely
"Top 20" teams, but where do teams 11-30 really fit?
-- Teams ranked in the top 20 have an incentive to intentionally lose
to teams in their own region who are just outside the top 20 in order
to get another bid for the region. This is not a made-up problem, it's
real.
Re: Nationals Bids Chaos [message #111820 is a reply to message #111818] Thu, 08 March 2012 08:41 Go to previous messageGo to next message
sswang
Messages: 29
Registered: June 2010
Junior Member
Franzia wrote on Thu, 08 March 2012 07:03

My overall feelings is that the decay function is meant to weight games in late March heavier than games in January. The loophole we're discussing is unfortunate but I think teams go into the season knowing that a computer is rating them and any sorts of BS including rainouts could happen. If you can keep the weighting function and find a way to correct for this rare occurrence, that would be amazing.


Right, I have no problem with the idea of time-weighting, but the way it is currently implemented yields this problem. There is a simple technical fix which is to change the weight to something where time is in the exponent. That way the weight of games relative to one another remains constant no matter when time zero is and the problem goes away.


Re: the original proposal for bid allocation, I have two comments.

First, it doesn't seem that including the top team from each region in the calculation is a good idea. This allows for the problem whereby one really good team can potentially earn a second bid for its region even when other regions are more deserving. For instance, take two regions, one with the #1 team and the #30 team, and one with the #20, #23, #25, #28 and #29 team. According to Ariel's proposal, region 1 has a total of 30 + 1 = 31 points, whereas region 2 has a total of 11 + 8 + 6 + 3+ 2 = 30 points, so region 1 would be higher priority for a second bid, when I think most people would think region 2's 2nd-place team would be much more likely to be stronger.

Secondly, I think that resorting to "place in rankings" as the numbers the algorithm calculates with is kind of silly, when we already have numerical ratings to work with. Theoretically, there could be a larger difference between the #5 and #6 team than there is between the #15 and #30 team. Why not use UPA power ratings instead?

Re: Nationals Bids Chaos [message #111822 is a reply to message #111820] Thu, 08 March 2012 09:39 Go to previous messageGo to next message
aghesquiere
Messages: 36
Registered: October 2008
Member
I think an easy answer to this problem (implementable immediately!) would be reducing the weighting for In-Region games, similar to how the weighting for January games is reduced relative to April games. If In-Region games count for a fraction of out-of region games the benefit of tanking to in-region teams is greatly reduced.
Re: Nationals Bids Chaos [message #111824 is a reply to message #111822] Thu, 08 March 2012 10:42 Go to previous messageGo to next message
Big_Jim
Messages: 111
Registered: August 2010
Location: Madison WI
Senior Member
Being able to game the system is a problem. It presented itself last year in the Harvard-Tufts game. However, let's examine what your "solution" would have done to the Nationals field from last year:

North Central, which had 4 of the top 10 regular season teams, and placed all 4 of their Nationals bids in Quarterfinals, 3 in Semis and both Championship teams, would lose a bid.

North West, which had 2 of the top 7 and 3 of the top 18 regular season teams, and who had a Quarterfinal team and 9th place team at Nationals, would lose a bid.

Great Lakes, whose best team was 17th in regular season (one ahead of NW's 3rd) and whose only team at Nationals placed 17th, would gain a bid.

South West, who had no teams in the top 10 regular season, but placed one Quarterfinal team at Nationals (their other team tied for last) would gain a bid.

I'm sure the ranking algorithm needs some find tuning. But your biggest fear, that throwing games to increase your regions total number of bids, isn't even solved by your algorithm! The North East would still get a strength bid in your algorithm! Subtracting a full 30 points for each strength bid seems to be the reason why: it penalizes multiple strength bids going to the same region, even if that region (the North Central) is stacked and deserves all the strength bids.
Re: Nationals Bids Chaos [message #111825 is a reply to message #111824] Thu, 08 March 2012 11:18 Go to previous messageGo to next message
thefan
Messages: 1059
Registered: September 2008
Senior Member
interesting that in a sport where participants are held to the standard of making honest, impartial decisions in the middle of a game with their adrenaline and blood pumping can't be trusted not to sit around and pre meditate ways to game the system.
Re: Nationals Bids Chaos [message #111828 is a reply to message #111820] Thu, 08 March 2012 13:16 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
sswang wrote on Thu, 08 March 2012 11:41

Re: the original proposal for bid allocation, I have two comments.

First, it doesn't seem that including the top team from each region in the calculation is a good idea. This allows for the problem whereby one really good team can potentially earn a second bid for its region even when other regions are more deserving. For instance, take two regions, one with the #1 team and the #30 team, and one with the #20, #23, #25, #28 and #29 team. According to Ariel's proposal, region 1 has a total of 30 + 1 = 31 points, whereas region 2 has a total of 11 + 8 + 6 + 3+ 2 = 30 points, so region 1 would be higher priority for a second bid, when I think most people would think region 2's 2nd-place team would be much more likely to be stronger.



I'd have to run a sample with the exact numbers that you use, but I'm fairly certain that in this case, neither region would be strong enough to earn a strength bid (2nd overall bid), which sounds reasonable to me. I'll run a test with your numbers and get back to you on that.

Quote:

Secondly, I think that resorting to "place in rankings" as the numbers the algorithm calculates with is kind of silly, when we already have numerical ratings to work with. Theoretically, there could be a larger difference between the #5 and #6 team than there is between the #15 and #30 team. Why not use UPA power ratings instead?



We discussed this while developing the Top30 system. I don't want to say that the decision was unanimous, but I decided not to rely on USAU's PR (power ranking) values. There have already been many issues brought up with the ranking system itself (both here and over at Skyd), and one of our original assumptions is that the ranking system is not perfect. By using points instead of PR, we are hedging against an imperfect ranking system.
Re: Nationals Bids Chaos [message #111829 is a reply to message #111822] Thu, 08 March 2012 13:21 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
aghesquiere wrote on Thu, 08 March 2012 12:39
I think an easy answer to this problem (implementable immediately!) would be reducing the weighting for In-Region games, similar to how the weighting for January games is reduced relative to April games. If In-Region games count for a fraction of out-of region games the benefit of tanking to in-region teams is greatly reduced.


I don't agree that this is an optimal solution. It would be kind of a quick fix, but I think that a better solution is to develop a system that is very difficult to manipulate by throwing games.

With your suggestion, we are throwing out data about relative team strength. 99%+ of intra-region games are played on legitimate terms (i.e. no attempt at gaming the system). Let's say that Washington plays Oregon and wins. Why punish Washington for playing well against a team in their region? This hurts Washington's ranking which sends ripples throughout the rankings, making them overall less accurate.
Re: Nationals Bids Chaos [message #111830 is a reply to message #111828] Thu, 08 March 2012 13:22 Go to previous messageGo to next message
Ryan Thompson
Messages: 364
Registered: September 2008
Senior Member
I do agree with Jim's point about the North Central getting screwed
out of bids because of the 30-point subtraction. Shouldn't we be
subtracting 20-11 points instead of 30-21?
Re: Nationals Bids Chaos [message #111831 is a reply to message #111824] Thu, 08 March 2012 13:40 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
Big_Jim wrote on Thu, 08 March 2012 13:42
Being able to game the system is a problem. It presented itself last year in the Harvard-Tufts game. However, let's examine what your "solution" would have done to the Nationals field from last year:

North Central, which had 4 of the top 10 regular season teams, and placed all 4 of their Nationals bids in Quarterfinals, 3 in Semis and both Championship teams, would lose a bid.



Actually, only 3 made quarters, Luther was knocked out in pre-quarters, but still a very impressive showing by the North Central. There is danger in using Nationals placement to determine whether the bid allocation is fair. Remember, the information we have to go on is the regular season results. We can evaluate whether a region over or under performed at Nationals, but if we are judging bid allocation, we need to compare it to regular season results, not Nationals performance.

I don't think you are looking at the correct rankings. At the end of the regular season (4/8/11), NC had the following teams ranked:
#1 Carleton
#7 Wisconsin
#12 Minnesota
#18 Iowa
#34 Luther
I count 2 in the top 10, not 4.

Quote:

North West, which had 2 of the top 7 and 3 of the top 18 regular season teams, and who had a Quarterfinal team and 9th place team at Nationals, would lose a bid.


Did the NW deserve 4 bids based on their regular season performance?

Quote:

Great Lakes, whose best team was 17th in regular season (one ahead of NW's 3rd) and whose only team at Nationals placed 17th, would gain a bid.


Michigan was ranked 8th at the end of the regular season. Michigan State was 20th and Illinois was 25th. Which rankings are you looking at? Post nationals perhaps?

Quote:

South West, who had no teams in the top 10 regular season, but placed one Quarterfinal team at Nationals (their other team tied for last) would gain a bid.


Again, we shouldn't look at Nationals performance to determine bids to Nationals. That is very difficult, logistically, to implement.

Quote:

I'm sure the ranking algorithm needs some find tuning. But your biggest fear, that throwing games to increase your regions total number of bids, isn't even solved by your algorithm! The North East would still get a strength bid in your algorithm! Subtracting a full 30 points for each strength bid seems to be the reason why: it penalizes multiple strength bids going to the same region, even if that region (the North Central) is stacked and deserves all the strength bids.


The algorithm does in fact stop teams from throwing games. In the Top30 system, the NE got a strength bid because the region earned it. In the USAU system they got it because they allegedly gamed the system. In the Top30 system, Harvard lost points when they lost to Tufts. It is nearly a wash, and almost impossible to determine before the game which outcome would best favor the region (which is why they cannot game the system).
Re: Nationals Bids Chaos [message #111832 is a reply to message #111830] Thu, 08 March 2012 13:52 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
Ryan Thompson wrote on Thu, 08 March 2012 16:22
I do agree with Jim's point about the North Central getting screwed
out of bids because of the 30-point subtraction. Shouldn't we be
subtracting 20-11 points instead of 30-21?


I tried 20-11 first. Here are the results it gave:
NC:3 NW:3 SC:2 SW:2

It depends on who you think the autobids should be assigned to. USAU has been assigning them to the strongest team in each region. I don't think that's the best way to look at it.

The STRENGTH bids should go the STRONGEST teams, not the weaker teams. The autobids go to the weaker teams to ensure regional diversity. In this light, the strength bids go to the top 10 region*teams at nationals (i.e. points 30-21) and the autobids go to region*teams 11-20.
Re: Nationals Bids Chaos [message #111833 is a reply to message #111832] Thu, 08 March 2012 13:58 Go to previous messageGo to next message
Ryan Thompson
Messages: 364
Registered: September 2008
Senior Member
I've got to disagree with you there - if a region has the #1 team and
the next best team is #50, the autobid clearly goes to the #1 team. If
the #1 team earns a strength bid before the autobids are handed out,
we've got the whole thing backwards.
Re: Nationals Bids Chaos [message #111834 is a reply to message #111819] Thu, 08 March 2012 14:14 Go to previous messageGo to next message
Reggie Fanelli
Messages: 1958
Registered: September 2011
Senior Member
> "But, for the time being, all I can see is that the top 20
> teams hold the 20 bids to nationals.  Isn't that what we've
> always wanted??? "
~~~~~~~

---nope....we've always wanted the best teams from all over the
country.
big difference.
Re: Nationals Bids Chaos [message #111835 is a reply to message #111833] Thu, 08 March 2012 14:18 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
Ryan Thompson wrote on Thu, 08 March 2012 16:58
I've got to disagree with you there - if a region has the #1 team and
the next best team is #50, the autobid clearly goes to the #1 team. If
the #1 team earns a strength bid before the autobids are handed out,
we've got the whole thing backwards.


Let me clarify. The wording on this is getting very tricky and I did a bad job in my previous post. I'm trying to use region*teams as a unit in the same way you might use people*miles/gallon to compare fuel efficiency of a bus to a car. It's tricky because we really want to assign bids to the strongest regions, not teams, but regions can obviously have more than 1 bid. If the NC gets the 3rd and 6th bid and the NW gets the 4th, how do we describe the relative strength of each region? The 3rd and 6th bid both go to the NC, not a specific team in the NC.

I should have said the STRENGTH bids go to the STRONGEST regions (or region*teams), instead of strongest teams.

In your 1-50 example, I agree that the autobid is applicable here. In the same way that a car gets better mpg than a bus, the #1 team is better than the other teams, but since it comes from such a weak region (in your 1-50 example) it is not the #1 region*team, just as the bus gets better people-mpg than a car (assuming reasonable capacity). Since the region*team that it represents is in the 11-20 field, it acquires the autobid and not a strength bid.

If someone can come up with better terminology than region*team, I'm all ears.
Re: Nationals Bids Chaos [message #111836 is a reply to message #111828] Thu, 08 March 2012 14:36 Go to previous messageGo to next message
sswang
Messages: 29
Registered: June 2010
Junior Member
Jackson wrote on Thu, 08 March 2012 13:16
sswang wrote on Thu, 08 March 2012 11:41

Re: the original proposal for bid allocation, I have two comments.

First, it doesn't seem that including the top team from each region in the calculation is a good idea. This allows for the problem whereby one really good team can potentially earn a second bid for its region even when other regions are more deserving. For instance, take two regions, one with the #1 team and the #30 team, and one with the #20, #23, #25, #28 and #29 team. According to Ariel's proposal, region 1 has a total of 30 + 1 = 31 points, whereas region 2 has a total of 11 + 8 + 6 + 3+ 2 = 30 points, so region 1 would be higher priority for a second bid, when I think most people would think region 2's 2nd-place team would be much more likely to be stronger.



I'd have to run a sample with the exact numbers that you use, but I'm fairly certain that in this case, neither region would be strong enough to earn a strength bid (2nd overall bid), which sounds reasonable to me. I'll run a test with your numbers and get back to you on that.

OK, if 31 points isn't enough (though I'm sure I could concoct a scenario in which it would be), how about
Region A: #1, #27, #28 (30 + 4 + 3 = 37 points)
vs
Region B: #18, #19, #20 (13 + 12 + 11 = 36 points)? These examples are pretty easy to come by.


Quote:

Quote:

Secondly, I think that resorting to "place in rankings" as the numbers the algorithm calculates with is kind of silly, when we already have numerical ratings to work with. Theoretically, there could be a larger difference between the #5 and #6 team than there is between the #15 and #30 team. Why not use UPA power ratings instead?



We discussed this while developing the Top30 system. I don't want to say that the decision was unanimous, but I decided not to rely on USAU's PR (power ranking) values. There have already been many issues brought up with the ranking system itself (both here and over at Skyd), and one of our original assumptions is that the ranking system is not perfect. By using points instead of PR, we are hedging against an imperfect ranking system.


I wouldn't say you're hedging against the ratings being imperfect. I'd say you're taking an imprecise ratings system and using it in a way that makes it even more imprecise.
Re: Nationals Bids Chaos [message #111837 is a reply to message #111817] Thu, 08 March 2012 15:25 Go to previous messageGo to next message
Krishna
Messages: 98
Registered: October 2008
Member
Ryan Thompson wrote on Thu, 08 March 2012 09:49


But back to the bid system and addressing Krishna's points:

A team ranked #8 losing intentionally to a team ranked #31 in the
proposed system does not have the same benefit as a team ranked #8
losing to a team ranked #21 in the current system.


I think you missed my point. In the current system #8 wants to lose to #21, whereas under your system it is unclear if #8 wants to lose to #31.

However, in the current system #35 wants to get blown out by #21, just as in your system #45 wants to get blown out by number #31.

You have fixed one of the two ways team might throw a match to game the system, but not the other.
Re: Nationals Bids Chaos [message #111838 is a reply to message #111789] Thu, 08 March 2012 16:06 Go to previous messageGo to next message
gabers20
Messages: 27
Registered: September 2008
Junior Member
Ryan, did both you and Ariel graduate from Stanford?
Re: Nationals Bids Chaos [message #111840 is a reply to message #111831] Thu, 08 March 2012 16:25 Go to previous messageGo to next message
Big_Jim
Messages: 111
Registered: August 2010
Location: Madison WI
Senior Member
Jackson wrote on Thu, 08 March 2012 13:40

Michigan was ranked 8th at the end of the regular season. Michigan State was 20th and Illinois was 25th. Which rankings are you looking at? Post nationals perhaps?

Ahh, it seems that I was. Now that I'm looking for them, I can't find regular season rankings ending on 4/8/11.

I still don't think this is a good idea. This system might work with a value other than 30 as the subtracting total. As it stands, it becomes extremely difficult for regions to get more than 2 strength bids, regardless of top 20 or top 30 rankings. Consider the following:

Region A has teams ranked:

1, 25 --> 35 total points

Region B has teams ranked:

4,6,8,10 --> 92 total points

Region A will get a bid for its first team. They have 5 points remaining. Region B gets 3 bids for its first three teams. They have 0 points remaining. At this point, their second strongest team, ranked 25 in the country, is in line to receive a bid over a team ranked 10. All that is required to beat out a region with teams ranked 4,6,8,10 is two teams who combine for 31 points or more.

Going off the rankings you cite from last year:

NC has teams ranked:

1,7,12,18 --> 86 total points

GL has teams ranked

8,20,25 --> 40

After NC gets 3 bids, they're at -4 points. I basically doesn't matter where their 4th best team is ranked. Even ranked 14, they'd be at 0. And it also doesn't matter where GL's 3rd best team is (since 8+20 gets you more than 30 points, enough to beat 0).

I think the current system works well: strength bids are about how good your second, third, fourth teams are. Does it suck for other regions if you have a highly ranked team in your region willing to throw games? Yes. So maybe we make margin of victory in games factor less into RRI. Maybe we make inner-conference matchups worth less. But a system where it's basically impossible to secure more than 2 strength bids just to prevent something as shitty as people mathcrafting their Nationals bid hopes is not a good system.
Re: Nationals Bids Chaos [message #111841 is a reply to message #111837] Thu, 08 March 2012 17:05 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
Krishna wrote on Thu, 08 March 2012 18:25

However, in the current system #35 wants to get blown out by #21, just as in your system #45 wants to get blown out by number #31.

You have fixed one of the two ways team might throw a match to game the system, but not the other.


You're right that #45 wants to get blown out by number #31, but the magnitude of this outcome is much much less in our system than the 21v35-current-system-example. In 21v35, their region can gain an entire strength bid with this outcome. In our system (31-45 example), their region gains only a point or 2. This is why having a gradient is strongly preferred to having a sharp cutoff (as they do in the current system).

I believe that a team will be tempted to throw a game when they are nearly guaranteed to gain an extra bid to nationals. I'm less persuaded that a team will throw a game for a very slight improvement in their chance at gaining an extra bid.
Re: Nationals Bids Chaos [message #111842 is a reply to message #111840] Thu, 08 March 2012 17:12 Go to previous messageGo to next message
Ray Illian
Messages: 32
Registered: October 2008
Member
What if teams are just blinded to the actual power ratings and bid-watch predictions during the regular season?

With some minor tweaks made to the algorithm before each season, the USAU could release the rankings each week in bins or tiers similar to how they do it on the Inside Breaks blog.

Tier I - top 5
Tier II - next 10
Tier III - next 10, ect

Teams are not ordered by rank within each tier, and their power rating is not revealed. It's less obvious which teams are hovering on the 20th cutoff and gaming the system becomes more difficult (but not impossible).

Since we already have Skyd Power Ratings, Inside Breaks Tiers, UOA Top 25, Rodney's RRI... it won't be a huge blow to not have these rankings that everyone complains about anyway.
Re: Nationals Bids Chaos [message #111843 is a reply to message #111840] Thu, 08 March 2012 17:28 Go to previous messageGo to next message
Jackson
Messages: 97
Registered: October 2008
Location: Stanford Bubble
Member
Big_Jim wrote on Thu, 08 March 2012 19:25

Ahh, it seems that I was. Now that I'm looking for them, I can't find regular season rankings ending on 4/8/11.

http:// www.usaultimate.org/competition/college_division/college_sea son/college_rankings.aspx

Quote:

Consider the following:

Region A has teams ranked:

1, 25 --> 35 total points

Region B has teams ranked:

4,6,8,10 --> 92 total points

Region A will get a bid for its first team. They have 5 points remaining. Region B gets 3 bids for its first three teams. They have 0 points remaining. At this point, their second strongest team, ranked 25 in the country, is in line to receive a bid over a team ranked 10. All that is required to beat out a region with teams ranked 4,6,8,10 is two teams who combine for 31 points or more.


Going off the rankings you cite from last year:

NC has teams ranked:

1,7,12,18 --> 86 total points

GL has teams ranked

8,20,25 --> 40

After NC gets 3 bids, they're at -4 points. I basically doesn't matter where their 4th best team is ranked. Even ranked 14, they'd be at 0. And it also doesn't matter where GL's 3rd best team is (since 8+20 gets you more than 30 points, enough to beat 0).

I think the current system works well: strength bids are about how good your second, third, fourth teams are. Does it suck for other regions if you have a highly ranked team in your region willing to throw games? Yes. So maybe we make margin of victory in games factor less into RRI. Maybe we make inner-conference matchups worth less. But a system where it's basically impossible to secure more than 2 strength bids just to prevent something as shitty as people mathcrafting their Nationals bid hopes is not a good system.


You've calculated the points incorrectly. A=36, B=96, but that isn't very important to your point. Also, you correctly calculated the NC and GL points, so I assume it was just a single error. However, you are definitely implementing the system incorrectly. No region will ever get to negative points.

Your main concern, that it will be very difficult for a region to get more than 2 strength bids, is a valid one. I will look into this.
Re: Nationals Bids Chaos [message #111844 is a reply to message #111842] Thu, 08 March 2012 17:50 Go to previous messageGo to next message
Krishna
Messages: 98
Registered: October 2008
Member
Ray Illian wrote on Thu, 08 March 2012 20:12
What if teams are just blinded to the actual power ratings and bid-watch predictions during the regular season?

...

It's less obvious which teams are hovering on the 20th cutoff and gaming the system becomes more difficult (but not impossible).



This seems pretty feasible. They would have to change the ranking system and keep it secret though. It would take about half a day of work for someone to write a program that scrapes all the scores from score reporter and then runs the algorithm. The worst possible solution is if only a few teams know what the rankings are.
Re: Nationals Bids Chaos [message #111845 is a reply to message #111789] Thu, 08 March 2012 18:13 Go to previous messageGo to previous message
dwrenn
Messages: 127
Registered: November 2008
Senior Member
if you win regionals it doesn't really matter where you are ranked, now does it.
Previous Topic:Huge Sky D by Wildcat #7
Next Topic:Yale Cup 2012: March 24-25
Goto Forum:
  


Current Time: Tue Dec 18 19:06:15 PST 2018
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.0RC2.
Copyright ©2001-2009 FUDforum Bulletin Board Software