View
51
Download
4
Category
Tags:
Preview:
DESCRIPTION
ASSESSMENT OF SOCCER REFEREE PROFICIENCY IN TIME-SENSITIVE DECISION-MAKING. Nathan Jones Andrew Cann Hina Popal Saud Almashhadi. Context Problem & Need Statement Design Alternatives Simulation Simulation Output Utility Analysis Conclusions Management. Agenda. Introduction to Soccer. - PowerPoint PPT Presentation
Citation preview
Nathan JonesAndrew CannHina PopalSaud Almashhadi
ASSESSMENT OF SOCCER REFEREE PROFICIENCY IN TIME-SENSITIVE DECISION-MAKING
2
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
3
Introduction to Soccer
Soccer is the world’s most popular sport.Generates the most revenue:
• In 2009-2010 season the English Premier League generated roughly 3.2 billion dollars
• European soccer generated 21.6 billion dollars
Highest average attendance for international club competitions:
• FIFA World Cup• UEFA Champions League
Information taken from: http://www.economist.com/blogs/gametheory/2011/09/ranking-sports%E2%80%99-popularity
European Soccer
NFL MLB NBA0
5
10
15
20
2521.6
97.2
4.1
Professional Sports vs. 2009-2010 Generated Revenue
Generated Rev-enue ($ Billions)
4
Introduction to Soccer
• The game is played by two teams (11 vs. 11).
• Field dimensions:
115 by 74 yards
• 2 – 45 minute periods
• 3 Referees – 1 main referee and 2 assistant referees• Responsible for upholding
the integrity of the game
MR
AR
AR
5
Referee Responsibilities
Upholding the integrity of the game:
• Make accurate calls • Make calls that don’t interrupt
the flow of the game• Be in proper position, to
assess, process, and identify correct call
Referees are categorized as either junior referees (entry level) or senior referees (advanced level).
Current MLS referees make 86.1 % correct calls. (USSF)
6
Acknowledgement of Sponsor
Metro DC Virginia State Referee Program (MDCVSRP) oversees all soccer referees in the Commonwealth of Virginia (over 5400 referees)
Responsibilities:
1) Train and evaluate junior and senior referees
2) Assign Referees to officiate games
3) Promote high quality referees to senior ranks
Responsibilities 2 and 3 depend heavily on ability to assess referee call accuracy
7
Referee Call Making Process
8
Referee Assessment
On-Field Assessments
Written Exam on Knowledge of the
Game
Fitness Test
9
Referee Assessment is Broken
Junior referees do not undergo fitness tests or on field assessments
(Preventing evaluation of Fitness or GFU attributes)
The evaluation process for referees is broken:
• 96% of total MDCVSRP Referees (junior level) do not receive assessment in two of three attributes.
Referee Attributes Assessment MethodFitness Fitness Test
(senior referees)
Call Decision Making (CDM) Written exam on rules (All referees)
Game Flow Understanding (GFU) Indirectly using on field assessment (senior referees)
10
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
11
Problem Statement
96 % of MDCVSRP referees (Junior level) do not receive assessment for Game Flow Understanding and fitness attributes as predictors of call accuracy.
12
Need Statement
An assessment method is needed to evaluate referee accuracy in a cost effective manner utilizing fitness and/or Game Flow Understanding (GFU).
Scope: Our analysis will focus on determining the best system concept for assessing MDCVSRP junior referees.
Specifics of design and implementation are considered future work.
13
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
14
Design Alternatives
# Alternative Description Tests Total Cost
(5,139 Referees)
1 Fitness Test
A baseline fitness test equivalent to those
administered at senior grades
Fitness $26,990
2 Game Flow Evaluation
Video performance assessments
conducted by official assessors
GFU $337,995
3 Combined EvaluationCombination of first
two evaluationsFitnessGFU $341,870
4 No AssessmentNot conducting any referee evaluations
(status quo)None $0.00
Costs defined as physical + implementation cost for one time evaluation of all junior referees.
15
Evaluation Of Alternatives
Utility of each alternative defined as:
Expected call accuracy of the top 100 referees identified using each alternative within junior referee pool (5000 referees).
To determine utilities, a two part analysis was conducted:
1) Function for call accuracy based on fitness and GFU levels developed using discrete soccer game simulator.
2) Using part 1 function, expected call accuracy of top 100 referees selected by each alternative computed through Monte Carlo analysis.
16
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
Simulation: Input / Outputs
17
18
Expansion on Prior Work
Simulation was re-designed and re-coded from scratch.
Simulation Element Solomon, et al. (2011) This Project
Probability Maps 1 map for all teams, all time, and all score 19 maps dependent on team, time, an score
Ball Position Function 1 event 4 state cycle scaled to time
Referee Position Function 1-D, chase ball on left diagonal 2-D, based on GFU, scaled to time
Fitness 3 levels 5 levels
GFU None 5 levels based on probability maps
Call Grids None Survey 16 senior state referees
Call Event Trigger Simple probability Calls grids and position in cycle
Distance vs. Call Accuracy Function Estimated Figure of MeritSurveyed 16 senior state referees and
generated regression
Number of Teams in Game Home vs. Home Home vs. Away (4 Options)
Number of Teams Simulated 1 4
Team Strategy Changes Never Time / Score
Referee/Ball Movement Scaled to Time No Yes
19
Simulation – Ball and Referee Position
• In the discrete event simulation, a soccer field is divided into a fine grid of cells.
8510 cells
Each Cell 1x1 yd
Cell Groupings:
• 60 Movement Polygons
• 24 Call Grids
• 0.5 s refresh rate (game time)
20
Two Teams - Possession Shifts
The ball shifts possession between two different teams, each executing its own unique strategy . Changes in possession occur due to failed passes or shot events.
21
Simulation: Ball Movement
22
Cycle of Events
23
Ball & Referee Movement Algorithm
• At any time in simulation, ball moving to set destination in straight line.
• Destination changes during dribbling / passing.• Time taken for ball to move incrementally to destination (#
Refreshes) is reflective of ball speed and distance:
24
Shot Events
Whenever ball finishes dribbling, probability determines if ball is shot at goal. Shot either results in goal or turnover.
25
Pass Events
• If no shot, Ball passed between polygons controlled by movement probability maps indicating destination and chance of success.
• Polygon (n+1) = Polygon (n) * Prob. Map• When new polygon selected, destination is set to random cell
within polygon
Map sets areformulated for:
• Manchester United• Arsenal• Wigan Athletic• Stoke City
26
Probability Maps
• Ball movement and shot data were gathered from the Guardian Chalkboard Website. 80 total games (over 35,000 pass & shot events) recorded for Stoke City, Manchester United, Arsenal, Wigan.
Data was analyzed for strategy and used to produce shot and movement probability maps
27
Probability Maps – Team Strategy
Two way ANOVA Analysis: Time + Score + Time*Score
• Time has an effect on pass accuracy: Arsenal(p = 0.777); United(p=0.142); Stoke (p=0.001); Wigan (p=0.001)
• Score has an effect on pass accuracy:
Arsenal(p = 0.231);United(p=0.001);Stoke(p=0.000); Wigan (p=0.000)
• Score*Time has an effect on pass accuracy:
Arsenal(p = 0.338);United(p=0.000);Stoke(p=0.000);Wigan(P= 0.116)
Strategy Analysis conducted to determine when strategy maps should be changed (Metric = % completed passes)
28
Simulation: Referee Movement
29
Referee Profile DefinitionTo determine the effect of Fitness and Game Flow Understanding on call performance:
• 25 referee “profiles” defined as combinations of fitness and game flow understanding.
Referee Game Flow Understanding
Referee Fitness
0 25 50 75 100
///////// 0.25 0.41 0.58 0.74 0.9
02.023 yds / s Ref 1,1 Ref 1,2 Ref 1,3 Ref 1,4 Ref 1,5
252.495 yds / s Ref 2,1 Ref 2,2 Ref 2,3 Ref 2,4 Ref 2,5
502.967 yds / s Ref 3,1 Ref 3,2 Ref 3,3 Ref 3,4 Ref 3,5
753.439 yds / s Ref 4,1 Ref 4,2 Ref 4,3 Ref 4,4 Ref 4,5
1003.911 yds / s Ref 5,1 Ref 5,2 Ref 5,3 Ref 5,4 Ref 5,5
30
Simulation – Ref movement
One main referee running within left hand diagonal route area.
Referee movement speed depends on fitness level of profile tested.
31
Simulation - Ref Movement
At each refresh rate (0.5 s), referee will compute desired location relative to ball using one of 2 movement scripts:
1) No Prediction – Referee will set destination to closest cell within 11 – 13 yds of ball’s current location.
2) Prediction – If dribbling: Referee will set destination to closest cell within 11 – 13 yds of next most probable pass destination.
Once destination is set, referee will begin moving to destination (rate = speed).
Process repeats at each refresh
32
Simulation – Ref Movement
• Proportion of time referee utilizes script 2 depends on GFU level. • Referee with (GFU = 0.75) with remain in script 2
75% of time.
GFU also includes an ability of referee to recognize a build up to a call:
Probability that predicting referee anticipates the call and switches to script 1 until the call occurs.
33
Simulation: Call Events
34
Call Events
• Call grid probabilities used to generate events based on ball location whenever new cycle begins. Further probabilities determine where in cycle event occurs.
Source: Senior MDCVSRP referee surveys (n = 16)
Roughly 90 events per game
Passing: 0.21Dribbling: 0.44En-route: 0.15Receiving: 0.21
35
Simulation - Call Accuracy
• Whenever a call event occurs, referee must make a decision regarding the nature of the event (infraction, no infraction).
• The probability that he makes the correct call depends on the distance from the ball.
36
Referee Call Accuracy Function
Source: Senior MDCVSRP referee surveys (n = 16)
Distance > 20 ydsDistance <= 20 yds
Accuracy Peaks at 11 – 13 yds
37
Simulation: Output
38
Simulation - Output
Simulation output :• Each profiles simulated through 2,000 games (200 per team comb.)• Referee call accuracy was calculated for each game.
39
Simulation Demo
40
Validation of Simulator
STATISTIC Simulation Professional Soccer
Average Goals per game 0.8266 ̴� 1.553(EPL 4 team Average) [1]
Average Team Passes per game
449 ̴� 424 (EPL 4 team Average) [2]
Average Referee Distance Run per game (yds)
11, 686 11, 289 (NZFC) [3]
[3] - D.R.D. Mascarenhas et al. (2009) "Physical Performance and Decision Making in Association Football Referees: A Naturalistic Study" [online]. Available: http://www.benthamscience.com/open/tossj/articles/V002/1TOSSJ.pdf
[1] - http://soccernet.espn.go.com/stats/_/league/eng.1/year/2010/barclays-premier-league?cc=5901
[2] - http://www.whoscored.com
41
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
42
Simulation - Call Accuracy Results
010
2030
4050
6070
8090
100
010
2030
4050
6070
8090
1000.7
0.71
0.72
0.73
0.74
0.75
0.76
GFU
Call Accuracy(Fitness,GFU)
Fitness
Average2000 games per profile
43
Simulation Results - Regression
Accuracy (Fitness, GFU):
0.713491 + 0.000923486 *Fitness + 1.28791e-005*GFU 6.4846e-005*Fitness^2 + 1.12504e-006*GFU^2 + 1.26193e-006*Fitness^3- 6.75305e- 009*Fitness^4
R-Sq = 99.51%
Fitness, GFU nonlinearNo interaction (p = 0.813)
44
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
45
Defining Referees for Utility Analysis
• Referees are defined as a combination of two independent traits (Fitness, GFU)
• Each trait is scaled from worst (0) to best (100) possible• The distribution of referees for each trait is Normal at mean
50 and st. dev 15
Call accuracy for each referee defined using Call Accuracy Regression
46
Utility Analysis Method – Monte Carlo
• 5000 Referees (Junior level) were generated .• For each alternative, a cutoff was defined on each attribute assessed where
if a referee preformed above the cutoff on all attributes, he would be selected by program.
• Cutoff developed using Normal CDF to ensure top 100 referees selected
Alternatives assumed to have perfect ability to identify if referees make the cutoff
Alternative Attributes Cutoff Avg. # Referees Chosen
Fitness Test Fitness Fitness > 81 97
Game Flow Evaluation GFU GFU > 81 97
Combined Evaluation Fitness, GFU Fitness >66 & GFU > 66
102
No Assessment N/A N/A 100
47
Analysis Method – Monte Carlo
For each alternative, referees are identified that meet the selection cutoff. The average call accuracy of referees selected (% correct calls) is used to
determine alternative utility.
48
Utility Analysis Results
Alternative Cutoff Avg. Call Accuracy 95 % Half-Width
Call Accuracy
Fitness Test Fitness > 81 0.74926 0.00012
Game Flow Evaluation GFU > 81 0.72693 0.00028
Combined EvaluationFitness >66 &
GFU > 66 0.74174 0.00021
No Assessment N/A 0.72099 0.00004
Based on n = 30 trials
49
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
50
Alternative Cost vs. Benefit
“Fitness Test” dominates all other assessment based alternatives.
51
Recommendations for MDCVSRP
Recommendation: It is not cost effective to implement assessments on junior referees within MDCVSRP.
Fitness Test vs. No Assessment (status quo)
Marginal Cost Fitness Test: $26,990
Marginal Utility Fitness Test: Accuracy improvement of 2.8% for top 100 referees identified
52
Further Findings – Impact of Teams
Impact of different team strategies on game flow has noteworthy effect on referee performance
53
Impact of Teams – Call Distance
776655443322110
9
8
7
6
5
4
3
2
1
0
Call Distance
Perc
ent
Call Distances (United vs. United)
847260483624120
4
3
2
1
0
Call Distance
Perc
ent
Call Distances (Stoke vs. Stoke)
Same Referee Profile (GFU = 50, Fitness = 50)
500 Simulated games (30,000 calls) per team combination
Team combination has substantial effect on distribution of call distances.
54
Additional Findings – Recommendation for USSF
• When comparing the quality of multiple referees based on in-game performance, match difficulty in terms of game flow and team combination must be taken into consideration.
55
Agenda
1. Context
2. Problem & Need Statement
3. Design Alternatives
4. Simulation
5. Simulation Output
6. Utility Analysis
7. Conclusions
8. Management
56
Work Breakdown Structure
57
Work Breakdown: Systems 490
58
Work Breakdown: Systems 495
59
BudgetTask Predicted Velocity Cost
Research 135 hours $4,050
Referee/Game Data 244 hours $7,320
Referee Evaluation Simulator 303 hours $9,090
Formulation of Conclusions 40 hours $1,200
Communication of Results 415 hours $12,450
Project Management 330 hours $9,900
Total 1467 hours $44,010
60
Earn Value Management
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 350
200
400
600
800
1000
1200
1400
Earned Value Chart
Planned ValueActual CostEarned Value
Week (starting 8/29/2011)
Cos
t (h
ours
)
Cost Performance Index = .9289Schedule Performance Index = .954
61
Sponsor Testimony
“ The analysis done by the students has been incredibly eye-opening. They have changed the way our management at MDCVSRP think about referee development and where to use our budget.”
-Pat Delaney
MDCVSRP Chairman
Recommended