Upload
vucong
View
237
Download
1
Embed Size (px)
Citation preview
#EATPconf
EPSO – Gilles Guillard
Hudson – Nikola Trbovic
Prometric – Nicoletta Vullo
#EATPconf
Presentation outline
1. Starting situation- About EPSO, Prometric & Hudson
- Two single-provider collaborations
- Original candidate trajectory
2. Need for a new candidate trajectory- Insertion of intermediate eliminatory test
- New trajectory
- Different providers, different solutions
– Hudson’s E-tray
– Prometric
3. In practice- Multi-provider collaboration process
- Reliability
- Equal opportunities
- What worked well
- Lessons learned
28-30 September 2016 2
#EATPconf28-30 September 2016 3
#EATPconf28-30 September 2016 4
European Personnel Selection OfficeEPSO selects highly qualified staff for
#EATPconf28-30 September 2016 5
European Personnel Selection OfficeThe EU Institutions
#EATPconf
European Personnel Selection OfficeA few figures
28-30 September 2016 6
Since the creation of EPSO in 2003:
Over 600,000 candidates have participated in EPSO
selection processes
Over 510,000 EPSO candidates have taken a
computer-based test delivered via Prometric
Over 70,000 CBT exams were delivered in 2015
Between 1 % and 2 % of all candidates in EU open
competitions are successful and can be recruited
Some competitions have attracted up to 52,000
applicants for the approx. 300 positions offered
#EATPconf
Present Prometric
28-30 September 2016 7
About Prometric
Wholly owned subsidiary of ETS
A trusted market leader in technology enabled testing and assessments
Comprehensive suite of services, including test development, test delivery and data management capabilities
350+ clients in every major
industry segment
Over 9 million exams
delivered annually
160 countries
26 languages
#EATPconf
Starting situation:
Two single-provider collaborations
28-30 September 2016 8
Prometric has been providing Computer-Based Admission
Testing to EPSO since 2005
Over the years, Prometric has implemented innovative
and tailored solutions to meet EPSO’s specific testing
needs
Prometric exam delivery for the EPSO includes:
Unproctored internet-based testing in candidates’ locations;
Proctored Multiple Choice exams of various types, Case
Studies, Audio-Video exams in the Prometric testing network;
In all 24 official EU languages and across all 28 EU Member States and worldwide
#EATPconf28-30 September 2016 9
About Hudson
#EATPconf28-30 September 2016 10
About Hudson Belgium
Since 1982 (°De Witte & Morel): more than 30 years experience
Market leader with a strong reputation
6 regional offices: Brussels, Gent, Antwerp, Kortrijk, Hasselt & Louvain-La-Neuve
Over 200 people including over 100 consultants
More than 85% loyal customers
Worldwide Research & Development Center located in Belgium
Partnerships with universities & top business schools e.g. Vlerick Business School, Université
catholique de Louvain, Louvain School of Management, UGent, …
REWARD
MANAGEMENT,
COMPENSATION &
BENEFITS
ASSESSMENT &
DEVELOPMENT
CENTERS
CAREER GUIDANCE
INTERIM
MANAGEMENT &
CONTRACTING
RECRUITMENT &
SELECTION
COMPETENCY &
PERFORMANCE
MANAGEMENT
LEADERSHIP
DEVELOPMENTJOB GRADING
#EATPconf
Starting situation:
Two single-provider collaborations
Hudson as trusted advisor for EPSO — Assessment Centres
Develops new test material (and test variations) and
accompanying documentation (marking grids, manuals,
guidelines)
Adapts pre-existing tests and provides off-the-shelf exercises
Trains permanent and non-permanent Selection Board
members, markers and EPSO support staff
Provides advice to EPSO on the further development of its
Assessment Centres
Provides IT tools, support and services relating to the
Informatics tools used by EPSO
28-30 September 2016 11
#EATPconf
Starting situation:
Original candidate trajectory (AD-cycle)
28-30 September 2016 12
MONTHS1 9
1 month 4 to 6 weeks 2 to 3 months
33,837
candidates25,544
candidates
504
candidates
185
laureates
#EATPconf
2. Need for a new candidate trajectory
28-30 September 2016 13
#EATPconf
New candidate trajectory
Intermediate test: E-tray
28-30 September 2016 14
Decision to insert an intermediate eliminatory test between pre-selection tests (CBT) and Assessment Centre: invite 10 x the number of candidates sought to take the e-tray in order to reduce invitees to AC to 2 x the number of candidates sought
Invite only the very best candidates to AC Cost effective (financial & human resources) Improved gender balance Assesses generic competencies at an early stage Automated scoring Limited scope for training
#EATPconf
New candidate trajectory:
Need for a multi-provider collaboration
28-30 September 2016 15
#EATPconf
New candidate trajectory
28-30 September 2016 16
40,666
candidates
27,388
candidates
2,190
candidates541
candidates235
laureates
MONTHS1 9
#EATPconf
New candidate trajectory:
Different providers; different solutions
28-30 September 2016 17
Implementing the new trajectory was not without challenges:
EPSO’s needs Hudson Prometric
Testing facilities and administrators in 28 MS
E-tray to measure competencies
IT platform for test administration
(need for alignment)
Equal opportunities for all candidates in all MS
Parallel tests and randomisation of
test material
Standardised testing conditions
in all MS
#EATPconf
New candidate trajectory
Description of Hudson E-tray
28-30 September 2016 18
Computer-based test using documentation provided in the form of an email inbox
Duration: 50 minutes
Assesses 4 general competencies:
Analysing & Problem-solving;
Delivering Quality & Results;
Prioritising & Organising;
Working with Others
#EATPconf
New candidate trajectory
Hudson E-tray
9/20/2016 19
Step 0:
EPSO request
Step 1:
Development planning & check
NoC
Step 2:
Contact Subject Matter Expert
Step 3:
Meet Subject Matter Expert
Step 4:
Develop exercise
Step 5:
Selection Board validation & adaptations
Step 6:
Linguistic revision and translation
Step 7:
Trial session
Step 8: Psychometrics
Step 9:
IT programming
#EATPconf
New candidate trajectory
Hudson E-tray
28-30 September 2016 20
#EATPconf
New candidate trajectory
Prometric solution
9/20/2016 21
Prometric’s Web-launch technology supports clients who have their own browser-based test driver but who require a secure, proctored environment for test delivery
Web-launch acted as a proxy for Hudson’s website, providing an interface with Prometric’s administrative system
Hudson’s website had to operate using the infrastructure in place at the Prometric testing centres
In summary, using Web-launch required integration between Prometric’s and Hudson’s systems
#EATPconf
Prometric solution
Web-launch functionality
Start the browser and navigate to Hudson’s start URL
using agreed parameters passed in the URL
Interface to the administrative system to set the status
of test delivery
Maintain a test seat session timer
Ensure that candidates navigate only to known
internet domains
9/20/2016 22
#EATPconf
Prometric solution
Web-launch functionality
Monitor for pages that indicate conditions such as
end-of-test or error
Monitor for a loss of internet connectivity
Support restart of a candidate’s sessions
Close the browser and return control back to the
administrative system
9/20/2016 23
#EATPconf
Prometric solution
Hudson’s browser
Supported the Start and End control pages
Supported checkpoint and recovery of testing sessions
Supported test-specific timing (for ex-time extension
for Testing Accommodations)
9/20/2016 24
#EATPconf
3. In practice
28-30 September 2016 25
#EATPconf
In practice:
Multi-provider collaboration process
28-30 September 2016 26
Explore the capabilities of Hudson and Prometric
Define the requirements
for information flow between
the three parties
IT implementation
(platform integration)
Training of test administrators by Prometric
Trials on dummy candidates in
Brussels
First roll-out:
1,500 candidates
6 days
28 MS
#EATPconf
Hudson and Prometric readiness for Web-
launch implementation
Prometric and Hudson determined parameters to be
passed from Web-launch to Hudson’s website
Prometric and Hudson determined Start and End URLs
for QC and Production
Prometric published service scripts for Hudson tests,
representing parameters, URLs, and other
configuration decisions
9/20/2016 27
#EATPconf
Hudson and Prometric readiness for Web-
launch implementation
A test harness was created and Prometric and Hudson
performed integration testing in the QC environment
Four dry-runs were held in Brussels, one of which used
real candidates’ eligibilities
An escalation matrix was established between EPSO,
Prometric and Hudson prior to going live.
9/20/2016 28
#EATPconf
Prometric readiness for test delivery
in the Prometric network
Information was gathered on the ISP bandwidths of all EU28 testing centres
Hudson’s exams were downloaded in Prometric technical support labs in order to issue knowledge-based documentation
Beta testing was conducted in all participating EU28 MS testing centres (38 locations in total)
Hudson verified all results
A Command Centre was put in place during go-live, and bi-hourly reports were issued during testing
9/20/2016 29
#EATPconf
In practice:
Challenges
28-30 September 2016 30
The process was not without challenges:
A few limits to the integration of both IT platforms (example: abort functionality not possible)
Candidates had some questions on the scoring (need to publish more information on scoring methods on EPSO website/ombudsman enquiry)
Isolated technical issues encountered during testing (restarts, loss of connectivity, etc.)
#EATPconf
In practice:
Reliability
28-30 September 2016 31
E-tray test is stable and reliable.
Analysing &Problem Solving
Delivering Quality &
Results
Prioritising &
Organising
Working with
others
Cronbach's
Alpha.528 .567 .504 .575
#EATPconf
In practice:
Equal opportunities
28-30 September 2016 32
In terms of equal opportunities, no adverse impact for gender
ANOVA
Sum of Squares
dfMean
SquareF Sig.
APS_C Between Groups 1.220 1 1.220 .508 .476
Within Groups 1867.294 777 2.403
Total 1868.514 778
DQR_C Between Groups .810 1 .810 .223 .637
Within Groups 2818.667 777 3.628
Total 2819.477 778
POR_C Between Groups 1.758 1 1.758 .794 .373
Within Groups 1719.903 777 2.214
Total 1721.661 778
WWO_C Between Groups 5.989 1 5.989 2.452 .118
Within Groups 1898.246 777 2.443
Total 1904.236 778
Count
Gender_Female 341
Gender_Male 438
Total 779
#EATPconf
In practice:
Equal opportunities
28-30 September 2016 33
WITH E-TRAY INTERMEDIATE STAGE
Increased gender balanced achieved:
WITHOUT E-TRAY INTERMEDIATE STAGE
#EATPconf
In practice:
Equal opportunities
28-30 September 2016 34
In terms of equal opportunities, no adverse impact for test language
Count
language DE_BE 18
EN_INT 702
FR_BE 59
Total 779
ANOVA
Sum of Squares
df Mean Square F Sig.
APS_C Between Groups11.687 2 5.843 2.442 .088
Within Groups 1856.827 776 2.393
Total 1868.514 778
DQR_C Between Groups19.409 2 9.705 2.689 .069
Within Groups 2800.068 776 3.608
Total 2819.477 778
POR_C Between Groups1.483 2 .742 .335 .716
Within Groups 1720.178 776 2.217
Total 1721.661 778
WWO_C Between Groups5.019 2 2.509 1.025 .359
Within Groups 1899.217 776 2.447
Total 1904.236 778
#EATPconf
In practice:
Equal opportunities
28-30 September 2016 35
In terms of equal opportunities, no adverse impact for test day
Count
Testingday 09/21/2015 212
09/22/2015 270
09/23/2015 297
Total 779
ANOVA
Sum of Squares
df Mean Square F Sig.
APS_C Between Groups11.225 2 5.612 2.345 .097
Within Groups 1857.289 776 2.393
Total 1868.514 778
DQR_C Between Groups2.711 2 1.355 .373 .689
Within Groups 2816.766 776 3.630
Total 2819.477 778
POR_C Between Groups1.112 2 .556 .251 .778
Within Groups 1720.549 776 2.217
Total 1721.661 778
WWO_C Between Groups13.750 2 6.875 2.822 .060
Within Groups 1890.486 776 2.436
Total 1904.236 778
#EATPconf
EPSO’s idea in practice
What worked well
28-30 September 2016 36
Interaction between the three parties was very positive,
prompt and transparent
Dry-runs were effectively used to test and validate the exam
prior to delivery and helped identify and solve a number of
issues
Command centre structure and daily testing reports were
effective
Operations worked well for a first-time delivery, with no
candidates displaced and five re-tests in total
Escalation path and communication with Hudson worked
very well
What worked well
#EATPconf
EPSO’s idea in practice:
Lessons learned
28-30 September 2016 37
The multi-provider collaboration was a positive experience
Will be retained for future competitions
Further improvements are being introduced to enhance the
process
A dashboard was created to facilitate follow-up in
real time by EPSO
28-30 September 2016 38