Initial consultation event: Timing and publication

A feedback survey for taught postgraduatesWorkshop 2: Timing and publication

Catherine Cameron

Senior HE Policy Adviser, HEFCE

Richard Puttock

Head of Data and Management Information, HEFCE

Future Inn, Bristol City Centre

21 September 2017

Session Outline

1. Survey coverage (10 mins)

2. Survey timing (20 mins)

3. Data linking (10 mins)

4. Publication levels (20 mins)

Survey coverage

Initial starting point was whether there were any groups of PGT students who should be excluded

Our proposal

All students on PGT courses which lead to a qualification

Excepting those on integrated Masters

Questions or comments?

Survey timing

Key consideration is to provide representative and meaningful information

Options

• Single survey window

• Post completion survey

• Multi-point survey

• Rolling survey

Survey timing

End date analysis

Whilst June/July and September/October cover most students, there are differences for:

• Particular providers

• Some students on unstructured courses

• Some students studying particular subjects

End month Percentage of students

Jan 3%

Feb 2%

Mar 3%

Apr 2%

May 4%

Jun 12%

Jul 16%

Aug 5%

Sep 34%

Oct 10%

Nov 6%

Dec 3%

Survey timing

Conclusion

• A multi-point or rolling survey is the only way to achieve responses which reflect the majority of a students’ academic experience and a sufficiently high response rate.

• Running the survey at four points in the year should be sufficient to capture a significant majority of the PGT population close to the end of their period of study.

Survey timing - questions

Your questions ….

Our questions

a. What are the challenges for institutions in delivering a multi-point survey?

b. Is there anything we haven’t considered that we should have?

Data linking

• In order to be able to analyse and represent survey responses by student characteristics we must be able to link back to individualised student data

• This will also allow us to link to other datasets: NSS, Graduate Outcomes for analysis

• Data futures – will introduce in-year data collection and potentially allow us to generate target lists

Questions or comments?

Publication

• Intend to consult on some elements of publication approach now and some following successful piloting

• The key things that we think that we will need to consider are:

• Publication routes

• Publication thresholds

• Publication level

• Breakdown of responses

• Benchmarking of responses

• Detailed approach to presentation of information

Publication

Publication routes

Envisage publishing outcomes in similar way to NSS:

• Publishing responses on the Office for Students (OfS) website

• Publishing on the central information source to support decision-making being provided by the Office for Students and the other funding bodies (assuming continuation)

• Publishing an open dataset so that other information providers can make use of this information

• Making responses available to providers through a secure system.

Publication

Publication thresholds

Propose to adopt same thresholds as used for publication of NSS

• At least 10 students

• 50 per cent of sample

Question

Our analysis is based on these thresholds and we propose not to consult on them. Are there any concerns about this?

Publication

Potential coverage

However

Expected gaps for some particular types of provision, particularly at FECs, due to very small cohort sizes

Publication level Estimated coverage (70% response rate)

Estimated number of students included

Course 19% 67%

HECoS CAH level 3 57% 94%

HECoS CAH level 2 70% 98%

Publication

Breakdown of responses

We also need to consider the course characteristics by which the survey responses should be broken down to best meet its purposes

For example, mode of study.

But …. we must be careful not to fragment to an extent where we cannot publish data

Questions

• Are there ways in which is would be essential to break down the data to make it meaningful?

• Are there ways in which it would be desirable to break it down?

PublicationBenchmarking

• We will consider benchmarking as part of the second stage of developing proposals.

• HEFCE is currently leading a fundamental review of the approach to benchmarking used in the UK HE Performance Indicators and the TEF to ensure it meets best practice.

• This will influence our approach to benchmarking of survey outcomes.

Other questions or comments?

How to find out more

website www.hefce.ac.uk/lt/PGT/