For the Executive Committee Item ExCo_30_1_2019_0830th January 2019

From: Professor Andrew Jones and colleagues

Subject: Preparing for the Research Excellence Framework 2021

1.0 Background

Research is central to City’s future, and the Research Excellence Framework (REF) exercise is an important way in which this work is recognised and rewarded. The REF result greatly influences our competitive position and, thereby, affects our ability to continue to attract excellent staff and students. The REF result also determines how ‘QR’ money is distributed across the HE sector, and this money funds staff salaries, research and impact even when it is not supported by a research grant or contract. High quality research underpins our education, our reputation and our contribution to society. Given the value of this exercise, it is important that we are well prepared to ensure our submission properly represents the research excellence at our university.

The next Research Excellence Framework (REF2021) must be submitted by 27 November 2020, with the results due in December 2021. It will be based on a census of staff who are in post with a significant responsibility for research on 31 July 2020. City performed well in the last exercise (2014), receiving an increase in QR funding. This time, however, the rules are different and we face increasing competition from other Higher Education Institutions (HEIs). Our institutional preparations are underway, with a census of outputs taking place and eligible impact case studies being identified.

There are four main REF panels sub-divided into 34 disciplinary based Units of Assessment (UoA). The REF sub-panels, comprising national and international experts, will assess the quality of research outputs (e.g., journal articles, monographs and performances), research impact and research environment submitted by each UK university. Summary results in the form of a quality profile will be published at the UoA level for each university.

It is a strategic priority for City to increase the number of research-excellent staff submitted into the REF (precise definition varies according to the % 4* aspiration of each UoA), and this aligns with the new REF2021 requirement for institutions to submit ‘all staff with a significant responsibility for research’. The university is committed to supporting academic staff to become 'REF ready' (with internationally significant 3* quality, or better, outputs). This is central to School strategic plans that Deans and others have been implementing over the past couple of years.

The outcomes of each REF are so important to City that we require a REF strategy. The REF may also be thought of as not only a periodic measure of our research quality but also as a guide to informing each School’s approach to research.

This document outlines how City will prepare for the exercise. It sets out what information will to be collected and what will be written. It provides some details as to who is involved and what timetable we are working to. The document also deals with the most important aspect of REF2021 planning, namely, staff inclusion. The first iteration of UoA modelling, based on existing ARQM data, is also presented.

1

2.0 Components of the REF2021 Submission

a. Staff details (‘REF1’): Information on all academic staff in post on the census date (31st July 2020) and who have ‘a significant responsibility for research’. Such staff are to be determined by a combination of contract type (must be a contract that is ‘Research Only’ or ‘Research & Education’) plus some form of auditable documentation where such a contribution is expected during this REF period (e.g. role profile, career pathway or workload model).

b. Research Outputs: (‘REF ‘2) Details of 1-5 research outputs produced by each member of submitted staff during the publications period (1st January 2014 to 31st December 2020). All journal articles and conference papers submitted to the REF must have been placed into Open Access (e.g., deposited into the City Research Online publication repository) at the time of acceptance, from 1st April 2016 onwards. There will be more reliance on metrics than in REF2014. Perhaps 11 out of 34 sub-panels will use citation counts as compiled annually by the Journal Citation Reports (produced by Clarivate Analytics’ Institute for Scientific Information (ISI) and where used, citation counts will support, not supplant peer review. Sub-panels will not use journal quality metrics such as journal impact factors or other journal rankings such as the Association of Business Schools Academic Journal Guide to make quality decisions.

c. Impact case studies (‘REF3’): Case studies describing specific examples of impacts achieved during the assessment period (1st August 2013 to 31st July 2020), underpinned by excellent research in the period (1st January 2000 to 31st December 2020). The number of case studies required is 2 for up to 20 Category ‘A’ FTE staff submitted plus 1 more case study for each additional 15 (or fewer) FTEs.

d. Research Environment data (‘REF 4a/b/c’): Data on research doctoral degrees awarded each year in the period (REF 4a); the amounts and sources of external research income for each year in the period (REF 4b); and the amount of research income-in-kind for each year in the period (REF 4c); where the ‘period’ is 1st August 2013 to 31st July 2020.

e. Research Environment template (‘REF 5’): Textual information about the research environment on a template that is more structured than last time. Section 1. Unit context and structure, research and impact strategy; Section 2 People including staffing strategy and staff development, training and supervision of, PGR students; and evidence of how the submitting unit supports and promotes equality and diversity. Section 3. Income, infrastructure and facilities including those facilities pertaining to research and research impact. Section 4. Collaboration and contribution to the research base, economy and society including relationships with key research users, beneficiaries or audiences; and the wider activities and contributions to the research base, economy and society. This again relates to the period 1st August 2013 to 31st July 2020.

3.0 REF2021 Pre-Planning Phase

2

3.1 Establishment of REF2021 decision-making bodies

The Research & Enterprise Implementation Plan 2016-2021 states that City will establish two REF2021 groups: a small strategy group and a larger operations group. The separation of strategy and operations is a key lesson from REF2014. A third group comprising UoA leaders will also be established to promote knowledge sharing between those writing the REF submissions (Table 2).

(i) REF Strategy Group (Table 1)

The REF Strategy Group is a sub-committee of Research & Enterprise Committee: it is responsible for overseeing City’s strategy for, and submission to, the REF2021. The purpose of the Research & Enterprise Committee is to advise the Vice-President (Research and Enterprise) on progress towards the achievement of City’s strategic aims relating to research and enterprise.

The REF Strategy Group brings together staff with previous REF panel experience and subject representation from all four main REF Panels: Physical Sciences, Biological Sciences, Social Sciences and Arts & Humanities. Over time, if our nominees become REF2021 sub-/panel Members they too will be invited to join the REF2021 Strategy Group.

Table 1: REF2021 Strategy Group

REF 2021 Strategy Group: Meeting Dates for 2018/19

Date Time Room

Tuesday 2nd October 2018 2:00pm – 4:00pm A103

Thursday November 29th 2018

11:00am – 1:00pm A103

Thursday February 21st 2019 2:00 – 4:00pm A103

Thursday April 18th 2019 2:00 – 4:00pm A103

Thursday June 20th 2019 2:00 – 4:00pm A103

Table 2: REF2021 Unit of Assessment Leaders Group

REF 2021 Unit of Assessment Leads Meeting Dates for 2018/19

Date Time Room

Wednesday October 17th  2018

2:00pm – 4:00pm College Building, AG03

Thursday January 17th  2019 11:00 – 1:00pm College Building, AG04

Thursday March 21st 2019 2:00 – 4:00pm Tait Building, C103

Thursday May 23rd 2019 2:00 – 4:00pm College Building, AG04

3

(ii) The REF Operations Group

The REF Operations Group takes direction from and reports to the REF Strategy Group. It comprises mostly Professional Services staff charged with translating the outcomes from the Strategy Group, mining systems and spreadsheets and tracking the REF timetable. The Group is much larger than the Strategy Group with representatives from Research & Enterprise Office (main contact for REF2021), Strategic Planning and Performance Unit (modelling draft submission data), Library Services (open access), and Human Resources (staff submission, REF code of practice); Research Finance (research income); and Graduate School (PGR graduate data). The REF Operations Group conducts most of its business by email but meets when necessary.

4.0 Developing City’s Code of Practice

Each institution making a submission is required to develop, document and apply a Code of Practice (CoP) on the fair and transparent identification of staff with significant responsibility for research; determining who is an independent researcher; and the selection of outputs.

The CoP will aid institutions in promoting equality and diversity, complying with legislation and avoiding discrimination when preparing submissions to the REF.

Research England has developed guidance based on advice from their Equality and Diversity Advisory Panel which is intended to assist HEIs in drawing up a CoP that frames their decision-making processes in relation to REF 2021. City submitted a CoP on staff selection as part of the 2014 REF exercise. While the changes to the submission process for REF 2021 require institutions to focus their Codes differently, there will be aspects of our previous Code that we can build upon for REF2021.

We can also consult the overview report which identified areas of good practice found in the 2014 exercise:

(www.ref.ac.uk/2014/pubs/refcodesofpracticegoodpracticereport/#d.en.75885)

City also carried out an equality impact assessment after making our 2014 REF submission and we need to look at any actions that City committed to from that process.

4.1 Main actions

The CoP expects universities to consult Trade Unions on the processes to be followed in selecting staff and outputs for REF. This gives affected staff some assurance over how we will determine who has and who does not have ‘a significant responsibility for research’. The consultation on the principles underpinning our REF CoP will be underway once we receive the final CoP Guidance document which is due in January 2019. This will enable us to incorporate feedback received from City staff into City’s draft REF CoP.

The REF CoP will be drafted by staff in the Research & Enterprise Office with significant input from the Equality and Diversity (‘ED’) Team in Human Resources. A timetable has been developed (Table 3) and the deadline for submitting the CoP is 7th June 2019. The REF Equality and Diversity Advisory Panel within Research England will advise the UK funding bodies on the Codes adherence to the guidance, prior to approval and publication.

4

All submitted and approved Codes will be published before the submission deadline, provisionally in December 2019.

Table 3: Timetable for developing City’s REF Code of Practice

Activity When WhoDrafting City’s REF Code of Practice

December 2018 – January 2019 R&E, HR (ED)

Consultation with School Deans

February-March 2019 R&E, HR (ED)

Approval of the Code of Practice

April – May 2019 REF Strategy Group

Submission of Code of Practice to Research England

June 2019 R&E

ED Training of internal staff and output selection panel members

July, September – December 2019 HR (ED) with R&E

Internal REF Panels use Code of Practice when finalising the selecting staff and outputs

Throughout 2020 School REF Panels

Staff appeals against REF exclusion

February 2020 – June 2020 School Deans, R&E, HR (ED)

Equality and Diversity Audit post REF submission

January - April 2021 HR (ED) with R&E

5.0 Developing City’s REF2021 Outputs Submission ‘Model’

For City to be able to identify how well it is likely to perform in REF 2021, the REF Strategy Group commissioned the Strategic Planning and Performance Unit to analyse how each Unit of Assessment (UoA) would fare (as of January 2019) based on the REF 2021 rules.

The modelling exercise involves using the Annual Research Quality Monitoring (ARQM) dataset of research outputs for the period 2014-2018 but adjusted for REF2021 rules e.g. a maximum of 5 outputs per eligible staff member rather than 4 as per ARQM. (The ARQM is City’s internal exercise which assesses the quality of research outputs and it is carried out by peer review panels or sub-panels within Schools).

The Outputs Submission Model aligns the 2018 ARQM data against the REF guidance on staff inclusion criteria, research outputs, and REF panel structure and when complete will give us a reliable indicator as to the strength of our submissions if we had to submit tomorrow.

As we expect research income data and postgraduate research student completion data to be requested by the REF2021 Panels, these datasets will be added to the model for each UoA once they become available.

The first iteration of the Outputs Submission model is complete. It aims to allocate outputs from the 2018 ARQM exercise to academic staff within each UoA based on the current criteria for REF2021 inclusion. These criteria include the following key elements: a minimum of 1 output per academic, a maximum of 5 outputs per academic and an average of 2.5 outputs per FTE across the UoA. The model works by ranking all the eligible outputs from

5

ARQM data available for the REF2021 period to date and allocating each to an eligible academic based on three rules:

First, all single author 4* outputs are allocated to an academic since there is no other individual they can be allocated to and we will always want to include these outputs.

Second, an output is allocated to all remaining (current) academics that require one – NB. For departed academics we can use any of their outputs produced while at City (hopefully mainly 4* outputs), but they do not count towards the UoA’s FTE.

Finally, the remaining outputs (co-authored outputs and 3*, 2* & 1* single author outputs) are allocated, the highest rated outputs being allocated first.

All of these allocations occur until an academic reaches a maximum of 5 outputs – at which point, no more outputs are allocated.

The model has been designed in such a way that it can be re-run as and when required; for example, when new outputs are added or when ARQM 2019 data become available. While developing the model, several issues became apparent that could affect the results – relating to both staff and outputs. These are outlined in Section 6.3. As many as possible of these issues will need to be resolved or considered as we seek to finesse the model to improve the accuracy of further iterations. The more accurate the model, the better will be the decision making about changes we can put in place to optimise City’s eventual REF outcome.

6.0 The Mainstream Planning Phase (January 2019 to November 2020)

There are 6 key areas of the REF submission which require significant on-going work:

1. Determining the cohort of staff to be submitted: those with ‘a significant responsibility for research’

2. Determining the research outputs to be submitted3. Iterative testing of the City model (above) to track progress in optimising City’s

submission. The model will be run following the implementation of decisions taken to optimise the return of each UoA according to the REF2021 rules.

4. Development of Impact case studies5. Describing the research environment at UoA and institutional levels6. Run a ‘mock REF submission’ in late spring 2019 which is intended to cover not just

outputs but also the environment and impact components of the REF exercise.

6.1 Determining who has a ‘significant responsibility for research’

REF 2021 states that all staff with ‘a significant responsibility for research’ should be submitted. This will not be determined by employment contract type alone. Instead, at City ‘a significant responsibility for research’ will be determined by the member of academic staff’s academic role profile (ARP). The final REF Guidance will set out a ‘menu’ of what we consider may be appropriate indicators of ‘a significant responsibility for research’. REF2021 indicates four key elements to this within institutions:

(i) Institutional identification of staff in scope of the policy (all staff on Research & Education or Research Only employment contracts, except Research Assistants)

(ii) Those required to carry out research – auditable documentation where there is no expectation of this (e.g. workload model)

(iii) Identification of eligible staff requires agreement between the institution and staff

6

(iv) Institutions will have to set out a process for identification in their Code of Practice

At City, most academic staff are on a standard Education and Research employment contract, with the exception of a small proportion of academic staff who are on an Education contract. A considerable number of academic staff who are education-focused in their activities are therefore on an Education & Research contract. However, since 2014/15, staff have been aligned with an ‘Academic Role Profile’ (ARP) which describes their major responsibilities as either Education and Research, or Education. If the staff member is largely focussed on education for example, they will be described as ‘Education & Research contract; Education ARP’). All academic staff now have an allocated ARP, and revisions to the text of the 2014 version of the role profiles is in the final stages of agreement with the Trade Unions. The primary route for identifying academic staff with ‘a significant responsibility for research’ is via their ARP. New academic appointments will normally align ARP and employment contract.

For the whole of this REF period, Schools (led by the Deans) have developed academic staffing plans that outline their planned proportions of staff on each of these ARPs. A process has been undertaken to make sure academic staff are on the correct ARP in terms of their responsibilities. Several staff have moved or will move ARP from an Education & Research ARP to an Education ARP where they do not have ‘a significant responsibility for research’. This process is ongoing and accords with the timetable shown in Table 4.

Human Resources has also worked through the SAP database and clarified which staff are Research Fellows (and hence independent researchers) and which are Research Assistants (and as such, would not be submitted to REF). The HESA staff return will be aligned with these revisions from 2018/19. Importantly, for REF2021 a HESA ‘flag’ will be allocated to staff to identify those with ‘a significant responsibility for research’ and work is being undertaken to make sure City’s return correctly aligns with the ARPs.

The key issue and risk for City in relation to staff identification relates to the proportion of staff in each School who are currently on an Education & Research ARP, but have an ARQM Grade Point Average (GPA) below 3 (and no 4* output). City will have to return these staff to REF2021. As City’s approach to REF2021 is to confine the submission to 4* and 3* outputs as far as that is possible, submitting staff who have GPAs lower than 3 will reduce the average quality of each of our UoA submissions (more so in some UoAs than others) and compromise City’s overall REF submission. A separate paper to ET has set out the research intensity gap in each School. Tactical decisions will need to be taken by the REF Strategy and UoA Leaders Groups during 2019 to close the gap.

REF2021 retains an allowance for early career researchers (ECR) by allowing a reduction in the required output pool of up to 1.5 outputs depending upon when the ECR started in their academic post. This applies on a sliding scale from 2016 onwards. An ECR starting after August 2018, for example, can be submitted with 1 output without undermining the 2.5 per FTE average requirement).

Table 4: Timetable for determining who has ‘a significant responsibility for research’

7

Activity When WhoRefer to school academic staffing plans over the REF period but starting with the most recent.

March – July 2019 ADREs

Set up School REF Panels, to meet quarterly

Jan, April, July, October 2019Jan, April, July, October 2020

ADREs

Validate Academic Role Profiles (for HESA and REF planning purposes)

May – October 2019, May – October 2020

HR, School Deans

Modelling ARQM data against known REF unit structure and issue guidance to estimate staff included

May – July 2019 SPPU

Internal REF Panels use Code of Practice when selecting staff and outputs

Throughout 2020 ADREs, School REF Panels

REF1: List of staff submitted to REF validated by REF Strategy Group

July 2020 REF Strategy Group

Staff appeals about their exclusion or exclusion of their outputs

February 2020 – June 2020 School Deans, R&E, HR (ED)

6.2 Research Outputs (60% weighting in the submission)

Submissions must include a set number of items of research outputs, equal to 2.5 times the combined FTE of Category A submitted staff included in the submission (Figure 1).

8

Figure 1 – Determining the research output pool in REF2021.

6.2.1 Open Access research outputs

The core of the REF 2021 open access policy is that journal articles and conference proceedings must be available in an open access form to be eligible for submission to REF2021. This means that these outputs must be uploaded to an institutional or subject repository for example, City Research On-line (CRO).

Library services send monthly reports to Schools informing them of uploads to the CRO and the names of staff who have not uploaded REF-type publications. School Deans and Heads of Department are chasing staff about these.

Library Services along with Research and Enterprise are encouraging academic staff to obtain and use an ORCID identifier. The REF Guidelines strongly encourages but does not require academics to register for ORCID identifiers for the next REF. These are numerical codes used to identify a particular individual, and to distinguish them from other academics with the same or similar names.

6.2.2 Outputs of staff who have left City

Under the REF2021 principle of ‘Transitional approach to non-portability’, research outputs of former staff may be submitted by City if the output was demonstrably generated while they were here. ‘Demonstrably generated’ will be determined by the date when the output was first made publicly available. A full set of eligibility criteria for output submission will be issued soon. To collect the outputs of staff who have left City, we have set up a project to track down eligible publications.

6.2.3 Fractional-contract academic staff

Schools have been asked to give special attention to any staff on 20% contracts. First, like all others entered to the REF, these staff must upload the final author version of their REF publications to CRO or to another institutional or subject repository. Second, these staff should have a substantive connection with City (with a statement required). The substantive connection is best evidenced by the length of time they have been at City but could also

9

include their involvement in other research activities such as PhD student supervision or advice, teaching, joint publications, or any other role to show that people are working together. It is vital that all staff on 0.2 FTE contracts be mentioned in the Research Environment statement otherwise they will likely be deleted from the submission. If there is an unsubstantial connection with City we should not include them in our submission.

6.2.4 Sorting 3* outputs into High Quality 3* (3*plus) and Lower Quality 3* (3*minus) outputs

For the majority of City’s REF submissions, the quality threshold for research outputs will be within the three star outputs. It is therefore important to know which 3* should be included and which to exclude. We propose to ask School ARQM panels to consider this issue when they assess research outputs in both 2019 and 2020. The outcome of this will be a set of research outputs assessed using the REF grading system 4*, 3* etc. but with the 3* publications further divided into: ‘High’ ‘Medium’ and ‘Low’.

Table 5: Timetable for deciding which research outputs to submit

Activity When WhoSupply of current staff list to Schools for ARQM 2019

January 2019 HR

Academics nominate outputs for ARQM and REF planning

February 2019 Individual academics, ADREs

Selected outputs will be cross-checked against open access

February 2019 Individual academics

ARQM Panels meet and decide quality grading of selected outputs

April 2019 School ARQM panels

Appointment of REF Project Manager

April 2019 R&E

Deadline for staff to appeal against their ARQM scores

May 2019 School ARQM panels

Project to discover all the research outputs of staff who have left

January – March 2019 R&E

Sorting 3* outputs into ‘Strong’ ‘High’ or Medium 3* and ‘Low 3*’ outputs

May – July 2019 School ARQM/REF Panels

Modelling City’s Research Outputs against REF2021 assessment criteria, in time for REF rehearsals

December 2018 – May 2019 SPPU

REF Rehearsals – preliminary staff inclusion lists for each of the 14 UOAs

May – July 2019 R&E, REF Strategy Group

Final Modelling City’s Research Outputs against REF2021 assessment criteria (2020). Updating the 2019 project.

April – July 2020 SPPU

10

Papers %4* 517 48%3* 504 47%2* 46 4%1* 5 0%

1,072 100%

Total staff FTEcurrently due to be

submitted to REF 2021

426.4

6.3 Modelling City’s Outputs Submission

6.3.1 The Position at January 2019

From the initial modelling exercise, the following table shows the sum of the outputs to be submitted for each of the 12 UoAs and shows the current position regarding the proportion of outputs at each rating. The total staff FTE is the aggregate of all the Units’ FTEs. (Due to rounding differences across the UoAs 2.5 x 426.4 does not equal 1,072.)

The data below are derived by calculating the outputs required (2.5 outputs x the UoA FTE). A line is then “drawn” at this point down the list of allocated outputs where the output pool is complete. The proportion (%) of different grades of output within the output pool can be calculated to create an output pool profile.

Table 6: Outputs submission modelling by UoA at January 2019 compared to REF2014

UoA

Rating %2014 REF

%2014 REF

%2014 REF

%2014 REF

%2014 REF

%2014 REF

4* 65% 37% 55% 15% 58% 11% 44% 17% 70% 10% 6% 19%3* 33% 42% 42% 56% 30% 59%2* 1% 3% 0% 0% 0% 35%1* 1% 0% 0% 0% 0% 2%

GPA 3.63 3.15 3.52 2.8 3.58 2.65 3.44 2.90 3.70 3 2.73 2.71Staff FTE 107.9 78 63.3 68 21.8 32 13.3 16 43.4 54 33.5 21Power 391 246 223 190 78 85 46 46 160 157 92 57

UoA

Rating % 2014 REF

% 2014 REF

% 2014 REF

% 2014 REF

% 2014 REF

% 2014 REF

% 2014 REF

4* 30% 14% 35% 14% 44% 17% 36% 22% 29% 33% 36% 17% 32% -3* 68% 63% 54% 64% 71% 58% 64%2* 1% 2% 2% 0% 0% 7% 5%1* 0% 0% 0% 0% 0% 2% 0%

GPA 3.29 2.73 3.32 2.70 3.42 2.54 3.36 2.82 3.29 3.10 3.31 2.67 3.27 -Staff FTE 31.3 22 32.2 28 23.5 14 14.1 25 9.6 9 24.0 13 8.5 -Power 103 60 107 76 80 36 47 71 32 28 79 35 28 -

EnglishSociology Psychology Economics Journalism / Lib/Inf

Music Int Politics

Business / Management

Health Computing Mathematics Engineering Law

11

6.3.2 Refining the Submission Model

(i) Staff

Academics no longer at City, whose papers we can use - work is about to start that will ensure that we have gathered all the outputs from departed academics who have 4* and 3* outputs.

Role profiles – we know that the role profile of some academic staff will change prior to the next staff HESA return.

Staff with one output – Schools are engaged in supporting academic staff with an ARQM GPA below 3 but who have a 4* or 3* output. A key goal is to increase the number of outputs they have published prior to REF. Support and effort between now and December 2020 will be targeted towards any potential outputs staff which could be eligible.

New staff yet to be recruited – these are likely to be research-excellent staff and so will enhance the UoA profile.

Staff who may yet leave City – we will monitor this up to the REF census date and alter the list of current staff accordingly.

“Discounts” for Early Career Researchers (‘ECR’s), maternity or sick leave – these have currently not been accounted for – work needs to be done to ensure that we capture all absences or ECRs in order to maximise the proportion of 4* outputs in each UoA.

(ii) Outputs

Missing papers – there is a risk that some papers were not submitted / included in the data produced for ARQM 2018. We will need to ask academic staff to review their best publications to ensure that they have been included / considered.

Unrated papers – for each UoA, several outputs do not have a rating in ARQM 2018. This will be checked, but hopefully this is principally due to them being lower quality outputs.

Pipeline publications – to predict / determine City’s likely REF submission in Dec 2020 for each UoA, any likely 4* papers that are about to be published will be determined. The UoA Leaders Group and the REF Strategy Group will consider this during their deliberations.

Citation counts – no metrics have been included in the analysis of citation count – panels have been instructed not to consider citation, although it is believed that it is still likely that a highly cited paper will be viewed more favourably than a lower cited one. Citation data for marginal outputs will be used as an informative metric in deciding marginal outputs to include in the output pool.

Journal “weighting”– panels have been instructed not to consider the journal in which a publication appears, but we should at least consider favouring, at the margin, outputs that have appeared in higher-rated journals.

Re-rating of 3* papers – all of City’s UoAs will need to decide which 3* outputs will be included in each submission. Consequently, some work will need to be carried out that decides which 3* outputs are ‘high’, ‘medium’ or ‘low’.

12

Double-weighted outputs – e.g. books. No outputs have been given double weighting. Where an output is double-weighted (and 4*), one less 3* output will be included.

Output type exclusion – UoAs will have different criteria for output types submitted. It has been assumed that 2018 ARQM only includes those output types we would consider submitting to the REF. We have not ruled out the need to exclude some outputs from the analysis.

ARQM 2019 – we expect fresh data in late spring when we will be able to conduct an analysis on more recent data.

Rank of authorship (i.e., whether an academic is listed as 1st or 7th author on an output) – there are differing views as to whether this should or should not affect inclusion in our submission. Clarity will be required before we run another iteration.

Cross-unit outputs – Co-authored cross-UoA papers have been may have been included for more than one UoA. This will be checked at a later stage in the analysis.

Open access – no attempt has been made to check whether outputs have been placed on a suitable open access platform within the appropriate timescales. It has been assumed that all outputs included in previous and subsequent ARQMs will satisfy this criterion.

Development of Impact case studies (25% weighting in the submission)

The weighting of impact cases in REF2021 has increased to 25% since REF2014 and so they are now even more important. To support the development (but not the writing) of impact case studies, Schools have access to City’s Higher Education Innovation Fund (HEIF) allocation. HEIF supports a broad range of knowledge-based interactions between universities and the wider world, which result in benefits to the economy and society.

Over the past year, Schools have engaged impact consultants to give further support to authors for writing their REF impact case studies. These include Saskia Gent from Insights for Impact; Professor Mark Reed from Fast Track Impact and Saskia Walcott from Walcott Communications. The feedback from academics about these impact consultants has been positive. On the back of this and in line with the rest of the sector, City has recently approved funding for two Research Impact Officers – one focusing on SASS which hosts over 50% of our REF UoAs and the other shared across the other four Schools. These posts will enhance the impact support to UoA leaders.

Schools have compiled short lists of potential impact case studies and are working with those staff with the highest potential of delivering a case study of 3* quality or higher. Those departments that are likely to struggle to find the minimum number of impact cases are receiving special attention. Once appointed, the Research Impact Officers will work closely with Associate Deans for Research & Enterprise and Business Development Managers to support the full assembly of impact case studies.

2019 actions to support the development of impact cases include:

1) Specific development activities for certain impact case studies (will need some funds and project management support, working in conjunction with the Business Development Managers from Research & Enterprise)

13

2) Impact evidence gathering to support claims made in the case studies. This will be the main responsibility of the two newly appointed Research Impact Officers who we anticipate being in post by spring 2019 at the latest.

3) Marketing and promotion of our case studies which can lead to further engagement (impact) by research users. This may include reframing research and enterprise aspects of the website, producing short illustrative videos, targeted press, etc. Marketing and Communications will be brought in regularly to the REF Strategy Group to assist co-working.

4) Further support to impact authors for writing their case studies. This will be supported by the two Research Impact Officers as well as the Business Development Managers.

5) Finally, building on City’s 2013 Impact Strategy we will develop an ‘Institutional Approach to Impact’ document by 28th February 2019. This will draw together what City does to enable it to achieve a broad-ranging goal i.e., more impactful research at an institutional level. The Impact Strategy may also be a framework for measuring and reporting on progress, and will be a vehicle for raising awareness of impact and “making things happen”.

Table 7: Timetable for supporting the development of impact case studies

Activity When WhoAppointment of two Research Impact Officers

February - April 2019 R&E

Induction, training and integration of Research Impact Officers

February 2019 – June 2019 R&E

Writing the Institutional Approach to Impact

January – April 2019 R&E

Business Development Managers facilitate engagement between academics and research users

Throughout 2019 and 2020 R&E

Allocation of Higher Education Innovation Funding (to further business and community engagement)

January – February 2019, September – December 2019

R&E

Allocation of Global Challenges Research Funding to further engagement with developing countries

January – February 2019, September – December 2019

R&E

Checking impact case studies for eligibility (underpinning research, evidence of impact)

May 2019 R&E

REF Rehearsal Exercise, including panel that will evaluate and give feedback about the quality of impact cases.

April – July 2019 R&E plus academic panels

14

Research Environment (15% weighting in the submission)

The Research environment component makes up the remaining 15% of the assessment and describes the environment we have to support our research. This includes research strategy, staff development and support for postgraduate research students (PGR), collaboration both inside and outside academia and equality and diversity. It also includes data on research income and post-graduate degrees awarded.

The UoA-level environment element will be assessed on the basis of a more structured template, including the use of more quantitative data to evidence narrative content.

The REF Operations Group has started to compile and prepare various quantitative indicators that are mentioned or suggested in the draft REF Guidance. The data falls into three categories: University, School and UoA levels.

Postgraduate research student data are compiled once a year for the HESA Return in October. Data which are held in SITS are collated directly from students at enrolment, provided by School administrators or is imported from other parts of SITS – such as workload of PhD supervisors or module enrolments in the case of structured PhDs programmes.

It should therefore be possible to produce the REF2021 PGR metric – number of students enrolled (in HESA) and graduating (derived information) each year per UoA. However, the data must be checked and validated by Schools.

Institutional Research Environment (Piloted in REF2021 for inclusion in future REFs)

The introduction of an institutional level submission for the environment was a key recommendation of an independent review of the REF, led by Lord Stern. Institutional level information will be appended to the UoA level environment template and will be taken into account by the sub-panel when assessing the UoA level statement. The pilot of the standalone assessment of the discrete institutional level environment will draw on this submitted information. Outcomes from the separate pilot exercise will not be included in REF 2021 but will inform future research assessments. The panel is chaired by Professor Chris Day, Vice-Chancellor and President of Newcastle University, and will pilot the assessment of the research environment at institution level. Professor Andrew Jones from City is a member of the Panel.

As time goes on it is likely that the UoA research environment statements and the institutional research environment statements will co-evolve.

Table 8: Supporting the Research Environment component of REF2021

Activity When WhoTo produce and compute metrics and indicators for the research environment statement.

December 2018 – February 2019 REF Operations Group

Research income by UoA per year, 2014 - present

January 2019, September 2019, September 2020

Finance

PhD graduates by UoA 2014 - present

January 2019, September 2019, September 2020

Graduate School

Drafting UoA Research Environment statements as part of the 2019 REF rehearsals

April – July 2019 Unit of Assessment Leaders

15

Drafting the institutional research environment: 3000 words.

January – May 2019 R&E, VP (R&E)

Final versions of the UoA research environment statements and institutional research environment

September 2020 Unit of Assessment Leaders, ADREs, REF Strategy Group

Next Steps

The planning logistics leading up to submission in November 2020 are shown in Figure 2.

1. ARQM: The 2019 ARQM exercise will conclude during late spring 2019 and guidance for it will be augmented to reflect the REF2021 rules. The ARQM will continue to assess individual research performance based on four best outputs over four years. However, to facilitate REF2021 planning, staff will be asked to nominate up to 5 outputs for review over the REF2021 period (2014-18). Additionally, the ARQM guidance on special circumstances which until now was based on the REF2014 criteria will be revised and updated to reflect recent REF2021 rules thereby harmonising ARQM and REF2021 processes at City.

2. The REF2021 Submission Model: Data will be refreshed when 2019 ARQM exercise is complete.

3. Outputs pipeline: One of the critical elements to determining accurately what our percentage of 4* outputs will be is to consider what outputs are currently in progress and expected to be available for submission to the REF – the “pipeline”. These prospective outputs can be added to those gathered by the ARQM exercise and the model re-run in summer 2019 to estimate UoA performance.

4. CoP: Development of the City Code of Practice document will be underway imminently.

5. Eligible staff: Finalising ARPs for staff is on-going. 6. Impact: Further development of impact case studies in terms of gathering evidence

and submission decisions are on-going.7. Environment: Provision of templates, background information and data for compiling

research environment statements at institutional and UoA level is being co-ordinated by the Research and Enterprise Directorate.

John Montgomery (R&E), Mark Brace (SP&PU) & Karen Shaw

28 January 2019

16

Figure 2: Global Gantt Chart of REF Planning Activities

17

18