Extending the QALY: Face validity results summary (Stage 3 ... · summary (Stage 3) October 2019 . Face validity summary – October 2019 consultation . 2 . 1 Background The Extending

Face validity summary – October 2019 consultation

1

Extending the QALY: Face validity results summary (Stage 3) October 2019


2

1 Background The Extending the QALY (E-QALY) project aims to develop a broad generic measure of quality of life for use in economic evaluation across public sectors with a key focus on health, social care and public health, based on the views of users and beneficiaries of these services. The project has six stages to identify the potential domains, generate a long list of items (or questions), identify suitable items from the long list based on face validity and psychometric analysis, valuation and validation of the new measure (Figure 1). The aim is to develop both a long questionnaire and a shorter classification system, the latter which will be valued on a 0 to 1 scale QALY scale. Figure1: Extending the QALY stages

A conceptual model (Figure 2) was used to inform the development of the new measure. This model was an extension of the original model developed by Wilson and Cleary (1995)1 for health to the broader context of the E-QALY project. In Stage 1 of the project, a targeted qualitative literature review identified seven high level domains (with sub-domains): feelings and emotions, cognition, activity, self-identity, relationships and social connections, ‘coping, autonomy and control’ and physical sensations (Figure 3). The conceptual model was used to develop the extraction framework used in the literature review and to inform the synthesis and subsequent stages of the project.

1 Wilson, I.B. and Cleary, P.D., 1995. Linking clinical variables with health-related quality of life: a conceptual model of patient outcomes. Jama, 273(1), pp.59-65.

Stage V: Valuation Valuation surveys using EQ-VTv2 Plus deliberative exercises with NICE citizen’s council

Stage I: Identify Domains / Themes Qualitative literature review Psychometric analysis of MIC data

Stage II: Generate long list of items (~100 items) From other instruments, item banks & new items. Must meet pre-identified criteria.

Stage III: Face validity interviews Face to face interviews with carers, social care users and patients

Stage VI: Impact & Validity Analysis of instrument validity. Compare to EQ-5D etc. Apply to existing cost effectiveness studies

Stage IV: Select items and agree descriptive system Psychometric survey including proposed items and other measures. Psychometrics and IRT (Item response theory)

Replicated in 5 countries

Argentina Australia China Germany USA


3

Figure 2: Conceptual framework

Figure 3: Preliminary domains and sub-domains at the end of stage 1


4

In Stage 2, items from 30 existing health and wellbeing measures (458 items) and item banks (229 items) that covered the sub-domains were reviewed for potential inclusion based on a set of item selection criteria around: ease of completion, ensuring items are not value laden, ensuring good coverage of sub-domains and severity range and suitability for valuation. Item wording was refined to meet these criteria and ensure consistency. Potential recall periods were also reviewed and a 7-day recall period adopted. In Stage 3, the face validity of the selected items was assessed. The face validity, or ‘respondent validity’ is concerned with how appropriate, relevant and understandable the items on a questionnaire are for the individuals who complete them and it an important element of ensuring content validity. This report provides an overview of the face validity methods and a summary of the findings that informed which items were taken forward to the next stage.

2 Methods Table 1 summarises the methods for face validity. Interviews were undertaken with potential users of the final measure to assess whether the questions had face validity. Written informed consent was taken at the start of each interview. At the end of the interview, participants were compensated. Ethical approval was obtained from the Institutional Review Boards and relevant Ethics Committees. Results were used to identify items to take forward, modify or to drop with consultation with stakeholders, advisory groups and a public involvement advisory group. Table 1: Face validity methods summary

Aim To assess: - Interpretation of questions and whether this matched domain construct - Whether question was understood - Whether it was relevant to participant - Whether how the question was framed mattered - Whether the response options and other instructions were relevant

Sample - Patients, social care users, carers and members of the general population - Participants from each group in England but different groups in Argentina,

Australia, China, England, Germany and the USA - Sample size target: n= 50 England; n=20-25 all other countries - Age 18 and above, capacity to consent and able to read questions

Data collection

- Semi-structured interviews supported by a topic guide and formatted questions in a questionnaire

- Participants saw 30-40 items by domain (participants did not review all items) - Interviews were audio-recorded and interviewers made notes during and at the

end of each interview - Completed demographic and health questions - Questions were translated into Argentinian Spanish, German and Simplified

Chinese by a single company using back and forward translation with input from the teams. Teams translated topic guides

Data analysis

- Notes were used to summarise performance of items by sub-domain by teams in each country. Audio recordings were used for clarification

- Results were reviewed and recommendations made for which items to keep, modify or drop based on overall feedback


5

- Items were assessed relative to other items within the same sub-domain - Enough items were needed to ensure that appropriate testing could be

undertaken in the next stage - Where there were not enough items to take to the next stage, new items were

proposed - Some items that did not perform particularly well in face validity were still taken

forward in order to generate additional evidence on their performance

3 Results

3.1 Sample Table 2 shows the mix of participants that were recruited to take part in the face validity interviews across the six countries. Most countries recruited across two or more groups. Patients were recruited across those with different conditions drawn from physical health and mental health. Table 2: Face validity participants

General public

Carers Patients Social-care

users Total Age Female (%)

England 6 13 18 8 45 23-95 58

Australia 4 4 17 25 28-70 56

Argentina 8 8 8 24 24-91 63

China 13 17 30 18-71 60

Germany 12 8 7 27 21-80 70

USA 19 19 23-76 53

3.2 Common findings There were a number of overall common findings which were found across the groups interviewed as well as in the different countries:

- Participants preferred simpler layouts i.e. where there was less repetition of text (repetition happened where responses were embedded in the questions e.g. I had no pain, I had mild pain, I had moderate pain etc.

- However, they wanted more information on context or definitions in order to help them complete the questions e.g. they preferred questions with examples.

- Where participants were given options either in questions or response options, there was often no clear preference. In some cases, preferences were expressed without any concrete reasons for expressing the preference.

- Some questions worked well for one group but not as well for another group e.g. being able to look after yourself was an issue for patients but not carers who were not clear how the questions related to them.

- Participants often forgot or ignored instructions e.g. the recall period. This may reflect how they would use the questionnaire independently or it may be an artefact of discussing questions in the context of an interview.


6

3.3 Retained items The results from the assessment of the items are summarised by domain and sub-domain.

Activity domain (self-care, meaningful or enjoyable activity, daily activity, independence, mobility, vision, hearing) From the questions that were tested, 1 were dropped (Table 3). Communication had varied meaning and was not interpreted as intended in Australia and England. Questions around personal need or receiving help were difficult for carers to consider with regards to themselves as they were used to doing this for those they cared for and these were modified. Table 3 Summary face validity findings - activity

Item Interim summary Next stage. Include, drop or amendment

1 I enjoyed what I did Respondents tended to focus on specific events One of the carers excluded caring tasks from consideration and focused only on the positive activities that occurred

Include

2 I was able to do the things I valued Include

3 I did things I found rewarding Drop

4 I was bored Being bored was seen by some as a trait. Some found this was difficult to answer even though they understood it. Some perceived boredom a normal thing.

Drop

5 I did what I wanted to do

Some preferred ‘I could do’ to ‘I did’

Drop

6 I could do the things I wanted to do

Include

7 I did what I needed to do The distinction between wanted and needed was clear for some but not all – and often examples of activities were very similar.

Drop

8 I was able to do what I needed Include

9 I had no difficulty with my day to day activities/daily activities

Repeated layout option not preferred Amend layout

10 Given the help I had/received My personal needs were met (e.g. being washed, going to the toilet, getting dressed, having food when I needed)

Problems with wording ‘given the help I received’ when no help had been received. Carers also found it difficult to relate to this question

Include with modification

11 Given the help I had/received my self-care needs were met (e.g. being washed, going to the toilet, getting dressed, having food when I needed)

‘Personal needs’ (Q10) preferred as a term to ‘self-care’ (Q11). Self-care also had ambiguous meaning for mental health service users. Australia found some difficulty with the meaning of ‘self-care’

Drop


7

12 I was able to look after myself Some carers not able to meet personal needs due to circumstances (e.g. hospital stay) not own health

Include

13 I needed help with looking after myself

Some problems with the term ‘needed’. Mental health patients needed motivation and support. Some carers noted ambiguity in the term ‘needed’ in that that they would have benefited from help.

Drop

14 I was able to look after myself with no difficulty

Repeated layout option not preferred Amend layout to difficulty across the top

15 I had no difficulty with self-care activities

Prefer ‘personal need’s to ‘self-care’ Drop

16 I was able to get around inside my home with no difficulty

Include

17 I was able to get around outside with no difficulty

Some mental health patients related this to social anxiety Not clear what limits of ‘outside’ are (UK and Australia)

Include

18 How well did you communicate with others?

Communicate includes skills in communication (logging onto internet, being able to get point across to professionals) In Australia – communicate included prevalence of anxiety; low to no proficiency in English

Drop

19 I was able to communicate with others with no difficulty

Drop

20 Because of hearing and/or speech, how difficult did you find it to have a conversation?

Some confusion as to whether virtual conversation were included

Include

21 How well can you hear (using hearing aids if needed)?

Some reported they could hear very well because they should if they actually wore hearing aids More preference for simple layout Q21

Amend to …How well can you hear (using hearing aids if you usually wear them)

22 I had no difficulty hearing (using hearing aids if needed)

More preference for simple layout of Q22 Drop

23 How well can you see (using your glasses or contact lenses if they are needed)?

Include

24 I had no difficulty seeing (using your glasses or contact lenses if they are needed)

More preference for simple layout of Q23 Drop

25 New item “I was able to do the things I wanted to do” – using severity response options

New item


8

Relationships domain (Support, Positive relationships, Stigma, Belonging, Loneliness) In the relationships domain, 5 questions were dropped (Table 4). The questions about support brought some confusion about who ‘other people’ referred to – for some respondents, key groups were excluded (e.g. family, or friends). The items taken forward focused on the feeling of being support – so that all groups are included. Table 4 Summary face validity findings - relationships

Item Interim summary Next stage.

Include, drop or amendment

1 I felt supported by other people More preference expressed for Q1

For some ‘Other people’ was a restricted group (e.g. only family or only professionals), others didn’t know who they should be thinking about Some felt question was hard to answer if well supported by one group but not another

Drop

2 I felt unsupported Changed to “I felt unsupported by people”

3 Other people gave me support Drop

4 I had support when I needed it Include

5 I had disagreements and conflict with people

Disagreements were seen as neutral or even positive (UK and Australia). Disagreements and conflicts were seen as quite different.

Drop

6 I got on with people around me

The term “got on” was seen as too colloquial and did not work well in Chinese.

Changed to “I got along well with people around me”

7 I got along well with people I came into contact with

When the term ‘people I came into contact with’ was included some respondents did not think about close friends/family but rather more brief informal contacts.

Drop

8 I felt lonely Clear interpretation Include

9 I felt there was nobody I was close to Some ambiguity in the term ‘close to’ Include

10 I felt I had no one to talk to

Having people to talk to was linked by some to talking to professionals

Include

11 I felt isolated Some confusion over meaning of the term ‘isolated’ – some referred to physical remoteness some interpreted as being alone without support

Include

12 I felt people avoided me People tended to refer to specific incidents. Generally well understood and considered important

Include

13 I felt judged by others

This item had a wide variation in meaning (stereotypes, exams, reputation, meeting others expectations). Some felt the meaning was unclear. Some felt that being judged was normal and it would have been better to specify being negatively judged.

Drop


9

14 I felt accepted by others Respondents preferred this to Q13. Include

15 I felt excluded Include

16 I felt left out Some respondents considered excluded and left out as the same thing. Include

Cognition domain (concentration, memory, confusion) Two items (‘I found it hard to focus my thoughts’, and ‘I had trouble with my memory’) generated interpretations that were different to expectations and were dropped (Table 5). The item on confusion did not perform well although it was put forward for additional testing. Table 5 Summary face validity findings - cognition


1 I found it hard to concentrate

Generally, well understood. Minimal problems with language Some preference overall for Q1 over 2 and 3. The question was ambiguous in China

Include

2 I found it hard to focus my thoughts

Misinterpretations of Q2 in China – the term focus seen as too abstract.

Dropped

3 I found it hard to pay attention

Include

4 I had trouble thinking clearly

Include

5 I had trouble remembering Include

6 I had trouble with my memory

The memory item seems to trigger responses that cover beyond the last 7 days. Having a good memory could be perceived as a trait. Requires a very clear link to the last 7 days (Australia).

Dropped

7 I felt/was confused Many unclear about the context “confused with what” (China) Related to different things – medical appointments, complex work tasks, making a good decision, getting advice from the appropriate carer organisation. ‘Felt confused’ generally preferred to ‘was confused’

Include “I felt confused”


10

Self-identify domain (confidence, self-worth, dignity) The sub-domain ‘dignity’ did not work well with the open ended items required for a generic measure (Table 6). Some respondents identified this in relation to their own behaviour rather than how they were treated by others. Whilst this is an important aspect, and identified as important within the literature review, it was felt that any questions asking about this needed to refer to specific people – e.g. did a particular staff member treat you with respect/dignity. The items on ‘feeling valued and feeling useful did not perform well for some of the very elderly who perceived them as not relevant to their stage in life. Table 6 Summary face validity findings – self-worth


1 I felt confident in myself Q1 and Q2 covered broad range of meanings (feeling capable, feeling like you look good, being able to talk to new people, self-esteem, confident at work, confident within caring role) (UK and Australia) The addition of ‘in myself’ made little difference Some thought the term ‘feeling unsure’ was quite broad and it was unclear what was being asked (Australia). Others say this as the inverse of confidence These were all quite ambiguous questions in China.

Include

2 I felt confident Drop

6 I felt unsure about myself

Include

3 & 4 I felt I was treated with respect I felt respected

Q3 and Q4 were often linked to specific events (e.g. helping across the road, holding doors open) Social care respondents did not always consider social care staff when answering.

Drop

5 I felt like I lived with dignity

Many respondents linked this to their own behavior (manners, kindness, honesty etc.) (UK and Australia) Some people had difficulty understanding the term (UK and Australia)

Drop

7 I felt good about myself The item ‘feeling good about myself’ has a range of interpretations – some of which could be relevant (including physical appearance) some less so as they are more about feeling good physically (being physically well) – rather than linked to self-esteem (UK and Australia)

Include

8 I felt like a failure Include

9&10 I felt valued I felt useful

These items were not considered relevant by elderly social care respondents.

Drop


11

Autonomy domain (coping, control) Overall results and recommendations are summarised in Table 7. The term ‘coping’ worked better when given context. Respondents also expressed a preference for additional explanation about the term ‘control’. The items which provided the additional context were included for the next stage. Table 7 Summary face validity findings - autonomy


1 I felt able to cope Ambiguous. Respondents asked “cope with what?” (UK, Australia, China, Germany)

Amend to “I felt able to cope with my day to day life”

2 I felt unable to cope As above. Drop

3 I felt unable to cope with my day to day life Include

4

I felt overwhelmed by my problems

Difficulty with the term ‘overwhelmed’ (UK Social care) ‘Overwhelmed’ similar to coping. But considered an ‘exaggeration’ (Australia) and negative (Germany) Feeling overwhelmed may not be because of “my problems” rather due to the circumstances (UK Carers)

Amend to “I felt overwhelmed by the problems or situation”

5 I felt in control of my daily life Drop

6 I felt in control of my day to day life

Respondents preferred day-to-day over daily. Daily was seen as needing to occur every single day.

Include

7 I have as much control over my daily life as I want

Respondents reported that they liked to see the definition of the term ‘control’ as it is given in the ASCOT question.

Amend response options and layout to frequency. “I had control over my day to day life”

8 New item “I felt I had no control over my day to day life”

New item


12

Feelings and Emotions domain (unhappiness, hopelessness, unsafe, anxiety, anger) In the feelings and emotions domain, 9 questions were dropped (Table 8). There were some concerns with some questions around doing enjoyable things because respondents considered a time frame that was outside the 7 days. Some of the associated items were considered too negative but were included to represent the severe ends of the scale (Table 8). Table 8 Summary face validity findings - relationships


1 I felt happy Some concern with top end of the scale Include. Also include ‘I felt cheerful’

2 I felt unhappy Include

3 I felt depressed Clinical term. Some would not answer yes unless they had a diagnosis (UK, Australia, Germany) but some considered this ‘unhappy’ or flippant use. Refers to a state longer than 7 days (Australia)

Drop

4 I felt sad Transient or related to specific events Include

5 I enjoyed life Enjoyment with life occasionally to lead to responses that went beyond 7 days – rather focusing on life overall to date.

Drop

6 I felt content with my life

Ambiguity – meaning of content Also interpreted to reflect all of life up to now (Australia)

Drop

7 I thought my life was not worth living

This was seen as quite negative. Some concern over whether people would answer honestly.

Include

8 I felt that I had nothing to look forward to

This was seen as quite negative (UK, Australia, Germany) In Chinese ‘look forward to’ requires an object.

Include

9 I had nothing to look forward to

Q9 preferred over 8 or 10 (Germany). Interviewees focused on specific events e.g. holidays.

Drop

10 I looked forward to each day

Positive works better if thinking of longer period (Australia) Unrealistic, idealistic not part of normal day ‘to look forward’

Drop

11 I felt frightened Some respondents thought of specific things or events when answering questions about feeling frightened and afraid.

Include

12 I felt afraid Q12 preferred to Q11 (Australia)

Include

13 I felt safe Ambiguity – some context needed for ‘safe’ (Australia)

Include

14 I felt unsafe Safe and unsafe not always interpreted as the same Include

15 I felt secure ‘Secure’ – ambiguous and mixed meanings (UK and Australia)

Drop


13


16 I felt anxious Include

17 My worries overwhelmed me

Linked to not coping with worries or not being in control [overwhelmed] (UK and Australia) Ambiguity – overwhelmed has no equivalent in Chinese

Drop

18 I felt worried Some considered worry stronger than anxious others considered it less

Include

19 I felt calm Calm was understood as being happy, bored, not worried, content

Include

20 I felt relaxed Relaxed was interpreted as a physical state of being relaxed but can still be anxious (Australia)

Drop

21 I felt irritable Ambiguity – for both irritable and irritated Include

22 I felt irritated Q21 preferred to Q22 (irritable vs. irritated) (UK and Australia) Irritable was seen as a mood, whereas irritated response to something happening (UK and Australia)

Drop

23 I felt angry Anger is quite extreme. Concern that people may not answer honestly

Include

24 I felt frustrated Q24 preferred to 23 or 25 Include

25 I lost my temper easily Preferred over 23 or 24 (Germany) Include


14

Physical sensations domain (pain, discomfort, energy, sleep) Respondents found the pain items fairly easy to understand, although many queried whether this included distress and psychological pain (Table 10). Respondents were divided on whether discomfort could be considered independently to pain – for some discomfort and mild pain were the same thing. Some respondents linked the items on energy to mental health state. In both cases items were amended to focus only on the physical aspects of pain and fatigue. Table 10 Summary face validity findings – physical sensations


1&2 I had no pain (mild pain etc.). How often do you experience pain?

Some uncertainty around whether this includes mental pain (distress) (Particularly mental health services users).

Amend to include the term ‘physical’

3&4 I had no discomfort (mild discomfort etc.). How often do you experience discomfort?

Some view as distinct to pain yet some view as mild pain. Other aspects included in discomfort (feeling fidgety, feeling too hot, socially awkward situations, embarrassment). Discomfort an ambiguous term even with examples (Australia)

Amend to include the term ‘physical’

5 I felt exhausted Straight forward interpretation

Include

6 I got tired easily Some linked more to a specific event, some requested more context, some linked to an underlying health problem

Amend to “I felt very tired”

7 I was too tired to do anything

Some saw as linking to persons mental state (UK & Australia)

Drop

8 I had problems with my sleep

Broad range of sleep problems identified Include

4 Discussion and conclusion A number of questions, including variations in how the same question was asked, were tested across the 6 countries. The results were used to inform which questions to take forward for the next stage. Testing across different countries and groups allowed us to gather useful information on which questions could work in different contexts, which was a strength of the validation process. We made some amendments and included new items based on the results as well as to support the next stage. However, these changes and additions were not subjected to face validity. The retained, amended and new items were taken forward to the psychometric testing which is summarised separately.

Documents

Extending the QALY: Face validity results summary (Stage 3 ... · summary (Stage 3) October 2019 . Face validity summary – October 2019 consultation . 2 . 1 Background The Extending