1

Guidance on evaluating or

developing a health app

October 2017

The global emergence of mobile health technology and health applications (apps) gives us

new opportunities to shape the way health care is managed and provided.

A health app is a piece of software that provides information, advice and feedback on health,

fitness or wellbeing. It can be downloaded onto a smart phone or other mobile device.

Apps can help maintain wellness, as well as assist with self-management and symptom

control of long-term conditions, such as diabetes, cardiovascular disease and chronic lung

disease. Personal health information monitoring can encourage behaviour change.

There are many health apps on the market, and it can be difficult for clinicians or consumers

to know which ones to recommend or use. Clinicians are often asked how effective different

apps are. This document focuses on two key areas:

Section 1: Guidance for clinicians and consumers – key points to consider for:

– clinicians if consumers ask about the effectiveness of a health app or if you wish to

recommend an app to them

– consumers wondering how to select an appropriate app to improve your health or

wellbeing.

Section 2: Guidance for app developers – key points to consider before deciding to

develop a new health app.

A second piece of work, to provide an assessment framework for clinical apps, is underway.

Section 1: Guidance for clinicians and

consumers about consumer-focused health

apps

1. Health Navigator app library

The Health Navigator website (www.healthnavigator.org.nz/app-library/) has a library of

consumer –facing health apps that have been reviewed by the Health Navigator review team

and an independent health professional. The library is funded by the Ministry of Health. The

purpose of the library is not to recommend apps or approve them but to provide clinicians

and consumers with a selection of apps and enough information to decide whether the app is

likely to meet their needs.

2

Apps are assessed on engagement, functionality, usability, privacy and security, clinical

relevance, and New Zealand relevance. The app library includes information on the purpose

of the app, the target audience, what features it has and its pros and cons. A brief literature

review also aims to identify: 1) if there is any relevant research or evidence for apps in the

category being assessed; 2) has the app been reviewed by any other reliable organisation.

Consumers or users are also able to review and rate apps.

There are limitations to this process. Firstly, the reviews reflect the opinion of the review

team. Secondly, assessment of data privacy and security can only take at face value what is

said on the app website and/or developer-provided documentation. Ideally, independent

testing to validate developer claims would be undertaken, however this is complex, time

consuming and expensive so it does not occur. Thirdly, not all reviews have a consumer

review. Ideally all reviews would include an assessment of the app by someone for whom the

app is intended.

2. Other independent health app review websites

Clinicians and consumers can also source reviews of consumer-facing health apps from the

following websites. These websites have been reviewed by the Health Navigator team and are

deemed to be credible, and independent, with a robust review process. However, most are

not New Zealand based and therefore are unlikely to consider the New Zealand relevance of

the app or include reviews of New Zealand-based apps.

Website Description

Books on Prescription This is a New Zealand website, managed by WellSouth. It has a small selection of app reviews, covering the clinical areas of heart disease, lifestyle and mental health. Many of the apps have been reviewed in partnership with Health Navigator, but they do have a few additional apps.

NHS App Library This website provides independent reviews of health apps for the NHS in the UK. It also includes some apps that have been ‘NHS approved’ and some that are ‘Being tested in the NHS’.

my health apps This website is maintained by PatientView, a UK-based research and publishing company. The app library has large number of app reviews across a variety of clinical areas. Reviews include input from users including patient groups.

Practical apps

This is a Canadian website comprising reviews that are conducted by a family physician with a special interest in virtual care and health technology. Their app evaluation framework includes clinical usefulness, usability, safety, privacy/security and accessibility. Apps are selected for review with a focus on apps available in Canada that relate to Canadian guidelines and, where possible, are Canadian-made.

Ranked This is a United States website that has a selection of apps related to managing and monitoring chronic conditions common in the US, including mental health, heart disease, diabetes, obesity and sleep quality. Additionally, apps were chosen due to their focus on broader issues impacting large populations of people, including fitness, medication adherence, symptom tracking, emergency/acute care and pregnancy and reproductive health.

AppScript Clinicians can sign-up to the AppScript website. This gives them access to app reviews – apps are assessed on six criteria – professional, endorsement, patient, developer, functional and clinical.

Healthy Living Apps Guide This website is managed by VicHealth, Australia and comprises reviews of apps promoting good health – apps focused on nutrition, fitness, smoking cessation, alcohol tracking and mental wellbeing.

3

3. Undertake independent app reviews

Anybody can undertake app reviews themselves. Below are some tools or frameworks that

have been used for mobile health apps:

App

assessment

tools

Description

Mobile App Rating Scale (MARS)

The MARS scale is a well-known standardised tool developed by the Queensland University of Technology by which health apps can be compared. It is designed to score apps on the criteria of engagement, functionality, aesthetics, and information quality. The MARS scale is attached as Appendix A.

App Chronic Disease Checklist (ACDC)

The ACDC assessment tool includes similar criteria used in the MARS, but also includes assessment of warnings of unhealthy values, user profile setup and features available offline. The resultant checklist has 4 constructs – engagement, functionality, ease of use, and information management.

Royal College of Physicians checklist

18-item checklist developed by the Royal College of Physicians, UK to help clinicians assess the structure, functions and impact of medical apps, so they can feel more confident about using medical apps themselves, about recommending them to their staff or prescribing them for patients.

Guiding principles for physicians recommending mobile health apps to patients

Guidance by the Canadian Medical Association, comprising seven general principles physicians need to consider when assessing the suitability of mobile health apps for their patients – endorsement by a recognised medical or professional organisation, usability, reliability of information, privacy and security and avoidance of conflict of interest.

Section 2: Guidance before deciding to

develop a new health app The Ministry of Health supports the digitisation of health information, however does not

itself generally get involved with the development of health apps. Developing an app is

expensive and time consuming. Before commissioning or developing an app, the Ministry

recommends you check the Health Navigator app library and online app stores to see if there

is an existing app that meets your/the consumer’s needs.

Other resources that provide guidance on health app development include:

Digital assessment questions (currently a beta version) and review process developed by

the NHS library that enables developers to provide information and evidence against

relevant national standards and best practices. At this stage, this appears to be the most

comprehensive and useful guidance on health app development. Through a series of

clinical and technical standards, it guides developers by assessing 9 core areas to be

considered in app development (clinical effectiveness, regulatory approval, clinical safety,

privacy & confidentiality, security, usability & accessibility, interoperability, technical

stability, change management).

The MARS scale (Appendix A), which can be used as a checklist and guide for the design

and development of new health apps.

Guidelines for creating healthy living apps, Vic Health. This provides a step-wise guide of

the app development process of healthy living apps starting with before you start building

an app and ending with updating and improving the app. This document also includes

other resources for app developers.

4

The Our Mobile Health App Library – Developers complete a self-assessment

questionnaire of about 250 questions, after which the app is assessed by a panel of over

150 independent expert reviewers from clinical and non-clinical backgrounds who assess

and provide a feedback report. Once the app meets the stipulated criteria, it is added to

the Our Mobile Health App Library.

Good Practice Guidelines on Health Apps and Smart Devices (Mobile Health or mHealth)

– Developed by the French organisation HAS.

Mobile Medical Applications – the Federal Drug Agency (FDA) provides guidance on

which mobile apps they will regulate and how. Using a risk-based approach they will

assess mobile apps that meet the regulatory definition of “device” and that are intended to

be used as an accessory to a regulated medical device, or transform a mobile platform into

a regulated medical device.

Mobile health app resources – a library of useful resources of mobile health apps put

together by West of England Academic Health Science Network.

Xcertia mHealth App Guidelines (work in progress) – aims to create standards for data

privacy and security, clinical effectiveness, safety, usability, interoperability, and other

categories to assist app developers. The founding members include the American Medical

Association, the American Heart Association, HIMSS, and DHX Group.

5

Appendix A

Mobile Application Rating Scale (MARS) App Classification

The Classification section is used to collect descriptive and technical information about the

app. Please review the app description in iTunes / Google Play to access this information.

App Name:

Rating this version: Rating all versions:

Developer:

N ratings this version: N ratings all versions:

Version: Last update:

Cost - basic version: Cost - upgrade version:

Platform: iPhone iPad Android

Brief description:

__________________________________________________________

Focus: what the app targets Theoretical background/strategies

(select all that apply) (all that apply)

Increase happiness/Well-being Assessment

Mindfulness/Meditation/Relaxation Feedback

Reduce negative emotions Information/Education

Depression Monitoring/Tracking

Anxiety/Stress Goal setting

Anger Advice /Tips /Strategies /Skills training

Behaviour change CBT - Behavioural (positive events)

Alcohol /Substance use CBT – Cognitive (thought challenging)

Goal setting ACT - Acceptance commitment therapy

Entertainment Mindfulness/Meditation

Relationships Relaxation

Physical health Gratitude

Other_______________________ Strengths based

Other______________________

6

Affiliations:

Unknown Commercial Government NGO University

Age group (all that apply) Technical aspects of app (all that apply)

Children (under 12) Allows sharing (Facebook, Twitter, etc.)

Adolescents (13-17) Has an app community

Young adults (18-25) Allows password-protection

Adults Requires login

General Sends reminders

Needs web access to function

7

App quality ratings The Rating scale assesses app quality on four dimensions. All items are rated on a 5-point

scale from “1.Inadequate” to “5.Excellent”. Circle the number that most accurately represents

the quality of the app component you are rating. Please use the descriptors provided for each

response category.

Section A

Engagement – fun, interesting, customisable, interactive (eg, sends alerts,

messages, reminders, feedback, enables sharing), well-targeted to audience

Entertainment: Is the app fun/entertaining to use? Does it use any strategies to

increase engagement through entertainment (eg, through gamification)?

Dull, not fun or entertaining at all

Mostly boring

OK, fun enough to entertain user for a brief time (< 5 minutes)

Moderately fun and entertaining, would entertain user for some time (5–10 minutes total)

Highly entertaining and fun, would stimulate repeat use

Interest: Is the app interesting to use? Does it use any strategies to increase

engagement by presenting its content in an interesting way?

Not interesting at all

Mostly uninteresting

OK, neither interesting nor uninteresting; would engage user for a brief time (< 5 minutes)

Moderately interesting; would engage user for some time (5-10 minutes total)

Very interesting, would engage user in repeat use

Customisation: Does it provide/retain all necessary settings/preferences for

apps features (eg, sound, content, notifications, etc.)?

Does not allow any customisation or requires setting to be input every time

Allows insufficient customisation limiting functions

Allows basic customisation to function adequately

Allows numerous options for customisation

Allows complete tailoring to the individual’s characteristics/preferences, retains all settings

8

Interactivity: Does it allow user input, provide feedback, contain prompts

(reminders, sharing options, notifications, etc.)? Note: these functions need to

be customisable and not overwhelming in order to be perfect.

No interactive features and/or no response to user interaction

Insufficient interactivity, or feedback, or user input options, limiting functions

Basic interactive features to function adequately

Offers a variety of interactive features/feedback/user input options

Very high level of responsiveness through interactive features/feedback/user input options

Target group: Is the app content (visual information, language, design)

appropriate for your target audience?

Completely inappropriate/unclear/confusing

Mostly inappropriate/unclear/confusing

Acceptable but not targeted. May be inappropriate/unclear/confusing

Well-targeted, with negligible issues

Perfectly targeted, no issues found

A. Engagement mean score =

Section B

Functionality – app functioning, easy to learn, navigation, flow logic, and

gestural design of app

Performance: How accurately/fast do the app features (functions) and

components (buttons/menus) work?

App is broken; no/insufficient/inaccurate response (eg, crashes/bugs/broken features, etc.)

Some functions work, but lagging or contains major technical problems

App works overall. Some technical problems need fixing/Slow at times

Mostly functional with minor/negligible problems

Perfect/timely response; no technical bugs found/contains a ‘loading time left’ indicator

9

Ease of use: How easy is it to learn how to use the app; how clear are the menu

labels/icons and instructions?

No/limited instructions; menu labels/icons are confusing; complicated

Useable after a lot of time/effort

Useable after some time/effort

Easy to learn how to use the app (or has clear instructions)

Able to use app immediately; intuitive; simple

Navigation: Is moving between screens logical/accurate/appropriate/

uninterrupted; are all necessary screen links present?

Different sections within the app seem logically disconnected and

random/confusing/navigation/is difficult

Usable after a lot of time/effort

Usable after some time/effort

Easy to use or missing a negligible link

Perfectly logical, easy, clear and intuitive screen flow throughout, or offers shortcuts

Gestural design: Are interactions (taps/swipes/pinches/scrolls) consistent and

intuitive across all components/screens?

Completely inconsistent/confusing

Often inconsistent/confusing

OK with some inconsistencies/confusing elements

Mostly consistent/intuitive with negligible problem

Perfectly consistent and intuitive

B. Functionality mean score = ____________

10

Section C

Aesthetics – graphic design, overall visual appeal, colour scheme, consistent

style

Layout: Is arrangement and size of buttons/icons/menus/content on the screen

appropriate or zoomable if needed?

Very bad design, cluttered, some options impossible to select/locate/see/read device display

not optimised

Bad design, random, unclear, some options difficult to select/locate/see/read

Satisfactory, few problems with selecting/locating/seeing/reading items or with minor

screen size problems

Mostly clear, able to select/locate/see/read items

Professional, simple, clear, orderly, logically organised, device display optimised. Every

design component has a purpose

Graphics: How high is the quality/resolution of graphics used for

buttons/icons/menus/content?

Graphics appear amateur, very poor visual design – disproportionate, inconsistent style

Low quality/low resolution graphics; low quality visual design – disproportionate,

stylistically inconsistent

Moderate quality graphics and visual design (generally consistent in style)

High quality/resolution graphics and visual design – mostly proportionate, stylistically

consistent

Very high quality/resolution graphics and visual design - proportionate, stylistically

consistent throughout

Visual appeal: How good does the app look?

No visual appeal, unpleasant to look at, poorly designed, clashing/mismatched colours

Little visual appeal – poorly designed, bad use of colour, visually boring

Some visual appeal – average, neither pleasant, nor unpleasant

High level of visual appeal – seamless graphics – consistent and professionally designed

As above + very attractive, memorable, stands out; use of colour enhances app

features/menus

C. Aesthetics mean score = ______________

11

Section D

Information – Contains high quality information (eg, text, feedback, measures,

references) from a credible source. Select N/A if the app component is

irrelevant.

Accuracy of app description (in app store): Does app contain what is described?

Misleading. App does not contain the described components/functions. Or has no

description.

Inaccurate. App contains very few of the described components/functions.

OK. App contains some of the described components/functions.

Accurate. App contains most of the described components/functions.

Highly accurate description of the app components/functions.

Goals: Does app have specific, measurable and achievable goals (specified in

app store description or within the app itself)?

N/A Description does not list goals, or app goals are irrelevant to research goal (eg, using a

game for educational purposes).

App has no chance of achieving its stated goals.

Description lists some goals, but app has very little chance of achieving them.

OK. App has clear goals, which may be achievable.

App has clearly specified goals, which are measurable and achievable.

App has specific and measurable goals, which are highly likely to be achieved.

Quality of information: Is app content correct, well written, and relevant to the

goal/topic of the app?

N/A There is no information within the app.

Irrelevant/inappropriate/incoherent/incorrect

Poor. Barely relevant/appropriate/coherent/may be incorrect

Moderately relevant/appropriate/coherent/and appears correct

Relevant/appropriate/coherent/correct

Highly relevant, appropriate, coherent, and correct

12

Quantity of information: Is the extent coverage within the scope of the app; and

comprehensive but concise?

N/A There is no information within the app.

Minimal or overwhelming

Insufficient or possibly overwhelming

OK but not comprehensive or concise

Offers a broad range of information, has some gaps or unnecessary detail; or has no links to

more information and resources

Comprehensive and concise; contains links to more information and resources.

Visual information: Is visual explanation of concepts – through

charts/graphs/images/videos, etc. – clear, logical, correct?

N/A There is no visual information within the app (eg, it only contains audio, or text).

Completely unclear/confusing/wrong or necessary but missing

Mostly unclear/confusing/wrong

OK but often unclear/confusing/wrong

Mostly clear/logical/correct with negligible issues

Perfectly clear/logical/correct.

Credibility: Does the app come from a legitimate source (specified in app store

description or within the app itself)?

Source identified but legitimacy/trustworthiness of source is questionable (eg, commercial

business with vested interest).

Appears to come from a legitimate source, but it cannot be verified (eg, has no webpage).

Developed by small NGO/institution (hospital/centre, etc.) /specialised commercial

business, funding body.

Developed by government, university or as above but larger in scale.

Developed using nationally competitive government or research funding (eg, Australian

Research Council, NHMRC).

13

Evidence base: Has the app been trialled/tested; must be verified by evidence

(in published scientific literature)?

N/A The app has not been trialled/tested.

The evidence suggests the app does not work.

App has been trialled (eg, acceptability, usability, satisfaction ratings) and has partially

positive outcomes in studies that are not randomised controlled trials (RCTs), or there is

little or no contradictory evidence.

App has been trialled (eg, acceptability, usability, satisfaction ratings) and has positive

outcomes in studies that are not RCTs, and there is no contradictory evidence.

App has been trialled and outcome tested in 1-2 RCTs indicating positive results.

App has been trialled and outcome tested in > 3 high-quality RCTs with positive results.

D. Information mean score = _____________ *

* Exclude questions rated as “N/A” from the mean score calculation.

App subjective quality

Section E

Would you recommend this app to people who might benefit from it?

Not at all – I would not recommend this app to anyone.

There are very few people I would recommend this app to.

Maybe – There are several people whom I would recommend it to.

There are many people I would recommend this app to.

Definitely – I would recommend this app to everyone.

How many times do you think you would use this app in the next 12 months if it

was relevant to you?

None.

1–2

3–10

11–50

>50

14

Would you pay for this app?

No

3 Maybe

5 Yes

What is your overall star rating of the app?

★ One of the worst apps I’ve used

★★

★★★ Average

★★★★

★★★★★ One of the best apps I've used

Scoring

App quality scores for

Section F

A: Engagement mean score = __________________________

B: Functionality mean score = __________________________

C: Aesthetics mean score = __________________________

D: Information mean score = __________________________

App quality mean score = __________________________

App subjective quality score = __________________________

App-specific

These added items can be adjusted and used to assess the perceived impact of the app on the

user’s knowledge, attitudes, intentions to change as well as the likelihood of actual change in

the target health behaviour.

15

Section G

Awareness: This app is likely to increase awareness of the importance of

addressing [insert target health behaviour].

Strongly disagree Strongly agree

1 2 3 4 5

Knowledge: This app is likely to increase knowledge/understanding of [insert

target health behaviour].

Strongly disagree Strongly agree

1 2 3 4 5

Attitudes: This app is likely to change attitudes toward improving [insert target

health behaviour].

Strongly disagree Strongly agree

1 2 3 4 5

Intention to change: This app is likely to increase intentions/motivation to

address [insert target health behaviour].

Strongly disagree Strongly agree

1 2 3 4 5

Help seeking: Use of this app is likely to encourage further help seeking for

[insert target health behaviour] (if it’s required).

Strongly disagree Strongly agree

1 2 3 4 5

Behaviour change: Use of this app is likely increase/decrease [insert target

health behaviour].

Strongly disagree Strongly agree

1 2 3 4 5