Selecting the right measures to understand and measure impact

Once whole-system outcomes are established, providers need to identify the measures that will enable them to monitor progress and make evidencebased judgements on the extent to which outcomes have been achieved.

Agree measures from the outset

For health and care systems, it is important to adopt and use a set of measures that align with the main elements of a national, regional or local strategy for person-centred, coordinated care. The complexity and the necessary variety in how integrated care strategies need to be developed means that outcomes and measures need to be chosen to suit local and national priorities.

To choose the right measures through which to evaluate and judge performance and progress in integrated care, there needs to be a clear understanding of:

• the core aims of integrated care that are framed in terms of the people and the systems the interventions are seeking to influence
• the range of desired outcomes that should result from the interventions, drawn primarily from the patient’s/service user’s perspective – measures need to be relevant and aligned with outcomes
• the time frame over which such outcomes can reasonably be expected to be achieved in order to understand which measurement categories have the potential to be improved
• how chosen measures can enable analysis of what has made a difference (attribution between the interventions developed and the outcomes observed)
• how the measures can help drive improvement activity in the system, and avoid perverse incentives
• the importance of simplicity and ease of measurement – where possible, data that is already being collected should be used
• who holds responsibility for achieving the targets set within each measure.

Why do we need measures?

Measures define the data that should be collected to develop an understanding about progress and impact. A mix of measures is usually needed to cover the complexity of projects and programmes, and ensure that evidence gathered is sufficiently detailed and meaningful for the wide range of stakeholders involved in, or affected by, the changes.

Good measures:

• help to collect evidence in a systematic way about what works and what should be improved or decommissioned
• enable judgement on progress towards outcomes
• set out clearly who ‘owns’ the measure and is responsible for achieving the goals
• reflect local priorities as well as national requirements
• are signed up to by partners and stakeholders involved and anchored in a common purpose: to improve health and wellbeing for individuals and communities.

Metrics for better care

10 key questions

1. What is being measured?

   Example: Prevalence of diabetes.

2. Why is it being measured?

   Example: It is a serious disease with serious consequences. Although it can be prevented and treated, it is still the leading cause of chronic disease globally and accounts for about 10 per cent of NHS costs.

3. How is this indicator defined?

   Example: From recorded levels in general practice.

4. Who does it measure?

   Example: 17+ only.

5. When does it measure it?

   Which day/month/year? In the diabetes example, data is collected annually.

6. Will it measure absolute numbers or proportions?

   Example: Proportions: Percentage of GP-registered population.

7. Where does the data come from?

   Example: Collection and collation from Quality Outcome Framework (QOF) data in general practice via NHS Digital.

8. How accurate and complete will the data be?

   Example: In 2016–17, the QOF dataset includes data from 95.4 per cent of GP practices in England that were open and active at some point in the reporting period.

   However, The accuracy of QOF information depends on:

   • clinical case finding by GPs; for example, information from QOF diabetes registers or about QOF diabetes indicators depends on people with diabetes being diagnosed
   • clinical coding; for example, when patients are diagnosed with diabetes, the quality of QOF data about people with diabetes depends on the GP practice maintaining accurate and coded clinical records.

   It is also worth noting that not everyone is registered with a GP – especially some groups with particular needs.

9. Are there any caveats/warnings/problems?

   Think about potential for errors in collection, collation and interpretation (such as an undersampling of ethnic populations, young people, homeless people, migrants and travellers).

10. Are particular tests needed, such as standardisation, significance tests or statistical process control to test the meaning of the data and the variation they show?

    For example, when comparing small numbers, in small populations, or to distinguish inherent (common cause) variation from special cause variation. Note that this is not related to the above diabetes example.

View: Guidelines for selecting and using indicators

Six key domains to consider when assessing integrated care

Dr Nick Goodwin has identified six key domains through which to assess progress on people-centred and integrated care. Consider using this framework as a basis when selecting your measures.

• System-level measures of community wellbeing and population health, including reductions in avoidable deaths for treatable conditions, improved mental health and wellbeing and the proportion of populations engaged in healthy lifestyle behaviour.
• Service proxies for improved health outcomes, such as avoidable admissions to hospital, lengths of hospital stay and reductions in adverse events.
• Personal health outcomes to people and communities, primarily relating to measures of improved quality of life, remaining independent and reducing risk factors to better manage existing health conditions.
• Resource utilisation that seeks to describe measures which demonstrate the reorientation of activities towards primary and community care – for example, in terms of the balance of financial and human resources.
• Organisational processes and characteristics that support evidence that systems to support high-quality peoplecentred and integrated services are in place – for example, in improving access to care, care planning, better care transitions, self-care support, care management and medication reconciliation.
• User and carer experiences of, for example, shared decision-making, care planning, communication and informationsharing, and care coordination.

Are measures SMARTER?

19. Specific: Measures can be clearly articulated to people with a basic knowledge of better care.

13. Measurable: Criteria for measuring progress towards the attainment of the goal are concrete.

1. Achievable: Measures are practical, achievable and realistic within operational constraints.

18. Relevant: Measures offer insight into better care that matters.

20. Time-bound: Clear time frames have been set and are evident to stakeholders.

5. Evaluated Evaluation is happening on a consistent basis.

18. Reviewed: Regular review cycles are planned for.

Getting a fuller picture

Qualitative and quantitative data can complement each other, with qualitative data giving meaning and richness to quantitative data. By combining the two, a fuller picture can be produced. How can we be sure that any one factor or service is directly responsible for any given effect or outcome? Although qualitative data cannot solve problems of causal connections, it is particularly relevant where there is ambiguity about terms and variables, and can help improve understanding of different contributions towards outcomes, with several advantages. See Building greater insight through qualitative research for more information on how qualitative information has become increasingly important.

Quantitative information can be characterised by the question, ‘What happened, where, when and who with?’ An example would be a patient experience survey in which multiple choice questions that can be measured on a scale (e.g. 1 to 5).

Historically, the focus has been on quantitative information as it is regarded as reliable and usually generalisable to a larger population.

However, quantitative information isn’t always enough: views and experiences matter.

Qualitative information can be characterised as ‘Factors or reasons affecting behaviour or outcomes – the how or why’. An example would be a patient experience survey which uses free text answers.

Patient and service user experience of care and support

In the publication Measuring patient experience, the Health Foundation articulates the strong evidence base supporting the measurement of patient experience: ‘Measuring patient experience is important, not only to guide service improvement, but also because people’s experiences of care may be linked to clinical outcomes and costs. A systematic review of 55 studies in primary care and hospitals found consistent positive associations between patient experience, patient safety and clinical effectiveness for a wide range of disease areas, settings, outcome measures and study designs.’

Tracking user experience enables you to:

• improve communication between communities, patients, service users, commissioners and providers
• allow performance to be monitored over time and improvements demonstrated
• give patients, carers and their families a better understanding of their conditions and treatment plans to achieve better outcomes
• increase understanding of patients and the public about health and social care services
• empower communities to have a say in the delivery of local services
• encourage better decision-making and leads to more effective service delivery: by involving communities in the design/ delivery of services they are more likely to be successful in terms of their relevance, usage levels and impact.

Patient/service user experience can be measured in the following ways:

• using an existing national measure such as the Family and friends test
• using an existing local measure from data which has already been collected (interviews, service reviews, surveys etc.)
• using a newly developed local measure i.e. new case studies, interviews, service reviews etc.

If providers are developing a new local measure, they can consult the Picker framework to access the 18 questions developed by the Picker Institute and Oxford University.

Developing person-centred health and wellbeing outcomes that matter most to people’ in Tower Hamlets

In 2015, Tower Hamlets became a Vanguard for a multispecialty community provider (MCP) new model of care. Tower Hamlets started by considering the strategic context, vision and goals for the MCP through a review of local strategy and policy documents, patient and public engagement reports, existing incentive schemes, and interviews with key stakeholders – including frontline staff and users – across the Tower Hamlets health and care economy. Based on this, four dimensions of outcomes were identified. These were:

• Experience of services
• Fairness and equity
• Clinical outcomes
• Personal and functional goals.

For each of these dimensions, ‘I statements’ were developed describing key outcomes from a patient and service user point of view. For example: “I have a good level of happiness and wellbeing”. Following this, key performance indicators were agreed, against which to measure progress towards the articulated outcomes in the ‘I statements’. The indicators were selected based on stakeholder insights, as well as validated national sources – e.g. Public Health outcomes framework (PHOF), NHS outcomes framework, local plans. For the statement “I have a good level of happiness and wellbeing”, for example, the PHOF measure on self-reported wellbeing (people with a low happiness score) was suggested and agreed. As part of the indicator development, it was considered what data is already collected, and whether local surveys are in place to avoid data collection burden.

An Outcomes Reference Group (ORG) was established to lead the development and implementation. This group had representation across provider organisations, including clinical practitioners, public health, the voluntary sector, primary care, the local authority, CCG, lay partners and service user representatives. Outside of this, the developing framework was discussed at public engagement events, attending local community groups, staff events, and a clinical engagement workshop. For more information please contact: Richard Fradgley, Director of Integrated Care from ELFT (Richard.

Checklist: Selecting the right measures

Your checklist has been saved successfully