Selecting the right measures to understand and measure impact

Once whole-system outcomes are established, providers need to identify the measures that will enable them to monitor progress and make evidencebased judgements on the extent to which outcomes have been achieved.

Agree measures from the outset

For health and care systems, it is important to adopt and use a set of measures that align with the main elements of a national, regional or local strategy for person-centred, coordinated care. The complexity and the necessary variety in how integrated care strategies need to be developed means that outcomes and measures need to be chosen to suit local and national priorities.

To choose the right measures through which to evaluate and judge performance and progress in integrated care, there needs to be a clear understanding of:

Why do we need measures?

Measures define the data that should be collected to develop an understanding about progress and impact. A mix of measures is usually needed to cover the complexity of projects and programmes, and ensure that evidence gathered is sufficiently detailed and meaningful for the wide range of stakeholders involved in, or affected by, the changes.

Good measures:

Metrics for better care

While there are many care integration initiatives in local areas and across sustainability and transformation partnerships (STP), the Better Care Fund is explicitly put in place to facilitate care integration.

Local areas are asked to agree and report metrics in the following four areas:

  • delayed transfers of care
  • non-elective admissions (general and acute)
  • admissions to residential and care homes
  • effectiveness of reablement.

For more information, see: 2019–2020 Better Care Fund Policy Framework.

The metrics, in isolation, will not provide a full view on progress and performance. Local areas will need to supplement these with other national or local metrics related to improvements in health and care, experiences of care and cost-effectiveness.

There is a chronic lack of evaluation and measurement on which to judge the performance of care coordination programmes. This is a fundamental weakness; far greater attention is required to measure, evaluate, compare and reflect on performance.

Report on Coordinated care for people with complex chronic conditions , The King’s Fund

Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it.

H. James Harrington, international author on performance

10 key questions

  1. What is being measured?

    Example: Prevalence of diabetes.

  2. Why is it being measured?

    Example: It is a serious disease with serious consequences. Although it can be prevented and treated, it is still the leading cause of chronic disease globally and accounts for about 10 per cent of NHS costs.

  3. How is this indicator defined?

    Example: From recorded levels in general practice.

  4. Who does it measure?

    Example: 17+ only.

  5. When does it measure it?

    Which day/month/year? In the diabetes example, data is collected annually.

  6. Will it measure absolute numbers or proportions?

    Example: Proportions: Percentage of GP-registered population.

  7. Where does the data come from?

    Example: Collection and collation from Quality Outcome Framework (QOF) data in general practice via NHS Digital.

  8. How accurate and complete will the data be?

    Example: In 2016–17, the QOF dataset includes data from 95.4 per cent of GP practices in England that were open and active at some point in the reporting period.

    However, The accuracy of QOF information depends on:

    • clinical case finding by GPs; for example, information from QOF diabetes registers or about QOF diabetes indicators depends on people with diabetes being diagnosed
    • clinical coding; for example, when patients are diagnosed with diabetes, the quality of QOF data about people with diabetes depends on the GP practice maintaining accurate and coded clinical records.

    It is also worth noting that not everyone is registered with a GP – especially some groups with particular needs.

  9. Are there any caveats/warnings/problems?

    Think about potential for errors in collection, collation and interpretation (such as an undersampling of ethnic populations, young people, homeless people, migrants and travellers).

  10. Are particular tests needed, such as standardisation, significance tests or statistical process control to test the meaning of the data and the variation they show?

    For example, when comparing small numbers, in small populations, or to distinguish inherent (common cause) variation from special cause variation. Note that this is not related to the above diabetes example.

View: Guidelines for selecting and using indicators

Six key domains to consider when assessing integrated care

Dr Nick Goodwin has identified six key domains through which to assess progress on people-centred and integrated care. Consider using this framework as a basis when selecting your measures.

Are measures SMARTER?

  1. Specific: Measures can be clearly articulated to people with a basic knowledge of better care.

  2. Measurable: Criteria for measuring progress towards the attainment of the goal are concrete.

  3. Achievable: Measures are practical, achievable and realistic within operational constraints.

  4. Relevant: Measures offer insight into better care that matters.

  5. Time-bound: Clear time frames have been set and are evident to stakeholders.

  6. Evaluated Evaluation is happening on a consistent basis.

  7. Reviewed: Regular review cycles are planned for.

Getting a fuller picture

Qualitative and quantitative data can complement each other, with qualitative data giving meaning and richness to quantitative data. By combining the two, a fuller picture can be produced. How can we be sure that any one factor or service is directly responsible for any given effect or outcome? Although qualitative data cannot solve problems of causal connections, it is particularly relevant where there is ambiguity about terms and variables, and can help improve understanding of different contributions towards outcomes, with several advantages. See Building greater insight through qualitative research for more information on how qualitative information has become increasingly important.

Quantitative information can be characterised by the question, ‘What happened, where, when and who with?’ An example would be a patient experience survey in which multiple choice questions that can be measured on a scale (e.g. 1 to 5).

Historically, the focus has been on quantitative information as it is regarded as reliable and usually generalisable to a larger population.

However, quantitative information isn’t always enough: views and experiences matter.

Qualitative information can be characterised as ‘Factors or reasons affecting behaviour or outcomes – the how or why’. An example would be a patient experience survey which uses free text answers.

Patient and service user experience of care and support

In the publication Measuring patient experience, the Health Foundation articulates the strong evidence base supporting the measurement of patient experience: ‘Measuring patient experience is important, not only to guide service improvement, but also because people’s experiences of care may be linked to clinical outcomes and costs. A systematic review of 55 studies in primary care and hospitals found consistent positive associations between patient experience, patient safety and clinical effectiveness for a wide range of disease areas, settings, outcome measures and study designs.’

Tracking user experience enables you to:

Patient/service user experience can be measured in the following ways:

If providers are developing a new local measure, they can consult the Picker framework to access the 18 questions developed by the Picker Institute and Oxford University.

Developing person-centred health and wellbeing outcomes that matter most to people’ in Tower Hamlets

In 2015, Tower Hamlets became a Vanguard for a multispecialty community provider (MCP) new model of care. Tower Hamlets started by considering the strategic context, vision and goals for the MCP through a review of local strategy and policy documents, patient and public engagement reports, existing incentive schemes, and interviews with key stakeholders – including frontline staff and users – across the Tower Hamlets health and care economy. Based on this, four dimensions of outcomes were identified. These were:

For each of these dimensions, ‘I statements’ were developed describing key outcomes from a patient and service user point of view. For example: “I have a good level of happiness and wellbeing”. Following this, key performance indicators were agreed, against which to measure progress towards the articulated outcomes in the ‘I statements’. The indicators were selected based on stakeholder insights, as well as validated national sources – e.g. Public Health outcomes framework (PHOF), NHS outcomes framework, local plans. For the statement “I have a good level of happiness and wellbeing”, for example, the PHOF measure on self-reported wellbeing (people with a low happiness score) was suggested and agreed. As part of the indicator development, it was considered what data is already collected, and whether local surveys are in place to avoid data collection burden.

An Outcomes Reference Group (ORG) was established to lead the development and implementation. This group had representation across provider organisations, including clinical practitioners, public health, the voluntary sector, primary care, the local authority, CCG, lay partners and service user representatives. Outside of this, the developing framework was discussed at public engagement events, attending local community groups, staff events, and a clinical engagement workshop. For more information please contact: Richard Fradgley, Director of Integrated Care from ELFT (Richard.

Checklist: Selecting the right measures

Your checklist has been saved successfully

To save your checklist, please login to your MySCIE account or register if you do not yet have an account.

To save your checklist, please login to your MySCIE account or register if you do not yet have an account.

How to... understand and measure impact of integrated care
Previous section | All sections | Next section