SCIE/NICE recommendations on looked after children: Promoting the quality of life of looked-after children and young people

Appendix B

Summary of the methods used to develop this guidance

Introduction

The reviews, primary research and commissioned reports include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it. The minutes of the PDG meetings provide further detail about the Group’s interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix D and are available at www.nice.org.uk/guidance/PH28

Guidance development

The stages involved in developing public health programme guidance are outlined in the box below.

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by the PDG to help develop the recommendations. The overarching question was:

• What strategies, policies, systems, structures, interventions or services are effective and cost-effective in promoting the physical and emotional health and wellbeing of looked-after children and young people?

The subsidiary questions were:

• Can current systems, frameworks and processes used to identify and monitor health, emotional and social outcomes for looked-after children, young people and their families be improved?
• How does placement stability and breakdown affect key outcomes, and subsequent care placements?
• How do the structure, type, continuity and length of care that children and young people receive affect key outcomes for children, young people and their families?
• How effective, and cost effective, are interventions and activities (including participatory approaches) that are used to maintain, improve or promote key outcomes, in different settings and at different levels of intervention?
• Are there key points or transitions in the life-course or care pathways of looked-after children and young people at which intervention may be particularly beneficial (or harmful)?
• What physical, emotional and social outcomes are important to looked-after children and young people and their families?

These questions were made more specific for each review (see reviews for further details).

Effectiveness reviews

Reviews of effectiveness were conducted using NICE methods, on:

• support services for transition to adulthood/leaving care (review 1)
• training and support for carers/professionals/volunteers (review 2)
• improving access to health and mental health services (review 3).

Identifying the evidence

The strategy adopted for these reviews combined systematic review searching undertaken by SCIE with more targeted searches undertaken by ScHARR. The following databases were searched for published literature (January 1990 to September 2008):

• Applied Social Sciences Index and Abstracts (ASSIA)
• Australian Family and Society Abstracts
• British Education Index (BEI)
• Campbell Collaboration C2 Library
• CERUK Plus
• ChildData
• Cochrane Library
• Cumulative Index to Nursing and Allied Health Literature (Cinahl Plus)
• EMBASE
• Health Management Information Consortium (HMIC)
• International Bibliography of the Social Sciences (IBSS)
• JSTOR MEDLINE
• PsycINFO
• Social Care Online
• Social Services Abstracts
• Zetoc (electronic tables of contents).

In addition, searches of reference lists and citation searches were conducted on all relevant papers included in the ‘Correlates review’ (review 4; see page 98). The reference lists of included papers were searched by hand, and citation searching was undertaken on all included papers. The PDG was also consulted for relevant literature.

Further details of the databases, search terms and strategies are included in the review reports.

Selection criteria

Inclusion and exclusion criteria for each review varied and details can be found at www.nice.org.uk/guidance/PH28 However, in general:

• Population:
  • Review 1: looked-after children and young people and/or adults who were previously looked after as children and/or young people.
  • Review 2: carers (including foster and residential carers), professionals (such as teachers and social workers) and approved volunteers (such as independent visitors, mentors) involved in the care of or working with looked-after children and young people.
  • Review 3: looked-after children and young people, or adults who were looked after if relevant information on their childhood was collected.
• Intervention:
  • Review 1: support services to prepare looked-after young people for the transition from foster or residential care to independent living or community care.
  • Review 2: training and support to enhance the skills of carers, professionals or volunteers involved in the care of looked-after children and young people. This included training and support for birth families, but not treatment foster care (also described as therapeutic foster care).
  • Review 3: any intervention designed to improve access to any specialist or universal service for children and young people physical and emotional health and wellbeing of looked-after children and young people during their time they were being looked after. Treatment foster care was not included.
• Comparison: usual or practice or no ‘intervention’.
• Outcomes:
  • Review 1: all reported outcomes, including housing, alcohol or drug misuse, employment, educational attainment, employment, offending behaviour and physical, mental and sexual health.
  • Review 2: all outcomes relating to physical and emotional health and wellbeing of looked-after children and young people, including placement stability and long-term outcomes.
  • Review 3: access to the service in question.
• Study types: Decisions on the type of evidence to be included were informed by the searching process. Papers that did not present quantitative data were excluded.
• Other: English language papers.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual ‘Methods for the development of NICE public health guidance’ (see appendix D). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the checklist criteria have been fulfilled. Where they have not been fulfilled, the conclusions are very unlikely to alter.

+ Some of the checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are unlikely to alter the conclusions.

– Few or no checklist criteria have been fulfilled. The conclusions of the study are likely or very likely to alter.

Summarising the evidence and making evidence statements

The review data were summarised in evidence tables (see full reviews).

Within each review, the findings were synthesised and used as the basis for a number of evidence statements relating to each research question. The evidence statements were prepared by the public health collaborating centres (see appendix A). The statements reflect their judgement of the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Other reviews

Two other reviews were conducted: a correlates review (review 4) and a qualitative review (review 5):

Correlates review

This review aimed to identify factors that are associated with outcomes for looked-after children and young people, and to present these as conceptual maps showing the strength of associations between factors, supported by evidence tables. It is not a full systematic review with exhaustive or comprehensive searches but it is a review of quantitative data, with inclusion and exclusion criteria, quality appraisal and a systematic and transparent methodology.

Identifying the evidence

Searches were done using free text and keywords/index terms, and several iterations were undertaken. Once an outcome or variable had been identified, further searches and sifting sought to identify new variables rather than to identify all the evidence for each relationship between variables. A thorough audit trail was maintained, to ensure that searches were transparent, systematic and replicable.

Selection criteria

Studies were included if the population was looked-after children and young people aged up to 25 years, or adults who had been looked-after children and young people. Explanatory factors were included if they could be defined as risk factors, protective factors or interventions (any intervention or activity that could affect these outcomes). Outcomes were as defined by individual studies.

Quality appraisal

Study quality was assessed using the checklists and guidance in ‘Methods for development of NICE public health guidance (second edition, 2009)’.

Summarising the evidence

Relevant information was extracted and studies were classified according to both study type and the main variables analysed. The findings were then mapped using software that allows associations to be presented graphically.

Qualitative review

This review used SCIE methods and aimed to identify qualitative research on and synthesise the views, experiences and preferences of children and young people, their families and carers, about the care system.

Identifying the evidence

The following electronic health and social care databases were searched:

• ASSIA
• CINAHL
• EMBASE
• IBSS
• MEDLINE
• PsycINFO
• Social Care Online
• Social Science Citation Index
• Social Services
• Abstracts Sociological Abstracts
• Google (and Google Scholar)

Further details of the databases, search terms and strategies are included in the review.

Selection

Studies were included if they were performed in the UK and their methods for data collection and analysis enabled them to reflect what is important to looked-after children and young people and their families and carers.

Quality appraisal

Included studies were critically appraised and rated for:

• soundness (internal methodological coherence), using an adapted version of the NICE checklist for qualitative studies
• appropriateness of the research design and analysis used for answering the review
• relevance of the topic (from the sample, measures, scenario, or other indicator of the focus of the study) to the review question.

Summarising the evidence

The pool of findings from all the studies was assessed for each of the three groups: looked-after children and young people; carers; and parents. Themes and subthemes were developed via a framework analysis approach. The detail of the findings was interrogated to answer, where possible, the research questions and evidence statements were developed.

Practice survey

This used SCIE’s practice survey methodology to identify innovative and emerging practice in assessing, maintaining and improving the physical and emotional health and wellbeing of looked-after children and young people. The survey sought views about the acceptability, accessibility and effectiveness of targeted and specialist, as opposed to universal, interventions.

Sampling

The PDG identified sites that were of particular interest because of their specific areas of practice – for example, multidisciplinary work, and specialist services for looked-after children. In addition, the Action for Children team who wrote the report identified a local authority that had ‘Beacon’ status for its engagement of young people and its targeted youth support initiatives.

Methods

Following initial approaches, all sites agreed to participate in the survey, and approval was gained from the Assistant Director of Children’s Services Research Team and ethical approval in each site. The first phase of the survey was to conduct a series of group and individual semi-structured interviews with managers, commissioners and practitioners from a range of social care, education and health organisations. These interviews took place between April and June 2009.

The second phase involved interviews with carers and looked-after children and young people.

The data collected from the survey was analysed, by reading and aggregation of data, measuring, identifying differences and similarities, identifying and acknowledging variables, contextualising, noting correlation, identifying themes and cross-cutting issues, comparing views within and across sample sectors and ordering and grouping findings to address topics and questions.

Summarising the evidence

The findings were structured around four topic areas: structure of care; delivery of care; interventions and activities; and the views of children, young people and their families. These were used as the basis for evidence statements.

Economic analysis

The economic analysis consisted of a review of economic evaluations and cost-effectiveness modelling.

Review of economic evaluations

A database supplied by SCIE was used to find papers containing evidence of cost effectiveness. This was supplemented by targeted searches of: Web of Science MEDLINE NHS EED EconLit. Inclusion criteria followed the same criteria used for the effectiveness reviews, for which the economic reviews were an accompaniment. Quality was assessed using the ’Methods for development of NICE public health guidance’ (2006).

Cost-effectiveness modelling

An economic model was constructed to accompany the three effectiveness and cost-effectiveness reviews (see appendix E). A satisfactory model was constructed for review 1. A model was attempted for review 2 but was rejected by the PDG because of data limitations. For review 3, a model was not attempted because of lack of relevant data.

The results of the model to accompany effectiveness review 1 are reported in: ‘Review 1: The cost-effectiveness of support services for transition to adulthood/leaving care on the adult outcomes of looked after young people’. Following the inability to produce a meaningful modelling analysis for reviews 2 and 3, the PDG was asked to use a cost-consequences framework to consider the cost effectiveness of the recommendations. This required members to consider the best available evidence on the costs and effectiveness of services of relevance to the recommendations under consideration and to draw on their expertise to make an appropriate recommendation where evidence gaps existed.

The PDG examined the recommendations to see whether any were:

• on the balance of probabilities, likely to do more good than harm
• likely to be cost saving or have a very low additional cost.

This was successful for recommendations where the available evidence and the opinions of experts and PDG members strongly suggested that their implementation would do more good than harm, would be of relatively low cost to implement and would probably produce cost savings in the long run.

What prevented this method being used more widely to determine whether a recommendation was likely to be cost effective was that many of the recommendations focus on system-level changes, multi-agency working, information sharing and training. For these recommendations, the lack of data and the resulting level of uncertainty were too great to support even a cost-consequences approach.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations were for practitioners and how feasible it would be to put them into practice. It was conducted with commissioners and practitioners who are involved in social care, health and education services for looked-after children and young people. This included those working in the NHS, local authorities and the wider public, private, voluntary and community sectors.

The fieldwork comprised:

• eight half-day workshops carried out in Manchester and London with 95 representatives from the public, private, voluntary and community sectors forty-four telephone interviews with practitioners.
• The fieldwork was commissioned to ensure there was ample geographical coverage. The main issues arising are set out in appendix C under fieldwork findings. The fieldwork report ‘Fieldwork on promotion of the physical and emotional health and wellbeing of looked-after children and young people’, is available at www.nice.org.uk/guidance/PH28

Consultation with looked-after young people and care leavers

A consultation was carried out with 30 looked-after young people and care leavers aged 11–24. The aim was to gather their views about the relevance, usefulness and acceptability of appropriate sections of the guidance.

The consultation was undertaken by Action for Children and comprised focus groups at four sites (two county councils, a district council and an inner city unitary authority).

The consultation report, ‘The physical, emotional health and wellbeing of looked-after children and young people’, is available at www.nice.org.uk/guidance/PH28

How the PDG formulated the recommendations

At its meetings in 2008 and 2009, the PDG considered the evidence reviews, commissioned reports, expert testimony and cost effectiveness to determine:

• whether there was sufficient evidence (in terms of strength and applicability) to form a judgement
• where relevant, whether (on balance) the evidence demonstrates that the intervention or programme can be effective or is inconclusive
• where relevant, the typical size of effect (where there is one)
• whether the evidence is applicable to the target groups and context covered by the guidance.

The PDG developed draft recommendations through informal consensus, based on the following criteria:

• Strength (type, quality, quantity and consistency) of evidence of effectiveness.
• The applicability of the evidence to the populations/settings referred to in the scope.
• Where available and relevant, the effect size and potential impact on the target population’s health.
• Impact on inequalities in health between different groups of the population.
• Ethical issues and social value judgements.
• Cost effectiveness (for the NHS and other public sector organisations).
• Balance of risks and benefits.
• Ease of implementation and any anticipated changes in practice.

Where possible, recommendations were linked to an evidence statement(s), report or expert testimony (see appendix C for details).

The draft guidance, including the recommendations, was released for consultation in February 2010. At its meetings in May 2010, the PDG amended the guidance in light of comments from stakeholders and experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in September 2010.