SCIE/NICE recommendations on looked after children: Promoting the quality of life of looked-after children and young people

Appendix B

Summary of the methods used to develop this guidance

Introduction

The reviews, primary research and commissioned reports include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it. The minutes of the PDG meetings provide further detail about the Group’s interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix D and are available at www.nice.org.uk/guidance/PH28

Guidance development

The stages involved in developing public health programme guidance are outlined in the box below.

  1. Draft scope released for consultation
  2. Stakeholder meeting about the draft scope
  3. Stakeholder comments used to revise the scope
  4. Final scope and responses to comments published on website
  5. Evidence reviews, economic analysis and practice survey undertaken and submitted to PDG
  6. PDG produces draft recommendations
  7. Draft guidance (and evidence) released for consultation and for field testing
  8. PDG amends recommendations
  9. Final guidance published on website
  10. Responses to comments published on website
Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by the PDG to help develop the recommendations. The overarching question was:

The subsidiary questions were:

These questions were made more specific for each review (see reviews for further details).

Effectiveness reviews

Reviews of effectiveness were conducted using NICE methods, on:

Identifying the evidence

The strategy adopted for these reviews combined systematic review searching undertaken by SCIE with more targeted searches undertaken by ScHARR. The following databases were searched for published literature (January 1990 to September 2008):

In addition, searches of reference lists and citation searches were conducted on all relevant papers included in the ‘Correlates review’ (review 4; see page 98). The reference lists of included papers were searched by hand, and citation searching was undertaken on all included papers. The PDG was also consulted for relevant literature.

Further details of the databases, search terms and strategies are included in the review reports.

Selection criteria

Inclusion and exclusion criteria for each review varied and details can be found at www.nice.org.uk/guidance/PH28 However, in general:

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual ‘Methods for the development of NICE public health guidance’ (see appendix D). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the checklist criteria have been fulfilled. Where they have not been fulfilled, the conclusions are very unlikely to alter.

+ Some of the checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are unlikely to alter the conclusions.

– Few or no checklist criteria have been fulfilled. The conclusions of the study are likely or very likely to alter.

Summarising the evidence and making evidence statements

The review data were summarised in evidence tables (see full reviews).

Within each review, the findings were synthesised and used as the basis for a number of evidence statements relating to each research question. The evidence statements were prepared by the public health collaborating centres (see appendix A). The statements reflect their judgement of the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Other reviews

Two other reviews were conducted: a correlates review (review 4) and a qualitative review (review 5):

Correlates review

This review aimed to identify factors that are associated with outcomes for looked-after children and young people, and to present these as conceptual maps showing the strength of associations between factors, supported by evidence tables. It is not a full systematic review with exhaustive or comprehensive searches but it is a review of quantitative data, with inclusion and exclusion criteria, quality appraisal and a systematic and transparent methodology.

Identifying the evidence

Searches were done using free text and keywords/index terms, and several iterations were undertaken. Once an outcome or variable had been identified, further searches and sifting sought to identify new variables rather than to identify all the evidence for each relationship between variables. A thorough audit trail was maintained, to ensure that searches were transparent, systematic and replicable.

Selection criteria

Studies were included if the population was looked-after children and young people aged up to 25 years, or adults who had been looked-after children and young people. Explanatory factors were included if they could be defined as risk factors, protective factors or interventions (any intervention or activity that could affect these outcomes). Outcomes were as defined by individual studies.

Quality appraisal

Study quality was assessed using the checklists and guidance in ‘Methods for development of NICE public health guidance (second edition, 2009)’.

Summarising the evidence

Relevant information was extracted and studies were classified according to both study type and the main variables analysed. The findings were then mapped using software that allows associations to be presented graphically.

Qualitative review

This review used SCIE methods and aimed to identify qualitative research on and synthesise the views, experiences and preferences of children and young people, their families and carers, about the care system.

Identifying the evidence

The following electronic health and social care databases were searched:

Further details of the databases, search terms and strategies are included in the review.

Selection

Studies were included if they were performed in the UK and their methods for data collection and analysis enabled them to reflect what is important to looked-after children and young people and their families and carers.

Quality appraisal

Included studies were critically appraised and rated for:

Summarising the evidence

The pool of findings from all the studies was assessed for each of the three groups: looked-after children and young people; carers; and parents. Themes and subthemes were developed via a framework analysis approach. The detail of the findings was interrogated to answer, where possible, the research questions and evidence statements were developed.

Practice survey

This used SCIE’s practice survey methodology to identify innovative and emerging practice in assessing, maintaining and improving the physical and emotional health and wellbeing of looked-after children and young people. The survey sought views about the acceptability, accessibility and effectiveness of targeted and specialist, as opposed to universal, interventions.

Sampling

The PDG identified sites that were of particular interest because of their specific areas of practice – for example, multidisciplinary work, and specialist services for looked-after children. In addition, the Action for Children team who wrote the report identified a local authority that had ‘Beacon’ status for its engagement of young people and its targeted youth support initiatives.

Methods

Following initial approaches, all sites agreed to participate in the survey, and approval was gained from the Assistant Director of Children’s Services Research Team and ethical approval in each site. The first phase of the survey was to conduct a series of group and individual semi-structured interviews with managers, commissioners and practitioners from a range of social care, education and health organisations. These interviews took place between April and June 2009.

The second phase involved interviews with carers and looked-after children and young people.

The data collected from the survey was analysed, by reading and aggregation of data, measuring, identifying differences and similarities, identifying and acknowledging variables, contextualising, noting correlation, identifying themes and cross-cutting issues, comparing views within and across sample sectors and ordering and grouping findings to address topics and questions.

Summarising the evidence

The findings were structured around four topic areas: structure of care; delivery of care; interventions and activities; and the views of children, young people and their families. These were used as the basis for evidence statements.

Economic analysis

The economic analysis consisted of a review of economic evaluations and cost-effectiveness modelling.

Review of economic evaluations

A database supplied by SCIE was used to find papers containing evidence of cost effectiveness. This was supplemented by targeted searches of: Web of Science MEDLINE NHS EED EconLit. Inclusion criteria followed the same criteria used for the effectiveness reviews, for which the economic reviews were an accompaniment. Quality was assessed using the ’Methods for development of NICE public health guidance’ (2006).

Cost-effectiveness modelling

An economic model was constructed to accompany the three effectiveness and cost-effectiveness reviews (see appendix E). A satisfactory model was constructed for review 1. A model was attempted for review 2 but was rejected by the PDG because of data limitations. For review 3, a model was not attempted because of lack of relevant data.

The results of the model to accompany effectiveness review 1 are reported in: ‘Review 1: The cost-effectiveness of support services for transition to adulthood/leaving care on the adult outcomes of looked after young people’. Following the inability to produce a meaningful modelling analysis for reviews 2 and 3, the PDG was asked to use a cost-consequences framework to consider the cost effectiveness of the recommendations. This required members to consider the best available evidence on the costs and effectiveness of services of relevance to the recommendations under consideration and to draw on their expertise to make an appropriate recommendation where evidence gaps existed.

The PDG examined the recommendations to see whether any were:

This was successful for recommendations where the available evidence and the opinions of experts and PDG members strongly suggested that their implementation would do more good than harm, would be of relatively low cost to implement and would probably produce cost savings in the long run.

What prevented this method being used more widely to determine whether a recommendation was likely to be cost effective was that many of the recommendations focus on system-level changes, multi-agency working, information sharing and training. For these recommendations, the lack of data and the resulting level of uncertainty were too great to support even a cost-consequences approach.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations were for practitioners and how feasible it would be to put them into practice. It was conducted with commissioners and practitioners who are involved in social care, health and education services for looked-after children and young people. This included those working in the NHS, local authorities and the wider public, private, voluntary and community sectors.

The fieldwork comprised:

Consultation with looked-after young people and care leavers

A consultation was carried out with 30 looked-after young people and care leavers aged 11–24. The aim was to gather their views about the relevance, usefulness and acceptability of appropriate sections of the guidance.

The consultation was undertaken by Action for Children and comprised focus groups at four sites (two county councils, a district council and an inner city unitary authority).

The consultation report, ‘The physical, emotional health and wellbeing of looked-after children and young people’, is available at www.nice.org.uk/guidance/PH28

How the PDG formulated the recommendations

At its meetings in 2008 and 2009, the PDG considered the evidence reviews, commissioned reports, expert testimony and cost effectiveness to determine:

The PDG developed draft recommendations through informal consensus, based on the following criteria:

Where possible, recommendations were linked to an evidence statement(s), report or expert testimony (see appendix C for details).

The draft guidance, including the recommendations, was released for consultation in February 2010. At its meetings in May 2010, the PDG amended the guidance in light of comments from stakeholders and experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in September 2010.