Critical appraisal: assessing research quality
One you have selected a range of journal papers and other evidence to guide your work, it is important to read through the material carefully to check that it is not only relevant to your research focus, but also trustworthy in its procedures and findings. As Rutter et al. (2010: 50) point out:
The relevance of a study to the review topic, and the appropriateness of design to address the review question, are two aspects of quality; the integrity of the methods used in the study, and the confidence we can have in its findings, are others.
There are many different tools available to help you assess qualitative and quantitative studies and they may look at different aspects of the research project to determine 'quality': Harden (2004) assessed 31 aimed at only qualitative research, and found 545 'domains of quality' between them (in Rutter et al 2010 p51).
For most student and practitioner researchers, the TAPUPAS framework devised by Pawson et al. in 2003 (see Long et al. 2006) can provide the thinking space to assess quality.
- Transparency: the process of knowledge generation should be open to outside scrutiny. For knowledge to meet this standard, it should make plain how it was generated, clarifying aims, objectives and all the steps of the subsequent argument, so giving readers access to a common understanding of the underlying reasoning.
- Accuracy: all knowledge claims should be supported by and faithful to the events, experiences, informants and sources used in their production. For knowledge to meet this standard, it should demonstrate that all assertions, conclusions and recommendations are based upon relevant and appropriate information.
- Purposivity: the approaches and methods used to gain knowledge should be appropriate to the task in hand, or 'fit for purpose'. For knowledge to meet this standard, it should demonstrate that the inquiry has followed the apposite approach to meet the stated objectives of the exercise.
- Utility: knowledge should be appropriate to the decision setting in which it is intended to be used, and to the information need expressed by the seeker after knowledge. For knowledge to meet this standard, it should be 'fit for use', providing answers that are as closely matched as possible to the question.
- Propriety: knowledge should be created and managed legally, ethically and with due care to all relevant stakeholders. For knowledge to meet this standard, it should present adequate evidence, appropriate to each point of contact, of the informed consent of relevant stakeholders. The release (or withholding) of information should also be subject to agreement.
- Accessibility: knowledge should be presented in a way that meets the needs of the knowledge seeker. To meet this standard, no potential user should be excluded because of the presentational style employed.
- Specificity: the knowledge must pass muster within its own source domain, as perceived by its participants and proponents.
(Pawson et al. 2003 in Long et al. 2006: 210)
Long et al. (2006) point out that while the TAPUPAS framework is generally applicable, not all of the standards are suitable for different types of study.
Orme and Shemmings (2010: 75) suggest a set of 'key questions' that should be asked as part of a critical appraisal of social work research. These are:
- How relevant is the study to the review question?
- How much information does it contribute?
- How trustworthy are its findings?
- How generalisable are its findings?
- Was it conducted ethically?
The Critical Appraisal Skills Programme (CASP) has a set of questions that can be asked of a range of studies to help to appraise their quality, including randomised controlled trials, systematic reviews, case-control studies, qualitative studies and economic evaluations. CASP has both a UK websiteand one targeted at an international audience.
Select a study that you would like to use in your research. Read it closely to find out what kind of a study it is. Is it qualitative or quantitative? What is the research design?
Then use these three different quality assessment methods - the TAPUPAS framework, Orme and Shemmings' five questions and one of the CASP appraisal checklists to look at its strengths and limitations.
Which of these do you find most useful? Do you think it would still be the most useful if you were looking at a different kind of research?
Bear in mind when using prescriptive tools such as these that your intention is not to score and exclude research (unless you discover that it is untrustworthy), but to be able to assess the strengths and limitations of what you find, so that you can use your judgement to reflect on its merits in relation to your topic. As a research-minded student or practitioner, your focus is always going to be on the service user and practice problems. Being able to bring your knowledge into the assessment of research quality will help ensure that your choices of evidence to inform your practice are critically assessed, and remain centred on the context of provision.