Formulate a Question > Find the Evidence > Appraise the Evidence > Apply the Evidence > Evaluate the Results > Disseminate the Research

Appraising the Evidence

Critical appraisal is the process of assessing and interpreting evidence by systematically considering its relevance, validity and reliability. This can sound daunting, but it doesn't have to be. Most of us do this every day when listening to the news or reading the paper.

Suppose you read about a survey of 10 doctors and one of its findings was that kids who spend a lot of time in front of video screens have a higher risk of diabetes. You might have concerns about various things - for example, you might feel that that the method of the study (survey/expert opinion) is inappropriate to make a determination on this issue, or you might feel that the sample is too small. Chances are, if you were interested in the question, you'd give this study very little weight when considering the larger body of evidence.

If you were looking into the research on this question, it's likely you'd evaluate the available research based on the following criteria:

a) relevance: Does the study touch directly on the question of time in front of video screens as an independent factor in children's risk of obesity? Do the results provide illumination on the question?

b) validity: Is the study design appropriate to the question? Is the study well-designed and as free from bias as possible?

c) reliability: Is the method of the study clear enough to allow it to be reproduced? How likely is it that similar results would be arrived at by another researcher using the same methods?

The Hierarchy of Evidence - one tool for streamlining the appraisal process

It seems obvious once stated, that a really good meta-analysis precisely on your topic will have done much of the work for you (comparing and interpreting evidence from numerous studies). So if you find one of these, great! To be thorough, all you have to do is make sure that no key research is missing (later work, or important work that was missed), and that the quality of the meta-analysis is high.

Similarly, if you find a randomized, controlled trial on your topic with a large enough population and good methodology, you can be pretty confident that the study will be reliable. As a general rule, start with the highest level of evidence you can find that is relevant to your topic, and continue your investigation, as time allows, until you feel you have a handle on the major research on your topic. Of course you may find a case study that is particularly relevant, in which case you may want to place special importance on it - there are always exceptions to any rule! Using the hierarchy of evidence just encourages you to use your time and energy efficiently, and to place greater emphasis on the most valid and reliable research possible.

Making Appraisal Systematic - tools and checklists

Lindsay Glynn's Critical Appraisal Checklist.

• EBLCriticalAppraisalChecklist.pdf

See also Glynn, Lindsay. "A Critical Appraisal Tool for Library and Information Research." Library Hi Tech 24.3 (2006): 387-399.