Hierarchy of Evidence

  1. Meta analysis
  2. Systematic reviews
  3. RCTs
  4. Cohort Studies
  5. Case-control studies
  6. Case studies / reports
  7. Background Info / Expert Opinion



Defintions

  • “Process of systematically locating, appraising and synthesising evidence from scientific studies in order to obtain a reliable overview” - Helen Worthington

  • “Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies, thereby making the available evidence more accessible to decision makers.” - CRD (2009) p.V

  • “A systematic review can be defined as “a review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.” - http://www.cochrane.org/glossary



Why might a review be needed?

  • Is there a need to summarise existing information using an unbiased method?

  • Are more general conclusions possible to be drawn from a SR over multiple single studies?

  • Is evidence needed to inform a research project or answer previous unanswered questions in a topic area?



Points to Consider Beforehand

  • Why is this question being asked?

  • Do we already know the answer e.g. what other evidence is there related to this?

  • Is a review the best method?

  • Does the research question now need to be refined?



Narrative Review vs Systematic Review

  • Narrative Review

    • Often more broad, less or undefined method of searching, critiquing and synthesising data (unlikely have meta analysis)

  • Systematic Review

    • Focused research question, rigorous method, searching, appraising, critiquing and then synthesising data (qual and quant)



Pros and Cons

  • Advantages

    • Reduces Bias
    • Replicable
    • Can provide understanding of conflicting findings across studies
    • Helps inform decision making
    • Can combine data from studies to perform a meta-analysis to determine effect of interventions

  • Disadvantages

    • Can still lead to inconclusive results and recommendations for more evidence
    • May have an empty or very limited review
    • Study quality may be poor
    • Can take a long time for evidence to impact practice/policy



Protocol Development

Check!

  • Has the review or a similar review already been conducted?

    • If so when/where – is it in need of updating/different location?

  • Has it been conducted using one method but there is scope to use a different one (e.g. qual/quant or review of reviews)

  • Are there likely to be studies in the review

  • Has the title or a similar study been registered as a protocol (e.g. on PROSPRO)


Why Develop a Protocol?

  • Provides a framework – ensure transparency and reproducibility

  • Reduces bias

  • Provides a brief background and rationale for the review

  • Describes how papers will be identified, selected and evaluated

  • Sets out the objectives

  • Sets out the selection criteria

    • What types of studies and why?
    • Types of participants and rationale
    • If relevant types of interventions and comparisons –> what criteria must they meet (e.g. setting, time, who deliver)
    • Types of outcomes measures
    • Describe how studies will be identified, selected and evaluated


Planning Considerations

  • What is the title of your review?

    • should be a statement not a question

  • Who are the participants/target audience your interested in and why?

    • (e.g. age, gender, ethnicity, pre existing conditions)

  • Intervention/programme characteristics of importance?

  • Comparisons interested in? (e.g. intervention vs. control)

  • Outcomes likely to be used within papers and interested in?

  • Any studies you know of that would fit in the review?


Protocols should consist:

  • Background and rationale for the study

  • Research questions/objectives

  • Search strategy

    • key words, data bases with years, any scoping

  • Study selection

    • inclusion & exclusion, how criteria will be applied how ensure bias is kept to a minimum

  • Study quality

    • how will this be checked and why that tool

  • Data extraction

    • how will the details needed to answer research objectives be taken from studies

  • How synthesis planned

    • any expected analysis and type of analysis (qual or quant or meta analysis), any sub groups interested in

  • What and who you are aiming to contribute to through conducting the review

  • A provisional timeline

  • CRD guide to a review protocol


Formulating the review questions

  • Should relate to specific patient/topic problem

  • Likely to be linked to an intervention, aetiology, diagnosis or prognosis

  • E.g To estimate the accuracy of … or To determine the effectiveness of …

  • May be helpful to use the PICOC acronym

    • Population
    • Intervention
    • Comparison
    • Outcome/evaluative measures
    • Context/setting

  • Other acronyms include:

    • PEO, SPICE, SPIDER, PICo, PEO

  • Useful links:

  • Is it a broad or narrow question?

    • Broad questions:

      • e.g. Do behaviour change interventions work in reducing back pain in office workers?

    • Narrow questions:

      • Is CBT effective in treating moderate depression in females aged 30-40 years old?


Steps of Review Process

  1. Formulate question
  2. Develop search strategy
  3. Conduct comprehensive search
  4. Title and abstract screening - using predesigned questions
  5. Full text screening of possible included papers and unclear papers
  6. Final set of papers obtained
  7. Quality assessment of papers and data extraction
  8. Synthesis of data



Jigsaw Pieces

  • Piece 1: Develop and Conduct comprehensive search

    • MeSH (Medical Sub-Headings)
    • Descriptors
    • Subject Headings

  • Piece 2: Screen Titles, Abstracts and Full Papers

    • Inclusion and Exclusion criteria



Websites to explore:

Cochrane Collaboration (good manual for systematic reviews and meta-analysis)

Campbell Collaboration

Joanna Briggs Institute (good manual for many different types of reviews)

Eppi Centre

Free Training from Cochrane




Systematic Review vs Non Systematic Review

Systematic Review

  1. Description of how data will be synthesised
  2. It is objective and reproducible
  3. The question is clearly defined
  4. Criteria for inclusion of studies are stated
  5. The literature search is comprehensive
  6. Studies included are appraised for quality

Non-Systematic Review

  1. Provides an overview or update on a topic
  2. Criteria for inclusion are not stated
  3. It may be a subjective view from an expert
  4. No appraisal criteria are stated
  5. The search is not explained and/or is limited
  6. There is no systematic synthesis



Different Types of Systematic Reviews

SR with or without meta analysis Qualitative/Narrative reviews Review of reviews Behavioural intervention reviews Rapid SR
Aims to reduce bias Focus on synthesis of findings Combine outcomes multiple reviews Aim to identify aspects to improve int. design Streamlined approach
Follow standard structure Look at experience/implementation Search limited to other types of reviews Tend to look at how can combine multiple behaviours Evidence more timely
Best available evidence for a specific question More likely to include grey literature Helps provide overarching review of evidence Same as SR process with specific requirements May limit sources, search
Synthesis data from multiple studies Themes identified across studies Used to synthesis findings for evidence briefs Influenced by literature and need to streamline interventions More descriptive summaries, categories
Transparent Can help provide a deeper understanding of an area Can combine meta analysis May involve multiple searches of different behaviours Findings more limited/cautious
Detail procedure in advance Aid dialogue, reflection and use of narrative Aims to maintain rigour
Explicit search, inclusion/exclusion criteria No minimum reporting standards
May include meta analysis Come out of clinical/tech/time requirements



SR With and Without Meta-Analysis

Good methods guide

Good non-Cochrane/structured example

Qualitative/Narrative systematic reviews

Good example

Good methods paper

Good methods paper

Review of reviews

Good methods paper

Good example paper

Behavioural intervention reviews

NOTE - If you wanted to undertake this type of review for your dissertation you would need to first speak to you supervisor due to their complexity and more advanced nature of the review.

Good example‘s of two different types and ways of categorising behaviours in interventions - (there are 3 other papers linked of this one)

Another example

Rapid systematic reviews

NOTE – this type of review is NOT suitable for your dissertation.

Good methods paper

Good paper



Steps of Review Process

Steps of Review Process



Why do we do Data Extraction?

  • Studies are often published in different formats and using different reporting styles

  • Allows information needed to be standardised, helps us know about the main elements of interest, can help us start to see patterns and things relevant for analysis.

  • Aims to extract the findings from each study in a consistent way to allow synthesis

  • Once complete, you should require minimal reference to the original paper during data synthesis

Data Extraction Forms

  • By this stage, you will have identified which papers contain evidence relevant for your review question.

  • Process of taking relevant data from included papers and putting it into a single usable format.

  • All forms likely to be different in some way – needs to work for you and your review (what data do you want to extract).

  • Think of the order of information in articles when designing the form (tip record p. no for ease of referencing).

  • Interested in two things – the descriptive data and the analytical outcome data

  • What you extract will be informed by your research question/aim

CRD on Data Extraction

  • “The first stage of any data extraction is to plan the type of analyses and list the tables that will be included in the report”

  • “Standardised data extraction forms can provide consistency in a systematic review, whilst reducing bias and improving validity and reliability”

  • “The form should be unambiguous and easy to use in order to minimize discrepancies.”

Things to consider for data extraction

  • What is your objective? What do you want to know?
  • What was your inclusion criteria?
  • What was the intervention? (time, session, setting, content etc)
  • Evaluation tools and outcomes.
  • Adverse events
  • Participant characteristics

Suggested Sections of the Form

  • Methods – design, length intervention or study, groups, blinding, attrition rates etc
  • Participants – n= for control and intervention, age, geographical location, gender, setting, inclusion/exclusion criteria etc
  • Intervention/main area of interest – design/pilot, intervention description, target, delivered by, involvement, aspects relating to specific factors etc
  • Outcomes – Key statistics of interest or the quotes and interpretations, conclusions etc
  • Notes – any linked papers or background papers, sponsors etc
Steps of Review Process



Developing your data Extraction Tool



Quality Assessment

“A systematic research synthesis is attempting to answer a question using the findings of research evidence. It is therefore important that the evidence is both trustworthy and relevant.” EPPI (2010)

“Recording the strengths and weaknesses of included studies provides an indication of whether the results have been unduly influenced by aspects of study design or conduct (essentially the extent to which the study results can be ‘believed’). Assessment of study quality gives an indication of the strength of evidence provided by the review and can also inform the standards required for future research.” CRD (2009)

Risk of Bias and Quality Assessment

Taken from Cochrane review training:

  • Quality = “did they do the best they could?”
  • Bias = “should I believe the results?”


Consider using a pre-validated tool, for example

  • Cochrane risk of Bias tool
  • CASP


Tools likely to consider

  • Appropriateness of study design
  • Choice of outcome measure and quality of reporting
  • Quality of intervention delivery and reporting
  • Generalisability

Choosing a Tool - Considerations

  • Study design – qual/quant, RCT etc
  • Level of detail required
  • Ease of the tool and generalisability of tool
  • Previous use of the tool in similar style studies
  • May need to use a batch of tools if different study types

Different Tools

Example Qualitative Tools

CDR critical appraisal for reviews

  • “Was the review question clearly defi ned in terms of population, interventions, comparators, outcomes and study designs (PICOS)?
  • Was the search strategy adequate and appropriate? Were there any restrictions on language, publication status or publication date?
  • Were preventative steps taken to minimize bias and errors in the study selection process?
  • Were appropriate criteria used to assess the quality of the primary studies, and were preventative steps taken to minimize bias and errors in the quality assessment process?
  • Were preventative steps taken to minimize bias and errors in the data extraction process?
  • Were adequate details presented for each of the primary studies?
  • Were appropriate methods used for data synthesis? Were differences between studies assessed? Were the studies pooled, and if so was it appropriate and meaningful to do so?
  • Do the authors’ conclusions accurately reflect the evidence that was reviewed?”



Analysis/Synthesis

Expectations

  • To provide added value beyond just descriptive mapping and reporting and providing a bibliography
  • Provide a synthesis of the studies = collation, combination and summary of individual studies into one analysis

  • Looking to work through three steps:

    • The analysis
    • Producing the narrative, numerical, thematic or graphical presentation
    • Finalising the report

  • Provide a PRISMA Flow diagram

  • Use tables to present data e.g. table of included studies, to make it easier that lots of text for the reader

  • For your dissertation - Provide an extract of your title and abstract screening process in the appendices

  • Present initial scene setting data e.g. countries and number of participants

  • Provide a synthesis of the outcomes and analysis

Narrative Synthesis

  • Presenting your results in a written manner (but not just repeating what is in tables)

  • Used to report overall summarised findings and explained in words

  • Should help you present similarities and differences in your results

  • It is not a meta-analysis: narrative synthesis can support a meta-analysis or be used when meta-analysis is not possible

  • In this section you are only presenting (not discussing) your results

Results and Data Synthesis

  • Results of studies
    • Summary assessment methodology and interventions
    • Summary of results
    • Meta analysis if possible

Qualitative Synthesis

  • Aims to look for consistencies and differences in qual. literature
  • Can identify gaps in themes
  • Steps in analysis can be less prescribed than qualitative reviews
  • May use framework analysis or thematic analysis

Discussion and Conckusion Points to Consider

  • Adapted from Boland, Cherry & Dickson (2014)
    • Did you find the evidence you thought your would?
    • Were you able to answer your objectives
    • How do your findings fit with previous literature
    • Did the studies have any particular strengths and weaknesses
    • Did the review process have any particular strengths and weaknesses
    • How generalizable are the findings
    • What conclusions are you able to make from your review
    • What recommendations and what are the implications of your review for practice, future research and policy?



Useful Resources