Image

Transcript

Key steps in a systematic review brought to you by Evidence Synthesis Ireland and Cochrane Crowd. In this module we're going to look at asking the right question. 

“The wise person doesn't give the right answers, they pose the right questions.” 

When making decisions about health, it is important not to rely on the results from a single study. A systematic review aims to carefully combine the results of multiple studies investigating the same thing. 

[Music] 

A critical step in the process of producing a systematic review is making sure the question is right. Setting out with a dodgy question is like building a house on dodgy foundations. But what do we mean by asking the right question? The right question needs to be : needed, doable, clear and relevant. 

A question that is too big or too broad in scope will be hard to answer well and may take a very long time to do. Here's an example, “is exercise good for you?”. This question is far too broad. What kind of exercise? What does good for you mean? Is it about losing weight, improving strength or preventing dementia? Who is the you in the question, a child, a teen, a middle-aged woman, someone over 65? However, a question that is too narrow in scope may fail to be relevant to anyone. For example, is high intensity rowing before breakfast good for reducing blood pressure in retired Olympic athletes? 

Luckily help is at hand thanks to the PICO framework 

P stands for patient or population 

I stands for intervention 

C stands for comparison and  

O stands for outcomes 

A carefully thought through PICO will set the review up on solid foundations 

[Applause] 

Image

Transcript

Key steps in a systematic review brought to you by Evidence Synthesis Ireland and Cochrane Crowd.

[Music]

Module 2, finding the evidence. Leave no stone unturned, said Euripides.

Euripides: “Ouch!”

Once you have defined your question, you can begin the search for the answer. This means finding all the right studies that match your question’s PICO. Getting the search right is critical. Done badly the review may end up being inaccurate. As they say garbage in, garbage out.

[Music]

Conducting a good search for a review involves having a plan, being systematic, searching a range of sources and documenting what you are doing as you go along.  

Systematic reviewer: “Okay I think I'm ready.”

“Peer-reviewed search strategies, check. List of sources to search, check.  Search expertise on hand, check. Search report form, check.”

“Right, wish me luck, I'm going in.”

Some studies are harder to find than others but it is important to try and find all the evidence. Above all “cherry picking”, choosing studies because they look good, must be avoided. Experts in the topic area might know about some studies but don't just rely on them as your only source of information.

Dr Herrypick: “My study, uh I mean this study is really the only one you need”

Systematic reviewer: “Thank you, Dr Herrypick, I'll add it to the pile. Help yourself to a cherry”

The search forms a critical part of producing a systematic review. A flawed search will result in a flawed review. So, no cherry picking!

Module 2, finding the evidence.  

Euripides: “Oh ,there it is found it!” 

Image

Transcript

Key steps in a systematic review brought to you by Evidence Synthesis Ireland and Cochrane Crowd. Module 3, assessing the evidence.

“You know my methods, Watson. There was not one I did not apply to the inquiry. It ended in discovering traces, but very different ones than I had expected said.” Sherlock Holmes to Watson.

Once you have found the studies for inclusion within the review, each one must be carefully examined. Details about each study need to be recorded accurately and in a consistent way. Studies often have lots of publications and reports about them. These need to be grouped together so that the same study does not get included in the review more than once.

It's critical to get a clear idea of how each study was conducted. The methods section of a study paper should provide enough detail for readers to understand how the study was conducted. There should also be a section that describes any limitations or weaknesses of the study.  

No study is perfect. A badly designed study may mean the results of the study are not reliable or accurate.  

But how is the quality of the study assessed?

It's reported on the news it must be high quality. Sorry but no.

It was published in a high-quality journal so it must be high quality. Not necessarily.

The results were amazing so it must be high quality, right? Afraid not.  

Quality assessment of studies is done using checklists. There are different quality assessment checklists for different types of studies. Quality assessment of a study aims to identify any risk of bias. Bias is when an error occurs that skews the findings of the review. Assessing the conduct and quality of the studies is a critical stage in producing a systematic review.

Careful scrutiny may reveal some important factors that might have affected the results of the study.

[Music]

Watson: “So, it was Cochrane all along.”

Sherlock Holmes: “Elementary my dear Watson”

[Music] 

Image

Transcript

Key steps in a systematic review brought to you by Evidence Synthesis Ireland and Cochrane Crowd. Module 4, synthesizing the evidence.

The whole is greater than the sum of its parts. Once you have found and assessed the studies that are eligible for inclusion within your review, the next stage is to synthesize them.

Sorry, Dave. Not that kind of synthesizing.

Synthesizing the evidence means carefully combining the results data from each of the individual studies. Combining the results of each study in this way will help provide a more reliable answer to our review question.  

You don't have to be a mathematical genius to do this. Luckily statistical programs exist to help you synthesize the evidence. Results data from each study need to be carefully entered into the computer program. The statistical program then combines the results data from each individual study and produces a summary, or overall result. The result of combining, or should I say synthesizing, data in this way is often displayed as a special graph called a forest plot.

Sorry, Dave. Not that kind of forest

A forest plot is a type of graph that displays the results of each individual study included in the review, and the summary result from combining each study's results. Each study is shown in a new horizontal row. The results of each study are then plotted on the chart.  

If the intervention had a positive effect the results appear on the left. If the intervention had a negative effect the results appear on the right. Results which touched the line mean the intervention had no effect. The length of the lines tell us how precise the results of the study are. The shorter the line the more confident we are of the results.  

Finally, the diamond is added showing the synthesized result by combining the result from each individual study.  Synthesizing the evidence by combining the results of individual studies and producing a forest plot with a summary result is an important step in review. This is then followed by interpreting the result and making the result available for others.

Sometimes it isn't possible to synthesize the results of your included studies in this way. If the studies are very different then combining their results is a bad idea and will potentially produce an invalid overall result.

It's a bit like comparing apples with oranges

Oh, Dave!

[Music]

Module 4, synthesizing the evidence.