Skoči na glavni sadržaj

Screen4Me

A service to help review teams to screen search results

What is Screen4Me?

Screen4Me is a service designed to help review teams complete the screening of search results much more quickly, without compromising on quality. It consists of three components: Known Assessments, the RCT Classifier, and Cochrane Crowd.

  1. Known Assessments are records that have been through Cochrane’s citizens science platform, Cochrane Crowd, and have received a final classification of either describing a randomised controlled trial (RCT) or not.
  2. The RCT Classifier is a machine-learning routine that assigns records a 'likelihood' score, based on whether the machine thinks the record is describing an RCT or not.
  3. Cochrane Crowd is Cochrane’s crowdsourcing platform, where contributors from around the world help to identify randomised trials and other types of healthcare-related research.

Please note: Screen4Me is currently only available for Cochrane intervention reviews that intend to include evidence from randomised controlled trials only.

How does Screen4Me work?

Once the review team has conducted the searches, the de-duplicated search results are loaded into the Cochrane Register of Studies. Once imported, the Screen4Me workflow can be applied.

The first part of the workflow will identify any already known RCTs; the second stage will remove very obvious non-RCT records from the set via machine learning; and the final stage sends the remaining records to be assessed by volunteers via Cochrane Crowd. At the end of the process, the author team should be left with a much smaller batch of records to assess.

Screen4Me projects take on average two weeks to complete. The search results set is expected to reduce by 50-70%.

How do I access it?

There are two main ways to access Screen4Me:

  1. Contact the Cochrane information specialist you are working with. Many Cochrane information specialists have been trained to run the Screen4Me service. They will do this using the Cochrane Register of Studies application.
  2. If you are not working with a Cochrane information specialist or they do not have access to Screen4Me, contact Cochrane Support to arrange the Screen4Me service for you.

Report Screen4Me in Cochrane protocols and reviews

If you have plan to use Screen4Me in your review, or if you have used it, you need to report this clearly in your methods. See below for reporting guidance.

Screen4Me reporting guidance

Additional references

When reporting use of Screen4Me in your Cochrane protocol or review, add these references to your additional references in RevMan first, so that you can link to them when inserting the template text below.

Noel-Storr AH, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, Wisniewski S, McDonald S, Murano M, Glanville J, Foxlee R, Beecher D, Ware J, Thomas J. An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. J Clin Epidemiol. 2021 May;133:130-139. https//doi.org/10.1016/j.jclinepi.2021.01.006. Epub 2021 Jan 18. PMID: 33476769.

Thomas J, McDonald S; Noel-Storr AH, Shemilt I, Elliott J, Mavergames C, Marshall I. Machine learning reduces workload with minimal risk of missing studies: development and evaluation of an RCT classifier for Cochrane Reviews. Journal of Clinical Epidemiology 2020; https://doi.org/10.1016/j.jclinepi.2020.11.003

Noel-Storr AH, Dooley G, Wisniewski S, Glanville J, Thomas J, Cox S, Featherstone R, Foxlee R, Cochrane Centralised Search Service showed high sensitivity identifying randomised controlled trials: A retrospective analysis. Journal of Clinical Epidemiology 2020; 127: 142-150 https://doi.org/10.1016/j.jclinepi.2020.08.008

Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide. Res Synth Methods. 2018;9(4):602-614

Protocols

RevMan section: Selection of studies

Add the following template text at the start of this section:

"We will be using Cochrane’s Screen4Me workflow to help assess the search results. Screen4Me comprises three components: known assessments – a service that matches records in the search results to records that have already been screened in Cochrane Crowd and been labeled as an RCT or as Not an RCT; the RCT classifier – a machine learning model that distinguishes RCTs from non-RCTs; and if appropriate, Cochrane Crowd – Cochrane’s citizen science platform where the Crowd help to identify and describe health evidence.

For more information about Screen4Me and the evaluations that have been done, please go to the Screen4Me webpage on the Cochrane Information Specialist’s portal. For more detailed information regarding evaluations of the Screen4Me components, see [add in-text citations to additional references Noel-Storr 2021, Thomas 2020, Noel-Storr 2020, Marshall 2018]."

Reviews
RevMan section: Selection of studies

Add the following template text at the start of this section:

"We used Cochrane’s Screen4Me workflow to help assess the search results. Screen4Me comprises three components: known assessments – a service that matches records in the search results to records that have already been screened in Cochrane Crowd and been labeled as an RCT or as Not an RCT; the RCT classifier – a machine learning model that distinguishes RCTs from non-RCTs, and if appropriate, Cochrane Crowd – Cochrane’s citizen science platform where the Crowd help to identify and describe health evidence.

For more information about Screen4Me and the evaluations that have been done, please go to the Screen4Me webpage on the Cochrane Information Specialist’s portal. For more detailed information regarding evaluations of the Screen4Me components, see [add in-text citations to additional references Noel-Storr 2021, Thomas 2020, Noel-Storr 2020, Marshall 2018]."

RevMan section: Results of the search

Add the following template text at the start of this section:

The search identified a total of [insert number] search results. In assessing the studies we used Cochrane’s Screen4Me workflow to help identify potential reports of randomised trials. The results of the Screen4Me assessment process can be seen in Figure x. We then assessed the remaining [insert number] records left in after Screen4Me. [Author team then continues to describe their screening process e.g. assessing remaining records based on a title and abstract assessment.]

RevMan section: Figures

1. Screen4Me summary diagram
We recommend that you export the Screen4Me summary diagram from your Screen4Me project and import it into the Figures sections of the review. Figure type: Other figure. Caption: Screen4Me summary diagram. Here is an example. Screen4Me produces a flow diagram per search so if you use S4M multiple times for a single review, you will have generated multiple Screen4Me flow diagrams.

2. PRISMA flow diagram example

In the example below, the key information about what was excluded at the Screen4Me stage of the screening process is detailed in the Screening exclusion box.

RevMan section: Acknowledgements

If you use the Cochrane Crowd component of Screen4Me, you must acknowledge by name those Crowd contributors who screen 250 or more records for your Screen4Me review/project. You can see who should get acknowledgement within the Screen4Me project summary screen. You can name Screen4Me volunteers in the Acknowledgements without seeking permission to name them individually. Volunteers give this permission when agreeing to take part in the Crowd task.

Template text for the Acknowledgements section:

We would like to acknowledge and thank the following people for their help in assessing the search results for this review via Cochrane’s Screen4Me workflow: [insert names].

Updates

Some users will make use of Screen4Me for updates of reviews where Screen4Me has not been used previously. Here is an example of how to reflect the use of Screen4Me in these circumstances:

RevMan section: Figures - PRISMA flow diagram example

Multiple uses of Screen4Me for an individual review

Some users will use Screen4Me more than once for a specific review. Here is an example of how to reflect the use of Screen4Me in these circumstances:

RevMan section: Figures - PRISMA flow diagram example

Communicating with Cochrane Crowd

If you have used Cochrane Crowd, we would like to alert Crowd contributors when reviews they have contributed to are published. Please contact us when your review is published. The Cochrane Crowd team will send an alert to the Crowd contributors and flag the review as published on the Cochrane Crowd platform.