Dual screening is a best practice for conducting systematic reviews, but there’s often not enough time or resources to have two people screen each reference. Good news: DistillerSR’s new automated screening tool, DistillerAI, can be used as the second screener. That’s right – you now have a robot at your disposal (and you don’t even need to pay it!) Here’s how to use DistillerAI as a member of your screening team.
Step 1: Screen 100 References
Before DistillerAI can be used for automated screening, it needs to learn what constitutes an included and excluded reference. This typically requires about 100 references to be screened manually. To ensure the best results, set your level settings to require two screeners per reference. Have both reviewers screen the same 100 references, then make sure you resolve inclusion and exclusion conflicts. Voila! You have just created a high quality training set.
Step 2: Rank Your References
Once you’ve trained DistillerAI to recognize your inclusion and exclusion criteria, the next step is to use it to rank the remaining references by likelihood of inclusion.
To rank your references in DistillerSR, go to the DistillerAI Toolkit and click the AI Preview and Rank tab. You should select only the Naive Bayes Classifier (i.e. uncheck the SVM Classifier), which provides a more granular scoring of references that works better for automated screening. Then, click the AI Preview and Rank button to assign a score to each reference. Once the ranking process is complete (you will see a green note appear in the top right of your browser screen when it’s done), click the Tag References button to tag the references with their respective scores.
Step 3: Screen Automatically
Now that DistillerAI has determined which references are most likely to meet your inclusion criteria, you’re ready to screen – or rather, to let the robot screen for you! This step uses the scores that DistillerAI created in step two to automatically complete your screening forms.
To start the automatic screening process, head to the DistillerAI Toolkit and click the AI Review tab. Next, choose the screening form and question that you want the robot to “answer”. Note that DistillerAI can only answer one question on a screening form, even if that form has multiple inclusion/exclusion questions, so pick the question and answers that are most relevant for inclusion, exclusion and unsure.
When using DistillerAI for automated screening, It’s important to remember that the robot screener does not know why a reference was excluded or included. It is simply looking for references that strongly resemble the references that were included in your training set. During automatic screening, the robot screener is simply selecting an answer on your screening form according to the score that was assigned to that reference.
That leads us to inclusion/exclusion thresholds. You will need to tell the robot the minimum score for a reference to be included, and a maximum score for a reference to be excluded. Anything in between will be assigned to a “Can’t Tell” option on your screening form.
Once your thresholds are set up and the inclusion/exclusion/can’t tell options assigned, click the AI Review button, and let the robot do the rest.
DistillerAI will screen all unreviewed references and assign the inclusion/exclusion/can’t tell responses that you specified on your screening form, according to the score thresholds you have set. Once the automated screening is complete, you can have a human screener go through and be the second screener on those references.
The robot’s work is recorded in DistillerSR under the username “DistillerAutomatedReviewer”. You can run conflict reports and Kappa scores that include the robot screener, just as you would with any other reviewer.
As we all know, systematic reviews can be extremely time and resource intensive. Automated screening can help you adhere to best practices for conducting reviews and free up your team for other review tasks. Will your next hire be a robot?
