AI for the Evidence Synthesis

AI to Support Evidence Synthesis

The guide and the map of the AI tools.

Start

by Dr. olga Koz, alumni of ESI, 2021

Introduction

This guide focuses on AI applications in evidence synthesis (meta-studies). It follows the systematic review protocol to map recently developed AI tools. Due to the rapid development of AI, this is a living guide that is constantly updated. I hope it helps to answer the following questions: - What is our (evidence synthesis librarians) future in the era of AI? - Should we be prudent and wait for the AI hype to be over? - How and where should we integrate AI tools?

Index

AUTOMATION OF SR

Proposal

This guide maps AI tools to the evidence synthesis or a systematic review process.

EVIDENCE RETRIEVAL

SR SOFTWARE

COMPARE AI TOOLS

MY RESPONSE

Survey of Health Professionals about AI use in SR

Replace

AI will fully replace humans in LR

Query

To significant extent

Screening

Extraction

To some extent

Meta-Analysis

'Use this space to put a great sentence'

'Your content is liked, but it hooks even more if it is interactive'

Minimally

Creating Reports

AI & Humans work together

Needs in automation

Future AI in 5 years

Source

Next

Concerns

Credibility Reliability (replicability) Confirmability Transferability Transparency The source of the evidence Evidence Retrieval

Next

Meta-analysis vs Meta-Synthesis Evaluation Criteria Shaheen, N., Shaheen, A., Ramadan, A., Hefnawy, M. T., Ramadan, A., Ibrahim, I. A., … & Flouty, O. (2023). Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. Frontiers in Research Metrics and Analytics, 8.

AI and regulatory criteria for research

AMSTAR CASP RoB-2 (Risk of Bias)

Evaluation Matrix (Research Criteria)

Next

Concern 1

Concern 3

What does it mean?

Criteria

AConcern 2

Transperancy, Explainability

Is AI an author?

Transperancy, Explainability

The authority of the source or training dataset

Credibility, trustworthiness

Modifying questions for LLMVariations of prompts

Dependability, Consistancy

Modifying LLM anwers (RAG, Fine-tuning). Report the datasets and prompts.

Showing that discovery is consistent and could be repeatedMethods, protocol

Black box

Confirmability/Neutrality

+info

Measuring Risk of Bias with AI (RoB)

Degree to which the research findings are objective and free from researcher bias

Source & Data Bias

Next

Learn about AI apps: sources for data, LLM, and AI techniques.

Learn about AI techniques (RAG, Fine-Tuning, and so on)

Artificial Intelligence

Learn your LLMs

PROTOCOL

Evidence preparation

Design

Synthesis Writing

Evidence Collection

Analysis

Evidence Retrieval

Natural Language Understanding (NLU). Query meaning, context, and content.

Lexical Search

Hybrid or Neural

Field Search, Thesaurus Browsing, Syntax and Operators, Filters, Pre-indexed Databases

LLM enhances search and creates text with citations. Augmenting LLM with RAG, SLM, Fine-Tuning, and embeddings

Semantic search

...

AI Tools 4 LR

Datasets

Elicit

Semantic Scholar

Consensus

Undermind.ai

Open Alex

LitMap

Keenious

Scite.ai

Scite.ai

https://bit.ly/LR-AI

by Dr. Olga Koz

A bar graph showing the concerns of AI in literature review software AI Viewed as Inevitable, Despite Concerns

Compare

AI to assist with the systematic review

Source: Jimenez et al, 2022

SR Software with AI

Next

AI

AI for Citation Analysis and literature mapping

https://libguides.kennesaw.edu/litMap

Embedded AI

Link

Guide: AI for discovery & Literature review

Course: LR for EdD students

Scoping Review with AI

CREATING AI RESEARCH ASSISTANT

Next

Reference

Cierco Jimenez, Ramon & Lee, Teresa & Rosillo, Nicolás & Cordova, Reynalda & Cree, Ian & González Wong, Angel & Indave, Blanca. (2022). Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Medical Research Methodology. 22. 10.1186/s12874-022-01805-4.

Jardim, P.S.J., Rose, C.J., Ames, H.M. et al. Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Med Res Methodol 22, 167 (2022). https://doi.org/10.1186/s12874-022-01649-y

Mozgai, S., Kaurloto, C., Winn, J., Leeds, A., Heylen, D., Hartholt, A., & Scherer, S. (2023). Machine learning for semi-automated scoping reviews. Intelligent Systems With Applications, 19, 200249. https://doi.org/10.1016/j.iswa.2023.200249

Spillias, S., Tuohy, P., Andreotta, M., Annand-Jones, R., Boschetti, F., Cvitanovic, C., … & Trebilco, R. (2023). Human-ai collaboration to identify literature for evidence synthesis.. https://doi.org/10.21203/rs.3.rs-3099291/v1

Please follow up!

Post - presentation survey. I appreciate your answer to 3 questions

okoz@kennesaw.edu https://libguides.kennesaw.edu/AI4LR

Your content is likable, but it only engages if it's interactive. Capture your audience's attention with an interactive photo or illustration.

Your content is likable, but it only engages if it's interactive. Capture your audience's attention with an interactive photo or illustration.

With this function... You can add additional content to excite the brains of your students: videos, images, links, interactivity... Whatever you want!

Your content is appreciated, but it only engages if it's interactive. Capture your audience's attention with an interactive photo or illustration.

Did you know... In Genially you will find more than 1,000 templates ready to showcase your content and 100% customizable, which will help you in your classes?

Did you know that... We retain 42% more information when the content is dynamic? It is perhaps the most effective resource to capture the attention of your students.

Tip: Interactivity is the key to capturing the interest and attention of your students. A genially is interactive because your group explores and interacts with it.

Did you know... In Genially you will find over 1,000 templates ready to input your content and 100% customizable, which will help you in your classes?

Your content is liked, but only engages if it is interactive. Capture the attention of your audience with an interactive photograph or illustration.

Did you know... The window allows for adding more extensive content? You can enrich your genially by incorporating PDFs, videos, text... The content of the window will appear when clicking on the interactive element.

Got an idea?

Bring it to life with an interactive window. Create a new layer with all the Genially features; include text, images, videos, tables, PDFs... Even interactive questions!

¡QUIZ!

Got an idea?

Bring it to life with an interactive window

Create a new layer with all the Genially features.

• Generate experiences with your content.
• It’s got the Wow effect. Very Wow.
• Make sure your audience remembers the message.
• Activate and surprise your audience.

With this function... You can add additional content that excites the brains of your students: videos, images, links, interactivity... Whatever you want!