AI for the Evidence Synthesis

The guide and the map of the AI tools.

Start

AI to Support Evidence Synthesis

by Dr. olga Koz, alumni of ESI, 2021

This guide focuses on AI applications in evidence synthesis (meta-studies). It follows the systematic review protocol to map recently developed AI tools. Due to the rapid development of AI, this is a living guide that is constantly updated. I hope it helps to answer the following questions: - What is our (evidence synthesis librarians) future in the era of AI? - Should we be prudent and wait for the AI hype to be over? - How and where should we integrate AI tools?

Introduction

This guide maps AI tools to the evidence synthesis or a systematic review process.

Index

Proposal

AUTOMATION OF SR

EVIDENCE RETRIEVAL

COMPARE AI TOOLS

SR SOFTWARE

MY RESPONSE

Source

'Use this space to put a great sentence'

'Your content is liked, but it hooks even more if it is interactive'

Survey of Health Professionals about AI use in SR

Future

Needs

Replace

Minimal

Screening

Extraction

CredibilityReliability (replicability)ConfirmabilityTransferabilityTransparencyThe source of the evidenceEvidence Retrieval

Concerns

Next

Meta-analysis vs Meta-Synthesis Evaluation CriteriaShaheen, N., Shaheen, A., Ramadan, A., Hefnawy, M. T., Ramadan, A., Ibrahim, I. A., … & Flouty, O. (2023). Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. Frontiers in Research Metrics and Analytics, 8.

AI and regulatory criteria for research

AMSTARCASPRoB-2

Next

Criteria

Concern - Response

AConcern -Response

Concern - Response

Dependability, Consistancy

Confirmability/Neutrality

confidence in the 'truth' of the findingsaccuracy and credibility of data in LLM

showing that the findings are consistent and could be repeated

The extent of a researcher bias,

The authority of the source. LLM treated as a source of evidence

Next

Not consistent (black box, algorithm)

Source & Data Bias

LLM or GPT is not a search engine for evidence

Modifying questions for LLMReport prompts

Is AI an author?

Modifying LLM anwers (RAG, Fine-tuning). Report the datasets and prompts.

+info

What does it mean?

Credibility

Measuring RoB with AI

Should we trust the LLM based "findings"?

Learn your LLMs

Next

Next

ArtificialIntelligence

Learn about AI apps: sources for data, LLM, and AI techniques.

Learn about AI techniques (RAG, Fine-Tuning, and so on)

Learn your LLMs

Next

Synthesis Writing

5

Analysis

4

Evidence preparation

3

Evidence Collection

2

Design

1

PROTOCOL

Field Search, Thesaurus Browsing, Syntax and Operators, Filters, Pre-indexed Databases

Natural Language Understanding (NLU). Query meaning, context, and content.

Hybrid orNeural

...

Semantic search

LLM enhances search and creates text with citations. Augmenting LLM with RAG, SLM, Fine-Tuning, and embeddings

Lexical Search

Evidence Retrieval

1

2

Open Alex

Elicit

Consensus

3

Scite.ai

Datasets

AI Tools 4 LR

Semantic Scholar

Keenious

Scite.ai

LitMap

by Dr. Olga Koz

https://bit.ly/LR-AI

Undermind.ai

A bar graph showing the concerns of AI in literature review software AI Viewed as Inevitable, Despite Concerns

Compare

Automation of SR

Source: Jimenez et al., 2022

AI to assist with the systematic review

Source: Jimenez et al, 2022

SR Software with AI

Next

AI

AI for Citation Analysis and literature mapping

https://libguides.kennesaw.edu/litMap

Scoping Review with AI

Next

Guide: AI for discovery & Literature review

Embedded AI

Link

Course: LR for EdD students

CREATING AI RESEARCH ASSISTANT

Next

The article

Mozgai, S., Kaurloto, C., Winn, J., Leeds, A., Heylen, D., Hartholt, A., & Scherer, S. (2023). Machine learning for semi-automated scoping reviews. Intelligent Systems With Applications, 19, 200249. https://doi.org/10.1016/j.iswa.2023.200249

Cierco Jimenez, Ramon & Lee, Teresa & Rosillo, Nicolás & Cordova, Reynalda & Cree, Ian & González Wong, Angel & Indave, Blanca. (2022). Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Medical Research Methodology. 22. 10.1186/s12874-022-01805-4.

Mozgai, S., Kaurloto, C., Winn, J., Leeds, A., Heylen, D., Hartholt, A., & Scherer, S. (2023). Machine learning for semi-automated scoping reviews. Intelligent Systems With Applications, 19, 200249. https://doi.org/10.1016/j.iswa.2023.200249

Reference

Spillias, S., Tuohy, P., Andreotta, M., Annand-Jones, R., Boschetti, F., Cvitanovic, C., … & Trebilco, R. (2023). Human-ai collaboration to identify literature for evidence synthesis.. https://doi.org/10.21203/rs.3.rs-3099291/v1

Jardim, P.S.J., Rose, C.J., Ames, H.M. et al. Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Med Res Methodol 22, 167 (2022). https://doi.org/10.1186/s12874-022-01649-y

Post - presentation survey. I appreciate your answer to 3 questions

Please follow up!

okoz@kennesaw.eduhttps://libguides.kennesaw.edu/AI4LR

5

Your content is likable, but it only engages if it's interactive. Capture your audience's attention with an interactive photo or illustration.

5

Your content is likable, but it only engages if it's interactive. Capture your audience's attention with an interactive photo or illustration.

With this function...You can add additional content to excite the brains of your students: videos, images, links, interactivity...Whatever you want!

1

Your content is appreciated, but it only engages if it's interactive. Capture your audience's attention with an interactive photo or illustration.

Did you know...In Genially you will find more than 1,000 templates ready to showcase your content and 100% customizable, which will help you in your classes?

Did you know that... We retain 42% more information when the content is dynamic? It is perhaps the most effective resource to capture the attention of your students.

Tip:Interactivity is the key to capturing the interest and attention of your students. A genially is interactive because your group explores and interacts with it.

Did you know...In Genially you will find over 1,000 templates ready to input your content and 100% customizable, which will help you in your classes?

3

Your content is liked, but only engages if it is interactive. Capture the attention of your audience with an interactive photograph or illustration.

Did you know...The window allows for adding more extensive content? You can enrich your genially by incorporating PDFs, videos, text... The content of the window will appear when clicking on the interactive element.

Bring it to life with an interactive window. Create a new layer with all the Genially features; include text, images, videos, tables, PDFs... Even interactive questions!

Got an idea?

¡QUIZ!

• Generate experiences with your content.
• It’s got the Wow effect. Very Wow.
• Make sure your audience remembers the message.
• Activate and surprise your audience.

Create a new layer with all the Genially features.

Bring it to life with an interactive window

Got an idea?

With this function...You can add additional content that excites the brains of your students: videos, images, links, interactivity... Whatever you want!