3. Search for evidence: AI or not AI

AI for Discovery & Literature Review https://libguides.kennesaw.edu/AI4LR/about

Literature or Evidence Review

Searching for evidence: AI or not AI

Presented by Dr. Koz

https://libguides.kennesaw.edu/lit_search

"Systematized traditional review" Systematic Review

Concept Paper; Research Planning

Phases

•Set up Articles Alerts *Topical journals and conferences

Monitoring

•Plan and structure the process of searching •Maintain consistency in searching •Document the search process and results

Systematic Search

•Develop a topic or research question, •Get familiar with topical concepts and sources

Preliminary Search

Strategies

Phases of Searching for Evidence

Data, information, evidence, types of publications, documents, and level of evidence

Search techniques & engines

Indexes, Databases, Repositories, LLM

What

How

Where

How?

Where?

Citation indexes, collections, repositories, and databases

Planning the Search (What? How? Where?)

Academic Search Engines

Citation Indexes

Citation Analysis

AI for Question & Topic Development

LRD tool

Meta-synthesis

Meta-Analysis

More about QUAN studies

More about meta-synthesis

More about QUAL

Experimental Studies

Quantitative Studies

Qualitative Studies

What? Evidence level by the research design

More about meta-analysis

Read the definition

Hybrid, Neural, Agentic

Lexical search

What? Data

LLM

Knowledge Graphs

Semantic search

Thesaurus browsing /Subject search

Curated data - indexes

Not curated data

Sources & Search Engines

What? Datasets

What? Filters

Some hybrid and semantic search engines allow basic filters. Check Advanced search

Filters based on Inclusion/Exclusion Criteria

Filters - Limit to Research

Search

AI for LR

College of Education Library portal

Subject directory of sources for the College of Education, organized by the typical search strategy:

Type of Search

Keywords

Most of databases and traditional search engines (keywords from a title, abstract)

Thesaurus terms

Indexes based on the listing of terms/concepts relationships, variants and synonyms,

Syntax & Filters

To construct search query and limit search results

Lexical Search

Semantic & Hybrid Search

Semantic

Natural Language Processing (NLP) understand the meaning of a query

Conversational

Asking questions, communicating (prompts) with LLM or Research Assistant

Hybrid

Lexical, semantic and supported by LLM

Hybrid

Semantic

Generative AI

Scite.ai, Consensus, Ellicit

Gemini, Bing, CoPilot

Semantic Scholar, Dimensions, Google Scholar

AI-based Search Engines

Summary

Filters

Scite.ai

Natural Language

Pearl Growing

Semantic

Boolean

Citation Chaining

Thesaurus browsing

Keywords

https://libguides.kennesaw.edu/search_tech

How?Search Techniques

List of databases with a thesaurus and tutorials how to do this search by a database

https://libguides.kennesaw.edu/search_tech/concept

ThesaurusBrowsing

Related Theories & Methods

Pearl- Growing

Source or Database

Journals

Concepts & Subjects

Pearl (A relevant article)

Pearl (An article)

References & Citations

Search for similar documents based on the ”pearl” metadata. From the initial pearl or a seed (article), grow (collect) other pearls.

Research Rabbit - pearl growing or a seed paper

Citation Chaining

Click in Cited by to see all works that cited the original publication

Learn more about citation chaining https://libguides.kennesaw.edu/search_tech/chain

Indexes such as Web of Science, Scopus, or a search engine like Google Scholar allow you to see the number of citations and find who cited the publication to find seminal works and the newest research.

Citation Chaining using AI tools

*Recall – a measure of the quantity of relevant results a search returns. *Precision – a measure of the quality of relevant results *Level of evidence - the relevant value of the evidence *The source of the evidence is important!

Prompt engineering might improve information or knowledge retrieval from raw LLM but not evidence retrieval.

Raw LLMs or GPT are not intended to search for "evidence." However, hybrid search engines, which capitalize on NLP and ML, offer different experiences in evidence retrieval.

LLM & GPT for evidence search: myths

Comparison of AI-based academic search tools

When to stop searching?

Protocol

Systematic Search

• Not an exhaustive review
• Selective, representative

When to Stop to Search?