Evalation Heuristic & Testing

Evalation Heuristic & Testing

Levelling Text w/ AI tools

Scenario

Our hypothetical user is:

A high school social studies teacher with a diverse class.

62% of their students are born outside of Canada

31% of students in their class DO NOT speak any English at home

Their textbook is written significantly above the comprehension level of their students

What Can They Do?

How can they assess if the tool is right for them?

Focus

Language Accessibility

Comprehension Support

Bias & Representation

Ethical Use

Environmental Impact

Pedagogical Usefulness

Rationale for Heuristic

Video made using the animate characters function on Adobe Express

Section

ANALYSIS

Khanmigo Ai and Magic School text levellers tested according to the heuristic

Key Takeaways

Procedure

Focus: stuff gets lost

🔍

Accessibility: language vs. clarity

🧠

Bias: erasure through simplification

⚖️

Statistics from coded responses

see all responses

Key Takeaways

Comprehension Support: what helps us understand?

💡

Ethical Use: do we guide responsibly?

🧐

Environmental Impact: are we accounting for the hidden costs?

🌍

Key Takeaways

Pedagogical Usefulness: prep time vs quality?

🧑‍🏫

🏅

⏱️

♻️

Consistency

Time Saving

Reusability

Final Verdict

Khanmigo does better, but a silver bullet text leveller is a mirage

Video made using the animate characters function on Adobe Express

References

Bogaerds‐Hazenberg, S. T. M., Evers‐Vermeul, J., & Bergh, H. (2020). A meta‐analysis on the effects of text structure instruction on reading comprehension in the upper elementary grades. Reading Research Quarterly, 56(3). https://doi.org/10.1002/rrq.311 Bucher, T. (2025). Beyond the hype: Reframing AI through algorithms and culture. Journal of Communication, 75(1), 81–84. https://doi.org/10.1093/joc/jqae048 Coleman, B. (2021). Technology of the surround. Catalyst: Feminism, Theory, Technoscience, 7(2). https://doi.org/10.28968/cftt.v7i2.35973 Common Sense Media. (2024a). Common Sense privacy evaluation for MagicSchool.ai. https://privacy.commonsense.org/evaluation/MagicSchool.ai Common Sense Media. (2024b, August 8). Khanmigo AI product review. https://www.commonsensemedia.org/ai-ratings/khanmigo Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press. Magic School, Inc. (2024, March 12). Student data policy. MagicSchool.ai. https://www.magicschool.ai/privacy-security/student-data-policy Magic School, Inc. (2025, February). Environmental impact statement. Magicschool.ai. https://www.magicschool.ai/privacy-security/sustainability Winterson, J. (2021). 12 bytes : How we got here. where we might go next. Grove Press. World Wildlife Fund UK. (2022). Understanding carbon offsetting and your footprint. WWF. https://www.wwf.org.uk/myfootprint/challenges/understanding-carbon-offsetting-and-your-footprint

Accessibility: language vs. clarity

🧠

• Readability improved for both versions, but both were still many levels higher than requested.
• MagicSchool used significantly less passive voice
• Khanmigo was the easiest in terms of syntax and general readability, but hard to see why things happened.

Takeaway - accessible language shouldn't make critical thinking more difficult.

FOCUS

Key Question

Does the output preserve

• key ideas
• themes
• vocabulary

from the original text

Environmental Impact

Key Question

• Does the tool provide any transparency about its energy use, model size, or sustainability practices?

• Does the tool allow for the export and/or saving of results, to reduce the need to complete this task again in the future?

Focus: stuff gets lost

🔍

• Khanmigo kept 3 out of 8 key vocabulary words
• MagicSchool did better, keeping 6 out of 8
• Core themes like colonialism or systemic injustice were removed from the text, or softened significantly

🚩

Takeaway - simpler text doesn't necessarily have to mean the loss of important themes... but it makes it more likely without mindful consideration

Bias: erasure through simplification

⚖️

• Khanmigo removed all mentions of
• Colonialism
• Resistance
• Agencies enforcing power

• MagicSchool retained more of this
• Khanmigo's passive voice made responsibility unclear.

Takeaway - it was simpler... but at what cost?

Comprehension Support: what helps us understand?

💡

• Khan Academy offers no lists, bold terms, or visuals by default.
• It can be improved through editing in their Khanmigo panel, but only with extra effort.

• MagicSchool lacks these supports entirely, and interactive editing is paywalled.

🚩

Khanmigo added features

Takeaway - Without scaffolds, even simplified text can stay hard to truly understand.

Procedure

aka how I analyzed the text for focus, accessibility and bias

Original Source

1. Read and pre-coded text, marking key ideas and themes important to understanding the concept within its subject area.2. Put the original text through Khan Academy's Leveler + Magic School Text Leveler apps at a Grade Five Level. 3. Ran for each passage through Microsoft Word 4. Coded outputs using the original code list.

readability statistics

BIAS & REPRESENTATION

Key Question

• Does the simplified version preserve fair, respectful, and inclusive portrayals of people, groups, and issues
• Does it avoid erasing important context or reinforcing stereotypes?
• Can the tool be guided to

improve representation when prompted thoughtfully?

• Khanmigo: Saves every output in “My Documents,” so you can reopen, tweak, and export (Word, PDF, Google Drive) without rerunning the model for each class.
• Magic School: One-click export exists, but only after the free trial or a paid upgrade; free users must copy-paste or regenerate from scratch.

🚩

Paywalls and extra clicks push teachers to rerun prompts—or skip thoughtful edits—undercutting the very efficiency these tools promise.

Environmental Impact: are we accounting for the hidden costs?

🌍

• Magic School cites “100 % renewable” Azure / Google Cloud / AWS partnerships and a carbon-negative-by-2030 pledge (Magic School, Inc., 2025).

• Khanmigo offers no published sustainability data, but relies on OpenAI infastructure (Common Sense Media, 2024).
• Magic School allows for Export to Docs/Forms, but only after the paywall - is that enough?
• Khanmigo's “My Documents” lets any teacher download to Word, PDF, Drive, etc.—free tier advantage.

🚩

“AI is neither artificial nor intelligent; it is made from minerals, energy, and water.”

—Kate Crawford, Atlas of AI (2021)

COMPREHENSION SUPPORT

Key Question

Is understanding enhanced through the use of:

• visuals
• formatting
• scaffolds
• other text features

ETHICAL USE: IP & DATA

Key Question

Does the tool:

• State if third-party or copyrighted materials are uploaded
• Give guidance on proper IP use and licensed resources
• Explain if user data is stored, shared, or

used for training

• Provide options to control or delete user

data

LANGUAGE ACCESSIBILITY

Key Question

Is the output appropriate for the level of the learner, including:

• clear syntax
• tiered vocabulary

• ZPD appropriate readability score

• Khanmigo: Keeps conversation history, so tone and examples stay aligned as you iterate—but thematic omissions and reading-level drift still creep in; every version needs an expert eye.
• Magic School: Spits out a polished, one-shot draft tagged “Grade X,” yet never shows the readability formula.

No transparency on leveling metrics means teachers have to double-check accuracy and appropriateness every time.

Ethical Use: do we guide responsibly?

🧐

• Both tools include warnings telling teachers to review content and use professional judgment. Khan Academy’s notice is clear and thorough and significantly more prominent
• Common Sense Media (2024b) rates the non-profit Khan Academy and Khanmigo as low to minimal risk across data use, fairness, and safety. It does not allow OpenAI to train on data and prioritizes transparency and moderation.
• MagicSchool, a for-profit corporation, holds a Privacy Verified Seal (Common Sense Media, 2024a) and commits to strong privacy standards. De-identified data is used for training their algorithms (Magic School, Inc., 2024).
• No blocks on uploading copywrited work exist.

Content Review Warnings from each tool (click to expand)

🚩

Pedagogical Usefulness

Key Question

Does the tool help the teacher meet the needs of students better than manual adaptation? Consider:

• time-taken
• reusability
• consistency

Khanmigo: Keeps the same chat thread open, so you can fire off quick follow-ups—“shorten the intro,” “swap to tier-2 vocab,” “add local examples”—and see tweaks quickly, if you know what to ask for. Magic School: Generates a slick first draft, but free users restart the process (or upgrade to Plus) for any wording changes, adding extra clicks and minutes.

🚩

“Generate & Go” looks instant, yet the first draft still needs teacher review—speed can tempt you to skip that critical editing pass.