DLI Project

Begin

DLI Professional Diploma in Digital Learning Design

Samantha Markham

Assignment: Identifying Bias in Academic Research

06/10/18

All images used in this project were sourced from Pexels via Genially unless otherwise credited.

UX & UDL

Evaluation Plan

Analytics

REFLECTIOn

Engagement

launch PLAN

Platform

StAFFING

REFLECTION

PROTOTYPE

STORYBOARD

SCREENPLAN

WiREFRAME

SCOPE

Reflection

Part 3

Part 2

Part 1

How?

What?

Why?

Who?

Analysis & LX Design

01

The 'Who' of Learning

Learner Personas

David Lee

Aisha Khatri

Maria Mootry

• Likes short Youtube videos or TikToks, and infographics.
• Needs a flexible timetable and access to course material online.

Learning Preferences

• Has an Android tablet and mobile phone.
• Access to computers in the university library.
• Internet connection in her student house can get crowded if everyone is using it.

Main goals and drivers:

• Wants to learn practical strategies for identifying bias in literature quickly and efficiently, so she can complete her assignments on time.
• Needs clear deadlines and time-saving resources specific to literary analysis

Barriers to learning:

• With a lot on her plate, she might find it difficult to stay focused.
• Could struggle with the fact that literary analysis can be quite a lengthy process.

Perception of digital learning:

• She finds it appealing due to its flexibility and accessibility.

Technology

Goals & Aspirations

Name: Maria MootryShort bio: First in her family to attend university. She's juggling multiple responsibilities: work, family commitments, and a heavy course load. Has limited time for research. Who is it?

• Age: 18
• Studying: Literature
• Location: Living off campus with other students.

Personal Data

• Favours clear step-by-step explanations and practical examples.
• Likes small group settings, especially with international peers, where she can ask questions and receive personalised feedback.

Learning Preferences

• Owns a new laptop with Windows operating system.
• Access to high-speed internet on campus.

Main goals and drivers:

• Wants to avoid plagiarism, and complete assignments to a high standard.
• Craves clear instruction and reassurance that her understanding of research and bias aligns with the expectations of her course.

Barriers to learning:

• Cultural differences in communication style might make her hesitant to ask questions.
• Noisy environment in halls, so will need to do self-paced work in the library.

Perception of digital learning:

• Cautious about the potential for cultural differences in online learning and the lack of hands-on guidance.

Technology

Goals & Aspirations

Name: Aisha KhatriShort bio: First-year international student. Excited about her studies, but overwhelmed by the university environment and adapting to a new culture. Has basic research skills, but struggles with academic language and referencing styles in the UK. Who is it?

• Age: 18
• Studying: Biology
• Location: Halls of residence.

Personal Data

• Enjoys interactive learning activities.
• Would like to have the opportunity for independent research on specific topics related to historical bias.

Learning Preferences

• Tech-savvy, he owns a brand new laptop, and a tablet for on-the-go learning.

• Utilises various productivity tools and extensions to streamline workflow.

Main goals and drivers:

• Wants to learn advanced techniques and become a more critical consumer of historical information.
• Interested in explaining how historical context influences bias and how it shapes our understanding of the past.

Barriers to learning:

• May find basic explanations repetitive.
• Might be sceptical of authority figures and challenge the course content or methodologies.

Perceptions of digital learning:

• Positive about the independence of self-paced digital learning.

Technology

Goals & Aspirations

Name: David LeeShort bio: A confident, independent learner who is very comfortable with technology. He enjoys questioning information and wants to develop a deeper understanding of how bias manifests in historical narratives. Who is it?

• Age: 19
• Studying: History
• Location: Lives with family, local to the university.

Personal Data

After analysing the learner personas, some key considerations are:

• Predominantly young, tech-savvy cohort.
• Course flexibility is crucial to cater to different timetables, workloads and non-university commitments.
• Choice over how to consume the information is also important to appeal to different learning preferences.
• Needs will differ depending on the student’s degree course and type of research involved.
• Some learners will need and/or want additional support, while others will favour exclusively self-directed learning.

Observations

The 'Why' of Learning

The Problem & The Need

The Need

The Problem

Bloom's Taxonomy (Image Created by the Author)

Considering the broad spectrum of students who would benefit from the course, and the diverse subjects they're studying, the learning gap varies. Some students lack even lower-order thinking skills (understanding and applying) regarding bias in academic research. However, for others, the gap lies in higher-order thinking skills, such as analysing and evaluating. The gap sits between understanding (knowing what bias is) and evaluating (judging the overall impact of bias in research). The aim of the course will be for all students to be at least capable of successful analysis-level thinking about bias in academic research texts.

The Learning Gap

5. Develop a research plan that minimises bias in the learners' own work. CREATE

4. Critically evaluate the credibility of sources used in academic research. EVALUATE

3. Effectively communicate their findings while acknowledging potential limitations due to bias. APPLY

2. Analyse research articles for potential bias and its impact on conclusions. ANALYSE

1. Identify and explain different types of bias in academic research.UNDERSTAND

Learning Outcomes

MODULE 1: Introduction to Bias in Research

LOs 3 & 5

LO: 2

LOs: 2 & 4

LO: 1

MODULE 4: Mitigiating Bias in Your Own Work

MODULE 3: Evaluating Source Credibility

MODULE 2: Analysing Research Methods

The 'What' of Learning

Course Structure

• Lesson 1: Bias-Free Designing Strategies for designing research questions and hypotheses to minimise bias.
• Lesson 2: Diverse Methods Importance of selecting a representative sample and diverse research methods.
• Lesson 3: Blinding Techniques Techniques for blind analysis.
• Lesson 4: Know Your Limitations Methods for acknowledging limitations and potential biases in your research.

• Lesson 1: The Source Landscape Types of academic sources - their strengths and weaknesses.
• Lesson 2: Credibility Evaluating the credibility of online sources.
• Lesson 3: Sources to Avoid Identifying and avoiding predatory journals and misleading websites.
• Lesson 4: Diversity and Perspective Importance of source diversity and considering alternative perspectives.

• Lesson 1: Reading Critically Techniques for reading (SQ3R method, active reading strategies).
• Lesson 2: Bias in Questions and Methods Identifying bias in questions, hypotheses, methodology, etc.
• Lesson 3: The Power of Words Recognising biased language and framing.
• Lesson 4: The Author The importance of considering author background and potential conflicts of interest.

• Lesson 1: What is Bias? Definition of bias and its different types.
• Lesson 2: How Bias Shapes Research How bias can influence research design, methodology, and interpretation.
• Lesson 3: Impact on Research Conclusions How bias affects research conclusions.
• Lesson 4: Real-World Consequences Examples of bias in famous historical studies.

Four: Mitigiating Bias in Your Own Work

Three: Evaluating Source Credibility

Two: Analysing Research Methods

One: Introduction to Bias in Research

Modular Framework

Optional Extras

Optional Extras

Optional Extras

Optional Extras

Lesson 4

Lesson 4

Lesson 4

Lesson 4

Lesson 3

Lesson 3

Lesson 2

Lesson 3

Lesson 3

Lesson 2

Lesson 2

Lesson 2

Lesson 1

Lesson 1

Lesson 1

Lesson 1

Module 4

Module 3

Module 2

Module 1

Course Content Map

Additional optional elements

Module Content Map

Knowledge check

Knowledge check

Knowledge check

Knowledge check

Workshop

Intro

Lesson 1

Lesson 2

Lesson 3

Lesson 4

Intro

Intro

Intro

Intro

Content

Activity

Content

Content

Activity

Activity

Activity

Content

Resources

Forum

Synchronous

Asynchronous

Modes of Delivery

Live Workshops

Interactive Forum

Downloadable Toolkit

Online Content

Formative assessment

• Quizzes
• Peer Assessment
• Checklists
• Skills assessment

Summative assessment

• Research analysis assignment

Feedback

• Automated
• Peer
• Tutor (workshop only)

• Knowledge check MCQs
• Case study - CCAF
• Peer discussion
• Self-reflection

• Virtual classroom (workshop only)
• LMS
• Collaboration tools

Facilitation Technique

Assessment & Feedback

Building Blocks

Activites

• Multimedia eLearning
• Instructional videos
• Interactive presentations
• Infographics
• Reading materials
• PDF summaries

Content

Forum

Toolkit

Workshop

Lesson materials

Module 4

Module 3

Module 2

Module 1

The core course content will follow a linear pathway - dealing with progressively more challenging knowledge and skills acquisition.Each module, meanwhile, offers a core and spoke approach - providing the main digital learning material alongside optional additional resources and activities.

Learning Flow

Module

Core Content

This digital course on identifying bias in academic research is a concept piece, designed for a fictional university. However, it is influenced by my experiences working in tertiary education. Throughout the project, I use “student” to refer to someone attending the university and “learner” to refer to a participant of the course. Structuring the course began with building the learner personas to help identify the kinds of first-year students who might benefit from the material. These personas were based on my observations working as a university lecturer and helped me to 'gain insights into what [the] learners truly need and how [to] meet their expectations.' (Ordorica, 2023) Considering the different needs, motivations and preferences of the learner personas, there are some key takeaways mentioned in the 'Observations' slide of this presentation. The next step was to identify the learning gap. As with the personas, the learning gap was based on my real-life observations and experiences. First-year university students often struggled when it came to choosing reliable sources for their research, and found it difficult to read between the lines of those sources to spot bias.

Reflection

Utilising Cathy Moore’s “Will Training Help?” flowchart, it seemed clear that the problem in the vast majority of cases was a lack of knowledge and skills, rather than student motivation or any issues related to the university environment. The learning outcomes were written to address these knowledge and skills gaps. Bloom’s taxonomy was helpful in determining the LOTS (lower order thinking skills) that all learners would need to master, and the HOTS (higher order thinking skills) that some learners might want to only get acquainted with, while others would want to practice and develop further. Action verbs were used to ensure that the learning outcomes were clear, specific and observable. As Julie Dirksen points out, ‘Ultimately, since we care not about what they know but about what they can do, the learning objectives should reflect that.’ (Dirksen, 2016). As mentioned in the presentation, the SMART approach was also adopted for the learning outcomes. The course design takes a top-down approach, taking into account the different levels of prior knowledge and aiming to make the course as efficient as possible, so learners can immediately put any newfound knowledge to use in their assignments. As the course develops into the latter stages, micro-level understanding will help learners to deepen their understanding and focus on the details within their specific degree subjects. The modular framework was developed with this in mind.

The structure is intended to be learner-centric; keeping the format simple and the content relevant. This is intended to ensure the course is efficient and useful, and that learners are engaged throughout. The course has been divided into four modules: 1. A macro view of bias; how it impacts research and conclusions, and real world examples of bias in studies. 2. Beginning the analysis process, encouraging learners to ask questions about their academic research materials. 3. Discussing credibility, giving learners a deeper understanding of how to identify credible sources of information. 4. A deeper look into how learners can mitigate bias in their academic research and their own work. This framework offers a comprehensive yet succinct approach, addressing the foundational knowledge and specific needs of each learner persona. By including dedicated modules for analysing research methods, source credibility, and mitigating bias, the course equips learners with a well-rounded skillset for critical evaluation of academic research.The choice over mode of delivery was relatively straightforward. The learners are a predominantly young, digital-native cohort and, with their differing timetables, it seemed logical to offer the bulk of the material as self-paced, digital content. However, it was also important to acknowledge the importance of social learning and the preferences of learners who favour more support. The inclusion of an online peer forum and tutor-led workshops would fill this gap.

Selecting a variety of content and activities was important to ensure that learners are offered multiple means of engagement, action and expression, and representation - following universal design for learning guidelines. Often, during the modules, it would be possible for learners to acquire the same knowledge as their peer by taking a different approach. The autonomy of the learner is key to enhancing their motivation and encouraging independent thinking. With this in mind, the modules needed to take a core and spoke approach. The core consisting of the asynchronous digital content, while the spokes would offer the online forum, the additional optional resources, and the workshop. Meanwhile, the course as a whole would adopt a linear flow to ensure that the content is scaffolded. Increasing the level of difficulty gradually as leaners move through the modules would ensure that even those with little to no prior knowledge are confident in their abilities by the time they reach the more challenging tasks.

CAST (n.d). About Universal Design for Learning. CAST.org. https://www.cast.org/impact/universal-design-for-learning-udl Dirksen, J. (2016). Design for How People Learn (Kindle edition). New Riders; Pearson Education. Digital Learning Institute. (2020–2023). Persona Template [PDF]. Copyright by Digital Learning Institute. Moore, C. (2017). Will Training Help? [PDF]. Retrieved June 28, 2024, from https://s3.amazonaws.com/cathymooremedia/will-training-help-A4.pdf Ordorico, A. (2023). Bring Your Learners to Life Through Personas. The Times Higher Education. Retrieved July 1, 2024, from https://www.timeshighereducation.com/campus/bring-your-learners-life-through-personas

References

Micro Design & Development

02

For Module 2:

• Analyse research articles for potential bias and its impact on conclusions.

• Critically evaluate the credibility of sources used in academic research.

First-year university students who wish to improve their knowledge of bias in academic research. Typically young adults who are transitioning from secondary to higher education. They are likely to be:

• New to academic research
• Developing critical thinking skills
• From diverse backgrounds
• Enthusiastic about exploring their fields
• Open to new ideas

Learning Outcomes

Target Audience

Why?

Who?

Scope

The Author

Bias in Questions and Methods

The Power of Words

Reading Critically

'Analysing Research Methods': Lessons

Module 2: Sections & Topics

Quiz

Critique

Case Study

The following are optional, so learners can elect to do whichever actitivity/activities best suit their learning preferences.

End of Module Activites

Adapted from the Digital Learning Institute's Multimedia eLearning Course Wireframe Template.

Wireframe

Adapted from the Digital Learning Institute's Sample Screenplan.

Screenplan

These are the storyboard pages for the slides that were prototyped. Click on each for a full screen view. The rest of the storyboard can be found on the following page.

Storyboard

These storyboard pages do not have any image mockups yet, as they are not part of the prototype.

Storyboard

The micro design and development phase of the project posed many challenges, but it was also very rewarding. The first hurdle was to determine which part of the course to focus on for the prototype. In the end, I opted for a multimedia portion of the course because, as mentioned in the DLI’s "Welcome to Module Four", it remains the most widely-used format for online learning (Digital Learning Institute, 2024). In addition, it would allow me to create a range of design documents, and provide good practice in working with an authoring tool – in this case, Articulate Storyline. The micro design process began with creating a scope for the second module of my course. This involved revisiting the learning personas and learning outcomes, then determining what topics needed to be covered in order to meet those learning outcomes. One of the main challenges here was to find a balance between providing enough information without overloading the learner. One of the most effective ways to overcome this was to ask myself a few questions, such as, 'Does this content align with the course objectives?' (SHIFT eLearning, n.d). Nevertheless, there were several iterations of the "Sections and Topics" list. Wireframing was the next task and, as I had never completed one before, this was a slightly intimidating prospect. However, being able to use the template provided by DLI simplified the process immensely.

Reflection

Similarly, the DLI’s template for a screenplan was invaluable to me as a newcomer to crafting design documents. It was, nonetheless, quite a time-consuming task. Moreover, I found that I needed to revise some of the screenplans once I had got into the storyboarding phase after realising my initial drafts could be improved upon. Storyboarding was by far the most challenging aspect of the micro design. I worked with a predominantly text-based approach, accompanied by some simplistic monochrome images/screen outlines. It seemed logical to focus on the core content and, as I would also function as the developer on the project, there was no necessity to include a wealth of developer notes regarding colours, fonts, etc. However, if I were to do it again, I might try a rapid prototype approach. Although a storyboard is often compared to the 'blueprint for a house', (The eLearning Designer's Academy by Tim Slade, 2021) with a text-based storyboard, it was difficult for me to visualise what the screens would look like. And when it came to developing the course, I found that I sometimes strayed quite far from what I’d originally planned on the storyboard. That said, I recognise that these iterations are part of the process. Additionally, as I was building a prototype, rather than a final version, deviation from the storyboard was not hugely problematic. Building the prototype was a particularly satisfying part of the process, although it was not easy. I enjoyed the creative nature of this part of the work, and it was immensely gratifying to see the course begin to come together.

Reflection

Graphic and visual design are not areas I have any expertise in, so I relied on colour theory to choose my palette, and opted for a primary colour of dark blue 'for its ability to convey a sense of professionalism and reliability' (Color Psychology, 2024). I also referred back to CRAP principles often to try to ensure that the content would be visually appealing. Upon reflection, I think there are ways the visual aspects of the course could be enhanced. For example:

• Some slides are too text-dense.
• Generally, more white space would help reduce the possibility of cognitive overload.
• In a bid to stay consistent with colour, I may have overused the primary colour — leading to the whole visual experience being a little too dark.

Graphic and visual design are certainly areas I can and will improve upon as I gain more experience.Other changes I would make to the prototype include:

• Using real voices, although I was impressed by how natural the principal AI voice that I used sounded.
• If I were to build the whole course, I would also be sure to use a range of voices — different genders and accents to ensure better representation.
• Including tailored rather than generic feedback for the assessment.

Reflection

Color Psychology. (2024, April 16). Navy Blue Color Psychology and Meaning. Retrieved September 28, 2024, from https://www.colorpsychology.org/navy-blue/ Digital Learning Institute (2024). Welcome to Module 4 [video]. Module 4: Multimedia eLearning Design Digital Learning Institute. (2020–2023). Multimedia eLearning Course Wireframe Template [PDF]. Copyright by Digital Learning Institute. Digital Learning Institute. (2020–2023). Sample Screenplan [PDF]. Copyright by Digital Learning Institute. SHIFT eLearning. (n.d.). Avoid learner overload: Five rules for eLearning course design. Retrieved October 1, 2024, from https://www.techstructional.com/content-overload-elearning/ The eLearning Designer's Academy by Tim Slade. (2021, August 3). Storyboarding vs. rapid prototyping for eLearning [Video]. YouTube. https://www.youtube.com/watch?v=qIB1T9NSstI

References

Implementation & Evaluation

03

Advantages

Technological

Information sourced from Blackboard Learn. Retrieved Sept 17, 2024 https://www.blackboard.com

Pedagogical

Learner Needs

Learning Platform

Staffing

Rollout Plan (Adapted from DLI Rollout Plan)

The university's LMS admin team can incoporate this project into their current workload.

Tech Admin

The university has a team of moderators and community managers. For this project, the team can provide one community manager and one moderator.

Moderation

The University has an established learning design team. For this project, the team can provide two instructional designers and a UX designer. The role of tutor for the workshops can be filled by a member of the 'Writing Centre', 'Academic Support Staff', a librarian or research specialist.

Content Management

Going Live

Testing Plan

Content and UX

This 'Going Live' plan provides a comprehensive overview of the strategies and timeline for the successful launch and delivery of the course.

Launch Plan

Accessibility, universal design and UXDL are all crucial for any digital learning experience. However, examining the learner personas, there are some particular areas of accessibility and design that should be kept in mind for this course: Accessibility Multimedia: Provide downloadable audio versions of readings, and captions for videos to accommodate different learning preferences (e.g., Maria may benefit from listening to materials while commuting).Mobile Optimisation: Ensure all course materials, including quizzes and readings, are mobile-friendly. This supports Maria and David, who rely on mobile devices for on-the-go learning. Content Formats: Providing diverse content formats caters to different learning styles and preferences. Assessment Formats: Offering a variety of assessment options allows learners to demonstrate their knowledge in ways that suit them best. UXDL Intuitive Navigation: Create a clear, intuitive course layout using consistent labeling and chunked modules so Aisha, who may find new platforms overwhelming, can navigate confidently. Engaging Content: Using interactive elements, multimedia, and gamification can enhance the learning experience and keep learners motivated. Personalised Learning: Offering opportunities for learners to customise their learning paths and pace to cater for individual preferences and needs.

Learner Engagement

Pathways & Mobile

Nudging

Gamification

Marketing

Engagement Tactics

By combining all three approaches, we can create a comprehensive evaluation framework that ensures the course is achieving its goals while continuously improving delivery and identifying long-term impacts.

Evaluation Types

Outcomes-Based

Process- Based

Goal- Based

By combining these models, we ensure a comprehensive evaluation of the course:

• Kirkpatrick’s model focuses on outcomes and answers: Did the course work?
• CIPP evaluates design and delivery and answers: How well was the course planned and implemented?
• Brinkerhoff uncovers qualitative insights and answers: What worked, what didn’t, and why?

For this course, after researching various evaluation models, I focused on comparing the benefits of three:

• Kirkpatrick's
• CIPP
• Brinkerhoff's Success Case

We could consider using a blended approach to evaluation — Using Kirkpatrick's model with elements of CIPP and Brinkerhoff's. Kirkpatrick’s model is excellent for tracking and assessing how well the course achieves its intended goals, particularly in terms of student learning and application of skills. For a course on identifying bias in academic research, this model provides a clear, step-by-step framework to evaluate its impact. The CIPP model complements Kirkpatrick's by providing a broader, systemic evaluation of the course’s design, implementation, and overall alignment with learner needs. Brinkerhoff’s Success Case Method provides a way to capture the human stories behind the data. These stories can highlight successes and challenges in a way that resonates with our stakeholders, and informs actionable changes.

Evaluation Model

This evaluation plan uses Kirkpatrick's Model with the 'Context' and 'Input' elements of CIPP and Brinkerhoff's Success Case. The plan is adapted from the DLI's 'Programme Evaluation Planning Template' (2020-2023)

Evaluation Plan

Created by the author using Canva

Image created by the author using PowerPoint

The Honeycomb Model of UX Evaluation, proposed by Peter Morville, is a framework that visually represents the key facets of a high-quality user experience. These attributes guide designers in creating and evaluating experiences that are both functional and user-centred. Benefits of Using the Honeycomb Model to Evaluate This Course

• Holistic Evaluation: The focus is not just on learning outcomes but also on learners’ overall experience, from engagement to accessibility.
• Iterative Improvement: Regularly evaluating the course using this model allows for refinement of aspects like usability or desirability over time.
• Aligning Stakeholders: The model aids in addressing both learner needs and institutional goals by balancing value and credibility with accessibility and usability.

UX Evaluation

To assess the design of the course, the DLI's UCDL Honeycomb Model Checklist (2022) will be used to identify whether the course platform, content, and overall learning experience meet learners' needs.

Building a Community

Communication & Connection

Reflective Design

UX Evaluation Checklist

Behavioural Design

Elements

Images

Text

Universal Design for Learning focuses on creating inclusive learning environments that meet the diverse needs of all learners. By applying the three UDL principles — Engagement, Representation, and Action & Expression — to the evaluation process, we can assess whether the course provides equal access to learning and supports all learners effectively.

UDL Evaluation

Action & Expression

Representation

Engagement

UDL Checklist

This UDL Checklist is taken from the DLI's UDL Checklist for Digital Learning Evaluation (2022)

Executive functions

Expression and communications

Physical action

Comprehension

Language and other expressions

Perception

Provide options for self-regulation

Provide options for sustaining effort and persistence

Provide options for sustaining interest

Action & Expression

Representation

Engagement

Title here

Image created by the author using PowerPoint

Learner Analytics

I had gone into the final phase of the project believing that the implementation and evaluation steps would, in many ways, be the most straightforward. I was, however, quickly disabused of that assumption. While studying the course materials for modules 10 and 11, I felt I had a clear visualisation of the ADDIE method, and found that it is not (and should never be viewed as) a linear process in which you complete one phase before moving to the next. For example, the implementation can be quite complex, it requires a great deal of planning that might be in progress even in the early stages of the design and development of the course. The evaluation phase, meanwhile, is an ever-constant feature throughout each step; ensuring that the learning designer is always assessing and refining their work to ensure the best quality learning experience. It is also important to remember that, at every step, the learning designer should be reflecting on the information gathered during the assessment phase and returning to refine the design consistently. Implementation Because my course for this project is being designed for a fictional university, it made sense to imagine that the institution already has an LMS and/or VLE.

Reflection

After a Google search, I discovered that the most popular VLEs at UK universities are: Blackboard, Moodle, Canvas and Brightspace. I chose Blackboard for the VLE of choice for the project’s fictional university for a number of reasons: Firstly, it seemed to meet the 8 key features of an LMS highlighted by Brandt (2022) on his website. Secondly, it is, according to Google, the most common VLE for UK universities, so it felt as though I was creating a realistic scenario. And thirdly, as I have worked with Moodle in the past, I decided to utilise the opportunity to learn more about another system. By assuming the institution already has a VLE in place, I was able to simplify some aspects of the rollout plan (as noted in the presentation), and imagine that areas of the launch plan would also be streamlined. The fictional university would also, of course, already have trained members of staff in place – given that many educational institutions are embracing digital learning. There is no question that being able to imagine the situation made it easier for me to formulate an implementation plan. Nevertheless, it was clear that there are many more considerations than I had initially believed. I, therefore, relied heavily on the DLI’s Module 10 Toolkit to determine the roles, the tasks that need to be completed, and checklists of considerations. It was also interesting to note that, during this phase, the learning designer needs to still examine accessibility and UX. As mentioned above, the ADDIE method can be seen as cyclical in nature.

Reflection

Evaluation 'Even though it is the final letter in the ADDIE acronym, most of the evaluation phase does not come last.' (Gardner, 2011). Before embarking on this course, I had thought that the evaluation step of ADDIE only really focused on the bottom line – assessment results or KPIs. It has been invaluable to discover that it is an essential facet of each and every step of the method. Constant and mindful evaluation can help ensure that the learning experience is meeting its aims and fulfilling its raison d'être. In many ways, I regret not appreciating this fact sooner. Had I been armed with the knowledge I gained from Module 11 of the DLI’s course at the beginning of the project, I believe I would have been able to create a better learning experience. When considering the evaluation for my project, I first looked at the various approaches and methods. As mentioned in the presentation, I saw value in looking at all three approaches: goal-based, process-based and outcomes-based. I was already somewhat familiar with the Kirkpatrick model, so I was keen to research a little deeper and could see the potential benefits of adding some elements of CIPP and Brinkerhoff’s Success Case.

Reflection

It is possible that CIPP alone could have met the needs of evaluating this particular course, but I favoured the notion of utilising Kirkpatrick as a foundation and weaving some additional elements. I was mindful, though, of the possibility of needless repetition in combining models. And, subsequently, I believe I have managed to avoid this in my evaluation plan. It was fascinating to delve into UX and UCDL, as these are areas that I am less familiar with. They are, therefore, potential areas of growth in terms of knowledge and skills. When it came to planning the UX and UD evaluations, though, I depended upon the DLI checklists to guide my process. Noting that there is also a UDL checklist to the evaluation phase has given me a greater understanding of how fundamental UDL should be to every single phase of digital learning design. I intend to use it as a touchstone for all my future learning design, and I will consistently ask: 'Could [we] offer more flexibility in the way [we] present content, the way students engage in learning in [the] course, and the way they are assessed?' (Center for Teaching Innovation, n.d.) Finally, learner analytics was another area about which I knew relatively little. As with much of this final section of the project, I was reliant on the DLI materials to give me a framework for analytics. The source, data, monitor, insights, action format provided me with a clear roadmap.

Reflection

From there, I was able to imagine, within the scenario of my university course, where I might retrieve data from, what that data would be, how it could be monitored, what insights could be gleaned from it and, ultimately, what actions should be taken as a result of those insights. This exercise, although somewhat hypothetical in nature, was hugely beneficial in getting to grips with learner analytics. And I now feel confident that I have a working knowledge of how to handle analytics in a real-life situation.

Reflection

Brandt, D. (2022, July 30). 8 Key Features of any Learning Management System (LMS). LMSchef. Retrieved October 12, 2024, from https://lmschef.com/key-lms-features/ Center for Teaching Innovation. (n.d.). Universal Design for Learning. Retrieved October 12, 2024, from Center for Teaching Innovation website: https://teaching.cornell.edu/teaching-resources/universal-design-learning Digital Learning Institute (2024). Module 10 Toolkit. Module 10: Rollout and Universal Design Digital Learning Institute (2020-2023). Programme Evaluation Planning Template [PDF]: Copyright by Digital Learning Institute. Digital Learning Institute (2022). UCDL Honeycomb Model Checklist [PDF]: Copyright by Digital Learning Institute. Digital Learning Institute (2022). UDL Checklist for Digital Learning Evaluation [PDF]: Copyright by Digital Learning Institute. J. Clark Gardner. (2011, October 18). The ADDIE Evaluation Phase [Video]. YouTube. https://www.youtube.com/watch?v=CBoI0wBo4vw James Pann. (2021, October 14). The CIPP Evaluation Model with Guili Zhang [Video]. YouTube. https://www.youtube.com/watch?v=OsKwL7_M_Eo

References

• Are the icons meaningful and do they add to the learning?
• Are the elements consistent in their use?
• Are the elements in alignment?
• Is there consistency in the design?
• Have you used brand guidelines effectively?

Elements

A personalised, rather than predefined, learning path is the best option for this course, as learners are coming to the material with a range of prior knowledge and different learning goals. A personalised learning path will offer participants:

• A tailored learning experience
• Flexibility
• Enhanced learning outcomes

It's also important to take into account mLearning needs, because students are highly likely to access the material from mobile devices. The benefits of catering to this need include:

• Flexible Scheduling - Many first-year students have busy schedules, and mobile learning allows them to fit learning into their lives.
• Real-World Application - Mobile devices can be used to access real-world news articles and research papers, allowing students to apply their knowledge to real-world situations.

There will be four modules and each module will have four lessons. The course is designed to be self-paced, so learners are permitted to take as long as they need to complete. However, the optional workshops will be conducted once a fortnight. So, learners wishing to participate in these sessions should aim to complete a whole module's material every two weeks in order to get the most value from the workshops. If learners are able to work through a module per fortnight, the course should take eight weeks to complete.

Student satisfaction surveys demonstrate an increasing number of students find expectations surrounding research for their first-year assignments confusing. Lecturers have reported issues surrounding first-year students' research, specifically finding credible sources and knowing how to assess research critically. Subsquently, the teaching staff has felt pressure to devote lecture time to explaining these core concepts.

Frustrated students and pressure on lecturers

• Are the images used to support learning?
• Are there any redundant images?
• Have you ensured the images are not offensive?Do the images help in the learning?
• Are you cropping any images?

Images

This section will delve into the importance of considering the author's perspective and background when evaluating research. Students will learn to assess author credibility, identify potential conflicts of interest, and understand how authorial biases can impact research findings.

Blackboard Learn's status as the university’s established VLE ensures a stable, well-supported digital infrastructure.

• Device Compatibility: Mobile-friendly design and accessibility across laptops, tablets, and phones.
• Customisable Tools: The VLE supports tech-savvy learners by integrating extensions, productivity tools, and analytics that enhance the learning experience.
• User Support: University IT services provide ongoing support for both students and tutors, ensuring a consistent and smooth user experience.

• Blackboard
• xAPI-enabled activities
• LRS
• Course forums
• Quizzes and knowledge checks
• Digital surveys (integrated in Blackboard)

Source

Raw data will be sourced from the following:

This section will focus on the role of language in shaping research and public discourse. Students will learn to critically analyse how word choice can influence meaning, and how to identify potential bias in the use of language.

Additional considerations include: Accessibility - Ensure all platforms and materials are accessible for those with sensory disabilities or learning differences. Technical Support - Provide clear instructions and offer technical support for all digital platforms and tools.

Blackboard caters to a diverse range of learners by providing flexible tools that support independent, tech-savvy students like David, while also accommodating those juggling heavy workloads, such as Maria, or navigating cultural and academic transitions, like Aisha.

• For Maria: Tools like clear deadlines, accessible learning resources, and mobile-friendly content.
• For Aisha: Blackboard's structured modules, self-paced options, and feedback features ensure she feels confident in meeting academic standards.
• For David: Advanced discussion boards, independent research capabilities, and critical analysis tools empower him to challenge ideas and explore historical bias in depth.

The platform's flexibility, accessibility, and personalised learning features ensure that all learners are supported on their academic journey.

Goal-based evaluation measures whether the course achieves its stated learning outcomes. For this course, it would involve assessing each of the five learning objectives:

1. Identify and explain different types of bias in academic research: Use quizzes or written assignments to test students' ability to define and differentiate types of bias.
2. Analyse research articles for potential bias and its impact on conclusions: Evaluate leaners' ability to identify specific biases and articulate their impact on the article’s findings.
3. Effectively communicate findings while acknowledging limitations due to bias: Review written or verbal presentations where learners highlight bias in research and describe limitations clearly and professionally.
4. Critically evaluate the credibility of sources used in academic research: Use exercises requiring students to assess sources for credibility, reliability, and potential biases.
5. Develop a research plan that minimises bias in learners’ own work: Assess student-created research plans for strategies addressing bias prevention.

Strengths of Goal-Based Evaluation: Clear alignment between learning objectives and assessment tasks. Provides concrete evidence of whether students achieved the intended outcomes.

Blackboard Learn supports diverse teaching approaches to accommodate various learner needs and aspirations.

• Personalised Learning Pathways: Features like adaptive release ensure Maria and Aisha can access content at their own pace, helping them focus on what they need most.
• Engagement and Critical Thinking: Interactive tools such as discussion boards, peer reviews, and collaborative projects engage learners like David, encouraging them to question and analyse content critically.
• Multimodal Resources: Tutors can incorporate videos, infographics, and practical examples to suit learners like Aisha who prefer step-by-step guidance and Maria who benefits from time-saving resources.
• Supportive Feedback Mechanisms: Inline grading and private feedback options provide the clarity Aisha needs to feel reassured, while open-ended assignments and project-based tasks offer David the chance to explore advanced topics independently.

A blended approach has been chosen for the course; the bulk is asynchronous, catering to different learning styles and the needs of independent learners (like David), and learners with busy schedules (like Maria). The inclusion of synchoronous workshops offer greater support and scaffolding for learners who favour a more tutor-led approach and opportunities for social learning (like Aisha).

• Have you checked if all Learning Outcomes are addressed in the design?
• Are all Learning Outcomes assessed formatively or summative?
• Have you designed macro (holistic) and micro–Learning Outcomes for your learners
• Are learners aware of Learning Outcomes?

Reflective Design

• Are your learners aware of all communication channels with instructors?
• Are your learners aware of all communication channels with other learners?
• Have you used discussion forums and FAQs for generic queries?
• Have you set up a space for sharing opinions and ideas within the programme?

Communication & Connections

• Did you design and develop a learning experience that is easy to navigate?
• Is it easy to access all materials?

Behavioural Design

This section will introduce students to the concept of critical reading, providing them with essential skills and strategies to actively engage with texts. By the end of the lesson, students will be able to identify key arguments, evaluate evidence, and recognize potential biases in written materials.

• Have you monitored the learning patterns and how the community evolves?
• Have you been fostering communities online?
• Has there been moderation of online discussions?
• Have you provided space for your learners to develop connections? For example: introductions?
• Have you been checking in with students who are falling behind in the course?
• Have you been acting on queries, feedback, on the course/programme?

Building a Community

1. Present a complex research case study.
2. Learners analyse the case for potential biases in research questions, methodology, data analysis, and interpretation.
3. They should propose strategies to address identified biases.

Learners can post their responses on the forum for peer feedback.

The university wants to help equip students with the tools and strategies necessary to assess the credibility of academic sources. This includes examining:

• Author credentials.
• Publication sources.
• Looking for evidence of balanced perspectives.

Evaluate source credibility

• Simplification: Steps 3 and 7 are streamlined, since the university’s VLE already has a defined UX and built-in communication tools.
• Focus on Content: Analysing, structuring, and testing course materials (Steps 1, 2, 4, and 5).
• Onboarding Learners: We should not underestimate Step 8. Even though learners may be familiar with the VLE, a good orientation must not be overlooked.

Content and UX Steps

Using an already established platform, like Blackboard Learn, simplifies several of the Content and UX steps, as some tasks would either already be done at the institutional level or need less customisation.

Insights

Examples of the kinds of insights that might be gained from this data include:

• Learners may avoid dense reading sections or overly complex module layouts.
• Stakeholders may identify design bottlenecks.
• Learners may find certain interactive elements confusing or unengaging.
• Stakeholders may request consistency in activity design.
• Learners may spend more time on external research tools than course-provided resources.
• Stakeholders may prefer better alignment between tools.
• Learners may find forums intimidating or irrelevant.

• Learners may misunderstand complex questions or skip challenging content.
• Stakeholders may notice gaps in assessment coverage.
• Learners may highlight repetitive or unclear instructions..

The university recognises that the vast amount of information readily available online can be overwhelming, making it difficult for students to distinguish between credible sources and biased ones. This can exacerbate the spread of misinformation, further hinder students' ability to conduct objective research, and lead to lower than expected grades.

The information age presents challenges

As the university's official learning platform, Blackboard Learn provides a familiar and cost-effective solution for both students and staff. Its established presence means no extra expenses for new systems or tools, ensuring financial resources are focused on enhancing the learning experience. With seamless integration into existing university systems, students can easily access course materials, grades, and announcements.

Outcome-based evaluation focuses on the measurable results of the course. It tracks whether students apply what they’ve learned to real-world or academic situations.

• Learning Objective Outcomes: Use post-course assessments to measure mastery of the five LOs.
• Behavioural Changes: Track whether students apply their knowledge in future coursework or research projects. For example: Are students reducing bias in their own work? Are they using credible sources and effectively communicating their findings in other assignments?
• Impact on Academic Success: Analyse whether students perform better in modules that require critical reading or analysis after completing this course.
• Qualitative Feedback: Collect data through reflective essays or focus groups where students describe how the course influenced their understanding of bias and critical evaluation skills.

Strengths of Outcome-Based Evaluation: Provides tangible evidence of the course’s impact on students. Aligns well with institutional goals, such as improving academic performance.

• Is the text in an accessible font with enough colour contrast between itself and the background elements?
• Is the text used effectively?
• Are there headings and subheadings used for visual hierarchy?

Text

The course employs a top-down approach. We begin by exploring the overarching concept of bias and its potential impact on research findings. This broad overview is then followed by a deeper dive into specific research methods and strategies for identifying and mitigating bias. This approach fosters a holistic understanding of the topic. It scaffolds the content, and also ensures that learners' new knowledge can be put to use in their academic studies immediately.

• An assessment, which can be assessed automatically and provide learners with immediate feedback.
• Test learners' understanding of key concepts from the module.
• Include a mix of question types (multiple choice, true-false, fill-in-the-blank, matching, ordering, etc.) to assess different learning levels.

Action

Some of the actions that may be taken as a response of these insights might include:

• Simplify navigation.
• Adjust content layout to make key resources easier to access.
• Reorganise menus for better usability
• Provide clearer instructions for interactive elements.
• Revise difficult or less engaging activities.
• Integrate external tools into Blackboard for easier access.
• Adjust content to encourage use of course-provided materials.
• Offer discussion prompts aligned with learners' interests.
• Add a "starter guide" for effective participation.

• Rewrite confusing questions.
• Add hints or supplemental resources for frequently missed questions.
• Simplify instructions.
• Improve accessibility (e.g., better captions, clearer navigation for screen readers).

Process-based evaluation focuses on the quality of course delivery, instructional strategies, and learner engagement. This method ensures the course is implemented effectively and aligns with the intended goals.

• Course Design: Evaluate the structure and usability of the eLearning platform. Is the content logically sequenced, easy to navigate, and visually engaging? Are interactive elements included to enhance comprehension and maintain learner interest?

• Digital Content: Assess the clarity and relevance of multimedia resources. Are the videos, readings, and examples tailored to help learners meet the learning outcomes?

• Automated Feedback: Review the effectiveness of automated feedback in assessments. Do quizzes or interactive tasks provide meaningful, immediate feedback that helps learners reflect and improve?

• Instructor Presence in Optional Workshops: Analyse participation in optional workshops. Do these sessions add value by clarifying complex concepts or providing practical applications of learning?

• Support Resources: Check availability of additional learning aids (FAQs, discussion boards, or instructor office hours), to ensure learners can seek help when needed.
• Learner Engagement: Use analytics to track learner progress and time spent on different modules. Are learners completing activities, or are there areas where drop-off occurs
• Content Delivery: Ensure that asynchronous content accommodates diverse learning styles.
• Feedback Loops: Use end-of-module surveys or a mid-course feedback form to collect insights from students about what’s working well and what could be improved in the asynchronous delivery format.

Strengths of Process-Based Evaluation: Helps identify and address issues with delivery or instructional design. Can improve student engagement and overall course quality.

This section will explore how research questions and methodologies can be influenced by bias. Students will learn to identify potential biases in research design, data collection, and analysis, and understand how these biases can impact research outcomes.

1. Learners choose a research article or a piece of creative work (e.g., novel, film, artwork) to analyse.
2. They will apply the module's concepts to evaluate the chosen work for evidence of bias.

This option can cater to students in a variety of academic fields. Learners can post their response on the forum for peer feedback.

Data

The data that will be sourced will include the following:

• Digital Body Language: Clickstream data (e.g., navigation patterns, time on page, and clicks on resources).
• Activity-specific data: Micro-interactions with videos, simulations, or quizzes (e.g., pause/play frequency, retries).
• Comprehensive Learning Histories: Statements tracking learner progress across Blackboard and external tools (e.g., video platforms)
• Frequency of posts/replies.
• Sentiment and topic analysis from discussions.

• Quiz performance metrics: Average scores, most/least attempted questions, and time spent per question.
• Mid- and post-course feedback on design, engagement, and accessibility.

The university has noticed a growing number of first-year students lack the critical thinking skills to identify and evaluate bias in academic research. This can lead to:

• Misinterpretations of information.
• Plagiarism by unknowingly incorporating biased sources.
• Inability to form well-supported arguments in their own work.

First-year students struggle to identify bias

The university recognises the need for students to learn how to analyse research for potential bias, including understanding different types of bias (confirmation bias, cultural bias, etc.) and how these can influence research methods, results, and interpretations.

Develop critical thinking skills

The course will have four modules: Each module consisting of four asynchronous digital lessons, a toolkit and an optional workshop. The variety aims to meet learners' preferences, availability and personal objectives. The toolkit will offer additional resources for those learners who need extra support or who wish to take their learning further. The workshop will provide support for those students who need it or prefer collaborative learning.

The learning outcomes aim to address the different learning gaps. Some LOs focus on the LOTS (understanding and applying), while others aim for the HOTS that more advanced learners require (analysing, evaluating and creating).The LOs are SMART:Specific - Clearly state what learners will be able to do by the end of the course. Measurable - Can be assessed with quiz questions, assignments and participation in discussions. Achievable - Focus is on realistic skills for first-year university students. Relevant - Equips learners with critical thinking skills necessary for evaluating information and structuring well-supported arguments. Timely - The LOs are achievable within the timeframe of the course, with increasing complexity as learners progress.

The university realises that students need to understand how to mitigate the impact of bias in their own research. This involves employing:

• Diverse sources.
• Acknowledging limitations in existing research.
• Forming their own well-reasoned arguments based on evidence.

Develop strategies to mitigate bias

Monitor

Monitoring of learner analytics will include:

• Use engagement heatmaps to identify popular or underused areas of the course.
• Analyse patterns of disengagement (e.g., exits or bounces).
• Compare success rates and engagement time for different activities.
• Analyse patterns of retries to identify difficult content areas.
• Aggregate xAPI statements to detect trends in how students interact with both internal and external course content.
• Analyse engagement levels (e.g., active participants vs. lurkers).Use sentiment analysis to gauge emotional tone or barriers to learning.
• Identify knowledge gaps by analysing incorrect answers and skipped questions.

• Track time-on-task trends for quizzes.
• Analyse trends in satisfaction ratings across cohorts.
• Track differences between demographics or learning preferences.