Hopscotch 4-SoTL

Introductory Video

Generate your SoTL Design

Step 1: Who I Am as a SoTL Researcher

Although many researchers new to SoTL are eager to dive straight in to their research design, the first step in the process of generating a SoTL study design should involve reflection about who you are as a SoTL researcher. The basic set of beliefs that guides your action as both a teacher and a researcher (Berenson, 2018; Haigh & Withell, 2020) will inform the decisions you make about your SoTL study. SoTL researchers bring to their studies their particular teaching perspective as well as their way of understanding how things work in our world and the way knowledge is constructed (worldview). The worldview of the SoTL researcher as well as their adscription to a particular Interpretive Community have a deep impact on the decisions and inquiry procedures they will put in practice (Hamilton & McCollum, 2024).

Check the following resources to learn more about this initial step:

Pragmatism

Constructivism

Transformative

Post-positivism

Resources & Suggestions

Worldview as a SoTL researcher

Next

Step 2: Problem/Need I Want to Study in My Class

The second step in the process of generating a design for your SoTL study has to do with reflecting on what aspect of learning or teaching you would like to study and the goals that will be driving your inquiry. We propose that you think about the following key areas.

Check the following resources to learn more about the second step:

Defining what you would like to study

Goals driving your study

Interest in studying this particular topic

Resources & Suggestions

Step 3: Evidence Supporting the Problem/Need I Want to Study in My Classroom

The third step in the process of generating a SoTL design has to do with reflecting on your topic of interest and its relationship with what others have already studied. It is important to explore the literature early in the process of designing your SoTL study; however, you should also expect to return to it later as your study evolves. For helpful suggestions on searching and reviewing the SoTL literature, see these excellent articles by Healey & Healey (2023a, 2023b). Good quality SoTL is grounded in context (Felten 2013), both in the literature it references and the classroom scenario it describes. To adequately frame the situational context, a SoTL study might draw on three bodies of literature: 1) scholarship from the discipline, or from higher education in general if the study is intended for an interdisciplinary audience; 2) scholarship on specific pedagogical topics; and 3) more general teaching and learning theory (i.e., the science of learning). To illustrate these three areas, we will consider the study by Tran, Verezub, & Fisher (2025) that investigated the effects of a flipped classroom approach on a select group of student's reading comprehension.

Scholarship on specific pedagogical topics

Scholarship from the discipline

More general teaching and learning theory

Resources & Suggestions

Step 4: How I Will Study the Problem/Need

The fourth step in the process of generating a SoTL design has to do with deciding on the research design you will use in the study. Almost any quantitative, qualitative or mixed-methods research design could be used in a SoTL project; however, there are some special considerations that set SoTL studies apart from other research studies.   SoTL has blurry boundaries with related concepts like research on the science of learning and Disciplinary-Based Educational Research (DBER), which can cause some confusion among those new to educational research. All are focused on teaching and learning, but each can differ from the others in aims, scope, and methodology. Faculty in fields like Psychology and Education may conduct research on teaching and learning as part of their disciplinary research agendas, with an aim toward building theory about how people learn. This large body of literature, collectively known as the “science of learning,” informs and is informed by SoTL, but it is not SoTL in the way most scholars define it. Rather than being focused inward on the researcher's own context--an important attribute of SoTL--theoretical research on the science of learning is focused outward, with a goal of producing generalizable theory (Larsson, et al., 2020). The fact that SoTL is conducted in the classroom gives the researcher less control over some aspects of the design (for example, students cannot typically be randomly placed into course sections), but it gives the research more ecological validity because the data is being collected in the authentic environment where the learning takes place. SoTL, then, can serve as the proving ground for theories built by research on the science of learning (Daniel, 2012)

Continue reading

Step 4: How I Will Study the Problem/Need

Review the following research designs at the original Hopscotch site to decide which fits with your topic of interest, theoretical perspective, and personal paradigm. Post-positivists are often drawn to quantitative designs, constructivists to qualitative designs, and pragmatists to mixed method designs.

Qualitative Research Traditions

Quantitative Research Designs

Mixed-methods Research Designs

Resources & Suggestions

Step 5: What Questions Will Be Driving the Study of the Problem/Need

What pervasive questions about teaching continue to puzzle you? Where are the bottlenecks in student learning in your course? Do you wonder whether a popular technique would work in your particular classroom context? As discussed in earlier Hopscotch 4-SoTL steps, SoTL studies begin with these kinds of questions—questions that arise from our intuition or experience and drive us to seek an answer (Poole, 2018). Often our questions begin as areas of interest that may be quite broad. Therefore, your first goal in planning a SoTL study is to narrow your area of interest into a question that is testable (perhaps using quantitative methods) or can be explored further (perhaps using qualitative methods). Clearly defining your research question at the outset of your study will allow you to choose an appropriate research design.

Research questions in quantitative SoTL studies

Research questions in qualitative SoTL studies

Questions to reflect upon when establishing your RQs

What to avoid when defining your research question

Resources & Suggestions

Step 6: What Data I Will Collect

The sixth step of the process has to do with defining the data collection methods that you will be using in your SoTL study. The following two questions might be of help in doing so:  What evidence do you need to convince yourself that you’ve answered your question? What tools do you use in your course that would provide that evidence? For example, if you are interested in measuring student learning, what evidence would convince you that students have learned? Their grades? Their scores on a standardized end-of-course assessment? Or might their evidence of learning come through in a term paper or other artifact? These will be the sources of your data. Much of this data will already be available to you in your course. However, in most cases you must receive consent from your students (under the oversight of an Institutional Review Board) to use the data in research. This oversight process will be discussed in Step 9 of the Hopscotch-SoTL model. Below are some of the most common sources of data for SoTL projects:

Learning Management System Tools

Surveys

Classroom Assessments

Interviews and Focus Groups

Observations & Reflections

Resources & Suggestions

Step 7: How I Will Analyze the Collected Data

The seventh step of the process has to do with organizing and making sense of the data that you will be using in your SoTL study. This advice in Frances Rust and Christopher Clark's guide to action research also applies to SoTL. They offer the following: "Analysis is the heart of making sense of your experience with action research. Analysis is fun and messy. It always begins with your data. Data never speak for themselves. Please remember this. Data never speak for themselves. Your mind is the most important analytical tool that you have. Analysis is a process of telling a convincing story about the sense that your data led you to make. As well, you must persuade a skeptical audience that the story that you tell and the sense that you make are supported by evidence" (p. 14). There are two major sources of support for your evidence:

• The first is the data you have collected and the patterns that you see.

• The second is equally important. It is what others have learned about this topic. If you haven’t already read other research and theory on your topic, now is the time to revisit Step 3. This is critical to situating your work. If, for example, you find that the action you took has results that are very similar to those of other researchers, then you know your analysis is in the right ballpark. Essentially, you can borrow from the authority of others that have come before you to strengthen the claims that you will make for the action that you took. If, however, your results contradict prior research, then you are well on the way to forming a provocative new question about why your study yielded such different results. You have something interesting to talk about with colleagues and with other researchers. Either way, what you learn locally can become part of a larger conversation among educators and researchers.

Continue reading

Step 7: How I Will Analyze the Collected Data

Your data analysis will look very different depending on the methods you use (for an in-depth look at the diversity of methods in SoTL studies, see Divan et al. (2017)). However, as a general rule, you should prepare to:

• Describe the action(s) that you took.
• Reflect on the evidence you have collected.
• Count. Look for patterns.
• Share the evidence with colleagues.
• Examine what different explanations could explain the data (draw on prior research).

• Revisit assumptions about the learning situation.
• Formulate a trial explanation.
• Develop an argument with evidence and claims.
• Check if your evidence support your claims: Does the evidence as you describe it support your claims? Do your colleagues (critical friends) find your argument credible? How does the argument fit into ongoing debates and conversations? What is unique about your setting or context? Will others find your argument useful?

In a quantitative study, you’ll be analyzing numbers. You may report some descriptive statistics, but you’ll likely be using inferential statistics, with the most common for SoTL studies being t-tests, analysis of variance (ANOVA), correlation, and multiple regression. Here are some things to consider when analyzing quantitative data:

• Because student data doesn’t usually fit a normal curve, you may decide to use non-parametric statistics. This is a conservative choice to make, because it increases your chances of making a Type II error (not seeing an effect when an effect is actually there); however, if your analysis reveals an effect when using non-parametric statistics, your claim is stronger.

Continue reading

Step 7: How I Will Analyze the Collected Data

• Many SoTL researchers are now choosing to report effect sizes, which give readers a better idea of the practical significance of the results. For more special considerations for drawing conclusions from quantitative datat in SoTL studies, see McGrath (2016).
• Commonly used software for quantitative data analysis includes SPSS and SAS, as well as R, which is freely available online.
• Contact your institution’s research center to identify resources and support for quantitative data analysis on your campus.

The data analysis process is quite different for qualitative data, and may require several iterations of analysis before completion. Developing and assigning themes and codes to the data is one approach (for example, see Braun and Clarke (2006)). An important concept in qualitative data analysis is triangulation—the inclusion of multiple sources of data in order to provide a richer, more accurate analysis. Some excellent examples of qualitative analysis in SoTL can be found in Brady, et al. (2022).

Resources & Suggestions

Step 8: How I Will Ensure the Trustworthiness of the Process

In a Scholarship of Teaching and Learning (SoTL) study, ensuring the trustworthiness of qualitative methods means establishing that your findings are credible, dependable, and meaningful to others. Strategies for doing this include triangulating your data sources (for example, using surveys, interviews, and classroom observations together), providing thick, rich descriptions so readers can understand the context of your study, and incorporating participant voices to ensure authenticity. Reflexivity is also essential—you should acknowledge your own perspective as a teacher-researcher and how it may influence both the data collection and interpretation. Peer debriefing and member-checking, where participants or colleagues review your interpretations, can further strengthen the transparency and trustworthiness of your work. When using quantitative methods, the focus shifts to ensuring validity and reliability so that your results are accurate and replicable. Validity can be strengthened by aligning your data collection instruments directly with your research questions and by using established, standardized measures whenever possible. Reliability can be supported by consistent application of procedures, clear operational definitions, and pilot testing your tools to ensure clarity and accuracy. Statistical techniques—such as checking for internal consistency, using appropriate sample sizes, and selecting the right analyses—also help confirm that your findings are not due to chance. April McGrath (2016) proposes some special recommendations for interpreting and reporting quantitative results in SoTL studies. Taking time to consider this information ensures that your study not only generates meaningful insights for your own teaching but also contributes trustworthy knowledge to the broader SoTL community.

Trustworthiness in Qualitative SoTL studies

Validity in Quantitative SoTL studies

Resources & Suggestions

Step 9: What Ethical Principles Will Be Guiding My Study

The term “ethics” derives from the Greek word “ethos” which means character. To engage with the ethical dimension of your research requires asking yourself several important questions:

• What moral principles guide your research?
• How do ethical issues enter into your selection of a research problem?
• How do ethical issues affect how you conduct your research—the design of your study, your sampling procedure, etc.?
• What responsibility do you have toward your research subjects? For example, do you have their informed consent to participate in your project?
• What ethical issues/dilemmas might come into play in deciding what research findings you publish? Will your research directly benefit those who participated in the study?

As with any research study that involves human subjects, SoTL studies are often under the purview of a university’s ethics board, sometimes known as the Institutional Review Board (IRB). We must protect the students who engage as participants in our study from unwanted breaches of privacy, coercion into participation, and potential psychological harm. However, because SoTL studies often involve normal classroom procedures, the process for applying for ethics board approval for a SoTL study is typically less involved than it may be for other research studies. In fact, some institutions have an umbrella policy for SoTL studies that requires a very limited amount of work on part of the researcher.For institutions without SoTL umbrella policies, you will prepare an application for ethics approval. This application will ask you to describe several aspects of your study. If this is the first time you have worked with human subjects in a research study, you may need to enroll in a training course, such as the one offered by CITI, before you begin your SoTL study. Please see your institution’s guidelines for more information.

Continue reading

Step 9: What Ethical Principles Will Be Guiding My Study

The major principles associated with ethical conduct in research are (Litchman, 2011):

• Do No Harm

• Privacy and Anonymity

• Confidentiality

• Informed Consent

• Rapport

• Intrusiveness

• Data Interpretation

Obtaining ecological validity within an ethical framework means sacrificing control. We usually cannot randomly place students into sections of a course in order to test a teaching method; this would be unfair to students, as they are typically able to choose their own course sections. It would also be a breach of ethics to expose some students to new and effective teaching techniques while condemning others to ineffective and outdated approaches. Students are our partners and participants in SoTL, and therefore must be treated with respect. You may even wish to involve students as co-researchers, a practice that is popular in the SoTL field (see Mercer-Mapstone, et al., 2017). At a minimum, you should consider your study from the student participants’ perspective, be as transparent as you can without unduly influencing your results, address any concerns, and above all else, do no harm.

Resources & Suggestions

Generate a SoTL Design

DirectionsTo generate your research design using Hopscotch 4-SoTL, please complete the form located on the left side of this page. For more information about each of the nine steps outlined in Hopscotch 4-SoTL, explore the interactive resources available by clicking on each step. Once you have completed the form, you will receive an email with your research design in PDF format.

Resources & Suggestions

References:

Bass, R. (2020). What’s the problem now? To Improve the Academy, 38 (1). http://doi.org/10.3998/tia.17063888.0039.102 Huber, M.T., & Hutchings, P. (2005) The Advancement of Learning: Building the Teaching Commons. San Francisco: Jossey-Bass. Maxwell, J. A. (2008). Designing a qualitative study. In L. Bickman & D. J. Rog (Eds.), The SAGE handbook of applied social research methods (Vol. 2, pp. 214–253). Thousand Oaks, CA: Sage. Poole, G. (2018) Using intuition, anecdote, and observation: Rich sources of SoTL projects. In N. Chick (Ed.), SoTL in action: Illuminating critical moments of practice. Stylus.

Trustworthiness in Qualitative SoTL studies

In qualitative research Guba (1981) proposes four criteria that should be considered in pursuit of trustworthiness:

• Credibility (in preference to internal validity): One of the key criteria addressed by positivist researchers is that of internal validity, in which they seek to ensure that their study measures or tests what is actually intended. According to Merriam, the qualitative investigator’s equivalent concept, i.e. credibility, deals with the question, “How congruent are the findings with reality?”

• Transferability (in preference to external validity/generalizability): External validity “is concerned with the extent to which the findings of one study can be applied to other situations”. In positivist work, the concern often lies in demonstrating that the results of the work at hand can be applied to a wider population. Since the findings of a qualitative project are specific to a small number of particular environments and individuals, it is difficult to demonstrate that the findings and conclusions are applicable to other situations and populations. Because of that we use “Naturalistic Generalization” (Stake, 2005).

• Dependability (in preference to reliability): In addressing the issue of reliability, the positivist employs techniques to show that, if the work were repeated, in the same context, with the same methods and with the same participants, similar results would be obtained. In order to address dependability in Qualitative research, the processes within the study should be reported in detail, thereby enabling a future researcher to repeat the work, if not necessarily to gain the same results. Thus, the research design may be viewed as a detailed “prototype model”.

• Confirmability (in preference to objectivity): Objectivity in science is associated with the use of instruments that are not dependent on human skill and perception.The concept of confirmability is the qualitative investigator’s comparable concern to objectivity. Here steps must be taken to help ensure as far as possible that the work’s findings are the result of the experiences and ideas of the informants, rather than the characteristics and preferences of the researcher.

Step 8: How I Will Ensure the Trustworthiness of the Process

Explore Additional Resources for Step 8

Step 5: What questions will be driving the study of the problem/need

Explore additional resources for step 5

Resources & Suggestions

Resources to help you develop Research Questions in Quantitative Studies

Price, P., Jhangiani, R., & Chiang, I. (2015). Research methods of psychology – 2nd Canadian edition. Victoria, B.C.: BCcampus. Retrieved from https://opentextbc.ca/researchmethHow to Structure Quantitative Research Questions: Cheat Sheet

Resources to help you develop Research Questions in Qualitative Studies

Agee, J. (2009) Developing qualitative research questions: a reflective process, International Journal of Qualitative Studies in Education, 22(4), 431-447, DOI: 10.1080/09518390902736512

Resources to help you develop Research Questions in Mixed-Methods Studies

Tashakkori, A., & Creswell, J. W. (2007). Exploring the nature of research questions in mixed methods research. Journal of Mixed Methods Research, 1(3), 207–211. https://doi-org.proxy.kennesaw.edu/10.1177/1558689807302814

Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking research questions to mixed methods data analysis procedures 1. The Qualitative Report,11(3), 474-498. Retrieved from https://nsuworks.nova.edu/tqr/vol11/iss3/3

Step 8: How I will ensure the trustworthiness of the process

Explore additional resources for step 8

Step 1: Who I Am as a SoTL Researcher

Explore Additional Resources for Step 1

Step 2: What is the problem/need I want to study in my classroom

The following AI tools can assist you in step 2 of the process of generating your design: Free AI Brainstorming Generator can be used to get help bringing your ideas to life with Grammarly’s AI-powered content idea generator, which helps you brainstorm ideas for all your business, school, and personal needs. Consensus could be used to assist users in the identification of relevant topics that have been published in peer-reviewed articles. Consensus is an AI-powered search engine designed to take in research questions, find relevant insights within research papers, and synthesize the results using large language models. It is not a chatbot. Consensus only searches through peer-reviewed scientific research articles to find the most credible insights to your queries. Carrot2 could be used to identify potential research topics. Carrot2 organizes your search results into topics. With an instant overview of what's available, you will quickly find what you're looking for.

Step 6: What data I will collect

Explore additional resources for step 6

CONSTRUCTIVISM Based on the ideas of Mannheim and from works such as Berger and Luekmann’s (1967) The Social Construction of Reality and Lincoln and Guba’s (1985) Naturalistic Inquiry, constructivism holds that there is not one objective truth; truth is socially constructed. Social constructivists believe that individuals seek understanding of the world in which they live and work, developing subjective meanings of their experiences. These meanings are varied and multiple, leading the researcher to look for the complexity of views rather than narrowing meanings into a few categories or ideas. The goal of the research is to rely as much as possible on the participants’ views of the situation being studied. The researcher’s intent is to make sense of (or interpret) the meanings others have about the world. Rather than starting with a theory (as in postpositivism), inquirers inductively develop a theory.

Step 2: Problem/Need I Want to Study in My Classroom

Explore Additional Resources for Step 2

Step 9: What ethical principles will be guiding my study

Explore additional resources for step 9

Step 2: What is the problem/need I want to study in my classroom

Explore additional resources for step 2

Step 3: Evidence supporting the problem/need I want to study in my classroom

Explore additional resources for step 3

Step 7: How will I analyze the collected data?

The following AI tools can assist you in step 7 of the process of generating your SoTL design: Qualitative Data: AI data analysis is on the rise. For instance, the AI module of Atlas.ti can be used to analyze qualitative data. You can also use Taguette, a free and open-source tool for qualitative research, or Highlight Tool, an extension for Google documents.

Quantitative Data: The One-Way ANOVA Calculator is designed to compare the means of three or more independent samples (treatments) simultaneously.

Visualization tools:

Interviews and Focus Groups

Qualitative and mixed-method research designs may necessitate the more in-depth data that can be gained from interviews or student focus groups. These data collection approaches may afford a deeper understanding of our questions, but also require more time in the collection and analysis of data. Question protocols can range from more structured to open-ended and informal, but the questions will almost always need to be reviewed by an ethics board ahead of time (see Step 9). In addition, it often helps to have a colleague moderating the interview or focus group so that students feel more comfortable in speaking freely.When conducting an interview or focus group, carefully note the details of when and where the session is taking place, ensure you have proper recording equipment, and above all, be a good listener.

Step 9: What Ethical Principles Will be Guiding My Study

Explore Additional Resources for Step 9

Step 3: Evidence supporting the problem/need I want to study in my classroom

The following AI tools can assist you in step 3 of the process of generating your design: ResearchRabbit is a scholarly publication discovery tool supported by artificial intelligence (AI). The tool is designed to support your research without you switching between searching modes and databases, a process that is time-consuming and often escalates into further citation mining; a truly unpleasant rabbit hole (and that's what inspired the name ResearchRabbit) 2Dsearch is a radical alternative to conventional 'advanced search'. Instead of entering Boolean strings into one-dimensional search boxes, queries are formulated by manipulating objects on a two-dimensional canvas. This eliminates syntax errors, makes the query semantics more transparent, and offers new ways to collaborate, share, and optimize search strategies and best practices. Connected Papers is a unique, visual tool to help researchers and applied scientists find and explore papers relevant to their field of work. Consensus can be used to assist students in the identification of relevant topics that have been published in peer-reviewed articles.

Questions to reflect upon when establishing your RQs

We recommend reflecting on the following questions as you establish the research questions for your SoTL study:

• Does my question describe “what is,” or answer “what works?” (Hutchings, 2000)?
• How might the methodology I have chosen for my study influence the way I state my question?
• If I am using quantitative methodology, are my variables clearly defined?
• If I am using qualitative methodology, is my question open-ended enough to allow additional questions to emerge from the data?
• Have I avoided compound questions, ambiguous terminology, and implications of causality?

Step 1: Who I am as Faculty SoTL-Researcher

Explore additional resources for step 1

Step 6: What data I will collect

The following AI tools can assist you in step 6 of the process of generating your design: We could use Google Gemini or Chat GPT to develop a draft of a data collection protocol for a given study. For instance, we could use the following prompt:

Generate an example of a thorough interview protocol with six questions for local participants involved in a qualitative case study research design focused on investigating the impact of river water quality on a local community it traverses, emphasizing how this environmental factor influences quality of life, the local economy, and business activities.

To practice your interview skills you could use Interview warm up:

How it works

You can also ask Chat GPT to be your interviewee so you can rehearse an interview:

Sample ChatGPT Script

Scholarship on Specific Pedagogical Topics

Once you have identified your topic of interest, you can begin to explore specific literature on your topic. The Hopscotch Literature Review Tool and the embedded AI resources can help you get started. In the study by Tran, Verezub, & Fisher (2025), the authors provide brief reviews of the literature on reading comprehension and the flipped classroom, two areas of focus in their study.

Defining what you would like to study

SoTL projects often arise from our anecdotal experiences, our intuition, and our observations (Poole, 2018). As curious teacher-researchers, we can draw from these sources to select a personally relevant topic of inquiry. Although many SoTL studies begin with a problem that requires investigation, others are motivated by the need to thoroughly describe a situation. In her Introduction to the book Opening Lines: Approaches to the Scholarship of Teaching and Learning (2000), Pat Hutchings, one of the early leaders in the SoTL field, describes SoTL questions that answer “What works?,” for example, an investigation of a new teaching method, and questions that answer “What is?,” which helps us better understand what’s already happening in our classrooms. Many of us immediately jump to a “What works” question, especially those whose disciplines emphasize hypothesis testing and a post-positivist paradigm. A “What Works” question focuses on establishing evidence of our teaching effectiveness on student learning. But sometimes we simply need to dive deeper into a particular classroom phenomenon—describing “What Is”—in order to understand it. Bill Cerbin (2018) describes a problem he had in his classroom where students were continually submitting low quality group assignments. After trying several different types of group work assignments with the same result, Cerbin realized he needed to better understand what was happening during the time students were working on their group assignments. He designed an observational, qualitative SoTL study to describe and interpret his students’ group work experience. What he found surprised him—what was actually happening during group work wasn’t at all what he thought was happening, and he was able to revise his assignments accordingly. As you begin to think about your SoTL topic, the following questions might guide your reflection:

• What pervasive questions about teaching continue to puzzle you?
• Where are the bottlenecks in student learning in your course?
• Are you curious about whether a popular technique would work in your particular classroom context?
• Who is learning, and who is being left out?

Step 9: What ethical principles will be guiding my study

The following AI tools can assist you in step 9 of the process of generating your design: Google Gemini or Chat GPT can be used to help you draft a solid IRB in compliance with US federal regulations.

Check Prompt

The following tool will assit you in drafting consent forms.

Worldview as a SoTL researcher

Guba (1990) describes a paradigm or worldview as "a basic set of beliefs that guide action.” That basic set of beliefs of the researcher is based on his ontological (What is the nature of reality?) and epistemological assumptions (What is the nature of knowledge and the relationship between the knower and the would-be known?). Therefore, how one views the constructs of social reality and knowledge affects how they will go about uncovering knowledge of relationships among phenomena and social behavior. Your ontological assumptions inform your epistemological assumptions which inform your methodology and these all give rise to your methods employed to collect data. From an ontological point of view, post-positivism understands that there is one reality which is knowable within a specific level of probability, while constructivism understands that the nature of reality is multiple and socially constructed. Pragmatism asserts that there is a single reality and that all individuals have their own unique interpretation of reality. Finally, those following a transformative worldview reject cultural relativism and recognize that various versions of reality are based on social positioning. From an epistemological point of view, post-positivists believe that objectivity is key and the researcher manipulates and observes in a dispassionate objective manner. Constructivists, on the contrary, believe that there should be an interactive link between researcher and participants, and that since knowledge is socially and historically situated, it needs to address issues of power and trust. Pragmatism posits that relationships in research are determined by what the researcher deems as appropriate to a particular given study. Finally, a transformative worldview acknowledges that since there is an interactive link between researcher and participants, and knowledge is socially and historically situated, there is a clear need to address issues of power and trust.

These ontological and epistemological assumptions have a direct impact on the methodology used in a given study. Post-positivism calls for interventionist quantitative studies, while constructivism is associated with qualitative hermeneutical studies, and pragmatism matches methods to specific questions and purposes of research by using mixed methods. In the case of researchers following a transformative worldview, qualitative methods deeply grounded in critical theories are most commonly used. For examples and further explanation of how these paradigms are expressed within a SoTL context, see Haigh and Withell (2020) and Hamilton & McCollum (2024). Your disciplinary traditions may also inform your paradigmatic positioning (Poole, 2013). For example, SoTL practitioners in the sciences typically approach research from a post-positivist perspective, whereas perspectives from the humanities may align more with constructivism. Before thinking about your research question and design, spend some time reflecting about your own worldview and any disciplinary traditions that may influence your perspective.

Surveys

Surveys are a common source of data for SoTL studies, allowing the researcher to quickly collect quantitative and/or qualitative data. As with other data collection methods, your survey will usually require pre-approval from your institution’s ethics board before distribution. If your study has well-defined variables that have been measured in previous studies (for example, “metacognition”), you may want to select a survey that has been empirically validated and published. Two good sources for SoTL surveys include A Compendium for Scales For Use in SoTL Research and the Hub for Introductory Psychology and Pedagogical Research. They include scales that measure self-efficacy, critical thinking, mindset, student engagement, epistemological beliefs, and more. If your study is very context-specific (for instance, you are gathering data on the student response to a particular assignment), you may want to create your own survey. If you create your own, in addition to considering the data needed to answer the research question, you should think about how to minimize potential bias and maximize the validity of the replies. The University of Toronto Centre for Teaching Support and Innovation has this guide for SoTL researchers who wish to create their own survey.

Step 4: How I Will Study the Problem/Need

Explore Additional Resources for Step 4

What to avoid when defining your research question

The Center for Educational Innovation at the University of Minnesota provides the following recommendations on what to avoid when generating your SoTL research questions:

• Compound research questions. For example: "Do new learning spaces inspire faculty members to employ new active-learning techniques that enhance student motivation and learning?" This question contains several distinct strands that should be disentangled and listed as separate questions.

• Ambiguous terminology.  For instance: “Are students more engaged after a service-learning experience than they were before?” Student engagement is a popular concept in current educational discourse, but its meaning is not entirely clear. Do we mean affective engagement (improved feelings or emotions)? Cognitive engagement (increased intellectual interest)? Social engagement (greater interaction with peers or instructors around class issues)? Specifying a way of measuring engagement will often lead to greater precision in terminology.

• Overly ambitious questions. For example: “Does using a problem-based approach to teaching evolutionary theory in introductory biology courses cause greater student acceptance of evolution?” Causal questions are among the most difficult questions to answer because an affirmative answer must not only document change over time, but also justify attributing that change to the putative causal factor. This can be challenging, particularly if circumstances do not permit you to use a comparative research design that controls for extraneous factors.

Resources & Suggestions

Resources to help you define your data collection methods

References

Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques. Jossey Bass Wiley. Visioli, S., Lodi, G., Carrassi, A., & Zannini, L. (2009). The role of observational research in improving faculty lecturing skills: A qualitative study in an Italian dental school. Medical Teacher, 31(8), 362-369.

An Overview of Quantitative and Qualitative Data Collection Methods Developing a Questionnaire/Survey Adopting or Adapting an Existing Instrument/Survey Additional Resources from the Interactive Research Methods Lab

Scholarship from the Discipline

Because SoTL studies are conducted by practicing instructors, they are often situated within the context of that instructor’s discipline. Citing scholarship about the unique challenges of, or approaches to, teaching a particular course provides a helpful connection for readers who are in the same discipline. For example, in the study by Tran, Verezub, & Fisher (2025), the authors begin the paper by discussing the important skills needed by students learning English as a foreign language and how these skills have been investigated in prior studies. As this is an article written for an interdisciplinary journal, this section is brief. In a disciplinary SoTL journal, this section might be expanded. As you begin thinking about your SoTL study, it is important to read examples of disciplinary and interdisciplinary SoTL studies, returning to these studies later when you are ready to dive deeper into your review of the literature. Kennesaw State University maintains a directory of SoTL journals that is searchable by discipline.

Resources & Suggestions

References:

Brady, A. C., Hensley, L. C., Sovic, D. M., Kulesza, A. E., Wolters, C. A., & Breitenberger, C. A. (2022). What makes a study strategy intervention impactful? An interview-based study. College Student Affairs Journal, 40(1), 17–31. https://doi.org/10.1353/csj.2022.0001 Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. Divan, A., Ludwig, L., Matthews, K., Motley, P., & Tomljenovic-Berube, A. (2017). Survey of research approaches utilized in the scholarship of teaching and learning publications. Teaching & Learning Inquiry, 5(2). https://doi.org/10.20343/teachlearninqu.5.2.3 McGrath, A. (2016). Searching for significance in the scholarship of teaching and learning and finding none: Understanding non-significant results. Teaching and Learning Inquiry, 4(2), 150–155. https://doi.org/10.20343/teachlearninqu.4.2.12

Step 7: How will I analyze the collected data?

Explore additional resources for step 7

Step 3: Evidence Supporting the Problem/Need I Want to Study in My Classroom

Explore Additional Resources for Step 3

Interest in studying this particular topic

In order to reinforce the previous aspects, we recommend you reflect on the interest you have in studying your topic. As a SoTL researcher, your personal connection to the study is an important part of the study’s context, which you will thoroughly describe for your readers. Samaras (2001) provides the following examples as an illustration:

• Why are you interested in studying this particular problem, issue, innovation, need?
• Example answer: This question is important to me because of my background as a Hispanic and the fact that I was an ESOL [English as a Second Language] student for a brief period of time.

• Who would benefit from addressing/studying the previous particular problem, issue, innovation, need?
• Example Answer: Many would benefit from addressing this question--ESOL [English for Speakers of Other Languages] teachers, students, parents, and other teachers. I want to work to improve ESOL students’ weak performance in school.

Resources & Suggestions

References:

Felten, P. (2013). Principles of good practice in SoTL. Teaching and Learning Inquiry, 1(1), 121-125. Healey, M., & Healey, R. L. (2023). Searching the literature on scholarship of teaching and learning (SoTL): An academic literacies perspective: Part 1. Teaching and Learning Inquiry, 11. Healey, M., & Healey, R. (2023). Reviewing the literature on scholarship of teaching and learning (SoTL): An academic literacies perspective: Part 2. Teaching and Learning Inquiry, 11. Steiner, H. H., & Hakala, C. M. (2021). What do SoTL practitioners need to know about learning?. Teaching & Learning Inquiry, 9(1), 79-85. Tran, Q., Verezub, E., & Fisher, R. (2025). Flipping EFL reading comprehension classes: Students’ learning achievement and perceptions. Teaching and Learning Inquiry, 13, 1–44

Step 4: How I will study the problem/need

Explore additional resources for step 4

Step 7: How I Will Analyze the Collected Data

Explore Additional Resources for Step 7

TRANSFORMATIVE This position arose during the 1980s and 1990s. It came from individuals who felt postpositivist assumptions imposed structural laws and theories that did not fit marginalized individuals. These inquirers felt that the constructivist stance did not go far enough in advocating for an action agenda to help marginalized peoples. Therefore, this worldview focuses on the needs of groups and individuals in our society that may be marginalized. Historically, the transformative writers have drawn on the works of Marx, Adorno, Marcuse, Habermas, and Freire. No uniform body of literature characterizes this worldview: it includes groups of researchers that are critical theorists, participatory action researchers, Marxists, feminists, racial and ethnic minorities, persons with disabilities, indigenous and postcolonial peoples, and members of the lesbian, gay, bisexual, trans-sexual, and queer communities. This worldview holds that research inquiry needs to be intertwined with politics and a political change agenda to confront social oppression at whatever levels it occurs (Mertens, 2010).

PRAGMATISM Pragmatism derives from the work of Peirce, James, Mead, and Dewey. There are many forms of this philosophy, but for many, pragmatism as a worldview arises out of actions, situations, and consequences rather than antecedent conditions (as in postpositivism). Instead of focusing on methods, researchers emphasize the research problem and use all approaches available to understand the problem. Individual researchers have a freedom of choice. In this way, researchers are free to choose the methods, techniques, and procedures of research that best meet their needs and purposes. As a philosophical underpinning for mixed methods studies, Morgan (2007), Patton (1990), and Tashakkori and Teddlie (2010) convey its importance for focusing attention on the research problem in social science research and then using pluralistic approaches to derive knowledge about the problem.

Observations & Reflections

Observational approaches allow us to gather quantitative data (such as number of hands raised in response to a question) and qualitative data (such as student quotes or reactions to a topic) that is not easily measured in another way. As a participant-researcher, your observations can be key to uncovering the nuances that aren’t readily apparent in the data. An excellent example of this approach can be found in Visioli et al. (2009).Your reflections on your teaching practice can be another source of data on their own, and can help situate your other data in context. We recommend keeping a notebook in which you write a brief daily reflection about what you notice in the classroom. You can decide later whether you want to include this data as part of your study.

Resources & Suggestions

References:

Daniel, D. B. (2012). Promising principles: Translating the science of learning to educational practice. Journal of Applied Research in Memory and Cognition, 1(4), 251-253. Larsson, M., Mårtensson, K., Price, L., & Roxå, T. (2020). Constructive friction? Charting the relation between educational research and the scholarship of teaching and learning. Teaching & Learning Inquiry, 8(1)

Validity in Quantitative SoTL studies

In quantitative research we usually consider two general dimensions in evaluating a measurement method: reliability and validity.Reliability is defined as the consistency of the measurements. To what level will the instrument produce the same results under the same conditions every time it is used? Reliability adds to the trustworthiness of the results because it is a testament to the methodology if the results are reproducible. The reliability is often examined by using a test and retest method where the measurement are taken twice at two different times. The reliability is critical for being able to reproduce the results, however, the validity must be confirmed first to ensure that the measurements are accurate. Consistent measurements will only be useful if they are accurate and valid. The term validity refers to the strength of the conclusions that are drawn from the results. In other words, how accurate are the results? Do the results actually measure what was intended to be measured? There are several types of validity that are commonly examined:

• Conclusion validity looks at whether or not there is a relationship between the variable and the observed outcome.
• Internal validity considers whether or not that relationship may be causal in nature.
• Construct validity refers to whether or not the operational definition of a variable actually reflects the meaning of the concept. In other words, it is an attempt to generalize the treatment and outcomes to a broader concept.
• External validity is the ability to generalize the results to another setting. There are multiple factors that can threaten the validity in a study. They can be divided into single group threats, multiple group threats, and social interaction threats.

For more information regarding the control of the validity of a quantitative study please read chapter 6 in Price, Jhangiani, & Chiang (2015).

Research Questions in Qualitative SoTL Studies

Research questions in qualitative studies are not as well-defined at the outset of the study. Being more open-ended and in-depth than quantitative studies, qualitative studies allow for additional questions to emerge during the data collection process. For example, a researcher taking this approach might be interested in the different ways her students learn in her Calculus II course. Her research question might be:

How do students who took an AP Calculus course in high school approach the study of Calculus II differently from those who took their first calculus course during college?

Although she has established a broad question in order to drive the collection of qualitative data, interviews or focus groups, her question might evolve as she proceeds. For example, early in the study if she notices that students who mention being taught metacognitive strategies in high school approach learning in a different way, she may decide to focus more carefully on this particular aspect. For additional guidance on developing qualitative research questions, see Agee (2009).

POST-POSITIVISM This tradition comes from the 19th-century (Comte, Mill, Durkheim, Newton, and Locke) and represents the traditional form of research (scientific method). Called post-positivism since it represents the thinking after positivism, it challenges the traditional notion of the absolute truth of knowledge (Phillips & Burbules, 2000). Postpositivists hold a deterministic philosophy in which causes (probably) determine effects or outcomes. It is reductionistic in that the intent is to reduce the ideas into a small, discrete set to test, such as the variables that comprise hypotheses and research questions. Its purpose is to test theories. The knowledge that develops through a postpositivist lens is based on empirical observation and measurement of the objective reality that exists “out there” in the world.

Learning Management System Tools

Most instructors, whether teaching online or face-to-face, use a learning management system (such as Canvas, Blackboard, or D2L Brightspace) in their courses. This system can be a rich source of data because many aspects of engagement are automatically captured, such as number of visits to particular pages, the amount of time a student spends looking at content, and comments on the discussion board.

Check with your institution for a guide to obtaining data from your learning management system. You may also have access to a data collection plugin like Terracotta, developed at Indiana University.

Step 4: How I will study the problem/need

The following AI tools can assist you in step 4 of the process of generating your design: Gemini or Chat GPT could be used to help users of Hopscotch 4-SoTL understand the differences between research traditions for a certain topic. Moreover, they can be used to understand how a thorough research design should look like. For instance, we could use the following prompt:

Generate a concise, coherent description of the key components of a qualitative SoTL design focused on investigating the impact of the use of the flipped calssroom model in a chemistry undergraduate class: Structure the response according to the nine steps of the Hopscotch Research Design Model: 1. Paradigmatic Positioning: Identify the researcher’s underlying epistemological and ontological assumptions guiding the inquiry, specifically within a SoTL framework. 2. Study Topics & Goals: Define the central themes of the study and its intended contributions to teaching practice and SoTL scholarship. 3. Conceptual Framework: Outline the educational theories or pedagogical models (e.g., active learning, Bloom’s taxonomy, self-determination theory) that inform the study’s analytical lens. 4. Research Tradition/Design: Specify the qualitative tradition (e.g., case study) and justify its suitability for exploring teaching and learning in authentic classroom settings. 5. Research Questions: Formulate 2–3 central, open-ended questions aligned with the study’s focus on the flipped classroom in undergraduate chemistry. 6. Data Collection Methods: Describe the qualitative methods (e.g., interviews, focus groups, classroom observations, analysis of course materials) used to gather data from students and the instructor. 7. Data Analysis Approach: Summarize the analytic strategies (e.g., thematic analysis, coding procedures) to interpret the findings in relation to engagement, learning, and instructional effectiveness. 8. Trustworthiness Strategies: Describe techniques to ensure credibility, transferability, dependability, and confirmability in SoTL research. 9. Ethical Considerations: Highlight ethical principles guiding the study, including informed consent, confidentiality, voluntary participation, and addressing potential power dynamics between instructor-researchers and students.

Step 5: What questions will be driving the study of the problem/need

The following AI tools can assist you in step 5 of the process of generating the research questions for your SoTL study: Consensus could be used to identify research questions that have been used in previously published SoTL studies. Consensus is an AI-powered search engine designed to take in research questions, find relevant insights within research papers, and synthesize the results using large language models. It is not a chatbot. Consensus only searches through peer-reviewed scientific research articles to find the most credible insights to your queries. Google Gemini could be used to identify potential questions for a particular research tradition or design. For instance, we could use the following prompt:

Generate examples of research questions that could be used to drive a SoTL study on the impact of using the flipped calssroom model in an undergraduate chemistry class:

The tool on the right from Monash University will help you develop and refine your research questions in an interactive fashion.

Resources & Suggestions

References:

Mercer-Mapstone, L., Dvorakova, S. L., Matthews, K., Abbot, S., Cheng, B., Felten, P., Knorr, K., Marquis, E., Shammas, R., & Swaim, K. (2017). A systematic literature review of students as partners in higher education. International Journal for Students as Partners, 1(1). Fedoruk, L. (2017). Ethics in the scholarship of teaching and learning: Key principles and strategies for ethical practice. Taylor Institute for Teaching and Learning Guide Series. Calgary, AB: Taylor Institute for Teaching and Learning at the University of Calgary. www.ucalgary.ca/taylorinstitute/guides

Goals driving your study

Maxwell (2008) states that goals for a study include motives, desires, and purposes—anything that leads you to do the study or that you hope to accomplish by doing it. These goals, which can be personal, practical, or intellectual, help guide your other design decisions to ensure that your study is worth doing, and are essential to justifying your study, a key task of a funding or dissertation proposal. In a SoTL manuscript, goals are often discussed in a “Rationale” section early in the paper. Some questions that could help you better define the goals of your SoTL study are:

• Why is your study worth doing?
• What issues do you want it to clarify, and what practices and policies do you want it to influence?
• Why do you want to conduct this study, and why should we care about the results?

Although the improvement of teaching and learning might be considered a “wicked problem” (Bass, 2020)—complex and multilayered, with no definable solution—SoTL can help address it in meaningful, small ways that lead to larger discoveries and theory-building. Spending time to reflect on your own topic of inquiry can contribute to a well-crafted research question, which is essential to designing your SoTL study.

Step 6: What Data I Will Collect

Explore Additional Resources for Step 6

Step 5: What Questions Will Be Driving the Study of the Problem/Need

Explore Additional Resources for Step 5

Research Questions in Quantitative SoTL Studies

In most quantitative studies, the research question is very narrow and specific, allowing the researchers to test their hypothesis through the use of inferential statistics. Often, the operational definition of the variable of interest is included in the question. For example, a Psychology instructor might be interested in whether his students develop self-regulated learning skills after completing a semester-long project. A narrow and specific research question that reflects this theme might be:

How do the self-regulated learning skills, as measured by the Motivated Strategies for Learning Questionnaire, of students in an introductory psychology class change after completing a strategy project assignment?

Scholarship on General Teaching and Learning Theory

Finally, we recommend you frame your SoTL study by drawing upon general literature on teaching and learning, perhaps using a theory or two from these fields as a theoretical framework. The breadth and depth of the teaching and learning literature often causes anxiety for novice SoTL practitioners, but it is not necessary to become an expert in educational psychology before beginning a SoTL study. Steiner and Hakala (2021) discuss several entry points to this literature that are accessible to those outside of the education and psychology disciplines. Tran, Verezub & Fisher (2025) frame their study with discussions of self-determination theory and cognitive load theory, which provides readers a better understanding of how their student participants changed as a result of their educational intervention.

Resources & Suggestions

Reliability and validity in quantitative studies

Price, P., Jhangiani, R., & Chiang, I. (2015). Research Methods of Psychology – 2nd Canadian Edition. Victoria, B.C.: BCcampus. Retrieved from https://opentextbc.ca/researchmethods/.Validity and Reliability Issues in Educational Research McGrath, A. (2016). Searching for significance in the scholarship of teaching and learning and finding none: Understanding non-significant results. Teaching and Learning Inquiry, 4(2), 150–155. https://doi.org/10.20343/teachlearninqu.4.2.12

Trustworthiness in qualitative studies

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education For Information, 22 (2), 63-75. Guba, E.G. Criteria for assessing the trustworthiness of naturalistic inquiries. ECTJ 29, 75 (1981). https://doi.org/10.1007/BF02766777

Classroom Assessments

A source of data for almost every SoTL study is the assignments that already exist in your course. You likely have a variety of ways you already assess student learning in your course. Depending on your research question, you may want to add additional assignments as an intervention. Assessments can include tests, quizzes, papers, projects, and informal “on-the-fly” assignments. For a thorough discussion of classroom assessment techniques, see the classic book by Angelo and Cross (1993).

Step 8: How I will ensure the trustworthiness of the process

The following AI tools can assist you in step 8 of the process of generating your SoTL design: Google Gemini and Chat GPT can be used to identify potential strategies we could implement as researchers to ensure the trustworthiness/validity of a given study. For instance, we could use the following prompt:

What strategies could a researcher use to ensure the trustworthiness of qualitative case study focused on investigating the impact of river water quality on a local community it traverses, emphasizing how this environmental factor influences quality of life, the local economy, and business activities.

Resources & Suggestions

References:

Haigh, N., & Withell, A. J. (2020). The place of research paradigms in SoTL practice: An inquiry. Teaching and Learning Inquiry, 8(2), 17–31. https://doi.org/10.20343/teachlearninqu.8.2.3 Hamilton, M., & MCollum, B. (2024). Moving From “Good” to “Great” SoTL: The Importance of Describing Your Research Epistemological and Ontological Traditions in Your SoTL Scholarship. Teaching and Learning Inquiry, 12, 1–15. https://doi.org/10.20343/teachlearninqu.12.31 Hutchings, P. (2000).Opening lines: Approaches to the scholarship of teaching and learning. Carnegie Foundation for the Advancement of Teaching. Maxwell, J. A. (2008). Designing a qualitative study. In L. Bickman & D. J. Rog (Eds.), The SAGE handbook of applied social research methods (Vol. 2, pp. 214–253). Thousand Oaks, CA: Sage.

Resource on "Critical Friends":

Baskerville, D., & Goldblatt, H. (2009). Learning to be a critical friend: From professional indifference through challenge to unguarded conversations. Cambridge Journal of Education, 39(2), 205–221. https://doi.org/10.1080/03057640902902260