Diving in to flipped learning research

An essential part of being an effective teacher is keeping your knowledge up to date, and one of the best ways to do this is to read and think about research on teaching and learning on a regular basis. Taking deep dives into peer-reviewed, published research on teaching and learning will give depth to your classroom instruction, introduce you to ideas about teaching and learning that you might not have considered otherwise, and spark questions about teaching and learning that deserve investigation in their own right.

Being on familiar terms with education research is especially important for instructors using flipped learning, because the landsape of flipped learning is evolving so quickly. Almost 200 peer-reviewed research articles on flipped learning were published in 2017 alone; over half of all research on flipped learning has been published in the last 18 months; and that body of research is currently doubling every 16 months^[1].

It's exciting that flipped learning is growing this rapidly as an area of research. But that explosive growth also makes it hard to keep up with the research. Most instructors have precious little time to devote to following the torrent of publications coming out every day, and many of us have limited experience with reading this kind of research, making it sometimes hard to really understand what the research is saying or know how best to apply its findings to our teaching. In this article, I want to share some tips for where to find research, how to decide what to read, how to read it, and then how to put it into practice for all of us busy instructors who struggle to find the time or direction to dive in.

What we mean by "research" and where to find it

By "research", we mean any sort of systematic inquiry that aims to answer questions, form new theories, or analyze existing knowledge about a topic. We don't mean published works like op/ed pieces, blog posts, trade publication articles, and so on. These are useful in their own ways, but a true research study -- where the authors pose a question or a problem, situate it in the context of what's already known or believed, and then investigate that question or problem in a rigorous way so that we can trust the results -- has a special richness of information to it that, if we are willing to invest the time and effort, can lead to outsized payoffs in our teaching.

We also usually consider only research that is peer-reviewed, meaning that before it's published, the study has been vetted and approved by a group of knowledgeable peers who have reviewed, often quite rigorously, the study and who vouch for its validity and significance; and we usually only consider works that have actually been published in a reputable print or online venue such as an academic journal. Many non-peer-reviewed publications can be very helpful, and conversely sometimes peer review doesn't rescue a bad study from seeing the light of day. But going through a rigorous peer review process adds a layer of trust that the author's writing is clear and unbiased, the methods used are sound, and the conclusions drawn are valid.

It's quite easy these days to tap into a torrent of peer-reviewed research on flipped learning. Two especially useful free tools that you can use to find research are:

• Google Scholar (https://scholar.google.com/). This dedicated Google search tool just for academic research works just like "regular" Google does, but its results are limited to (usually) peer-reviewed publications. There are only two caveats to note when using Google Scholar. First, like any Google search, you have to be specific and limited in you search query. For example, at the time of this writing, entering in "flipped learning" into Google Scholar yields about 119,000 results. Putting quotes around "flipped learning", so the search is forced to look for that particular two-word phrase, cuts the hits down to a little over 9,000. You can use tools within Google Scholar to narrow this search down more, for example by limiting the year of publication. If you are looking for a specific article or works by a specific author, you should be able to locate it easily. However, the second caveat is that even if Google Scholar returns a legitimate result, the paper itself may be behind a publisher's paywall. More on that below.

• Educational Resources Information Center (ERIC) (https://eric.ed.gov/). ERIC is a free database maintained by the United States Department of Education and functions similarly to Google Scholar. It provides advanced tools for narrowing article searches, for example by looking only for peer reviewed publications, only for articles with certain words or phrases in the abstract of the article, or a combination of these.

Unfortunately, many research journals place their articles behind a paywall, so when you find an article to read through one of the above searches, you may not be able to access it. If you are at a college or university, your institution may have a subscription to some of these journals or can order them for you through interlibrary loan. If not, you might still be able to find another copy of it by doing a web search. While more and more journals are becoming open-access, most still require exorbitant subscriptions to access their articles, and unfortunately this barrier isn't going away anytime soon. The best tactic if you are paywalled out of an article is to contact a colleague or friend at a university who can access it for you.

How to decide what to read (and what not to read)

A typical search for research on flipped learning will yield far more results than a normal person can process in their entire lifetime. Unless you have a small, definitely list of articles you want to read, when you set out to find research that interests you, you'll almost certainly have to decide what small fraction of your search results you'll actually read.

Possibly the simplest way to filter your research is to read the abstracts. These short, usually 100- to 200-word executive summaries give the questions, methods, and results of an article in a nutshell. You can use the abstract like you would a product summary on Amazon or a movie trailer --- to see if that article is something you're interested in exploring further.Use the abstract as a way to decide whether you'll commit the time needed to read and process the article. Many times, even if the article itself isn't necessarily something you want to read in full, the summary provided in the abstract is useful by itself. (Although be careful, because sometimes authors may exaggerate or oversimplify information in the abstract; more on that below.)

Be selective in what you choose to read, because your time is scarce and therefore valuable, and if you decide to put too much in your reading queue, it will likely never get done.

How to read a research article

Once you have found and selected an article to read, it's time to buckle down and read. This is nowhere as easy as it sounds. For many of us, research in education is tantamount to a foreign language. But like a language, education research has its own syntax, grammar, and idioms which, when understood, make for a rich conversation.

It helps to understand the structure of the typical research paper first. Although the specifics vary from journal to journal, most research articles have the following basic parts:

• An introduction that provides the setting for the research, sometimes including the research questions;
• A literature review where the authors situate their study of the research questions in the context of existing knowledge and theory;
• A methods section that describes the study in detail -- the population or sample being studied, the study design, the instruments (surveys, etc.) used, and the methods of data colelction;
• The results section, where the data and their analyses are presented;
• A discussion section where the results section is put into layperson's terms and analyzed on a big-picture level; and
• A conclusion that usually sums up the findings, presents any potential issues with the study, and suggests avenues for future work.

Armed with this knowledge of the parts of a research article, here is a workflow for processing and evaluating a research paper that I use both for my own reading and for times when I am reviewing papers for publication. It helps me get through the reading farly quickly but still allows me to soak up information that I need to learn. Remember this is the main reason we read research articles: to learn something that we can put into action.

1. Read the introduction with the goal of understanding the research questions. Ask yourself as you read the introduction: Why is this study being done? What problem is the study trying to solve, or what questions is it trying to answer? Sometimes it helps to start at the end of the introduction first, because that's where the research questions are often actually posed, and then loop back to the beginning and start over. I also have a rule that if the research questions are not made explicit in the beginning of the paper, I stop reading the paper. A research study that cannot clearly state its own research questions is likely to be a waste of time!
2. Read the literature review with the goal of understanding the big picture, but don't necessarily chase every reference that's mentioned. The literature review is in many ways the most important part of the paper. It sets up the context for the study, introduces vocabulary that will be used, and provides more research that you can read later. But it can also be overwhelming because of the torrent of quotes and references. Don't sweat the details of the references. Instead, look for the big picture: Given the research questions of this study, what seems to be known already, what nomenclature is used to describe the concepts, and where does this study fit into the overall tapestry of research already done? A well-written paper will make all of this clear in simple language, while also demonstrating thorough due diligence in checking what's already published. A paper that can't explain its own context, or which fails to find any prior related research as a point of reference, again is probably going to be a waste of time.
3. Read the methods section to understand the context of the study and start thinking about its limitations. Questions to ask when reading the methods section include: Who is being studied, and where? How are the questions being addressed? What are the overall logistics of how the subjects in the study were studied? What kind of surveys or other instruments were used? Here, it's important to start thinking critically about the study. Suppose the study is asking whether students in a flipped section of intro biology perform better on the final exam than students in a non-flipped section. All kinds of question arise: How many students were studied? Were the flipped and non-flipped sections comparable in terms of size, academic level, gender, time of the day, and so on? Were the two sections taught by the same person? Was the final exam written by that person? Was it graded only by that person or was there someone else? And on and on. A good methods section will address the validity of the study in as many ways as it can and provide enough information that an inspired reader could replicate that study at her own university or school.
4. Skim the results section but do not read it in depth. This sounds lazy and counterproductive. Something called the "results section" sounds like the one part of the paper we must read! But the fact is that much of the statistical esoterica that show up in the results section are generally not for the reader --- they are mostly for the peer reviewers. We readers do need to know, generally speaking, what the results were and if any quantitative results are statistically significant. But the details of the ANOVAs, t-tests, and so on are not meant to convince the reader that the conclusions (see next section) are right; they are there to convince the journals that the study employed good methods. Many lay readers of research get bogged down in the results section and give up, because they don't understand or are bored by the statistics. I discovered that I can still get actionable information out of a paper even if I skim, or even completely skip the results section, if I carefully read the other parts.
5. Read the discussion section with the goal of seeing what the authors think the results say. The discussion section is where the authors unpack the results, so this is worth reading in depth. As with the methods section, the discussion section is a place to put on our critical thinking caps. Ask yourself: Are the authors authentically and scientifically interpreting what their data say? Or, are they evangelizing for their own pet theory, or spinning the results to give a favorable impression of their hypotheses? You may have a very different interpretation of what the authors' data say than the authors themselves, having seen the context and methods of the section. In the example of the flipped vs. non-flipped sections of the biology class, the authors might have found a statistically significant difference in the final exam scores of the two sections with the flipped section performing better. That's good, but more care is needed: Is the result likely to be applicable to courses outside biology, or courses beyond the intro level? Were there any confounding variables introduced during data collection that might have influenced the results? Keep asking probing questions about what the authors are saying.
6. Read the conclusion section with the goal of gaining more context. The conclusions section is where many authors will provide insights to the limitations of the study that we didn't think of ourselves. It's a very good sign when this happens because it shows the authors are willing to be proven wrong. It's also a place where future research questions are posed, so potentially you could pick up where the authors left off.

As a final step, I often loop back around to the abstract and re-read it, having read the whole paper, to see if what the authors say the study found is the same as what the study actually found. Looking at the abstract with fresh eyes can often reveal a bias by the authors toward their own hypotheses, which should drive you to think even more critically about the data and the results.

Acting on what you learn

As we mentioned, the whole purpose of doing and reading research is to learn things you can put into action, especially in your own teaching. Once you've finished reading an article, take a few minutes to think about how what you read could be useful in your own teaching on a practical level. How will you use the results to inform your own practice? Or, could you possibly replicate the study in your own setting and add to the pool of knowledge? One of the great things about flipped learning research is that it's largely done by ordinary instructors in their own classes to understand how best to help their students, so it's wide open for people to join in.

Note: This article appeared first in a slightly different (= better edited) form in Flipped Learning Today, where it's the first installment in a regular column I'll be writing called "Researcher's Corner".