This week in *Forbes* magazine, Tom VanderArk renewed an ongoing and often intense debate about the role of second-year algebra in math curricula in high schools and colleges. It's far from the first time that people have questioned whether Algebra 2 should be required of all students in a school district or college setting. There was pushback in 2013 from the state of Florida and more recently in California. In fact right around 2012-2013, when the Common Core State Standards were emerging (and largely because of those standards, which mandated some items from Algebra 2), resistance to "algebra 2 for all" also started emerging both in these specific cases and in more general op-eds, like this article in Slate and this one in the New York Times, and the issue doesn't seem to be losing any steam.

I think this is an important issue because it addresses a fundamental question about math education in this country, namely: *What will the quantitative education of school kids and college students consist of?* In that sense, the question of Algebra 2 morphs into a battle for the very definition and purpose of mathematics itself. No wonder it draws out strong responses from all sides. Because this is such a vital issue, I wanted to write this post in which I will probably make everyone who reads it at least a little upset: Because while I agree with much of what Tom wrote, at the same time there are significant issues in his article, and more broadly I think all arguments about this issue are starting to veer dangerously away from the people who can contribute the most: **math educators**. So in the spirit of an *amicus* brief, I want here to affirm some things and call other things out to try to guide this debate to a more productive place.

The *Forbes* article in a nutshell says the following:

- Algebra 2 is currently a gatekeeper course, used as a requirement for graduation in many school districts and colleges.
- Its role as gatekeeper is a holdover from a bygone time, starting in the 1990s with concerns brought up by "A Nation at Risk" which then morphed into Common Core standards. But whatever needs used to be in place for requiring Algebra 2, are today not as valid (if not completely irrelevant).
- In fact there is reason to believe that Algebra 2 perpetuates inequities among some student demographics, and anyway the "vast majority" (Tom's words) of people don't actually use the rote symbolic manipulation skills of Algebra 2.
- So instead, students should study coding and
*computational thinking*. And there are some emerging examples in the real educational world where this is happening.

Before I get into the specifics, just a note: I met Tom VanderArk at the Steelcase Active Learning Symposium last fall, and I was really impressed by him. He's a smart guy with interesting ideas about the role of technology in our lives, including education, and I learned a lot from his keynote address. I was also grateful he took some time out to talk with me about my sabbatical at Steelcase, and he was gracious and generous, despite dealing with the usual last-minute chaos of getting ready for a big talk.

So as I point out some issues in the article, this isn't personal. In fact I agree with many of his points. But if we are going to try to make a push for better mathematical education for young kids and college students today, we have to get the details right. Passion isn't enough. So I'll first point out the issues, then talk about what I agree with and where we should go from here.

In my view there are three major issues with the *Forbes* article.

**Issue #1: Defining one's terms and using facts.** The article leads off with an unfortunate quote from venture capitalist Ted Dintersmith:

The tragedy of high school math. Less than 20% of adults ever use algebra. No adult in America still does integrals and derivatives by hand - the calculus that blocks so many from career paths. It remains in the curriculum because it’s easy to test, not important to learn. https://t.co/DS52yevTbR

— ted dintersmith (@dintersmith) January 20, 2018

"Less than 20% of adults ever use algebra" is not a fact. It is a made-up figure without supporting evidence. (If there *is* research out there that supports this number, I am more than happy to be corrected.) Insofar as that "20%" number *does* have any basis, it's probably in Dintersmith's personal experience, so it's more likely to represent confirmation bias than anything else, a reflection of the people he's around rather than "adults" in general. If we're going to make a compelling argument, we can't lead off with opinions disguised as facts.

Of greater concern to me is the use of the term *computational thinking*. I have written a lot about computational thinking in the past (here and here for starters), but despite this and how much I've worked computational thinking into my teaching, the fact is that what we actually mean by this term is very fuzzy today. Jeanette Wing's paper is the usual go-to reference for computational thinking basics, and various folks (including Google) have built on this. But as Lorena Barba --- an honest-to-goodness computational scientist --- has pointed out, many of these recent formulations have drifted away from the original intention of "computational thinking" which is due to Seymour Papert. She writes:

The contemporary message takes an orthogonal direction to Papert, ignoring his Power Principle. It has happened to many powerful ideas when taught in schools (like probability) that they becomedisempowered: “reduced to shallow manipulation that seldom connects to anything the student experiences as important.” [...] Combing through dozens of articles on computational thinking, the emphasis is on problem-solving (low on the problem–project dimension), understanding (low on power), an operational perspective (low on the object–operation dimension), and content priming over media.

The "Power Principle", deriving from a 1996 paper by Papert, is in her words: "What comes first, ‘using’ or ‘understanding’? The natural mode of learning is to first *use*, leading slowly to understanding. New ideas are a source of power to do something."

When the *Forbes* article speaks about "computational thinking", what does it have in mind? Is it Papert's idea of "progressively deepening understanding" --- and if it is, what exactly do we want students to understand? Or is it just teaching kids how to make animated cats in Scratch? Or something in between? VanderArk writes:

That starts with problem finding--spotting big tough problems worth working on. Next comes understanding the problems and valuables [sic] associated--that’s algebraic reasoning. But rather than focusing on computation (including factoring those nasty polynomials), students should be building data sets and using computers to do what they’re good at--calculations. [...] A little coding can be useful to set up big tough problems. A basic coding class or two can be helpful in this regard. The new approach, exhibited at Olin College and signaled by the launch of the Schwartzman school at MIT, is just-in-time coding, a computational resource available across the curriculum--learn the right coding to apply the right tools at the right time to solve the right problem.

To me, this feels poorly-defined and impoverished. Solving important, relevant problems is of course very important and criminally underused in many math classes today. But there is a *lot* more to learning mathematics than just having the ability to pick the "right tool at the right time to solve the right problem". Ironically, being able to choose the right tool at the right time for the right problem is exactly the same goal that some teachers have for learning the very rote symbolic manipulation that this article denounces. (Think "techniques of integration" in Calculus 2.) Do we really want a "computational thinking" that merely swaps one form of rote symbolic manipulation for another? Where's the four-part foundation of decomposition, pattern recognition, abstraction, and algorithm design in other formulations of this idea? And also, where's the *beauty* that is threaded throughout mathematics? Does everything have to be *useful* in order to justify keeping it around? What a dull world that would be.

The article mentions Olin College and the MIT Schwarzman College of Computing, This website at Olin gives a little more insight on what "computational thinking" might look like. But the Schwarzman College example seems hardly informative. Long story short, MIT is spending $1 billion to create a new college which will "lead the world in preparing for the rapid evolution of computing and AI." Fifty (!) new faculty positions will be created and it promises to "educate students in every discipline to responsibly use and develop AI and computing technologies to help make a better world". But how this will be done and whether it's an externally valid approach to teaching "computational thinking" (whatever this means) is unclear. For example, those faculty --- will they be actually teaching, or just doing more research? And students --- what will they be actually learning, when will they learn it, and how will they learn it? A $1 billion dollar price tag doesn't give the answer and throwing that much money into an already-rich institution for unclear learning outcomes doesn't strike me as a decisive blow toward making math education better in this country.

*To sum up*: If we are going to make a convincing case for a better approach to math education, it has to start with facts, clear definitions, and transparent strategies and learning outcomes. That gets me to the next issue:

**Issue #2: Conflating "Algebra 2" with symbol manipulation.** What exactly *is* Algebra 2? Usually it is a big course with a lot going on inside it. The most memorable (for better or worse, mostly worse) thing that happens in Algebra 2 is a lot of rote symbolic manipulation, like factoring cubic polynomials or other actions which can, I will readily agree, be charitably categorized as party tricks (for very special kinds of parties) and nothing more. In the article, and in many fellow-travelling articles, Algebra 2 is equated with symbol manipulation.

The problem is that this is too simple. Take a look around at people's Algebra 2 syllabi, and you will find Algebra 2 tends to be a mix of topics, some of which are neither clearly interesting nor especially useful (e.g. factoring cubics), along with others that many Algebra 2 detractors would say are vitally important for people to know today, such as:

- Probability and statistics
- Basic function types (linear, power, exponential, etc.) and transformations of, and modeling with those basic types
- Linear programming and linear systems
- Matrices

If you get rid of Algebra 2 to remove the burden of symbolic manipulation skill, you also get rid of the above ideas, unless you create something else in its place. But I don't see these ideas coming up in what's called "computational thinking" here, which remember is characterized as "understanding problems and valuables [*sic*] associated" along with "just in time coding". How is a student supposed to "understand" the structure of a problem without some formal study, at least self-study, of these Algebra 2 topics?

The quote from Ted Dintersmith at the beginning of the article said that very few people "use algebra", but what does "using algebra" mean? If I think critically through a complex problem in life, am I "using algebra"? If I am looking at a data set and trying to decide what kind of regression model to use --- linear vs. power vs. exponential etc. --- am I "using algebra"? Because I learned to think critically through complex problems and learned about function types in algebra.

To sum up: Algebra 2 is not just one thing, or as the article puts it, "a course on regurgitated symbol manipulation (Algebra 2)". It *can* be just mindless symbol manipulation, but it often isn't, because there are teachers out there who have the vision to take it to another level.

And that gets me to the third and perhaps most important issue:

**Issue #3: The lack of math educators' voices.** I haven't read every single article about Algebra 2 in the news, but in the ones I have reviewed for this post, there is a distinct and alarming absence of the opinions and voices of *actual math educators* in these public debates --- i.e. people who are now and have been in the trenches of education, working with students and dealing with the on-the-ground issues of teaching mathematics. The closest we get to talking to an actual mathematics teacher in the Forbes article is a quote from Dan Meyer (which was misattributed before it was fixed); venture capitalist Ted Dintersmith, who has to the best of my knowledge never set foot in a math classroom as a teacher, is quoted twice as much as Dan in the article. The main character in the Slate article is a professor emeritus of political science with zero math teaching experience. And so on.

This doesn't mean that Tom VanderArk or Ted Dintersmith have nothing useful to say about math education. (Far from it; keep reading.) But it does signal a frustrating and dangerous trend where the opinions of those who know best about the nuances of mathematics education in this country and who work daily on the human level to work those nuances out, are being ignored in favor of the opinions of venture capitalists, Silicon Valley types, and sometimes even celebrities.

To sum this up: If you want to make a compelling case for improving math education in this country, *talk to math teachers* and see what they are doing, what's working, and what's true. Linking to news articles about schools doesn't count. Get in there, up close with teachers and let them inform you.

And now that I've called out numerous aspects of the *Forbes* article, let me express how much I agree with its basic message. (I told you, this post will make *everyone* at least a *little* mad.)

Algebra 2 enjoys a privilege in our school and sometimes college curricula that it didn't earn and largely doesn't deserve. While it can contain some ideas that are vitally important (see above for a list) and while you can make an argument that the symbolic manipulation portions of the course are good for building critical thinking skills, or willpower or character or grit or whatever you want to call it, the fact remains that in many people's experiences, Algebra 2 is all about rote symbolic manipulations, and whatever else of value may be in the course are drowned out by the sheer pointlessness of having to factor cubics, simplify $(x^4)^{100} (x^6)^{1/2}$, and so on.

Algebra 2 enjoys this privilege because Calculus enjoys a similar state of unearned privilege, and in many cases Algebra 2 is a requirement because Calculus is seen as the apotheosis of the high school math curriculum; it's also usually the core of the college math major, although courses like linear algebra are probably more important today. David Bressoud among others has written powerfully about the drive to calculus that has reached a fever pitch at the high school level and consequently put college calculus at a crisis point. Calculus has a central position that is a holdover from the Space Race days; we math teachers tend to approach calculus primarily as symbol manipulation; neither of these two defaults are often questioned, and since Algebra 2 is considered the place where kids are supposed to learn the basic symbol-pushing skills "needed" for calculus, guess what Algebra 2 typically looks like.

But, there's hope: Bressoud also has been involved in changing calculus at his home institution of Macalester College so it's now a modeling-centered course with an increased focus on conceptual understanding and applications and a reduced emphasis on symbolic manipulation. It's possible, in other words, to dial back the rote mechanics of Calculus, reframe the subject as a *way of understanding the world* through modeling in context, and still learn all the things we say that symbolic skills teach --- critical thinking, attention to detail, grit, whatever.

If it can be done with Calculus, it can also be done with Algebra 2.

It's also very important to note Tom's reference to Conrad Wolfram and the injunction to "teach as if computers existed". I actually wrote about this over eight years ago, and reprised the idea in this post which eventually turned into a TEDx talk last year. No biology teacher would dream of teaching that subject without a microscope out of the belief that students need to learn critical thinking by doing observations without technology. Similarly, no astronomer would teach her courses without a telescope. And yet when it comes to the computer (and particular tools like SageMath, Python, and Jupyter notebooks) which is a basic tool for exploration and analysis in the study of mathematics --- the microscope/telescope of the math world --- we fear its use and worry that if we teach with it, if we teach students how to use the tool like a professional, that they will somehow be worse off. This is crazy and unprofessional and possibly unethical, and we need to turn it around and be braver and more creative than that.

The way forward from here on this debate can proceed like this:

- Let's be clear about what Algebra 2 is and is not, and also be clear about what we would like to replace it with, if anything --- or what parts of Algebra 2 should be phased out and what parts should remain.
- Let's focus our decisions based on sound research, enriched by the lived professional experiences of math educators who are getting it done with real students every day. Bring others into the mix in this debate but resist the temptation to be over-awed by rich people and thought leaders.
- Let's avoid buzzwords like the plague and insist on concepts that mean something.
- Let's explore how other people have solved this problem. At my university, for example, students have to complete a quantitative reasoning unit to graduate; College Algebra (basically Algebra 2) is
*one*way to satisfy it, but there are*eleven*other courses that do, including courses in logic or computer cartography. So perhaps there's a middle ground between forcing all students to take Algebra 2 and forcing them to learn to code. - Finally, let's try to get
*students*a little more involved. Missing from much of this debate is a diverse collection of student voices, some of whom might want math completely gone while others find something meaningful in that which we want to discard. Because if you center the debate on "What serves students the most?" then you will never go*too*wrong.

After doing speaking engagements and workshops for a few years now, I've decided that there is really only one criterion for deciding if what I'm doing with faculty is successful: **Behavior change**. If I fly into a campus and give a workshop that generates tons of enthusiasm, but then all the faculty's teaching goes back to the *status quo ante* as soon as I board the plane back home, it doesn't matter how well-received it was: I failed. I merely perturbed the system; I didn't actually change it. On the other hand, when I get an email from a workshop participant 2-3 months later who has been trying out the ideas of the workshop in their teaching and is curious about or stuck on some particular, I know that the workshop was a success, because someone's behavior has changed, and that will start a cascade of other changes that will make higher education better.

If individual behavioral change is the building block, then individual **habits** are the atoms that those blocks are made out of. In fact, I recently finished a book with this very idea in its title: *Atomic Habits* by James Clear. Clear's idea in the book is that behavioral change begins with small habit changes that lead to outsized results. Or as he puts it, our behaviors are lagging indicators of our habits. So if we want to improve teaching and learning in higher education, let's start with the habits of teaching that higher education instructors have. How do we do that? Clear, in his book, lays out some helpful frameworks for thinking about habits and how to alter them.

First, habits follow a consistent cycle of **cue -> craving -> response -> reward**. Some kind of cue sets off a craving; we respond to that craving with a behavior that will change our state to one of satisfaction; and the reward is that satisfaction. Habits are very efficient this way; Clear points out that we wouldn't have evolved the capacity for habit formation in the first place if habits weren't exceptionally effective at solving certain kinds of problems. But thankfully, that cycle admits the possibility for a hack. Clear suggests a formula for changing up how we respond to a craving, namely the simple sentence: **When I do x, I will do y.** To change a habit, or start one, identify the relevant cue, the response you *wish* to have, and write it down, like a contract. Then follow that rule until it sticks better than the one you replaced.

I think a lot of teaching at the college level today is done not because professors have looked at a variety of techniques, tried some out, and then made a rational and student-centered decision about it --- but merely because professors get stuck in habits. We walk into a classroom, or start preparing a lesson; that cue makes us crave the path of least resistance because we're overworked or have many other responsibilities; we then teach as we were taught. These habits are so strong that it takes purposeful effort to replace them with something better. This is why in my One Year Plan for flipping a classroom, Step 1 is to start a year in advance simply replacing unproductive habits of teaching with ones that better support active learning. What are those? Here are three suggestions that I think get us in the right direction.

In the literature review I wrote with Anat Mor-Avi on active learning classrooms, one of the things we discovered was the power that the physical layout of a space can exert over teaching and learning. When you walk into a classroom arranged in traditional row-by-column seating all focused forward on a large space only inhabited by an instructor, it creates an expectation of what's going to happen in that space: Students will sit and face forward while the instructor lectures. Anything else, while possible, is not the default and therefore produces a sense of wrongness.

So the first habit instructors can get into is simply **changing the space around to support active learning**. In our formula above:

When I enter my classroom, I will rearrange the furniture to get students in small groups.

This is often a really easy habit to enact. We don't have active learning classrooms in my building, but the furniture in the room I usually use consists of fairly lightweight chairs and rectangular tables; the tables can be put side-by-side to form a square. A couple years ago I started off the first day of classes by having students rearrange these tables into eight squares with four students each at them. The resulting configuration took about 30 seconds to attain (when we all pitched in) and made active learning a lot easier and more effective because students can communicate with each other better, and because I can reach students without having to squeeze down an aisle or row.

If the professor who uses the classroom after you doesn't like this kind of configuration, just end the class a minute early and move things back. If you teach in a room where stuff doesn't move, you can instead have a plan for getting students into groups by zones or proximity.

For those whose main teaching technique is lecture, it might feel like I and others just want you to jettison the lecture and go all-in on active learning right now. I actually recommend *against* that. The evidence is clear that active learning helps students, but I think the best argument for active learning is its own success, which can be realized through small, intentional changes that are more comfortably manageable from your current approach.

Jim Lang's book *Small Teaching* is the pre-eminent source for these simple, inexpensive modifications to traditional pedagogy; this handout and chart from the University of Michigan CRLT is handy as well. Whatever approach you choose to take for "small active learning", it takes intention to build it in. So using our habit formula:

When I plan out my lecture, I will build in breaks every 15 minutes for active learning, and also plan out the activities.

That 15-minute figure is arbitrary, so change it if needed. But, intentionally stop lecturing at reasonable intervals to let students *do something* with what you've lectured about. Don't just stick with the habit of writing a lecture outline and that's that; break the habit up by breaking your classroom time up. Also, give the activity the same level of planning (if not more) as the lecture, instead of just telling students "Break up into groups and discuss".

Another habit to go along with this is:

When I start my lecture in class, I will also start a timer for 15 minutes; and when the timer goes off, I will transition to the activity I planned.

Because it's one thing to *plan* an activity and another to discipline yourself to actually *do* it. It's too easy to just decide in the moment to keep lecturing and skip the activity. Make yourself stop.

One great way to implement small active learning in your class is through frequent, low-stakes quizzing, either on paper or electronically. The benefits of frequent low-stakes assessment are becoming more clear every day. They provide meaningful and effective practice for students, and just as importantly, they provide data for you on how well your teaching is going. And that leads me to the final suggestion:

After class is over, I will spend 15 minutes as soon as is practical to review the results of students' active work, reflect on the results, and decide what to do about those results.

In other words, make a habit not to rely on felt measures of enthusiasm but on what students actually know or don't know based on their work --- and then make real plans that affect your future instruction based on the results.

For example, let's say I teach a 50-minute calculus class that consists of 15 minutes of lecture, followed by a 5-minute ungraded quiz over the derivative rules of the day, followed by that same 15+5 minute block followed by a 5-minute exit ticket activity. (I know that doesn't add up to 50 minutes; I have slack time built in.) Immediately following the class --- or in the next available 15-minute block --- I am going to sit down, look through the results, and see how things went. (In some cases you might even be able to stop in the middle of a class and evaluate these data, for example if you do the low-stakes quizzing using clickers.) If students did well on the quizzes and seemed to articulate the concepts well at the end, I know my lectures were OK. If most students did horribly on those quizzes, then I know I need to do something about it --- change the lecture schedule, build in a practice day... *something* besides just moving on to the next lecture.

This is teaching like a scholar because we are using *evidence* to guide what we do, not feelings or faith. These have their own place in teaching, but they are horrible metrics for teaching effectiveness.

By focusing on building habits that line up with the identity we want to have as teachers --- and hopefully that means we want to be teachers who care about students and act accordingly --- I think meaningful change can come to higher education one faculty member at a time.

*Any other ideas for habits to form? Leave them in the comments.*

One of the most complex issues in teaching mathematics is how to handle examples. On the one hand, examples are important in mathematics because their construction is usually how we make sense of abstract ideas. On the other hand, students can get the wrong idea about examples. They can think that just by seeing enough examples done by a teacher, they will gain understanding of the subject; or that course assessments will be about completing examples just like the teacher did, and so they focus on replicating the teachers' examples instead of learning from them.

Last Fall semester I really felt this struggle as I taught my Modern Algebra 1 class. It's an upper-level course focused on number theory, ring theory, and fields. So there's a lot of abstraction and the only way to truly grasp the subject is to work with a *lot* of examples. Where I fell short in this class, and what I learned from the experience, is something obvious: *When I'm the one doing all the examples, the students aren't learning the math as well as they could.* Or as a colleague of mine put it, *the one doing the math is the one learning the math.* Whenever students ask to "see" more examples, I work them out, and students watch. But watching someone do a thing is not the same as learning the thing.

Why don't we have students generating their own examples more often? And how might such a strategy of student-generated examples look in practice? After my Fall teaching experience, I set out to see what research has been done on these questions, and I found this paper that I wanted to break open here today:

Anne Watson & John Mason (2002) Student‐generated examples in the learning of mathematics, Canadian Journal of Math, Science & Technology Education, 2:2, 237-249, DOI: 10.1080/14926150209556516

Link to paper: https://bit.ly/2SZ5CtX

This paper is a little different than other research articles I've written about here, in that it's a *qualitative* study. Qualitative research focuses not so much (or at all, in this case) on numerical measures of variables and their statistical differences, as it does on making careful observations of phenomena and then systematically analyzing what's observed. You'll often see qualitative research aimed at exploring questions that are difficult or impossible to operationalize, through anthropological-style observations, interviews, surveys, with the agenda of simply asking questions and making sense of the answers.

Some people think this makes qualitative research less rigorous than quantitative research. That's not the case. In my own research experience, doing qualitative research *well* is a lot harder than doing quantitative research; and doing quantitative research poorly is just as easy as doing qualitative research poorly. They're just complementary practices (and you'll often see them mixed together) and sometimes one is simply a better tool for the job.

Back to this study: The authors worked with kids (the study mentions 11- and 12-year olds in a few of the observations) on in-class exercises where students were asked to generate examples of five different kinds:

: Examples that involve executing and reversing processes, and "doing and undoing".**Experiencing structure**: Examples that elicit different kinds of examples of the same concept from different learners, or multiple representations of the same idea, or different questions that give the same answer.**Experiencing and extending the range of variation**: Examples that result in seeing a pattern in the examples that are produced.**Experiencing generality**: Examples asking learners to illustrate new concepts and invent notation or terminology to explain a phenomenon and then compare to standard mathematical notation and conventions.**Experiencing the constraints and meanings of conventions**: Examples that satisfy some conditions but not others, or those that exemplify "what is and what is not" or what cannot be done within specified constraints.**Extending example-spaces and exploring boundaries**

Each kind of example accesses different cognitive aspects of the example-making process. It's appropriate to give examples of each kind of example here.

The *experiencing structure* example given was about solving linear equations. They were asked to start with a value of $x$ stated as an equation (like $x = 5$) and then build up a linear equation by repeatedly doing the same operation to both sides. So start with $x=5$, then add 4 to both sides to get $x + 4 = 9$, then subtract $10x$ from both sides to get $x + 4 - 10x = 9 - 10x$, and so on. Then at some point they stopped and presented their equations and the rest of the class asked to solve them. The question came up --- if you were given this final equation and asked to solve for $x$, how would you do it? Well, for the group that created the example, it was easy --- just reverse all the steps that were used to build up the equation in the first place. For the rest of the class, the process was about figuring out what those steps were. Both groups experienced a structural process that generalizes to solving other linear equations.

The *experiencing and extending the range of variation* task had students give examples of multiplying multi-digit numbers together and making visible their thought processes for how this might work. I've seen this example myself in my own kids' school work. When asked to compute $89 \times 4$, some will compute $80 \times 4 + 9 \times 4$. Others will compute $90 \times 4$ and subtract another $1 \times 4$. By letting kids make the rules and then making their work visible, the entire group is exposed to a multiplicity of examples, some of which might "click" with a student who wouldn't have thought of it otherwise.

In *experiencing generality* tasks the researchers used a method they called "particular-peculiar-general". The specific task they gave students was:

- Write down a particular number that leaves a remainder of 1 when divided by 7.
- Write down a number that leaves a remainder of 1 when divided by 7 which is peculiar in some way.
- Write down a general form of a number that leaves a remainder of 1 when divided by 7.

Students mostly contributed small numbers for the first task like 8 or 15, then weirder ones like 700001 and 1 for the second task. The authors said that discussion ensued about how to handle negative numbers, and that "the third request followed easily from these contributions". (The latter I have to admit I'm skeptical about, because forming the right generalization for these numbers isn't easy.) This is an instance of using examples in a different direction --- not constructing particular instances from general concepts but using particular instances to *arrive at* the general concept, which is something that would be right at home in my Modern Algebra class.

In *experiencing the constraints and meanings of conventions* tasks, the teacher gave students a general idea --- in this case, to represent a function whose output is equal to the input plus 3 --- and have students generate their own representation of the idea. Students came up with some wildly different ways to think about these functions; the paper shows one result involving a sequence of nested boxes that eventually results in the graph of the function $y = x + 3$. After looking at student examples, the idea is to debrief the activity by comparing representations, discussing their pros and cones, and then comparing student representations to mathematically standard representations. The idea is that

If students have had to develop notations for themselves, and compared their usefulness, they are more likely to understand and accept the strengths and idiosyncracies of conventional notations.

And who among us math teachers has not had to deal with students struggling to understand the "idiosyncracies of conventional notations"? I'm looking at you, inverse functions and logarithms.

Finally, the *extending example-spaces and exploring boundaries* tasks are what I've had my students do in the past: Build a sequence of examples that satisfy increasingly strict constrains. In the paper, students were asked to draw a quadrilateral, then a quadrilateral with a pair of equal sides, then a quadrilateral with a pair of equal sides and a pair of parallel sides, then a quadrilateral with all these features and a pair of equal opposite angles. The idea with this kind of example is to explore the space of possible examples of a concept and discern what's possible and what's not possible.

So, what did the researchers find when they gave these kids all these example-generation tasks? Again, while no quantitative data were collected, the researchers uniformly observed that

Students were actively, noisily, and verbally struggling with attempts to reorganize what they knew to fit the kind of example the teacher was seeking. Students were led away from limited perceptions of concepts and towards wider ranges of objects. They restructured their ways of seeing and experienced the creation of mathematical objects and notations.

Some caveats are in order here: This is great, but it is a long way from a systematic analysis of student observations, and unfortunately it's pretty much the only general conclusion that the researchers draw. They also tend to align their observations with their own experiences as mathematicians: *In our own mathematical training, example-generation helped us, and look! It made these kids better too* --- which sounds to me like confirmation bias. I'd like to see this kind of study done again with tighter controls on the observations and analyses, and in fact this has been done --- actually Watson and Mason went on to write an entire book about this subject.

For me, the main importance of this article is that it validates the idea that while instructors may need to give examples to students at times --- and there is some reason to believe that instructor-led examples can be helpful in reducing cognitive load for students --- there is also great value in placing the main work of example generation into the hands of the students. It also sparks ideas for how we might do this on a regular basis in our teaching. Watch this space for some future posts on specific activities for different classes; and leave your own ideas in the comments.

]]>*Welcome to another 4+1 interview. This time, we're talking with Bonni Stachowiak, director of the Institute for Faculty Development at Vanguard University of Southern California and host of the popular and influential Teaching in Higher Ed Podcast. Bonni is someone in higher education who leads by example, and with the human element always at the forefront. I've been on her podcast a couple of times and I am really honored to return the favor now.*

**What got you started with the Teaching in Higher Ed podcast? What gave you the idea for doing the podcast, and what was it like getting the podcast up and running?**

My husband Dave had been running his Coaching for Leaders podcast for three years by that point. There really weren’t podcasts that focused exclusively on teaching in the context of higher education at that point, except a couple that also addressed other audiences like parents and students. Some of the big higher education news organizations had podcasts that talked about policy, but I was interested in conversations about teaching.

I had no idea what I was doing at first.

Except that I could go off of some of what had worked for Dave with Coaching for Leaders. One big boost in the beginning came from people who accepted invitations to be on a podcast that they hadn’t ever heard of before.

The early guests were also generous about recommending other people to be on the show. I had no idea what was in store and am eternally grateful for the transformation it has provided me.

**What are some things you've learned about teaching and learning from your interactions with the guests on TiHE that really stand out to you?**

While much has changed about the podcast over the years, one thing that has remained constant is that the start of each show, I talk about the show being a space to explore the art and science of facilitating learning.

What continues to stand out to me is just how much teaching is both an art and a science.

I enjoy those people who have helped me dive deeper into the scholarship of teaching and learning (SoTL). Yet, it is also invigorating to get to talk to someone who is playfully experimenting in their teaching and reveling in what Amy Collier (and Jen Ross) refer to as not yet-ness.

When Amy was a guest on episode #070, she stressed:

When you embrace not yet-ness, you are creating space for things to continue to evolve. – Amy Collier

Each student and every class community is different. We can build up a repertoire of approaches that have been demonstrated to improve learning (such as retrieval practice). This is a practice worth pursuing. Yet sometimes we will be playing the role of artist and creating something unique to a particular set of circumstances, without having the assurances that it is going to work.

**Are there any funny or embarrassing "outtake" moments from TiHE that you can share?**

The most memorable one comes from my conversation with Ken Bain. What the Best College Teachers Do was the first book I ever read about teaching in the context of higher education. I was incredibly nervous to speak with him.

At the end of the conversation, he was mentioning that he wished he could have said more. I joked that through the “magic” of podcasting, we could make that happen. Before I had a chance to press record, again, he started sharing the specifics about what he would like to add.

Fortunately, I type pretty quickly. I captured the bullet points of what he wanted to add and started to prepare to hit the record button, again. One big topic he wanted to do had to do with Eric Mazur, who had won The Minerva Prize for excellence in teaching.

I was unfamiliar with Mazur and also with the Minerva Prize.

Thus, when I pressed record and confidently resumed my inquiry, I started by asking him about the Manure Prize. I said it three times before Ken gently let me know that it was actually called the Minerva Prize. Autocorrect had changed Minerva to Manure and as I looked back over my notes, I didn’t know any different.

I could have just left that part of the interview out, entirely, when doing the edits. However, it was so illustrative of what I have always wanted to model about teaching with the podcast. I left it in and instead, eventually created the Manure Award to recognize people who have been vulnerable enough to experiment in their teaching and have experienced those inevitable failures that result from those endeavors.

**If you could have any one person, living or deceased, on the podcast to interview about teaching and learning, who would it be and why?**

Because the necessity for vulnerability is so central to what I believe about teaching, I would treasure the opportunity to speak with Brené Brown on the podcast. I just finished reading her latest book, Dare to Lead, and consider it to be essential reading for people who want to maximize the potential of people on a team.

**BONUS +1: What question should I have asked you in this interview?**

I mentioned earlier that I had wanted to start a podcast about teaching, yet I also have decided to include a focus on productivity. Some people might wonder why I considered it important to include that topic in some of the episodes, in addition to talking about teaching.

Robert, you wrote so powerfully conducting your trimesterly review after getting the news about your heart health issues. As we corresponded about me guest posting during your time of healing after the surgery, you said your friends were going to think you were nuts for spending time being sure stuff keeps getting posted.

When we regularly reflect on what is most important to us and have systems in place to continually be moving forward toward those aims, I truly believe we have more peace in our lives. It is a regular practice of aligning our sense of purpose with how we invest our time, energy, and attention.

**If you enjoyed this interview, check out these other ones from the past:**

- Josh Eyler on what he learned about teaching while writing a book on teaching and learning.
- Andrew Kim on the role of space and design in education.
- Lorena Barba on Jupyter notebooks and open science.
- T.J. Hitchman on inquiry-based learning.
- Linda Nilson on specifications grading and grading in general.

*I'm happy to reintroduce 4+1 interviews after a few months' hiatus, where I find someone who is doing something interesting in the world and then ask them four questions, plus a special bonus question at the end. This time I'm pleased to interview Josh Eyler, Director of the Center for Teaching Excellence at Rice University and author of the new book How Humans Learn: The Science and Stories Behind Effective College Teaching. *

**What was the catalyst for your book How Humans Learn?**

When I first moved into the world of teaching and learning centers, I did a lot of reading about pedagogical strategies, both general and discipline-specific techniques. There are some really wonderful resources out there about these methods--what works and how to implement them in your classroom. But one question continued to haunt me: *why* do some strategies work and others don't? I started doing some of the research that led to this book as a way to answer this question, and it took me into fascinating fields that I had never explored before--fields like developmental psychology and biological anthropology.

**Were there any issues or questions that you wanted to address, or wish you addressed, in the book but weren't able to?**

There is a point in my chapter on curiosity where I start to talk about our educational systems and why curiosity might fade into the background as students proceed through their educational careers. That section was originally much longer and explored all of the factors that might play into this, including teacher education programs and standardized testing. The section was eventually cut because of length, but I wish I had been able to keep it, because I think these are conversations we need to have as educators.

**What were some important things that you learned while writing How Humans Learn?**

I learned *so much,* which was one of the joys of writing the book. It took me 5 years to write this book, because I needed to dive deeply into these scientific disciplines that were largely unfamiliar to me and to become acquainted enough with their methods and findings that I could make claims that scientists felt were credible. That was really important to me. They didn't have to agree with me necessarily, but they did need to find what I was saying to be credible. In a sense, then, I was in a position similar to a student in a series of introductory courses--I needed to build frameworks and connections that were not yet in place, and that took quite a while. The things I was learning about were so interesting too! If I had to single anything out, I would have to say that learning more about evolutionary biology gave me a greater appreciation for the natural world and our fellow animals than I had before I started studying it. I'm grateful for that, and for everything else I learned along the way.

**Your "day job" is Director of the Center for Teaching Excellence at Rice, but you're also listed as an adjunct associate professor of humanities. What are some concrete ways that the lessons of your book have made their way into your own classroom teaching, and what are some aspects of your teaching you'd like to improve as a result of your research?**

Two big things have changed in my own teaching and lots of smaller things too. The big things, though, include even more prioritization of student agency. In the past, I've had students collaborate to design exam questions and things of that sort, but now we do an exercise at the beginning of the semester where we work together on a classroom compact--a document that lays out ground rules for discussions, guidelines for technology use, etc. Students generate the document together, and I give some feedback along the way. This gives them a sense of ownership over the class climate, and it sets the tone from the very beginning that I am interested in what they have to say and in their own contributions to the course. The other major thing that has changed has been my strategies for grading. My research on failure had a tremendous impact on me, and now all of my undergraduate courses are graded using the portfolio model (which aligns well with the writing courses I teach), and all of my graduate courses utilize contract grading. I want to continue pushing even more toward ungrading over the next few years. As for what I'd like to continue improving on, well, the answer is pretty much everything! I have so many ideas percolating now after finishing the book, but I'd like to pay some immediate attention to the assignments I'm developing and the ways in which they help my students engage their curiosity and take intellectual risks.

**BONUS +1: What question should I have asked you in this interview?**

How about: **what is the best part about writing a book?** My answer to that would be that the best part is being finished with it and getting the opportunity to have conversations with others about it. I've already learned a lot from those who have read it and talked to me about the extent to which the ideas fit in their own courses and teaching philosophy.

**If you enjoyed this interview, check out these other ones from the past:**

- Lorena Barba on Jupyter notebooks and open science.
- Eric Mazur (audio interview) on peer instruction, Montessori education, and Super Bowl picks.
- Diette Ward on how libraries support higher ed.
- Derek Bruff on directing a Center for Teaching and classroom response systems.

The longer I use specifications grading, and the more I see how differently students experience college courses that use mastery grading compared to courses that don't, the more I believe that the reform of our grading practices is an urgent ethical imperative. Like I said on Twitter last week:

Not just less important - it's clearer every year to me that grades are increasingly corroding education and student well being. The alarm bells are getting louder.

— Robert Talbert (@RobertTalbert) January 18, 2019

I switched from traditional, points-based, no-revision grading a few years ago to specifications grading because I had a strong sense that not only was traditional grading uninformative (large numbers of false positives and false negatives, and no clear link between the grade and what the students can do) but actively harmful to many students in many ways, one of the biggest being *motivation*. When I used traditional grading, students always seemed motivated not by the promise of learning the subject but by the inner game of scoring enough points in the right ways to get the grade they needed to move on --- or else they had no motivation at all.

This intution that traditional grading is demotivating was just that: An intuition. But a study I came across recently gives results about the real effects of traditional grading on motivation.

Chamberlin, K., Yasué, M., & Chiang, I. C. A. (2018). The impact of grades on student motivation. Active Learning in Higher Education, 1469787418819728.

Link to paper: https://journals.sagepub.com/doi/pdf/10.1177/1469787418819728

The authors in this study investigate how "multi-interval" grades (read: the A/B/C/D/F system) affect the basic psychological needs and academic motivation of students when compared with "narrative evaluation", where the instructor gives students verbal feedback both instead of, and in addition to multi-interval grades.

The theoretical basis of the study is self-determination theory (SDT). This framework is where we get the concepts of *extrinsic* and *intrinsic* motivation, where people are motivated to complete a task either by an external reward or for the sake of the task itself, respectively. (For more background, I wrote about SDT and flipped learning in this post.) According to SDT, there are three basic psychological needs that learners have while they are involved in a learning process: **competence** (the need to be good, or at least feel that they are good, at what they are learning), **autonomy** (having choice and agency), and **connectedness** or "relatedness" (being psychologically connected to others while doing the task). Essentially, the more these three needs are met in a learning process, the more intrinsic motivation the learner will experience; the lack of satisfaction of these needs leads to less intrinsic motivation, either in the form of extrinsic motivation or no motivation at all.

They studied 394 students at three different universities. One of those universities gave exclusively multi-interval grades in its classes; another had institutionally eschewed multi-interval grades and used only **narrative evaluations** in its courses. This is where instead of a grade, students get verbal feedback (that is honest, detailed, constructive, and actionable) on what they did and what they need to do. The third used a mix of narrative evaluation and letter grades. The students were given two surveys on academic motivation, and a subset of those underwent semi-structured interviews to dig deeper.

The results are a sobering indictment of traditional grading. Here are just a few that stood out.

Students were asked, among other things, about what information (if any) they got from their grades, whether their grades affected their decisions on what classes to take, and whether their relationship with grades had changed since high school. The prevailing opinion was that grades do *not* convey "competence-enhancing feedback" that can be used to improve; most students could not give any examples of how they used grades to improve their learning. Worse, the information that grades *did* give students tended to be negative signals about the students' self-worth. High-achieving students experienced pressure to achieve high grades; low-achieving students felt condemned by their low grades. All students associate the word "stress" with grades far more frequently than any other concept.

Moreover, traditional grades actively decayed students' sense of autonomy because many times the grade they get and what they have learned seem unrelated. As one student said:

And it was actually pretty frustrating because it felt like even in classes where I was really into the content and worked really hard I came out with a B+. And in classes that I didn’t care about and didn’t work very hard I still got a B+.

Grades worked against relatedness as well, as expressed by some students who described how their relationships with their parents suffered when their grades were poor.

The authors also noted that when discussing traditional grades, students readily adopted capitalist-style business language, for example referring to "cost-benefit analysis" and "payoffs" in describing how they approach class. That's strategic learning and extrinsic motivation taking hold.

The results from students who experienced narrative evaluation were almost completely the opposite of the results from multi-interval grading. Every "narrative evaluation" student interviewed expressed that narrative evaluation gave them usable information about their competence and were more useful than multi-interval grades. The study found strong links between narrative evaluation and enhanced competence, autonomy, and connectedness, and many of the students commented about how narrative evaluation built *trust* between the student and the instructor --- even if the feedback was largely negative.

These results came not just from the interviews but also from the quantitative results of the surveys, with statistically significant differences in measures of academic motivation found between students from traditional grading backgrounds versus narrative backgrounds (with narrative grading leading to higher indicators of motivation). Students from the university that used mixed grading methods experienced some of the benefits of narrative evaluation, but also some of the detractions of traditional grading --- and although the study didn't say this directly, it seems clear to me that the detractions happen because of the letter grades. (If you put a student in a "mixed" environment and give them good narrative evaluations followed by a "B+" grade, guess what the student will tend to focus on?)

So what do we do about this? For me, the course of action is clear: **We need to walk away from traditional grading** --- in which I include not only multi-interval letter grades but also grades based on statistical point accumulation. We've seen enough. Grades are harmful to students' well-being; they do not provide accurate information for employers, academic programs, or even students themselves; and they steer student motivations precisely where we in higher education do *not* want those motivations to go. There is no coherent argument you can make any more that traditional grading is the best approach, in terms of what's best for *students*, to evaluating student work. If we value our students, we'll start being creative and courageous in replacing traditional grading with something better.

Cue the objections about how this can't be done because of transfer credit issues, making non-traditional grading work at scale, etc. I agree partially, in the sense that this move is a long sequence of small steps. The article here is similarly pragmatic and gives some good advice:

Few universities are likely to abolish grades. However, universities should question the conventional use of multi-interval grades and consider their advantages and disadvantages in different departments, years of study, courses and learner types. For example, there may be specific courses or programs [...] in which cultivating deep learning and motivation may be more important than standardized communication of performance to external audiences. For such courses, greater use of narrative evaluations (as opposed to multi-interval grades) may be warranted. In addition, withholding grades from students or providing narrative/ written feedback several days prior to the grades may help students focus on mastery-related learning goals rather than extrinsic rewards.

I'd add the following ideas that I've learned from using specifications grading and hearing about how others use this and other forms of mastery grading:

- It's possible to keep the A/B/C/D/F system for reporting
*semester*grades, but use narrative evaluation and mastery grading instead of points and statistics to determine students' grades. Here's an example.^{[1]} - Do what the article suggests and start changing your grading practices over to something less focused on letters and points, in those courses where narrative evaluation and mastery grading make the most sense: Graduate courses, seminars, proof-oriented upper-level math courses, honors sections of courses, and so on.
- I think you could also make a strong case that introductory courses are also fertile ground for trying out narrative evaluation and/or mastery grading because these are where student motivation tends to be at its lowest point.
- Treat student work --- as much of it as possible --- like submissions to a journal. When we academics submit articles to a journal (or tenure portfolios, etc.) we don't get a point value or letter grade attached. We get verbal feedback with a brief summary: "Accept", "reject", "Major revision", "Minor revision" etc. followed by details. Then assign course grades, if you must have them, based on how much acceptable work the student was able to produce.

There are some practical issues at work here that can't be minimized, for example (and especially) large sections. The issue of scaling is a tough one, but it's not impossible. In my experience with specs grading, doing narrative evaluation takes no more time per student than traditional grading (which involves endless hair-splitting on how many points to give a response), so I don't think there's any reason to believe that nontraditional grading can't scale up.

Moving away from traditional grading could be one of those Pareto principle concepts where focusing intently on this one idea could usher in outsized improvements in many other areas of student learning. I think it could be a fulcrum for bringing about wholesale, even revolutionary change in higher education. Let's give it a try.

Although: I have to admit that recently, I've noticed that students in my specs-graded classes tend to focus laser-like on their grading checklist where they keep track of how many Learning Targets they've passed, rather than on what those Targets represent. In other words the specs end up becoming a proxy for letter grades and students fixate on those accordingly. I'm still thinking about how to handle this. ↩︎

Over the last few years, I think I've finally learned the obvious fact that *communication with students is the most important ingredient for a successful course*. Most of my failures in teaching can be traced back to failures in communication. Conversely most of the time when something goes right in my teaching, it's because I gave students the opportunity to speak their minds, and I listened.

For me, then, one of the most important aspects of designing a course is deciding on the tools students and I will communicate with each other. For a long time, this was just email. But as I started wanting more robust, full-group communication among students outside of class, I outgrew email, and I started using discussion boards like Piazza. But soon, even the best discussion board technologies weren't cutting it for me. The "discussions" were too *pro forma*, too much like something the teacher was telling you to use and not enough like real conversation. So a couple of years ago, I tried something different: I set up a Slack workspace for my class.

If you're not familiar with Slack, here's a video that covers the basics:

I introduced Slack in my Discrete Structures for Computer Science course, where the students are upper-level CS majors, with lots of tech savvy. Many of them had used Slack before in internships, jobs, and other classes, so it was no big deal for me to introduce it. Although I don't think I fully knew what I was doing and definitely didn't probe Slack's full potential, students liked it and used it, and it was a success.

This past fall, I taught a hybrid section of Calculus and two sections of Modern Algebra. I made up two Slack workspaces, one for Calculus and another for the two Modern Algebra sections. This time, however, the results were much less favorable.

I introduced Slack by setting up an "intro to Slack" assignment to be done in the first week that included watching tutorials on how to use Slack, posting a "hello world" message to a specially-created channel, replying to another person's message, and sending me a direct message (DM). Those are the absolute basics; students can learn more as they go, I decided.

Here's the channel setup that I used in Calculus (Modern Algebra was similar):

The #assignments channel is for discussion of assignments; #course is for discussing the course and the syllabus, #f2fmeetings is for followup discussion of what we did during the face-to-face meetings, etc. Technology questions are to be asked on the #tech channel; #updates is for course announcements and calendar events. I was posting course announcements to the #announcements channel as they occurred. (More on that below.)

The tools we had set up in the course were Blackboard for grades and files; Slack; and a Google Calendar. Calculus also used WeBWorK for online homework. Everything was linked on Blackboard so that all tools in the course were 3 clicks or fewer from the Blackboard main page.

So we were all set up for a very productive semester, right?

Although students seemed curious, if not enthusiastic, about using Slack and did use it initially, we ran into problems right away.

- Some students skipped the "intro to Slack" assignment. It only counted for participation credit, and so if a student just didn't want to do it, they could make up the credit it later. One student finally admitted in
*week 10*that he had never signed up for Slack. - A lot of students were not seeing the announcements for the course. There are several reasons for this, but one of them is inherent in Slack: Whereas Blackboard announcements are not only posted but also pushed to student emails, students had to proactively go to the #announcements channel on Slack to find their announcements. This one extra step was one step too many for a lot of students, which resulted in missed announcements and therefore missed assignments and therefore a lot of angst. I call this the "Yet Another Inbox Problem".
- Also with multiple announcements per day being posted, students needed to be logged into Slack constantly to catch them all. This is actually the way Slack wants you to use their product: be logged in on all devices at all times and listen for the ping. But that's not how students operate. (It might not be the best way for
*anybody*to operate.) Since students weren't always-on, they'd miss announcements. - Even those who went to the #announcements channel might have missed announcements because of overcrowding. I used a plugin for Google Calendar that pushed calendar events to the #announcements channel, giving students a daily digest of events for the day. Nice idea, but when the plugin drops 5-6 calendar items into the channel, the actual course announcements get pushed off the screen. Yes, there is a little alert telling you how many unread messages there are and an "All Unreads" aggregator that you can turn on. But it's easy to miss the unread messages alert, and the All Unreads feature is turned off by default.
- Students were confused by my channel structure. Many weren't sure where to post things and so put things in the "wrong" channels, e.g. technology questions in the #general channel, which meant I had to contact students to ask them to delete the original post and re-post it in the "right" channel. This got confusing/annoying enough to some students that they just disengaged from Slack altogether.
- Students were also confused by where things were located. I introduced the mantra
*If it's a file or a grade it's on Blackboard; if it's a message it's on Slack; if it's a date it's on the Calendar*but this wasn't enough for many students who were used to having everything just on Blackboard.

By week 4-5, we were settling into a habit of using Slack just for direct messages to the professor --- a feature students did use widely, and enjoyed using --- but nothing else. I didn't run the numbers, but I am sure that well over 85% of the posts on Slack were originated by me and have no replies.

I give informal course surveys in weeks 2, 6, and 9 to ask students what we need to start, stop, and continue doing. I knew from week 2 that Slack was an issue. In the week 6 survey, I asked the Calculus class whether we should keep announcements on Slack or move them elsewhere. They said:

Only 1 in 9 students wanted announcements to be on Slack! In response, I started putting announcements on Blackboard and not Slack and changed the name of the channel from #announcements to #updates, because I kept the Google Calendar plugin active. I thought I was making the right move here, but what this ended up doing was causing even more students to disengage from Slack.

After relocating announcements to Blackboard, student use of Slack (apart from DM's) dropped to essentially zero, even when I put activities on Slack that counted for credit or were requested by students. In the last two weeks of the semester, because some students were still using Slack while many others had dropped it, I needed to be sure that critical course announcements were being seen, so I was cross-posting announcements to both Slack and Blackboard. The way I had handled it, I couldn't be sure that students were using *either* of the communications methods I had set up. That's not good.

This issue was weighing on my mind all semester, so near the end, I gave a survey to collect student data about what they *did* want from a course communication tool. I got a surprisingly good response rate (46 students out of 72 total) and their replies were pretty illuminating.

When asked *How important is it to you, generally speaking, that your courses include an official electronic communications tool for communicating with the professor and classmates in between classes?*, 76.1% responded "important" or "very important". So it's not a case where students are just not interested in using these tools.

I asked which specific communications tools students used regularly, before taking my course. The top three were their university emails (98% of respondents), the Blackboard discussion board feature (56%), and personal emails (43.5%). Fourth and fifth place were Facebook (41.3%) and group text apps like WhatsApp and GroupMe (39%). Twitter was used by 26% and all other tools --- including Slack, Discord, Instagram, and Stack Exchange --- came in under 7% each. I think maybe I wasn't clear in my question here, because I wanted to know what tools students are using in *any* part of their lives to communicate with others. I think the students thought I meant in a classroom context only. I'm pretty sure Instagram is used by more than 7% of my students.

I then asked students about the kinds of features they'd want to have on a communications tool used for an academic course, and which of those features would be the their top-3 most important. They said:

- Notifications are pushed to my email (rated in the top 3 by 78.3% of respondents)
- The tool lets me send direct messages to my instructor (56.5%)
- The tool has mobile apps for my phone or tablet (50%)

I thought this was surprising because some of the features I thought would be must-haves, like the ability to share files easily, didn't crack the top 5. My students' top features are those that *actually facilitate communication* and not other bells and whistles.

The next question and its result are, I think, at the heart of my issues with Slack. I asked students to vote for which of the following situations they would prefer if they had the choice: (1) *Using an electronic communications tool that I already use or which is easy to learn and use, but which lacks many or most of the features that I find important*; or (2) *Using a tool that has most or all of the features I find important, but which is not something I already use and would take effort to learn and use.*

The results tell you everything you need to know:

This is a very economical way of expressing the problems we had with Slack. It's a tool that has a ton of excellent features but requires students to adopt a different paradigm of communicating and to check Yet Another Inbox. Half my students preferred this; the other half preferred the opposite. No wonder we couldn't figure out how to integrate it into the course.

The last question I asked had to do with just how many communications tools students would be willing to tolerate in the course. I asked students to vote on which of the following situations they would prefer if they had the choice:

- Having one tool for all course communication
- Using email for personal messages but then one tool for everything else
- Using Blackboard for announcements but then one tool for everything else
- Using Blackboard for announcements, email for personal messages, and a communications tool for everything else

Here are the results:

I was a bit surprised that the plurality of students, all things considered, would prefer to have one tool for everything, even if that tool requires effort to learn and use, rather than split the work of communication among multiple tools some of which may be very simple and familiar. Only a handful of students wanted to see three separate tools being used, even if two of them are email and Blackboard. It's as if they're saying, *if you're going to use a communciation tool like Slack, go the whole way*.

I don't regret having moved to Slack from straight-up discussion boards. I think Slack facilitates a more organic and free conversation, and it does make it easier for my students and I to communicate thanks to its immediacy and informality. But it's also a tool that requires a different way of thinking. That's not bad; but it does produce cognitive load for students that is not necessarily germane to their learning in the course.

In addition to everything I've presented, many of my students found Slack to be confusing, not in terms of ease of use, but rather in the chaotic nature of the communication process that Slack tends to promote. Yes, there is threaded discussion on Slack but even with threads, discussion topics tend to pile on each other with no clear way to disentangle them. This isn't helpful for those in an online or hybrid course, or in a flipped learning environment, or students in any situation who have learning disabilities, all of whom need structure to have a successful experience.

What the students are telling me is that they do see the value in course communications tools, but there's a balance to be struck:

- The tool needs to be simple and/or familiar, so students don't spend more time figuring out the tool than they do the math in the course.
- But it also has to have a good feature set, and many students are willing to put in the work to learn the software if there's a clear payoff in terms of better communication.
- The rule about low cognitive load does not distribute across multiple tools. That is, it's not OK to have several tools to use, all of which are simple; it's better to have one or two tools that have moderate complexity and a good feature set.

Slack *might* strike the right balance for some courses and not for others. Or, in some courses, Slack is the right fit but it will take guidance and modeling to help students learn how to use it; you can't just drop it into any class and expect success. Learning a new tool requires cognitive resources; you have to gauge your audience to know how much is reasonable to ask.

My next course is an asynchronous online Precalculus class in the summer, and I'm already thinking about the communications infrastructure. I might use Slack. I am also looking more closely at similar platforms like Discord or Twist, a promising new product called Campuswire, or even something like WhatsApp. Whatever I choose, I'm going to make sure that I will:

- Have students learn the platform and start using it early and often, through a week-1 assignment, this time with a lot more value and weight to it, to help ensure students don't treat it lightly. I'm considering capping students' course grades at a B if they don't complete it within the first 10 days of the course. Communication is
*that*important. - Make the structure of channels (or whatever the analogue is) muck simpler. If you can't tell what the channel is for just by reading the name, it's too complicated.
- Provide lots of support by finding excellent tutorials for it, and making the tutorials myself if I can't find any; and model the behavior I want to see by using the platform myself, also early and often.
- Lay out clear expectations for how and when to interact on the platform, and insist students abide by those expectations and make the platform a central, integrated part of the course experience.

I did some of these reasonably well in the Fall, some poorly and some not at all. I have a lot to fix myself when it comes to these tools.

]]>*A recent article in the *Chronicle* featured several university professors sharing their responses to the prompt: "Describe an experience that changed the way you teach, and what you do differently now as a result." There was a link in the article for readers to share their own turning points and experiences. I wrote the following. Who knows if it will ever appear in print, but I thought it might be worth sharing. *

I was teaching a Calculus class, giving what I considered to be a very clear, informative and well organized lecture. The lecture was clearly going well: Students were attentive, nodding along to my points, even smiling. As I ended one part of my lecture, I said, "So as you can see, it all makes perfect sense." Every student smiled and nodded and continued to take notes.

About 10 seconds into the next topic of the lecture, a student raised his hand and asked: "Professor, could you repeat again why what you just said makes perfect sense?"

I was speechless for three long seconds, thinking: *If I have to explain why it makes sense, then it doesn't make sense*. But I wasn't dumbstruck merely by the audacity of the question. I was struck by its truth. What I was doing in the class -- mostly lecturing, and doing it well by my peers' estimation -- was not leading to real learning. It was just playing school.

Since then, I've come to realize that I, as a professor, cannot cause sense to be made inside the minds of my students. Sense-making is something only the students can do. And indeed, it happens by *doing*, not listening alone. I've realized that human beings make sense out of ideas by actively engaging with those ideas, constructing and instantiating and testing them, bringing them to life by active work that is done and shared with other humans doing the same thing. And the more students *do*, the more sense gets made.

So here's what I do differently now:

- When building a course or a class session, I give active learning a privileged position. Lecture if it's appropriate, but realize that sense-making only happens through active work, in a social setting involving other learners. Put as much of this in every class as possible.
- Give frequent, ungraded formative assessment on key ideas several times per class session. This is much better than relying on body language, vague prompts for questions, or outright faith that students are "understanding". Don't ask if students understand; have them do something that will demonstrate what they understand and what they don't.
- Then, look at the formative assessment data and act on it -- make changes, plan a review, contact individuals -- whatever the data suggest. I try to teach like a scholar, in other words, instead of a performer.

This fall in Modern Algebra, I'm using mastery grading. I gave a full rundown of my mastery grading setup for Modern Algebra in this post and described a "T-shaped" experience I wanted students to have --- a deep understanding of foundational concepts coupled with a broad ability to connect those concepts to each other and to bigger ideas and applications. I'm assessing the foundations using what I call *Foundations quizzes*, and we have a *Proof portfolio* to gets students working in the creative space of connections and applications. Both Foundations quizzes and Proof Portfolio problems fit well in the mastery grading world, a fundamental pillar of which is that **students should have the ability to revise and resubmit assignments until their work reaches an acceptable level of competency**, rather than just being one-and-done.

However, when I was putting the course together in the summer, I realized something was missing --- namely the space between foundational concepts and high-level proofs and applications. I'm thinking of things like

- Compute $\gcd(5150, 90125)$ using the Euclidean Algorithm and then use the Extended Euclidean Algorithm to write this GCD as a linear combination of 5150 and 90125.
- Explain the steps of a written proof of the infinitude of primes, or fill in the blanks on a partial proof, but don't prove it from scratch.

Both of these occupy what I think of as the middle 1/3 of Bloom's Taxonomy --- "Application" and "Analysis" along with the top end of "Understanding" and the bottom end of "Evaluating". They require Foundational knowledge but do not directly assess it; they sometimes verge on proof construction but do not require it at the level of the proof portfolio. I introduced a timed test to gauge this knowledge, what I eventually called a *Connections exam* --- the name indicates it's specifically designed to probe the space in between the bottom 1/3 of Bloom (covered by the Foundations quizzes) and the top 1/3 (covered by the Proof Portfolio).

As I was putting the course together this summer, I began to wonder: **What about revision and resubmission? How would this work with Connections?** How do you make a timed test^{[1]} with a revision policy that gives students sufficient opportunities to improve, without causing a grading avalanche in the process?

One of the articles I stumbled across during my sabbatical leave was this blog post by Carl Wieman on **two-stage exams**. I read it, and the paper attached to it, and knew instantly that this was the answer to my revision/resubmission issue.

The idea behind two-stage exams is simple:

- In the first stage, students first work individually on an exam for a subset of the exam time.
- Then there's a short time during which students hand in their work and transition into groups.
- Then, in the second stage, students get a clean copy of the exam or a portion of the exam, and they work the rest of the time in groups to answer the exam questions again.

This idea isn't new, and it's a basic pillar of a lot of good teaching techniques like team-based learning. It struck me as something that would work well in the class on a number of levels, so knowing that I cannot leave well enough alone with my teaching, I decided to use it for Connections exams in Modern Algebra. Here's how they work. Note that my class meets Tuesdays and Thursdays in a 75-minute time block each day.

- Stage 1 of each exam is broken up into 4-5
*sections*, essentially one problem per section and each section focusing on one big concept. - Students get 45 minutes to work through Stage 1. I have a 45-minute timer running on the projector to keep things on track.
- When the 45 minutes are up, I start a 5-minute timer, collect students' Stage 1, and then put students into randomly generated groups of 3 or 4. Each group is given one copy of Stage 2, which consists of a subset of the sections from Stage 1, possibly slightly modified if needed to make sense in a group setting and shorter time limit.
- When the 5 minutes are up, students work in their groups through Stage 2 and write up one single group report. They get 25 minutes for this, at which point the exam is over.

In mastery grading, we usually don't use points for assessing work, but instead give grades that describe the work relative to quality standards that we discuss in the syllabus and in class. Here's how this works for Connections:

- Each section (which remember is roughly one problem) is graded
*Excellent*,*Satisfactory*,*Progressing*, or*Incomplete*--- both in Stage 1 and in Stage 2. **For any section that appears in both Stages, students get to keep the higher of the two ratings.**This way, students' work in groups won't bring down their individual work or vice versa. The exception to this rule is that if someone turns in work on a section that earns*Incomplete*(e.g., they leave a section blank otherwise skip big chunks of the section) then the highest they can earn on that section is*Progressing*, even if the group work is "Excellent". (So, students can't deliberately tank on a section in hopes of having the group work bail them out.)- Then, the exam itself gets an overall grade of
*Excellent*,*Satisfactory*, or*Progressing*depending on how the individual sections turned out:

Overall rating | Requirements |
---|---|

Excellent | All sections earned Satisfactory marks and at least one “Excellent". |

Satisfactory | All sections earned Satisfactory marks. |

Progressing | Not all sections earned Satisfactory marks. |

To earn a grade of "B" or "C" in the course, students have to earn Satisfactory or Excellent on all three Connections exams. To earn an "A", at least one of those exams must be "Excellent".

The two-stage approach makes revision more efficient, because in a way the group stage is the first round of revision. Students will have already made an attempt on all the sections; working in groups allows them to instantly revisit that work and pool their attempts with others. So before they even leave the testing session, they've already taken and revised their work once.

After the test is over, students are allowed *one* revision of any remaining section on the exam that isn't at the level they want. Students just look through their tests, pick the sections they want to redo, and then I will make up a new version of those sections with different content but the same concepts tested. This will be one new version for the entire class --- making up a new version for every single student just isn't feasible and it's why allowing unlimited retakes doesn't work for bigger tests where it might work for small quizzes. Then students work them out and submit them via Blackboard for regrading, and then the grade is final.

So this way, I limit the amount of regrading and making up new assessments to just one per exam, three times total during the semester, but students really get two chances at revision per test. It feels like getting two revisions for the price of one, which is perfect.

Last Thursday was our first Connections exam and the first time I'd ever done this. Here's Stage 1, and here's Stage 2.

I really liked the results. Students worked more or less as you would expect them to on a 45-minute timed exam. But when I got them into groups, magic happened --- students normally reticent were talking it up with their classmates, clever and creative solutions were being tried out, good and honest questions were being asked, and students were teaching each other mathematics. It was really the best group work I've seen in a long time and definitely the best collaboration we've had this semester.

I asked students to reflect on their experiences, and here are some of the things they said:

*I think the test went really well. I liked stage 2 of the test because it allowed me to revisit certain problems from stage 1 that gave me some trouble and discuss it with others.*.*I felt very rushed through the individual portion, but felt that there was plenty of time for the group portion.*.*I felt a little rushed for time on Stage 1, maybe one more fill in the blank proof would have evened it out a little more. I really liked how it was two stages though. That really helped me learn. Also knowing that I am able to do a revision gave me piece of mind.**Start with the group part of the test first.*(That's an interesting idea.)

Many of the other student takes on the exam just repeat these --- they were rushed on the individual part but felt fine on the group part. I don't think I have ever, in 21 years of teaching, heard a student say they had sufficient time on a timed individual exam no matter how many or how few questions are on it, so I take the comments about being rushed with a grain of salt. However, it's worth considering adding (say) 5 minutes to stage 1 and subtracting 5 from stage 2.

One thing I did not like about the two-stage approach is that the group stage can put certain students at a disadvantage, for example neurodiverse students who might get overloaded or distracted easily from the noise and bustle of Stage 2. (It did get quite loud in both sections.) I realized this issue belatedly, so I let any student choose to opt out of the group stage and spend that time working on Stage 1 in a quiet room. I had one student take me up on that offer, so I'm glad it dawned on me, but it's not an easy issue to route around.

One thing I am not sure about is how the revision process will play out, because I am still grading Stages 1 and 2. I am hopeful that the one revision will be enough; I wish I could open it up for more, but seriously, this is a lot of grading we're talking about and you can only take on so much before you lose the ability to give timely and meaningful feedback.

I'm also not sure how well this process works if you have a class that doesn't meet for 75 minutes. It seems like it doesn't scale down well --- for example in a 50 minute class, you'd be giving something like 30 minutes for stage 1 and then 15 minutes for stage 2, which just doesn't seem like enough time to assess anything more than 2-3 ideas.

I was very excited to come across the concept of two-stage tests, and overall I am very happy how it's working out so far. It keeps a layer of individual mastery on the testing environment while also injecting collaboration and peer-to-peer learning. This strikes me as one of those "small teaching" methods for doing something simple to get a lot of high-quality active learning going on in your class. I'm looking forward to seeing how it evolves for the next exam.

I considered not using timed testing at all, but the alternatives weren't really working for me. This is a flipped learning environment, so I didn't want to pile on graded homework on top of everything students are already doing; and grading students' in-class work seemed dicey. There's something to be said for the value of timed testing, especially if you can fortify it with revision and reassessment. ↩︎

Recently I enjoyed giving a flipped learning workshop to faculty at George Washington University. This took place just a few days before the start of Fall semester. Usually with workshops, the hope is that faculty will take what they've learned in the workshop and put it to work right away in their classes. But actually, I have been warning faculty to do the opposite: **don't implement flipped learning right away. Instead, give yourself a year**.

The reason is that, while flipped learning doesn't necessarily cost a lot of money, it does entail expenses in time and energy that, like payments on a mortgage, have to be spread out in order to be manageable. If a faculty member tries to go all-in on flipped learning just weeks after learning about it, the experience is likely to end poorly. Just ask me. In my book, I describe in detail how my first flipped learning experiment failed badly, in no small part because I tried to implement it right away without giving myself time and space to prepare.

At GWU, I proposed a **One-Year Plan** for converting one course into a complete flipped learning model. Under this plan, a faculty member who is interested in flipped learning, and who has a course in mind to flip, can start in the fall, work through the spring semester (here in Michigan, we call it "Winter" semester) and then finish up in the summer and in one year have a completely flipped design for the entire course, carefully researched and planned.

We begin one year away from actually flipping an entire class. The main goal in this phase is to **build habits of active learning and formative assessment that are at the core of flipped learning**. During this semester:

**Make a goal to include active learning elements in**This means that every class meeting (except test days, etc.) should somehow involve students actively in the construction of their own understanding of the material, beyond taking notes and asking questions. There are many simple ways to do this: short quizzes inserted at key points in a lecture, think-pair-share exercises, one-minute papers at the end of a session, and more. Start small if you need to and aim to increase the amount of active learning your students do as the semester progresses and as you get more comfortable using active learning.*every*class meeting (except for test days and the like).**Make a goal to include formative assessment in**. This just means that you should not simply present material and have students engage with it, but rather measure what students know as they are learning a concept. Again, there are simple ways to do this: short ungraded quizzes, one-minute papers, clickers, and so on.*every*class meeting (except for test days and the like)**Then, make a habit of actually using the formative assessment data you collect.**The purpose of formative assessment is to gather data about student knowledge "in the moment" so that you can make adjustments to your teaching. Make a real effort to make adjustments when necessary. Especially, consider whether you should be teaching differently if the data are consistently showing that your present method isn't getting the job done.

Just doing these three things on a regular basis will, I believe, bring about major improvements in student learning in your classes. But there's more to this phase of the one-year plan:

**Make plans to flip between one and three individual lessons during the semester.**These should preferably be in the last half of the semester. This is your "pilot study" as you look forward to flipping a whole course a year from now. Pick out lessons that seem like they'd especially benefit from flipping --- for example a lesson with a lot of terminology or informational content that you normally have to plow through in a lecture, leaving little time for application or synthesis activities in class. Pick those lessons; start creating materials for it early; and make plans to flip those. (See "Reading List" below for a guide.)**For each of the 1-3 lessons you flip, get student feedback on how it went and make a note of anything that went particularly well or particularly poorly**. Ideally if you flip more than one lesson, you can use the feedback from the first lesson to improve the second, and the second to improve the third. This is another good habit to nurture: Getting frequent informal student feedback and making changes to your teaching as a result.

Finally, read the following this semester:

- One of the following: Make It Stick: The Science of Successful Learning or Small Teaching. Both are great books that focus on simple improvements that any teacher can implement easily, , right away, and for little to no cost.
- Seven Steps to Flipped Learning Design. This is a workbook that I created for the workshops I give that lays out a step-by-step process for designing a single flipped learning lesson in a course. In my workshops, we work through the whole thing to arrive at a single flipped lesson. The workbook can help you do it on your own.

At this point you hopefully will have become a regular user of active learning and formative assessment, and flipped a handful of individual lessons and gotten student feedback. The main goal of Spring (Winter) semester is to **flip the final 1/3 to 1/4 of a course**. Here we are thinking about 3-4 *weeks* of a course. Do the following:

**Continue using active learning and formative assessment in every class, along with frequent informal feedback from students, and make corrections in your teaching when the data call for it**.**Plan on flipping the final 3-4 weeks of your course.**You can do this using a "gradual release of responsibility" model, which goes like this: The first four weeks of the course are traditionally structured. Then the second four weeks involve a small amount of flipping --- for example, have students do a little bit of preparation before class, and remove that material from your planned lecture, and then quiz them over it when they get there, then use the extra time for active learning. Then just gradually ramp up the amount of preparation you assign to students, until by week 8-10 the entire course is flipped.- As you are working with the flipped final 3-4 weeks remember to
**get student feedback regularly on what's working and what isn't, and make adjustments**.

In this phase there are also some forward-looking tasks for the big leap to a fully-flipped course in the fall:

**Start preparing any homemade materials you want to use in the fall**. Especially, if you plan on making your own video content for the fall, start NOW. Video-making is a time-consuming process and you do*not*want to start in August. Start in February and give yourself six months instead. (Also remember that you do not need to make or use videos.)- You can share your flipped learning experiences in the fall by collecting data on student performance and writing it up as a paper submitted to a journal that publishes scholarship of teaching and learning (SoTL). If this interests you, and hopefully it does,
**take the Spring (Winter) semester to read about SoTL research, read research done in flipped learning, and, especially, write up and submit a proposal to your university's Institutional Research Board**. Almost all SoTL projects require some form of IRB approval because they involve human subjects. Most SoTL projects end up in the Exempt category and are quickly processed, but still there could be a long line of research proposals that the IRB has to consider before they get to yours. If you want to make your flipped course into a SoTL project, don't wait until August to write a proposal. Do it now. - Also in the spring semester,
**get connected with communities of practice where flipped learning is discussed**. A great example is the Flipped Learning Slack community which is open to all (although you must request membership) where flipped learning issues are regularly and vigorously discussed. There are others; perhaps you could even start a group on your own campus. Plug in and start making a network that you can leverage in the Fall. - OPTIONAL:
**Get a certification from the Flipped Learning Global Initiative**. The FLGI's certifications consist of a series of videos that teach you all about the ins and outs of flipped learning and could be useful in a number of ways. (Disclosure: I am a research fellow for the FLGI.)

Finally, read the following this semester:

- Whichever one of Make It Stick: The Science of Successful Learning or Small Teaching you didn't read in the fall. (Good idea: Form a reading group with your flipped learning social network that you've formed.)
- I humbly suggest my book Flipped Learning: A Guide for Higher Education Faculty as a good thing to read at this point. A big part of the book is an expansion of the Seven Steps workbook, so really you've read pretty much 1/4 of it already.

During the summer the main goal is to get the Fall flipped course totally ready to launch. Do the following:

**Finish making all learning materials like videos, activities, projects, etc.**Faculty love to put things off until the last possible moment, but don't do that here. Videos made hastily suck; and they take more time than if you make them earlier because you make mistakes when you are under pressure. Get those things done before school starts.- Also
**finish the syllabus**-- which is more important than usual here because in the syllabus you'll need to think about how you are going to explain your flipped learning structure to students. **If you are doing a SoTL study in the fall, get anything prepared that needs to be prepared**(interview questions, measurement instruments, etc.)

I don't recommend preparing more than two weeks' worth of lessons for your flipped course in the summer. That's because, as Mike Tyson once said, everybody has a plan until they get punched in the face. That is, your flipped lesson plans may be expertly designed, but you may have to make major changes to your lessons and methodology once you actually deploy them to students. Make the *materials* well in advance but don't over-plan the *lessons*.

Then, when August rolls around, the magic happens. Students arrive and they get a great learning environment where active learning is king, you have real data in real time on their progress, and they are getting the maximum amount of help just when they are encountering the greatest difficulty. This is what flipped learning is all about.

This plan is just a suggestion, of course. It can be accelerated by people with more experience or slowed down by those who need a more gradual approach. It can be modified wholesale or completely ignored. A gradual approach is best, though, and by taking your time, you won't burn you or your students out.

]]>This is second of two posts on my use of specifications grading as I return to teaching in Fall 2018. In the first post, I described specs grading in general and then went into detail on how it's used in one of my courses, a hybrid-format Calculus 1 class. Here, I'm going to discuss my other course: Modern Algebra 1. I'm teaching two sections of this course, and as you might guess, the grading system turned out quite differently than Calculus.

Modern Algebra 1 is an upper-level course intended for mathematics majors and minors, focusing on theory of integers and integer arithmetic, rings, fields, and polynomials. It is called "abstract algebra" in some places; unlike many college courses on this subject we start with rings first, and then the second semester of the course is focused on the theory of groups. The course has a standard face-to-face format, and we meet twice a week for 75 minutes each.

Who takes Modern Algebra? Mostly mathematics majors; of the 46 students total in both sections of the course, 39 are majoring in mathematics. Of those 39 students, 23 of them are pre-service elementary or secondary mathematics teachers. That's half the entire enrollment, and it's no surprise – this course is one of the upper-level required courses for students seeking teacher certification, and it's one of the reasons we do rings first in the course, to get the theory of the course as close as possible to the algebra that these students will be teaching in middle and high schools.

The course has two prerequisites: Communicating in Mathematics (our introduction to proof course) and Linear Algebra. The first of these is especially important because Modern Algebra is theoretical in nature and so there are a lot of proofs. It's also a source of student stress, since proof is hard, and students often do not feel at all comfortable with proof writing despite making good grades in that course. (I know this because several of those students have already told me so.)

The course design for Modern Algebra diverged from the design for Calculus fairly quickly as soon as I started thinking about learning targets.

My first attempt at specifications grading was in Winter semester 2015 when I taught two courses: the second semester of Discrete Structures for Computer Science, and the second semester of *this* course, Modern Algebra 2. I wrote about the specs grading design part of those courses at the time. When I designed Modern Algebra 2, I did the usual thing: I wrote out specific learning targets for which students will accumulate work that shows they have hit those targets. Here's the list of targets I came up with for the course.

Do not adjust your set: That is a total of *67 learning targets* for the course. Of those, 30 are "concept check" objectives targeting foundational knowledge like stating definitions and theorem statements; the remaining 37 are "module" objectives that target higher-level tasks. And of the 37 module objectives, 24 are "core" module objectives that have a special place in the grading system, which you can read (if you dare) in the syllabus for the course.

Needless to say, this design concept is to be filed under "it seemed like a good idea at the time". I was staying true to the spirit of specs grading, I thought, but in practice the system became wildly overcomplicated. Just the very idea of keeping 67 Learning Targets straight in your mind, differentiated into CC and M and CORE-M objectives, is enough extraneous cognitive load to put off even some of the best learners. But the worst thing about having so many objectives was that **by breaking down the course into so many atomic-level objectives and assessing each one individually, students lost the big picture of the course . ** The course itself, which contains some of the most beautiful parts of mathematics, became dissociated as students focused almost entirely on the pieces I'd created and not on the whole, on how all those pieces connect.

This time around, I thought about my experiences in 2015 and decided that:

- I want students to become very fluent on the
**foundational**knowledge of the subject: Definitions, theorem statements, basic computations. - But I also want them to become skilled at
**connections**between the pieces of foundational knowledge: Applying the basics to new situations, doing nontrivial computations, making their own examples of structures, drawing conclusions using theorems, making conjectures, proving and analyzing written proofs of conjectures.

Thinking about this orthogonal relationship between foundations and connections brought to mind some discussions I had while on sabbatical at Steelcase last year about "T-shaped skills". Our discussions at Steelcase focused on the kind of educational experiences that learners need for the modern workplace, and how learning space design can support pedagogy that focuses on this.

I realized that this was also the idea I was trying, not very successfully, to capture with my specs grading setup in 2015. I want students to have deep functional knowledge of algebra, and the ability to connect that knowledge into a coherent whole both within the course and to application areas outside the course – including and especially, to teaching middle and high school mathematics. So this became the focus of the course design.

To capture this T-shaped idea, I designed the course so that the vast majority of our class meetings will be spent working on the early stages of *connections*: Taking foundational knowledge and finding relationships among the parts, including parts we'd previously studied and parts outside the course itself (again, especially any place where course concepts have something to say about middle- and high school math instruction). To make as much time for this as possible, the course is design using flipped learning, and we learn most of the *foundational* knowledge – definitions, theorem statements, basic computation, etc. – before class through structured inquiry assignments. Then we extend both the foundations and the early connections after class through creative work, particularly working on proof-oriented problems.

In particular, an ideal lesson design will get learners engaged in the mathematical process of *experimentation, conjecture, *and *proof *like so:

- Students learn the
*foundations*through pre-class activities and*experiment*with concepts, and bring in the results of their experiments to class. - Then in class, we make
*connections*by*conjectures*about what we are all seeing, then work on*proving*those conjectures together.

And my role in all this is to design the learning activities, manage questions, keep groups in class functioning well, and collect data on student comprehension as well as coaching on any creative work students might do, such as drafting proofs.

To structure this kind of experience, we've set up the following categories of assignments:

**Guided Inquiry**assignments to structure the students' pre-class experiences of learning the foundational knowledge. Here's the first one. It doesn't quite have the "experimentation" flavor to it I described because we are just getting started, but it gives the idea. It's basically a wrapper around a Google Form. The wrapper gives the overview, learning objectives, and resources while the form contains the exercises. These are graded*Satisfactory*or*Unsatisfactory*on the basis of completeness and effort ("Satisfactory" is awarded for giving good-faith responses to each exercise and turning the form in on time.)**Foundations quizzes**will assess fluency with foundational knowledge. Being fluent with definitions of terms, notation, etc. is critically important in algebra. These are graded*Satisfactory*or*Progressing*. Here's sample of the first Foundations quiz which will be given in a week or so; and here's a Google doc I am using to list the topics on each quiz and the criteria for "Satisfactory" for each one.**Connections**tests are timed assessments that get at higher-level learning tasks such as using theorems to draw conclusions given some situational data; doing more advanced computation; and outlining or analyzing proofs. These tests are broken into 3-6 sections each of which targets a specific idea. Then the tests are given using a two-stage concept: For the first 45 minutes, students work on their tests individually. Then their work is collected, students are put into groups of 3--4, and a new version of the test consisting of a subset of problems from the individual stage (suitably modified for time) is given and done in small groups. Each section is graded*Excellent*,*Satisfactory*,*Progressing*, or*Incomplete*. Students get to keep the higher of the two grades from the individual and group stage on each section. Then the grade on the test is Excellent, Satisfactory, or Progressing depending on the grades on the sections.- Finally there is a
**Proof Portfolio**students do, involving solving proof-related problems from different groups of topics.

We will also have a final exam in the course, and students earn experience points (XP) for completing tasks that signify engagement in the course (attendance, filling out surveys, etc.).

Here is the final syllabus for the course, which contains all the details about how the grades on individual items work. The Appendix in particular shows the "specs" for each individual form of work in the course.

As with calculus, finding the course grade is a two-step process. First, the *base grade* is determined, which is just the A/B/C/D/F grade without plusses or minuses:

The current calendar has us doing 23 Guided Inquiry assignments and 13 Foundations quizzes. So to earn a "C", a.k.a. baseline competency, you would have to satisfactorily complete 70% of the Guided Inquiry, 60% of the Foundations, *all* of the Connections, and do what I defined as "C-level work" on the proofs. The thought here is that a "C" indicates reasonably OK performance on the higher levels and minimally acceptable performance on the lower levels. To earn higher grades, students have to do more, and do it better.

One big difference between this table and the one for Calculus is that *there are no learning targets here. *I started to write out Learning Targets for this course but quickly felt like it was Winter 2015 again, and the course was losing its coherence because of the focus on micro-sized learning objectives. The closest we come to having Learning Targets are the topic lists for Foundations quizzes, and there I simply sample from the list for individual quizzes rather than try to test everything.

So this makes my system a lot more like *specifications* grading and less like *standards-based *grading. I'm sometimes asked to explain the difference between these two forms of mastery grading and now I think I know the answer. It was recently put very succinctly by my friend Josh Bowman in a talk, in which he said **"Standards reflect content, specifications reflect activity"**. In Calculus, the focus is on content (which for me includes both procedural and conceptual knowledge) In Modern Algebra, and I think in many other upper-division math courses, the focus is on *broad areas of skill, *again especially the dual categories of *foundations* and *connections*, not so much on individual pieces of content, which are important but not the point.

In Modern Algebra there's a more complex system for determining plus and minus modifiers once the base grade has been determined. The final exam and XP are the primary factors in finding the modifier, similarly to what I did in Calculus. But I also include some "near miss" situations; for example if you complete all the requirements for a base grade plus either the Connections or Proof Portfolio requirements for the next grade up, that gives you a "plus". The details are in the syllabus.

Finally, as with Calculus, there's a very robust system of revision and reassessment:

- Foundations quizzes can be retaken during two designated 75-minute sessions during the semester and at the final exam.
- Connections tests can be retaken
*once*through a take-home process. Part of the rationale behind using a two-stage setup here was that it sort of provides an automatic revision, in the group stage. So in some sense students get two revisions: Once in their groups, and again through take-home. - Proof Portfolio problems can be resubmitted as often as needed, with the restriction that no more than two proof submissions total (two new submissions, two revisions, or one of each) can be made in any given week. Students can spend a token to get a third submission. Also, each group of proof problems has a deadline, and problems whose initial submission comes after the deadline are accepted but may only be revised
*once*. In my Discrete Structures course I only had one final deadline, but I found students were waiting until week 8 to submit proofs from work done in week 2, so hopefully the incentive of "unlimited" revisions for work submitted by a deadline will mitigate that.

The main thing I like about this setup is that I think it will help students get more comfortable with writing proofs, particularly the role of failure in writing proofs. Most proofs are not correct the first time and need iteration. Many students still struggle, even though they are seasoned veterans in this course, with coping with turning in work that needs significant revision. A lot of them see writing a proof as if it were defusing a bomb, where everything has to fit together just so, according to a system of rules for logic and grammar that they don't fully understand, or else the thing explodes and takes them with it. I want them to realize that writing a proof is like writing anything else – it's a process of reflection and iteration, and they should be unafraid to engage in it.

I also like that this system seems to provide equal footing for both parts of the "T" that I am aiming for: both foundations and connections, and space to make mistakes and correct oneself on each. Also, I like the two-stage testing idea – I have never used it before but a 75-minute class is the perfect environment for it.

As with Calculus and other classes in which I've used specs grading, I dislike how complicated this all is. It could certainly be simplified, but it would come at the expense of accuracy and fidelity to students' learning. Finding the right sweet spot between complexity and accuracy is very difficult.

Things I am not sure about yet:

- This class always gets me worried about academic dishonesty. It's so hard to come up with somewhat original proof problems, and so easy to search them up online. I hope that the revision process disincentivizes cheating – why plagiarize when you can just revise? – but I am sure academic dishonesty will come up at least once. I just hope it's not an epidemic.
- I'm wondering if the group stage of the Connections tests are going to be overwhelming for students who are introverts or have different neurological needs. Or just whether it will be too loud to think.
- There's considerably more timed assessment in the class than I used to have in an upper-level course and I'm wondering if this is going to be a problem, e.g. every time there's a Foundations quiz or Connections test there will be some student who misses, and I have to turn into a lawyer to determine if the absence is legitimate or not and how to handle it.
- Finally, I am not sure what the unknown unknowns are. What should I be thinking about that I am currently not thinking about?

So that's the state of my mastery grading systems in Fall 2018. I'd be curious to know your thoughts, nitpicks, suggestions, and so on for any of this – just air them out on Twitter and @-reply me (@RobertTalbert) or email me to let me know.

]]>When I finished up my sabbatical in May, I turned my attention to two things: Shuttling my children back and forth to camps and sports practices, and trying to remember how to teach. I've gotten pretty good at the first of these. As for the second, we'll find out in two weeks, when for the first time in 15 months I will step back into the classroom as an instructor.

This Fall, I'm teaching a section of Calculus 1 and two sections of Modern Algebra 1. I started prepping these back in April, as the sabbatical was winding down, because I knew I would have a lot of rust and would need a solid 3-4 months not only to get back into the routine, but also find ways to make my teaching new and refreshed and not *merely* back into the old routines. The whole journey of rediscovering and reinventing my process for course design and preparation is worthy of another post. For now, I want to describe one particular aspect of my courses for this fall, namely the use of **specifications grading** in each of the courses I am teaching. I first wrote about specs grading three and a half years ago and have blogged about it off and on here as I've gone through several iterations in my classes. I also moderate a Google+ community on this subject^{[1]} and I can attest that there is a lot of interest in this alternative system of grading. In this post, I'll detail how I'm using specs grading in my hybrid Calculus section. In the follow-up, I'll write about how it's being used in Modern Algebra.

Specifications ("specs") grading is a species of mastery grading which has been around for quite some time in various forms. Linda Nilson is credited with coining the phrase "specifications grading" and her book on this subject has much, much more about it; this article provides an accessible but detailed overview. My interview with Linda back in 2014 also has a lot of her insights.

Specs grading is based on the following principles:

- Student coursework is evaluated not using a point system but rather using a simple two-level (i.e. Pass/Fail) rubric, according to whether the work meets or exceeds predetermined criteria for quality. (Nilson suggests that "Passing" should be the equivalent of "B" level work, although this is up to the instructor.)
- Those criteria come in the form of clear, detailed
*specifications*that are made public to the class (often fortified by examples of passing and non-passing work). - Since there are no points, there is no partial credit. Instead, (most) student work is allowed multiple attempts, with non-passing work given extensive instructor feedback that students can use to improve what they turned in. Resubmissions are graded according to the specs and the grades updated if there's improvement.
- The students' course grades are still A/B/C/D/F, but determined by the quantity and quality of the work they turn in that meets the specifications --- not a statistical formula that combines points. The higher the grade, the more work and/or higher quality of work the student must supply as evidence.

I've been using specs grading since 2015, and it's revolutionized my teaching. It's not always easy and students sometimes push back; but it's absolutely a net win for all of us.

The section of Calculus I'm teaching has some distinguishing characteristics. First and foremost, it's a hybrid section, meeting twice a week (11:00-11:50 Mondays and Wednesdays) and the rest of the course is asynchronously online. The two F2F hours will be used only for active learning tasks and for assessment. The rest of the time, students will be engaging in reading and viewing, working out online activities, and practice. The vast majority of the course in other words is individual with just a touch of group time together.

The 26-ish students in the course^{[2]} are almost evenly split between first-year and third/fourth-year students, and there are a lot of engineers and a lot of biomedical sciences majors in the course. Most of the third/fourth year people are biomedical. So it's a mix of new students who are still emerging from high school; and grizzled veterans who are getting their last few courses out of the way before med school or grad school.

Here's the syllabus for the course which has all the details of the grading system. The grading system is on pages 3--6. What follows here is a summary.

Students do four different kinds of work:

**Guided Inquiry**, which are structured assignments for students to use while learning new material on their own. We're a flipped learning environment, plus the course is hybrid, so students are doing most of the initial learning of concepts independently, and Guided Inquiry provides structure and guidance as students do this. Here's the first Guided Inquiry assignment in the course if you're interested. As you can see, each one contains text and video resources plus exercises that are submitted online prior to class. (I used to call these "Guided Practice"; I felt "Inquiry" better described these than "Practice".) Guided Inquiry is graded**Satisfactory**or**Unsatisfactory**on the basis of completeness, effort, and deadline-compliance only.- The course also has 24
**Learning Targets**that spell out the main content objectives of the course. Ten of these are**Core**Learning Targets which are (in my view) things that every student who claims comptency in Calculus need to demonstrate they can do. The other 14 are**Supplemental**Learning Targets and are important but not (IMO) essential tasks. You can see the whole list of Learning Target in the syllabus in Appendix B. Students show their skill with these Learning Targets through in-class quizzes, which are graded**Satisfactory**or**Progressing**based on specs that are different for each target. - Students also do
**online homework**sets in the class which give them further practice with basics. We use WeBWorK for online homework; I am planning on two sets per week with 3-6 problems each. These are the only thing in the class graded with points because I can't make the system do otherwise, but it's essentially Pass/Fail because each problem is 1 point if correct, 0 otherwise. - Finally, students do
**Labs**, which are extended application problems involving computer technology. These are actually standard for all of my department's sections of Calculus and are a good way to assess student skill at extensions and applications. These are graded**Excellent**,**Satisfactory**,**Progressing**, or**Incomplete**. The two new levels here are for work that is truly outstanding, and work that has major gaps, omissions, or systemic errors that render evaluation impossible, respectively.

There is also a final exam that will be based heavily on Learning Target quizzes and will be graded with a sort of hybrid of points and specs. Also, I will be awarding **Experience Points** for doing things to engage with the class, like engage in online discussions or do something useful in a class meeting.

As is the case with specs grading, almost everything is redoable. Learning Target quizzes can be retaken during later quiz sessions, during a few designated quiz-only class meetings in the semester, and with any leftover time at the final exam session. Quizzes can also be retaken during office hours through 15-minute appointments. Labs can be redone through a take-home process described in the syllabus. WeBWorK sets can be redone as often as you want before the deadline. (Guided Inquiry assignments aren't redoable since they are for class preparation.)

There are two parts to the determination of a grade: Finding the *base grade* (just the plain A/B/C/D/F with no plusses or minuses), and finding whether or not you have a plus or a minus on the base grade.

My belief is that the base grade should be determined and affected only by important stuff: Mastery of basic information, ability to extend and apply the basics, and preparing for class meetings. Other, not-as-important work --- such as the final exam^{[3]} and class participation --- should not affect the base grade but should be used to determine plus/minus modifiers.

You'll see that philosophy reflected in the requirements for each base grade, which are:

With Learning Targets, **Completing** a target means earning "Satisfactory" on one of the quizzes over that target. **Mastering** a target means earning "Satisfactory" on a *second* quiz over that target, which students can do during any time set aside for retaking Learning Target quizzes (of which there is a lot).

I built this table from the middle outward, by first asking: *What does baseline competency in Calculus look like?* That's what a "C" is. I happen to think my criteria for a "C" are *very* minimal almost to the point of feeling uncomfortable about it. But I decided that I'd err on the side of leniency rather than being too strict about it. For a B, students have to do more, and what they do has to be better than for a C. For an A, the same but moreso.

The base grade gets a + or - modifier depending on what happens with the final exam and XP:

- If the final exam grade is at least 85%,
*and*at least 85 XP are earned, add a "+" to the base grade. - If the final exam grade is less than or equal to 50%,
*or*50 XP or fewer are earned, add a "-" to the base grade. - Otherwise the course grade equals the base grade.

So blowing it on the final, or willfully disengaging with the class (while still getting required work done) will not kill your grade, but it won't be without effects either. On the other hand, doing really well on the final and staying engaged with the class gives you a bonus, but not a massive boost.

What I like about this system:

- It checks all the boxes for me that specs grading normally does. Students have clear expectations and guidelines; it promotes a growth mindset; it's relatively simple in terms of the moving parts, and there's no mysterious statistical formulae to contend with; and it should shift the narrative on grades from "I need to make at least $x$ on the final to get $y$ in the class" to "I need to improve on Learning Target $n$".
- It fixes a bug with previous incarnations of my specs grading system where a student could demonstrate competency on a topic in one part of the semester and show evidence of non-competency later. Having students "Master" Learning Target with two points of data, plus having a final exam, gives more confidence.
- It all stems from a simple theory about grading, that grades should be based on basic mastery, ability to extend the basics, and staying engaged with the course. The "why" of this is easy to grasp and explain.

What I don't like:

- Like every specs grading system I have ever seen or tried, it still feels overly complex and forbidding to students. In my own mind, this makes perfect sense, but what about everyone else?
- There are steep dropoffs for not making some of the requirements. For example, if you complete everything for an "A" but have only 69% on online homework, your base grade isn't an A- or B --- it's a D! I tried building in more plus/minus rules to handle near misses like this but it made things unreadably complicated. I decided to leave things alone and instead make concerted efforts get students to understand that there are severe consequences for missing the requirements, and acceptable work in one area doesn't "average out" with unacceptable work in others. But I have a bad feeling that in December I'll be dealing with at least one student who didn't get the message, and thinks he earned an A- when in fact he has a D.

What I'm not sure about yet:

- I'm not sure whether I have budgeted enough time in the semester for in-class reassessment on Learning Target quizzes. I think so, but I fear we'll be in week 12 and half the class will have only completed half the Core targets, and then things get really scary. But in a hybrid class, it's difficult to know when/how to add more face-to-face time.
- I'm not sure what kinds of wild edge cases will show up where the grade doesn't reflect the student's work. On the flip side, I'm not sure if there are loopholes that students can game to get grades they didn't earn.

One final thing I will say about the complexity of specs grading systems: *All* grading systems are complicated. Some of them are just more open and transparent about it than others. In a traditional points-based system, when you see the table in the syllabus that says there are three tests each worth 25% of the grade and a final that is also 25%, it seems simple, but actually it isn't. It's just hiding the complexity: Figuring out what will be on the test, how the tests related to the course objectives (if there are any), how the composition of the final compares to the tests, and so on. There's a lot that students don't know and won't know until it's test time, and then it's one-and-done, and if you have a bad day or are a bad test-taker, you're screwed. With specs grading, it looks complex but that's because everything is laid bare and the student has complete control over all of it. This is a tough sell to students sometimes, but at least it's sellable.

In the next post, I'll go through the specs grading setup for Modern Algebra, which is a very different beast.

Yeah, Google+. Forgot about that one, didn't you? Don't worry --- everyone else has forgotten about it too. ↩︎

We're still getting changes in enrollment and will probably get this right up until the middle of week 1. ↩︎

I do not believe that the final exam in the course is really all that important. It has the illusion of importance because we come from a tradition of assessment that places a huge proportion, sometimes 100%, of a student's course grade on a few high-stakes tests. Like most traditional assessment, this choice to emphasize high stakes testing doesn't seem to be based in any sort of data, or really based in anything at all except the desire for a few powerful professors to engage as little as possible with teaching. For me, true assessment is day-to-day, and the final exam --- which I only readopted in my specs grading system last year --- is there only to provide another layer of data to solidify assessment that has already taken place. So it's not worth much in and of itself. ↩︎

The biggest question, by a wide margin, I hear from faculty when we talk about flipped learning is: *How do I make sure students do the pre-class work?* It's a real concern. Flipped learning is predicated on the strategy of freeing up as much time as possible in students' group context (i.e., their class meetings) and then focusing that time on tasks that engage students *beyond* the basics. In order to do this, in flipped learning environments students cover the basics themselves, prior to class. And there's the potential issue: If students *don't* do this pre-class work, then this entire plan goes straight down the tubes.

To repeat: This is only a *potential* issue, albeit a serious one. It's not inevitable! There are strategies we instructors can employ to head this issue off, and in this article we'll look at what research says about the underlying issues and then translate those into practices that will help our students come to class equipped and ready to work.

Before we dive in, let's clarify: Nobody can "make sure" that students do the pre-class work in a flipped learning environment, whether it's watching videos or reading or anything else. You and I cannot "make" students complete any kind of assignment if they, the students, don't ultimately decide to do it. So the real question isn't *How can I make sure students do the pre-class work*: It's, **How do I design pre-class work that students want to do?** (That's not a bad way to think about *any* assignment we give.)

Two threads of research in psychology can shed some light on this issue: *self-determination theory* and *cognitive load theory*.

**Self-Determination Theory** (SDT) is a theoretical framework from psychology that has to do with *motivation*. It was originated by Richard Ryan and Edward Deci in the 1980s and was primarily expressed in a paper they published in 2000 (1). You may have heard of the terms *intrinsic motivation* and *extrinsic motivation*. Deci and Ryan gave us these. *Intrinsic* motivation refers to motivation to seek out challenges and improvements merely for their own sake. *Extrinsic* motivation, on the other hand, is motivation that comes from an outside force. When we go to the gym purely because we enjoy exercise, then this is intrinsic motivation; when we do it because our doctor told us to, or because we're being offered $100 to do so, this is extrinsic.

Deci and others determined that a person's inclination to intrinsic motivation is based on three factors: *competence*, *autonomy*, and *relatedness* --- respectively, a person's perception of being good at the task, having choice and agency in the task, and being psychologically connected to others while doing the task. People tend to find more intrinsic motivation for a task when any of these three aspects are increased. For example, if my doctor told me I need to work out more and therefore I am extrinsically motivated to do so, I'll feel more motivated to do it if I am part of an exercise class (relatedness), I get to choose which exercises I do (autonomy), or if my exercise instructor tells me I am doing a good job (competence).

These two kinds of motivation can interact and sometimes undo each other. Deci's research (for example, (2)) found that when people were given tasks for which they were intrinsically motivated --- in this case, solving a puzzle, with no rewards attached other than the fun of solving it --- and then later given monetary rewards for solving it, the subjects' intrinsic motivation for solving the puzzle decreased. We also know that outside inputs can alter the levels of a person's motivation to complete a task. A study by Jang (3) took 136 college students and gave them a task that was inherently uninteresting --- of course, it was a statistics exercise --- and then, later, one group was provided with a rationale given in non-controlling language for why they were doing the exercise ("Today's lesson will open the door for you to gain useful skills..."; see the original paper for the whole phrasing) while the other group received no rationale. The students with the rationale for their work showed greater interest, work ethic, and determination in completing the task than did students with no rationale given.

**Cognitive Load Theory** (CLT) is another framework from psychology that deals with the "cognitive architecture" of a human being and the amount of mental effort that goes into working memory. CLT was developed originated with John Sweller, also in the 1980's (see (4)) and distinguishes between three kinds of cognitive load: *intrinsic*, *extraneous*, and *germane*. Intrinsic load (not to be confused with intrinsic motivation) is the cognitive load that is irreducibly part of a task itself. Extraneous load is cognitive load which is not part of the task and does not actually lend itself to understanding or completing the task. Germane load is cognitive load that may not be inherent in the task itself but which is helpful for forming connections and schema for being able to work with similar or more advanced tasks later.

This illustration by Connie Malamed (source) is a good way to think about how these different kinds of cognitive load compare:

Let's suppose you're trying to bake a cake. There are certain cognitive parts to this that can't be removed: Checking to see if you have the ingredients, knowing how to operate your kitchen tools, making decisions about whether the cake is done. These can be made less burdensome through practice, but they can't be eliminated from the process because they are *part* of the process, i.e. intrinsic cognitive load. Extraneous load would be, for example, having to make a cake recipe where the units of measurement are in metric versus imperial units if you are in the USA, or vice versa everywhere else; unless you have kitchen tools that come in those units, you have to make room for the cognitive process of unit conversion which neither helps you make the cake or helps you build mental structures for later cake-baking. Finally, imagine that the recipe also includes some notes about the science behind the baking process; this is germane load because it's not strictly necessary, but if you can understand the science, then it could help you make sense of why the recipe is ordered the way that it is, helping you form a schema for baking.

So what does all this mean for instructors? How can self-determination theory and cognitive load theory help us create pre-class assignments (or any other kid of assignment) that students want to complete, especially when the entire flipped learning infrastructure is riding on that completion?

**Keep pre-class assignments minimal**. When writing your pre-class activities, ask:*What's the smallest amount of knowledge students need, to be successful in the in-class activities I have designed?*(Notice this means we design the in-class activity before the pre-class activity; see

for more on that.) Decide on a list of "Basic" learning objectives and design the pre-class assignment around*only that set*--- leave the rest of the learning objectives for class work. This practice focuses on eliminating extraneous cognitive load and emphasizing only intrinsic and germane load for student preparation. Also, ruthlessly eliminate any other extraneous load --- for example, asking two questions on the pre-class activity that assess the same thing --- and provide structure to help students manage the remaining load. Remember, we do not expect or need complete mastery before class, only*essential fluency on the most basic ideas*. Or as I tell my students, "Just Enough to Be Dangerous".**Promote intrinsic motivation by highlighting competency**. This doesn't mean "inflate students' self-esteem by telling them they're great for no reason" --- it means, give students plenty of opportunities to have a real sense of what they have learned from their work through feedback. If you use video, consider embedding mutliple choice quizzes inside them using tools like EdPuzzle. Or set up an online quiz using a Google Form that students can take following reading or video, not to be graded but for students to get feedback. Giving compliments to students who do well or improve their performances goes a long way too.**Promote intrinsic motivation by giving students autonomy**. Remember the example about working out? If I can pick which exercise to do, I am more likely to exercise at all, than if I am restricted to a single exercise. Similarly, we can find ways to let students pick*how*they want to learn in pre-class work. Instead of providing a list of videos and telling students, "Watch these videos", give students*learning objectives*instead and then providing a range of resources for learning --- text*and*video*and*simulations*and*anything else you can find --- and telling students: "Use whatever combination of the following resources works to help you gain fluency on the basic learning objectives". Stop worrying about whether all students watch all your videos. Instead shift that narrative from "making sure students watch the videos" to "providing ways for students to learn that make sense for them" and then assessing their learning.**Promote intrinsic motivation by giving students connectedness**. Students sometimes complain that in flipped learning, they don't have a way to ask questions while doing the pre-class work. In too many situations, this complaint is well justified: Connecting with other humans during pre-class work may be difficult, or even impossible or prohibited. SDT would tell us this is a bad thing, and that students will be more motivated if pre-class work has the promise of connectedness embedded in it. So, go ahead and let students work together on pre-class work. Give them robust systems for asking questions --- Slack, discussion boards, study sessions, etc. Make it clear that it's completely OK, even a good idea, to connect with others and ask questions when working on the assignment. Yes, some students may end up copying answers from others; I think we should consider this an acceptable risk, and explain to students the demerits of doing this and then trust them to do the right thing.**Give a non-controlling rationale to students explaining "why".**. Remember the study in which learner's motivations improved when they were simply given a friendly explanation of the learning task? The simplest way to improve motivation for pre-class work is just to tell students why they are doing the pre-class work in the first place, in non-controlling language --- i.e., avoid tough-professor talk such as "You are doing this because you'll be totally screwed during the in-class portion if you don't". Stick in a paragraph at the beginning explaining what they are about to learn, how it relates to what they already know, and why it's important. (And why it's important to learn it on their own.)**Take it easy on the grading**. As Deci's 1971 study shows, far from the notion that "if it's not graded they won't do it", extraneous rewards stunt the development of intrinsic motivation. So if you choose to grade pre-class work, do so lightly. I used to grade pre-class work on a 10-point scale and then give a 5-point entrance quiz over the material, because "accountability". Then I realized my students hated doing the work because of all that "accountability". So I started grading it*Pass/Fail on the basis of completeness and effort only*(i.e. not on correctness), and no start-of-class quizzes. And guess what: Students' completion rates skyrocketed and they showed up to class more prepared. So perhaps Deci was onto something. If you're feeling really radical, consider not grading pre-class work at all.**Consider ditching video**. Finally, keep in mind that pre-class work does not have to revolve around "delivering content" through video or text. Consider replacing video-watching or text-reading exercises with something more engaging, like a game, interactive demo (e.g. a Wolfram Demonstration), or even a physical, analog activity that gets students actively involved with exploring concepts in an open-ended way. A fascinating recent article from a pair of researchers at Stanford (5) suggests that having students interact with something tangible and*then*watching a video on a topic, is significantly more effective than watching the video first and then doing active work. I'll have a lot more to say about this study in another post, but suffice to say that flipped learning does*not*require video or reading before class, or even direct instruction or "covering material" first, and such activities could be a lot more interesting and hence more likely to be completed.

This idea of pre-class work isn't confined to flipped learning, of course. It can apply to any teaching method in which you want students to learn something before doing active work during class. We can't *make* students do the pre-class assignments (unless we have an army of enforcer goons at our disposal), but we *can* make our assignments more interesting by leveraging what we know about motivation and cognitive load.

**Do you have any tips for improving student success in doing pre-class work Share them in the comments!**

[1] Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. *American psychologist*, *55*(1), 68.

[2] Deci, E. L. (1971). Effects of externally mediated rewards on intrinsic motivation. *Journal of personality and Social Psychology*, *18*(1), 105.

[3] Jang, H. (2008). Supporting students' motivation, engagement, and learning during an uninteresting activity. *Journal of Educational Psychology*, *100*(4), 798.

[4] Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. *Learning and instruction*, *4*(4), 295-312.

[5] Schneider, B., & Blikstein, P. (2016). Flipping the flipped classroom: A study of the effectiveness of video lectures versus constructivist exploration using tangible user interfaces. *IEEE transactions on learning technologies*, *9*(1), 5-17.

Image: https://commons.wikimedia.org/wiki/File:Study_(16840395246).jpg

]]>One of the biggest issues professors face, especially when anything slightly nontraditional, is the risk of negative reactions from students --- "pushback" as it's usually called. And the instrument that causes student pushback to bloom into a full-blown, action-preventing fear is something we all know: The end-of-semester course evaluation. Student pushback can translate into poor course evaluations, and poor evaluations --- thanks to their outsized role in determining personnel decisions in higher ed --- can lead to losing your job.

I'm not writing today about the merits or demerits of course evaluations. Instead, I want to promote a very simple teaching practice that both mitigates the risk of poor course evaluations *and* helps faculty nip student pushback in the bud: **Give an informal evaluation to your students in the middle of the semester**, which is right about now for most folks. I've been giving mid-semester evaluations to my students for years and it has never produced anything but great results for me. It's simple, costs nothing, and produces loads of actionable information on your teaching that you can use to catch and address issues before they become irreparable.

The premise behind a mid-semester evaluation is what you think it is: You give some sort of instrument to students that asks them for input on what's working well, what's not working as well, and what changes should be made heading into the second half of the semester. Then you look at the responses and see what the good and bad points are, what issues are arising, and what changes you might make --- and then take action.

Here are a couple of common and simple methods for mid-semester evaluations:

- Laura McGrath describes in this article at GradHacker the
**start/stop/continue**method. Ask students three questions on a form:*(1) What should we STOP doing in the course? (2) What should we START doing in the course?*and*(3) What should we CONTINUE doing in the course?*This is fairly broad, but by casting a wide net, you can often get a good global view of how students are experiencing the course, and plenty of information to make course corrections heading into the second half. - In a related article at ProfHacker, George Williams expands this into four questions:
*(1) What's going well? (2) What needs improvement? (3) What can the students do to improve the class? (4) What can the instructor do to improve the class?*This is a nice variation on start/stop/continue in that it stresses that improving the class is a shared responsibility and gets students thinking in concrete terms about both sides of that responsibility.

My own practice for mid-semester evaluations includes a mix of numerical and open-ended items. The numerical items are a selection of items taken verbatim from the actual end-of-semester evaluations. I use some of the more important items, especially any item that has been an issue for me in the past. For example, one of the items on our evaluations is *The instructor treats students with respect* --- I always include this on my mid-semester evaluations because this is a really important item for me, and I want to know how I am doing. I also include the item *The instructor grades and returns assignments and exams promptly* because I've had lower scores than I want on this item in past classes, and I want to give it special attention to know if I need to work harder on it before the end of the course.

Along with those numerical items, I include three open-ended questions -- *(1) What are some aspects of the course and the instruction that are helping you learn so far? (2) What are some specific aspects in the course or the instruction that could be changed to help you learn better? (3) What other overall comments, concerns, or suggestions do you have about the course so far?* This is basically start/stop/continue and you could easily do that instead.

All of these items are packaged up and given to students as a Google Form, with the numerical items phrased as Likert-scale questions and the open-ended questions given as "paragraph" questions. Then just copy the link to the form and send it out or post it on the course LMS, and let students work on it until a preset deadline.

Google Forms helpfully dump student responses into a spreadsheet, so it's very easy to work with the data. Use the numerical results to triangulate the verbal results. Make histograms. Go crazy. And importantly, *do something* with the results. The whole point of mid-semester evaluations is to get advance warning that there are things to work on before they become unfixable.

To reiterate this last point, if you give an evaluation, then also make a sincere effort to work on areas that clearly need to be worked on. Here we can take an example from Starbucks. Baristas at Starbucks are trained in what's called the LATTE method for dealing with customer complaints^{[1]}. Applied to student evaluations, it would say:

**L**isten to what the students are telling you through their responses.**A**cknowledge any legitimate problems that are being reported --- and even the ones that are not really legitimate, for example complaints that come from misreading the syllabus or mishearing your expectations for student work.**T**ake problem solving action to address these problems.**T**hank the students for making you aware of their issues. Remember students are sticking their necks out to make their thoughts heard.**E**xplain why the problem occurred and what you are doing to fix it.

Ever since making mid-semester evaluations a regular part of my teaching, my *end*-of-semester evaluations have improved significantly. I believe part of that is that by simply giving students a way to be heard, they become more favorably disposed to the class and to me; and if there are real problems to address, I know about them *early* rather than after the course is over, so issues that arise on mid-semester evaluations rarely recur on end-of-semester evaluations. Mid-semester evaluations also help build a culture of trust and openness that makes your class a more welcoming place for your students.

If you've got ideas or variations on this idea of your own, leave them in the comments.

Image: https://www.pexels.com/photo/earphones-listen-music-590330/

No, I am not saying that we professors are like baristas, preparing and serving up products on demand to paying customers. I've written about this at length already. This is just an idea --- a very good one, if we have ears to hear --- that can be applied just as well in the client-consultant model of higher education as it can in your local Starbucks. ↩︎

The main focus of my sabbatical year has been the work I'm doing with Steelcase Education, but it's not the only project I'm working on. On the two days each week I am not on site at Steelcase, I'm working on two research projects related to flipped learning that were started a few years ago and for various reasons got put on the back burner; I'm taking 2/5 of my sabbatical time to put them on the front burner and get them moving again.

One of those projects is a study on the effects of flipped learning environments on students with learning disabilities. That term "learning disabilities" is broad, referring to any condition that introduces difficulty in learning, usually in the context of a neurological issue (as opposed to a physical disability). I'm studying this because although flipped learning proponents often *say* that flipped environments help LD students, in reality it's much less clear that it's a net benefit for those learners. A learner with ADHD, for instance, may benefit from having lecture content pre-recorded so they can pause, take breaks, and replay at will; but the same learner might be overwhelmed by the amount of activity that takes place in class, or have related issues with executive functioning skills that make it harder to manage their time and complete pre-class work. So, my collaborator (special education expert and GVSU colleague Amy Schelling) and I are developing flipped learning materials for one of our remedial mathematics courses, deploying them, and recruiting LD students to talk to us about their experiences learning with them so we can map out the benefits and challenges of flipped learning for these folks.

Although the research on flipped learning is still growing exponentially, I've found virtually nothing in the literature about flipped learning with LD students. However, it turns out there is a small but very interesting body of work on the experience of LD students with online and blended instruction, which is closely related to flipped learning. One of those studies really resonated with me, and that's the focus of this 1000-word lit review:

Madaus, J. W., McKeown, K., Gelbar, N., & Banerjee, M. (2012). The online and blended learning experience: Differences for students with and without learning disabilities and attention deficit/hyperactivity disorder.

International Journal for Research in Learning Disabilities, 1(1), 21-36.

First, some terminology. *Online* courses are those in which 100% of the teaching and learning take place through intentional online activities. *Blended* (sometimes called "hybrid") courses are those where some significant percentage, usually more than 20% but less than 100%, of teaching and learning is done online, and the rest is done through face-to-face meetings.

The questions that this paper addresses are:

- What are the barriers experienced by students with ADHD and other learning disabilities (hereafter, "ADHD/LD students") in online and blended courses?
- What are the opportunities afforded to ADHD/LD students with ADHD and other learning disabilities in online and blended courses?
- How can instructors and instructional designers build online and blended courses that address the needs of ADHD/LD students?

These are big and important questions. The authors note that in 2010, 31.3% of all US college students took an online course. In the same time frame, the percentage of US college students with ADHD was 19.1%. With more students taking online and blended courses *and* more students identified as having some form of learning disability, it becomes imperative to understand how these two conditions interact.

This is a qualitative study based on interviews with ADHD/LD students in online and blended courses. The researchers contacted instructors at their campus, who then spread the word to their students and gave them the choice to opt in to the interviews if they wanted. (The students were paid for their participation.) A total of 29 students were invited; 20 students participated. There were 10 students with ADHD/LD and 10 students without disabilities.

Students were interviewed using a structured interview protocol, meaning that each student was asked the same set of questions. The interviews were recorded and transcribed. The researchers used inductive analysis to examine the transcripts, which means that rather than approach the interviews with a predetermined set of phrases, they instead let the interviews speak for themselves and let the patterns and categories of responses arise naturally. Then the researchers used qualitative analysis software to do a frequency analysis of the data to refine and simplify their categories into a list with little to no overlap.

The results of the interviews revealed many benefits of online and blended courses for ADHD/LD students:

- Students cited the organization of online and blended courses, especially having course materials stored and organized through the course LMS, as a key benefit. This was mentioned by all students, not just ADHD/LD ones, but ADHD/LD receive more benefit from being able to lean on the LMS to help manage organization rather than doing it all themselves.
- Especially, students noted that having the course notes posted online made it easier for them to pause, take breaks, think, and annotate those notes. This echoes the benefit often cited for flipped learning with the ability to pause and replay video content.
- Interestingly, the ADHD/LD said that instructors are much more available in their online and blended courses than in their face-to-face courses. That's surprising since establishing and maintaining instructor presence is often a major problem in conducting online and blended courses (and this observation wasn't universal across all students; see below).
- ADHD/LD students were also more likely than the non-disability group to mention that online and blended courses offer an advantage in the form of accessing and learning from their peers. So, not only did ADHD/LD students feel they had more access to their instructors in online and blended courses, they also felt this way about access to classmates and that this access was beneficial to their learning and helped diminished the sense of isolation and anonymity that these students often face, even in face-to-face courses.

The interviews also revealed some challenges:

- ADHD/LD students noted that there can be a lack of clarity in online and blended courses, for example if a quiz is posted and students aren't aware of it. This can happen to any student, but for students with attentional issues, this can be especially problematic.
- Similarly, ADHD/LD students noted that if the course navigation on the LMS is unclear, it is a real problem. This is true for all students; but there's an outsized negative impact on ADHD/LD students who rely more on the LMS than do non-disability students.
- While some ADHD/LD students noted that online and blended courses decrease anonymity, others found that they were
*more*anonymous and isolated, often because their instructor was non-responsive to their emails or didn't give timely constructive feedback on their work.

Online and blended courses can be highly beneficial to ADHD/LD students, not only through the flexibility and autonomy they provide but also by the level of pre-packaged organization and enhanced communication their provide. This of course is *not* limited to online and hybrid courses. All of our courses ought to be flexible and organized, and we should always strive to give choice to students and to be highly responsive to them. But take note that the benefits of doing this are amplified for students with learning disabilities, even though we often may not see it.

Responsiveness and clarity are key issues with ADHD/LD students, and there are simple things we can do to enhance both: Reply to emails within a day or receiving them --- and don't make excuses about it. Clearly state what is expected on each assignment. Give frequent reminders about course expectations and due dates at key points in the course. Give the LMS a simple navigational structure with clearly-labeled areas. Don't leave important items out of your syllabus.

All of these pieces of advice were directly reported in the interviews; they ought to be common sense for every instructor and an essential part of being a professional in higher education. They are also echoed in emerging standards for course design such as Universal Design for Learning and Quality Matters, which are well worth exploring on your own.

The biggest weakness of this study, acknowledged in the study itself, is the sample size of 20 students, only 10 of whom were ADHD/LD. This limits the external validity of the results, although there are well-regarded studies with sample sizes in the single digits. So take it with a grain of salt; a replication of this study would be a big help in carrying the results to the next level.

I found this paper to be really enlightening, and I hope it will help me understand how these learners deal with flipped learning environments as well as just become a better, more responsive instructor to them. Share your own thoughts in the comments.

]]>