In this brave new world of big data and analytics in education, we have a lot of thinking to do.  Having a ton of data to inform what we do and analytics to help customize our instruction has the potential to be very powerful.  But massive amounts of data and clever algorithms won’t mean much if we aren’t asking the right questions and carefully thinking through what the answers mean.

A few days ago I reposted a post from eLearning Development News called “Tracking isn’t Learning.”  Gary Hegenbart, the author, makes some great points, differentiating “tracking metrics” from “performance metrics.”  The overall point is that all data are not the same and you have to think about what the data really mean.

So I’d like to talk about student errors today, particularly in the context of digital learning programs.  And I’d like to talk about types of errors in a signal detection theory paradigm, represented by the diagram here.

Imagine this scenario: a student is presented with the spoken word “dog” and is shown a picture of a cat and a dog.  The student’s task is to choose the image that matches the spoken word.  If the student chooses the dog picture, this response is not only a “hit” (given dog, chose dog), but it is also a correct rejection (given dog, rejected cat).  In effect, the student has made two correct responses.  Likewise, if the student chooses the cat picture, this response is not only a miss (given dog, failed to pick dog), but it is also a false alarm (given dog, chose cat).  In the case of the error, picking the image of the cat when presented with the spoken word “dog” demonstrates that the student knows neither the concept “cat” nor “dog.”

Imagine a more complicated scenario:  a chemistry student is asked to identify the elements represented by the symbolic notation, “H2O.”  On the first step, the student identifies Helium as the element represented by “H.”  The analysis of this error is the same as the cats and dogs.  The learner has failed to pick the correct element name, Hydrogen (“miss“), and has mistakenly chosen an incorrect element name, Helium (“false alarm”).

The problem in most digital learning programs is that the false alarm (choosing Helium) typically is not tracked.  Why is this a problem?  Because by choosing Helium when she should have chosen Hydrogen, the student has shown you that she doesn’t know the correct symbolic notation for Hydrogen…..but she also doesn’t know the correct symbolic notation for Helium.  Without tracking this error and adapting the program appropriately, the student may not get the remediation on “Helium” that she needs.  What you find, generally, in education, is that the left column of the signal detection matrix gets all the attention.  Everybody knows how to track and evaluate “hits” and “misses.”  But almost no one does a good job tracking and evaluating “false alarms” and “correct rejections.”  And that right column is SO critical because it tells you the rest of the story!

So this is kind of complicated, right?  I mean, tracking all of this manually is just not something possible for any mere mortal teacher!  But this is one of the great opportunities that digital learning programs provide.  As decision-making algorithms are built, the “right column” can be taken into account, not only to be tracked, but to be evaluated as part of the overall student pattern of performance.  And the programs that include both columns are the ones you want to buy.  The difficult part for customers is that most companies don’t explain how they perform their analytics.  You’re pretty much just supposed to trust that they know what they’re doing.

Which brings me around to the title of this post.  Don’t be a sheep.  Ask questions.  If the sales and marketing folks don’t know, ask to speak to a product manager or an instructional designer or even (if you’re really brave!) an engineer who is involved in the algorithms.  The power of big data and analytics is to do something really meaningful to customize student learning.  But it won’t be meaningful if it’s not built in a meaningful way.

What are some digital learning programs that you know of that use signal detection theory for error analysis?

The image above originally appeared on http://www-psych.stanford.edu/~lera/psych115s/notes/signal/