For most people this will probably be an incredibly boring post. You’ve been warned. For others, like me, who are very concerned with language in science, and precision of language, this is the post for you.
I’ve always described our review criteria on Balefire Labs as “objective.” But lately I’ve been noticing that a lot of other edtech review sites describe their criteria as objective. And I don’t think their criteria are all that objective. So I wanted to dig into that here and ask for your help in thinking it through.
To me, an objective criterion is one that is clearly defined in specific terms. The terms are specified sufficiently that two or more people can agree on whether or not the terms of the definition are met. So, for example, an objective criterion for an educational app might be “includes feedback for correct answers.” Subjective criteria, conversely, might be “fun and engaging.”
But I’ve been noticing that some edtech review sites, instead of using the term “objective” to mean the opposite of subjective, are using it to mean unbiased. For me, that is a completely separate thing having to do with whether or not a reviewer has an interest, monetary or otherwise, in seeing an app do well in the marketplace.
So here’s my dilemma that I need your help with and I’m going to get very nerdy and pedantic here. In science, we have another term that I could use instead of “objective,” given the clear differences in how people are using it. That term is “operational definition.” An operational definition defines something according to a specific process or set of validation tests that determine its presence or quantity. One Princeton University faculty member’s website explains its use thus: One might use definitions that rely on operations in order to avoid the troubles associated with attempting to define things in terms of some intrinsic essence. (Fun and engaging, anyone?)
Here is some more good stuff on operational definitions from that same website: Properties described in this manner must be sufficiently accessible, so that persons other than the definer may independently measure or test for them at will. The binary version [of the process] produces either the result that the object exists, or that it doesn’t, in the experiential field to which it is applied.
Okay. So you can probably see where I am with this. The way that we, at Balefire Labs, approach our reviewing is with operational definitions for our criteria. Essentially, we take a science-based approach to reviewing and we strive for our criteria to be unambiguous in their application by us and in their consumption by customers. Why do we think that’s important? Because we are striving to tell customers what they are getting in each app that comprises its instructional quality (based on best practices from educational research), not our subjective opinion of what is fun, engaging, or similar (unbiased though it may be).
Not to mention the fact that what is fun and engaging is pretty idiosyncratic and related to kids’ skill levels. But that’s another post.
I had been thinking that referring to our criteria as “objective” would simplify the marketing process so that we wouldn’t have to burden our customers with learning about operational definitions and why they’re important. But now I’m in a quandry about that because I think it’s confusing when people use the same word, in this case “objective,” to mean multiple things. Instead, should I take on the task of using “operationally-defined criteria?”
Need your opinions and advice on this. Have any of you had experience educating the public on operational definitions? And how has it gone? Thanks in advance for the help!
Related articles
- Operational Definitions and Health (petruchio71.wordpress.com)
- Measuring Academic Quality is Just Hard Work (assessingforlearning.wordpress.com)
disrupt learning! 
I’d go with operational definition. The word “objective”, isn’t no longer objective lol.
Excuse my double negative, I meant no longer.
Operational contains some excess baggage as well. The term comes from methodological behaviorism and is employed today to mean that there are observed indicators of unobserved processes. A score of 8 on the XYZ test of emotions indicates high anxiety, is an example. It operationally defines anxiety. What I think you are looking for is the explicit versus implicit distinction. There are explicit criteria, those that can be observed and directly measured and there are implicit criteria, ones that are assumed and not generally observed. One could then actually sort into each category. Explicit criteria: feedback to user, number correct to exit, etc.; implicit criteria: fun and engaging, (perhaps operationally defined as number of minutes of continuous play, etc.).
Joe, the explicit versus implicit is an interesting idea, and probably easier to make clear than operational definition. I’m less concerned about the methodological behaviorism baggage because I’m targeting a mainstream edtech audience. Right now I’m just thinking, practically, what is the best route to take. We’re not doing implicit criteria, but that would be a good way to differentiate what we are doing from what others are doing. Thanks for the suggestion!
How about specific and/or measurable criteria? The simpler, the better, I think.
You can’t get to the point where your website becomes awash in technical details, so I am with Fran. As much as possible try specific, measurable criteria, and let the rest fall by the wayside. Joe’s right, there is a giant thicket of potential issues with getting TOO operationally defined–and you will lose the folks you want most to engage. PERHAPS you could try a quick primer on what YOU mean by objective–have two columns with attributes on each side, but even that seems like it might open the door to more “how many angels can balance on the head of a pin” discussion. Read the Precision Teaching list to see how far this can take you from engaging potential customers!
So to clarify, on our website the criteria are defined in specific, measurable terms. And I can definitely appreciate that once a customer is on our site we can differentiate ourselves easily with how our objective criteria are defined. The issue is with customers who are not yet on the site. Is there a way that we can differentiate our description of our criteria in order to drive more people to our site on that basis?
How about “built in measurable behavioral criteria” ? Or “intrinsically programmed behavioral criteria”? Both mouthfuls and not so simple.
I think too, we want to avoid saying “behavioral,” per se, because many of them are behaviors of the app, not behavioral of the learner and I don’t want that to get confusing.We probably will just stick with objective, for now, and see how it goes over with users. We can always add more detail to the definitions, if necessary.