Monthly Archives: September 2017

When is a Measure of Oxytocin No Such Thing?

Leave a reply

Medical Technology class room shots. 4/27/05 By University of Delaware [CC BY 3.0] . http://creativecommons.org/licenses/by/3.0)%5D, via Wikimedia Commons. Laboratory demonstration photos are just the best.

A few days ago, I promised I would demonstrate how you can use Denny Borsboom and colleagues’ concept of validity to evaluate whether a scientific device is a valid tool for scientific measurement. To review, Borsboom and colleagues argued that we can claim that a scientific device D provides valid measurement of an invisible substance, force, or trait (which I will represent as T) when two conditions are obtained:

(1) T must exist;

(2) T must cause physical changes to D that can be read off as measurements of T.

Once you accept this definition of validity, evaluating whether a scientific device is actually a scientific measure becomes simple (not necessarily easy, but simple)—even fun: You need to concern yourself with trying to find answers to exactly two questions: First, does this thing that researchers call T even exist? In other words, is T what philosophers of science would call a “natural kind?” Second, if T does exist, are we justified in believing that the natural kind we have named T causes physical changes to the Device that can then be read off as measurements of T?

In this post, I’ll use this approach to think through the validity of a biological assay technique that is often used in hopes of measuring oxytocin in human body fluids such as blood plasma, serum, or saliva. I’ve written a bit on this blog about research in humans on the social causes and effects of oxytocin (for example, here, here, and here). My colleagues and I see signs that a lot of the enthusiasm for this research is being driven by wishful thinking about whether the devices that are being called oxytocin assays are actually valid measures. To gain purchase on this particular validity problem, Borsboom and colleagues’ concept of validity tells you everything you should want to know: First, you will want to know whether there is a natural kind in the world that corresponds to the concept that we have decided to call oxytocin. Second, you will want to know whether that natural kind that we are calling oxytocin is responsible for physical changes in the Device. That’s all you need to care about.

The consequence of accepting this simple but strict definition is liberating. Among other things, you can brush aside validity arguments that rest on claims that individual differences in measured levels of oxytocin are correlated (for instance) with self-ratings of social support, or scores on a measure of empathic accuracy, or how many Facebook friends people have. Sure, all of those correlations might fit with somebody’s theory of oxytocin, but validity arguments that rest on correlational claims like that are so 20th-century.

All you need to concern yourself with is (a) whether oxytocin exists; and (b) whether oxytocin is causally responsible for the scores that your device produces. You can quickly satisfy yourself that (a) is true: Sir Henry Dale extracted oxytocin from the human pituitary gland in 1909. The biochemist Vincent de Vigneaud identified its molecular structure in 1953. So, all that’s left to confirm is (b).

And how do we confirm (b)? Through experimental research, not correlational research. Quite simply, the question we want an answer to is this: Does the nine-amino-acid substance that we have come to call oxytocin exert causal effects on the physical states of a particular device that we can read off as measurements of that substance? If so, when you add known quantities of oxytocin to a container (or to an animal) that has zero oxytocin in it, the device should then undergo physical changes in proportion to the amount of oxytocin that you added. From those changes, it should be possible to work backwards and solve for the amount of oxytocin that was added in the first place. If you can’t do that (and, as a few of us have been arguing, with some of the most popular approaches to assaying oxytocin, you can’t), then you should doubt the validity of that particular assay. Indeed, it might be more accurate to view such a device as a very expensive random number generator.

The oxytocin assays I am referring to here fail Borsboom’s validity test because they fail on criterion (b): Changes in physical states of those assays cannot be read off veridically as changes in oxytocin. Other measures can fail the Borsboom test for a more interesting reason: The trait that they supposedly measure doesn’t exist in the first place.

I’ll look at that scenario in my next post.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.

Postscript: Denny Borsboom tells me that the reason that their paper is not visible to Google Scholar is that the citations for that paper are getting merged with another one of their papers. What a drag.

(Following an interesting interchange on Twitter with Cameron Brick and Dave Pietraszewski about essentialism in psychology and the hazards it creates for scientific progress, I thought I would re-post this 2017 blog entry, which might be useful and/or interesting to some, and perhaps even entertaining for an extremely small subset of that small group. I daresay the concerns I raise here aren’t any less concerning in 2021. ~M)

TWO years ago, I idly surfed my way to a harmless-seeming article from 2004 by Denny Borsboom, Gideon Mellenbergh, and Jaap van Heerden entitled The Concept of Validity. More than a decade had passed since its publication, and I had never heard of it. Egocentrically, this seemed like reason enough to surf right past it. Then I skimmed the abstract. Intrigued, I proceeded to read the first few paragraphs. By that point, I was hooked: I scrapped my plans for the next couple of hours so I could give this article my complete attention. This was a paper I needed to read immediately.

I’ve thought about The Concept of Validity every day for the past two years. I have mentioned or discussed or recommended The Concept of Validity hundreds of times. My zeal for The Concept of Validity is the zeal of an ex-smoker. The concept of validity in The Concept of Validity has led to a complete reformatting of my understanding of validity, and of measurement in general—and not just in the psychological sciences, but in the rest of the sciences, too. And those effects have oozed out to influence just about everything else I believe about science. The Concept of Validity is the most important paper you’ve probably never heard of.*

The concept of validity in The Concept of Validity is so simple that it’s a bit embarrassing even to write it down, but its simplicity is what makes it so diabolical, and so very different from what most in the social sciences of have believed validity to be for the past 60 years.

According to Borsboom and colleagues, a scientific device (let’s label it D) validly measures a trait or substance (which we will label T), if and only if two conditions are fulfilled:

(1) T must exist;

(2) T must cause the measurements on D.

That’s it. That is the concept of validity in The Concept of Validity.

This is a Device. There are invisible forces in the world that cause changes in the physical state of this Device. Those physical changes can be read off as representations of the states of those invisible forces. Thus, this Device is a valid measurement of those invisible forces.

What is most conspicuous about the concept of validity in The Concept of Validity is what it lacks. There is no talk of score meanings and interpretations (à la Cronbach and Meehl). There is no talk of integrative judgments involving considerations of the social or ethical consequences of how scores are put to use (à la Messick). There’s no talk of multitrait-multimethod matrixes (à la Campbell and Fiske), nomological nets (Cronbach and Meehl again), or any of the other theoretical provisos, addenda, riders, or doo-dads with which psychologists have been burdening their concepts of validity since the 1950s. Instead, all we need—and all we must have—for valid measurement is the fulfillment of two conditions: (1) a real force or trait or substance (2) whose presence exerts a causal influence on the physical state of a device. Once those conditions are fulfilled, a scientist can read off the physical changes to the device as measurements of T. And voila: We’ve got valid measurement.

Boorsboom and colleagues’ position is such a departure from 20th century notions of validity precisely because they are committed to scientific realism—a stance to which many mid-20th-century philosophers of science were quite allergic. But most philosophers of science have gotten over their aversion to scientific realism now. In general, they’re mostly comfortable with the idea that there could be hidden realities that are responsible for observable experience. Realism seemed like a lot to swallow in 1950. It doesn’t in 2017.

As soon as you commit to scientific realism, there is a kind of data you will prize more highly than any other for assessing validity, and that’s causal evidence. What a realist wants more than anything else on earth or in the heavens is evidence that the hypothesized invisible reality (the trait, or substance, or whatever) is causally responsible for the measurements the device produces. Every other productive branch of science is already working from this definition of validity. Why aren’t the social sciences?

For some of the research areas I’ve messed around with over the past few years, the implications of embracing the concept of validity in The Concept of Validity are profound, and potentially nettlesome: If we follow Borsboom and colleagues’ advice, we can discover that some scientific devices do indeed provide valid measurement, precisely because the trait or substance T they supposedly measure actually seems to exist (fulfilling Condition #1) and because there is good evidence that T is causally responsible for physical features of the device that can be read off as measurements of T (fulfilling Condition #2). In other areas, the validity of certain devices as measures looks less certain because even though we can be reasonably confident that the trait or substance T exists, we cannot be sure that changes in T are responsible for the physical changes in the device. In still other areas, it’s not clear that T exists at all, in which case there’s no way that the device can be a measure of T.

I will look at some of these scenarios more closely in an upcoming post.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.

*Weirdly, The Concept of Validity does not come up in Google Scholar. I’ve seen this before, actually. Why does this happen?