When is a Measure of Oxytocin No Such Thing?

Medical Technology class room shots. 4/27/05 By University of Delaware [CC BY 3.0] . http://creativecommons.org/licenses/by/3.0)%5D, via Wikimedia Commons. Laboratory demonstration photos are just the best.

A few days ago, I promised I would demonstrate how you can use Denny Borsboom and colleagues’ concept of validity to evaluate whether a scientific device is a valid tool for scientific measurement. To review, Borsboom and colleagues argued that we can claim that a scientific device D provides valid measurement of an invisible substance, force, or trait (which I will represent as T) when two conditions are obtained:

(1) T must exist;

(2) T must cause physical changes to D that can be read off as measurements of T.

Once you accept this definition of validity, evaluating whether a scientific device is actually a scientific measure becomes simple (not necessarily easy, but simple)—even fun: You need to concern yourself with trying to find answers to exactly two questions: First, does this thing that researchers call T even exist? In other words, is T what philosophers of science would call a “natural kind?” Second, if T does exist, are we justified in believing that the natural kind we have named T causes physical changes to the Device that can then be read off as measurements of T?

In this post, I’ll use this approach to think through the validity of a biological assay technique that is often used in hopes of measuring oxytocin in human body fluids such as blood plasma, serum, or saliva. I’ve written a bit on this blog about research in humans on the social causes and effects of oxytocin (for example, here, here, and here). My colleagues and I see signs that a lot of the enthusiasm for this research is being driven by wishful thinking about whether the devices that are being called oxytocin assays are actually valid measures. To gain purchase on this particular validity problem, Borsboom and colleagues’ concept of validity tells you everything you should want to know: First, you will want to know whether there is a natural kind in the world that corresponds to the concept that we have decided to call oxytocin. Second, you will want to know whether that natural kind that we are calling oxytocin is responsible for physical changes in the Device. That’s all you need to care about.

The consequence of accepting this simple but strict definition is liberating. Among other things, you can brush aside validity arguments that rest on claims that individual differences in measured levels of oxytocin are correlated (for instance) with self-ratings of social support, or scores on a measure of empathic accuracy, or how many Facebook friends people have. Sure, all of those correlations might fit with somebody’s theory of oxytocin, but validity arguments that rest on correlational claims like that are so 20th-century.

All you need to concern yourself with is (a) whether oxytocin exists; and (b) whether oxytocin is causally responsible for the scores that your device produces. You can quickly satisfy yourself that (a) is true: Sir Henry Dale extracted oxytocin from the human pituitary gland in 1909. The biochemist Vincent de Vigneaud identified its molecular structure in 1953. So, all that’s left to confirm is (b).

And how do we confirm (b)? Through experimental research, not correlational research. Quite simply, the question we want an answer to is this: Does the nine-amino-acid substance that we have come to call oxytocin exert causal effects on the physical states of a particular device that we can read off as measurements of that substance? If so, when you add known quantities of oxytocin to a container (or to an animal) that has zero oxytocin in it, the device should then undergo physical changes in proportion to the amount of oxytocin that you added. From those changes, it should be possible to work backwards and solve for the amount of oxytocin that was added in the first place. If you can’t do that (and, as a few of us have been arguing, with some of the most popular approaches to assaying oxytocin, you can’t), then you should doubt the validity of that particular assay. Indeed, it might be more accurate to view such a device as a very expensive random number generator.

The oxytocin assays I am referring to here fail Borsboom’s validity test because they fail on criterion (b): Changes in physical states of those assays cannot be read off veridically as changes in oxytocin. Other measures can fail the Borsboom test for a more interesting reason: The trait that they supposedly measure doesn’t exist in the first place.

I’ll look at that scenario in my next post.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.

Postscript: Denny Borsboom tells me that the reason that their paper is not visible to Google Scholar is that the citations for that paper are getting merged with another one of their papers. What a drag.

Advertisements

TWO years ago, I idly surfed my way to a harmless-seeming article from 2004 by Denny Borsboom, Gideon Mellenbergh, and Jaap van Heerden entitled The Concept of Validity. More than a decade had passed since its publication, and I had never heard of it. Egocentrically, this seemed like reason enough to surf right past it. Then I skimmed the abstract. Intrigued, I proceeded to read the first few paragraphs. By that point, I was hooked: I scrapped my plans for the next couple of hours so I could give this article my complete attention. This was a paper I needed to read immediately.

I’ve thought about The Concept of Validity every day for the past two years. I have mentioned or discussed or recommended The Concept of Validity hundreds of times. My zeal for The Concept of Validity is the zeal of an ex-smoker. The concept of validity in The Concept of Validity has led to a complete reformatting of my understanding of validity, and of measurement in general—and not just in the psychological sciences, but in the rest of the sciences, too. And those effects have oozed out to influence just about everything else I believe about science. The Concept of Validity is the most important paper you’ve probably never heard of.*

The concept of validity in The Concept of Validity is so simple that it’s a bit embarrassing even to write it down, but its simplicity is what makes it so diabolical, and so very different from what most in the social sciences of have believed validity to be for the past 60 years.

According to Borsboom and colleagues, a scientific device (let’s label it D) validly measures a trait or substance (which we will label T), if and only if two conditions are fulfilled:

(1) T must exist;

(2) T must cause the measurements on D.

That’s it. That is the concept of validity in The Concept of Validity.

This is a Device. There are invisible forces in the world that cause changes in the physical state of this Device. Those physical changes can be read off as representations of the states of those invisible forces. Thus, this Device is a valid measurement of those invisible forces.

What is most conspicuous about the concept of validity in The Concept of Validity is what it lacks. There is no talk of score meanings and interpretations (à la Cronbach and Meehl). There is no talk of integrative judgments involving considerations of the social or ethical consequences of how scores are put to use (à la Messick). There’s no talk of multitrait-multimethod matrixes (à la Campbell and Fiske), nomological nets (Cronbach and Meehl again), or any of the other theoretical provisos, addenda, riders, or doo-dads with which psychologists have been burdening their concepts of validity since the 1950s. Instead, all we need—and all we must have—for valid measurement is the fulfillment of two conditions: (1) a real force or trait or substance (2) whose presence exerts a causal influence on the physical state of a device. Once those conditions are fulfilled, a scientist can read off the physical changes to the device as measurements of T. And voila: We’ve got valid measurement.

Boorsboom and colleagues’ position is such a departure from 20th century notions of validity precisely because they are committed to scientific realism—a stance to which many mid-20th-century philosophers of science were quite allergic. But most philosophers of science have gotten over their aversion to scientific realism now. In general, they’re mostly comfortable with the idea that there could be hidden realities that are responsible for observable experience. Realism seemed like a lot to swallow in 1950. It doesn’t in 2017.

As soon as you commit to scientific realism, there is a kind of data you will prize more highly than any other for assessing validity, and that’s causal evidence. What a realist wants more than anything else on earth or in the heavens is evidence that the hypothesized invisible reality (the trait, or substance, or whatever) is causally responsible for the measurements the device produces. Every other productive branch of science is already working from this definition of validity. Why aren’t the social sciences?

For some of the research areas I’ve messed around with over the past few years, the implications of embracing the concept of validity in The Concept of Validity are profound, and potentially nettlesome: If we follow Borsboom and colleagues’ advice, we can discover that some scientific devices do indeed provide valid measurement, precisely because the trait or substance T they supposedly measure actually seems to exist (fulfilling Condition #1) and because there is good evidence that T is causally responsible for physical features of the device that can be read off as measurements of T (fulfilling Condition #2). In other areas, the validity of certain devices as measures looks less certain because even though we can be reasonably confident that the trait or substance T exists, we cannot be sure that changes in T are responsible for the physical changes in the device. In still other areas, it’s not clear that T exists at all, in which case there’s no way that the device can be a measure of T.

I will look at some of these scenarios more closely in an upcoming post.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.

*Weirdly, The Concept of Validity does not come up in Google Scholar. I’ve seen this before, actually. Why does this happen?

35 Reasons to Join Our Lab, Updated, with New Reasons (Thanks, Lab!)

  1. Our lab has 17 18 Papers in Nature (four 3 others in press).
  2. Our lab is currently funded by two NIH grants, an angel investor, and three crappy little NSF grants.
  3. Our lab partners with the private sector.
  4. Our lab has alumni organizations in Los Angeles, New York City, Boston, and London, and Paris (Sorry, Paris!).
  5. Our lab had 11 post-docs last year. 14 of them got tenure-track jobs.
  6. Our lab does strategic planning retreats.
  7. Our lab has a growler refilling station.
  8. All the students in our lab do New York Times op-eds as first-year projects.
  9. Our lab maintains a database of job offers we’ve turned down.
    Our lab is seeking an angel investor.
  10. Our lab can tell you every professor of every good cognitive science program from 1956 to 1978.
  11. Our lab already tested that.
  12. Our lab was there in 1994 at the first Science of Consciousness conference in Tucson.
  13. Our lab has a Michelin star.
  14. Did you ever slow down and really think about how such a thing could ever evolve? Our lab did. We actually checked it out empirically. It can’t.
  15. Our lab installed whiteboards on the outside of the building.
  16. Two years ago, our lab director got one MacArthur genius grant for his research and another to fix the nonprofit sector.
  17. Our lab was featured on Anthony Bordain: No Reservations.
  18. Our lab is a member of the National Academy.
  19. Our lab director has an amazing TED talk titled “Can attractive CVs change the world?”
  20. Our lab has its own Battle of the Bands.
  21. Tom Wolfe is writing a hit piece about our lab.
  22. Our lab has a condiments station.
  23. When our lab goes to scientific conferences, somebody stays behind as a designated survivor.
  24. I have a Science Review paper coming out on our lab’s Nature papers (embargoed)
  25. Our lab has a Tesla recharging station.
  26. This second-year dude in our lab used data from Google Scholar and Spotify to prove we’re in the same cultural clade as Charles Darwin, Marvin Minsky, and Bowie.
  27. Our lab director has no qualms about dialing it in.
  28. What did you think of Ethan Hawke’s performance in the biopic about our lab?
  29. Our lab knows a shoe can be both stylish and comfortable.
  30. Our lab has gone back to Blackberrys, fax machines, and AOL. They are more real.
  31. September through May, our lab hosts a “Second Saturdays” arts and culture event.
  32. As soon as our lab gets back from winter break, we’re going to disrupt six different research areas.
  33. This American Life’s is doing their next live event from our lab’s journal club.
  34. Our lab is a hub for Spirit Airlines.
  35. Our lab didn’t even bother to test that: It’s axiomatic.

~

33 Reasons to Join Our Lab

  1. Our lab has 17 Papers in Nature (four others in press).
  2. Our lab is currently funded by two NIH grants and three crappy little NSF grants.
  3. Our lab partners with the private sector.
  4. Our lab has alumni organizations in Los Angeles, New York City, London, and Boston.
  5. Our lab had 11 post-docs last year. 14 of them got tenure-track jobs.
  6. Our lab does strategic planning retreats.
  7. Our lab has a growler refilling station.
  8. All the students in our lab do New York Times op-eds as first-year projects.
  9. Our lab maintains a database of job offers we’ve turned down.
  10. Our lab is seeking an angel investor.
  11. Our lab already tested that.
  12. Our lab has a Michelin star.
  13. Did you ever slow down and really think about how such a thing could ever evolve? Our lab did.
  14. Our lab installed whiteboards on the outside of the building.
  15. Two years ago, our lab director got one MacArthur genius grant for his research and another to fix the nonprofit sector.
  16. Our lab was featured on Anthony Bordain: No Reservations.
  17. Our lab is a member of the National Academy.
  18. Our lab director has an amazing TED talk titled “Can attractive CVs change the world?”
  19. Our lab has its own Battle of the Bands.
  20. Tom Wolfe is writing a hit piece about our lab.
  21. Our lab has a condiments station.
  22. When our lab goes to scientific conferences, somebody stays behind as a designated survivor.
  23. I have a Science Review paper coming out on our lab’s Nature papers.
  24. Our lab has a Tesla recharging station.
  25. This second-year dude in our lab used data from Google Scholar and Spotify to prove we’re in the same cultural clade as Charles Darwin, Marvin Minsky, and Bowie.
  26. Our lab director has no qualms about dialing it in.
  27. What did you think of Ethan Hawke’s performance in the biopic about our lab?
  28. Our lab knows a shoe can be both stylish and comfortable.
  29. September through May, our lab hosts a “Second Saturdays” arts and culture event.
  30. As soon as our lab gets back from winter break, we’re going to disrupt six different research areas.
  31. This American Life’s is doing their next live event from our lab’s journal club.
  32. Our lab is a hub for Spirit Airlines.
  33. Our lab didn’t even bother to test that: It’s axiomatic.

~

Are We Just Going to Burn Illegal Ivory? How About Flooding the Market with Synthetic Ivory?

ON Saturday, Kenyan president Uhuru Kenyatta oversaw the destruction of over $100 Million in illegal ivory as a protest against the international ivory trade, which is slowly killing the African Elephant. As a heavily publicized international event (it was covered extensively by many news agencies, including in this piece in Sunday’s New York Times), it cannot be a bad thing to use this opportunity to raise awareness of the existential threat that poaching poses to Africa’s elephants, and I’m all for consciousness raising, but this is a half-measure at best. Can’t we do better than just grandstanding? How about driving down the value of poached ivory?

Attribution: Muhammad Mahdi Karim

Attribution: Muhammad Mahdi Karim

I think I read somewhere that the prices of goods tend to go down as the supply goes up. Is anybody talking seriously about just swamping the illegal ivory market with ivory we can make in the laboratory? I’m not just spinning a sci-fi scenario here. This company has figured out how to synthesize rhinoceros horn in the laboratory by taking the genetic code that built rhinoceros horn in actual living rhinos and using it as a set of instructions for 3-D printing. Is there some reason why the same approach could not be used to synthesize elephant ivory in the laboratory and then flood the market with it? Stiff penalties for poachers and ivory traders is certainly worth pursuing as a strategy, as is placing bans on ivory imports and exports, but why not deter poachers and traders by just flat-out removing their economic incentives?

Evidently, conservation experts are far from convinced that laboratory-synthesized rhino horn will displace the poaching industry by causing the prices to crash, and with so few rhinos left I suppose there isn’t much time for chasing options that won’t help.  Perhaps you can’t do everything at once when you’re facing a crisis as dire as the one that is facing the rhino. But honestly, it’s hard for me to see how a technological solution based on the laws of supply and demand don’t deserve some honest consideration as well as we think about saving Africa’s elephants.

The Generosity of Nations Before the Welfare State

Lange-MigrantMother02FOR the past four years or so, I have been working on a book about the evolutionary and cultural basis for humans’ generosity toward strangers. As I’ve worked to understand the major transitions in human generosity over the past few millennia, I’ve regularly lamented the fact that the quantitative data on formalized efforts to meet the needs of the poor and destitute are very poor in the historical record until around 1880. Nevertheless, I have continued to look on an almost weekly basis for new results that could help make sense of the history of human generosity before the welfare state.

This past week, I finally hit paydirt. In this new paper, Bas van Bavel and Auke Rijpma used data from a variety of historical documents to estimate the proportions of GDP devoted to formalized efforts to meet the needs of the poor in three European countries (Italy, the Netherlands, and England) going back as far as 1430. As far as I am aware, these are the oldest data on formalized poor relief that have ever been assembled. Between 1430 and 1850, it looks like these three nations had the will (and the ways) to devote about 1.5% of GDP to poor relief.

The numbers from Van Bavel and Rijpma are more or less directly comparable to the percentages of GDP that the modern OECD nations devote to so-called “social transfers.” The OECD numbers incorporate everything from health insurance to unemployment assistance to school lunches, and everything in between. Across all of the OECD nations, about 22% of GDP goes to social spending, but these numbers exclude private social spending (for example by corporations in the forms of health insurance and retirement pensions for workers, which are particularly important sources of social spending in the UK, the US, and Canada). Thus the OECD figures actually underestimate how much of GDP is going to social spending in the world’s richest nations today.

Compared to the 20-30% of GDP that most OECD nations currently devote to health, education, social security, and the like, the 1 or 2% expenditures in Italy, the Netherlands, and England from the Renaissance through the 19th century look paltry indeed. But it’s important to remember that most people were living at subsistence levels until the end of the 19th century anyway. You can’t expect people, or their nations, to take an abiding interest in the welfare of strangers in need until they have enough surpluses of their own to meet their own needs and the needs of their loved ones.

My Kid’s Middle School Teacher Teams Up with Richard Dawkins to Start The Teacher Institute for Evolutionary Science (TIES)

Last night, I got a text message from my kid’s middle school science teacher, Bertha Vazquez. She’s a hell of a teacher, and she’s also a huge fan of Richard Dawkins–particularly because of how effective he is at making evolution make sense. Some time ago Bertha told me that she was working on a project with Richard to create a project designed to provide teachers with better tools for teaching evolution in the middle-school curriculum.

The project has now been launched! The mission of Teacher Institute for Evolutionary Science (TIES), a project of the Richard Dawkins Foundation for Reason and Science, is to teach “middle school science teachers the most-up-to-date concepts of evolutionary science.” They have plans for workshops, web-based tools, and the quantitative measurement of learning outcomes. It’s a brand-new endeavor, and you can read more here.

You can watch a video introduction of TIES here.

If you want to make a donation (which the Louis J. Appignani Foundation will match dollar-for-dollar through the rest of 2015), visit the TIES fund-raising site on RocketHub.