TWO years ago, I idly surfed my way to a harmless-seeming article from 2004 by Denny Borsboom, Gideon Mellenbergh, and Jaap van Heerden entitled The Concept of Validity. More than a decade had passed since its publication, and I had never heard of it. Egocentrically, this seemed like reason enough to surf right past it. Then I skimmed the abstract. Intrigued, I proceeded to read the first few paragraphs. By that point, I was hooked: I scrapped my plans for the next couple of hours so I could give this article my complete attention. This was a paper I needed to read immediately.

I’ve thought about The Concept of Validity every day for the past two years. I have mentioned or discussed or recommended The Concept of Validity hundreds of times. My zeal for The Concept of Validity is the zeal of an ex-smoker. The concept of validity in The Concept of Validity has led to a complete reformatting of my understanding of validity, and of measurement in general—and not just in the psychological sciences, but in the rest of the sciences, too. And those effects have oozed out to influence just about everything else I believe about science. The Concept of Validity is the most important paper you’ve probably never heard of.*

The concept of validity in The Concept of Validity is so simple that it’s a bit embarrassing even to write it down, but its simplicity is what makes it so diabolical, and so very different from what most in the social sciences of have believed validity to be for the past 60 years.

According to Borsboom and colleagues, a scientific device (let’s label it D) validly measures a trait or substance (which we will label T), if and only if two conditions are fulfilled:

(1) T must exist;

(2) T must cause the measurements on D.

That’s it. That is the concept of validity in The Concept of Validity.

This is a Device. There are invisible forces in the world that cause changes in the physical state of this Device. Those physical changes can be read off as representations of the states of those invisible forces. Thus, this Device is a valid measurement of those invisible forces.

What is most conspicuous about the concept of validity in The Concept of Validity is what it lacks. There is no talk of score meanings and interpretations (à la Cronbach and Meehl). There is no talk of integrative judgments involving considerations of the social or ethical consequences of how scores are put to use (à la Messick). There’s no talk of multitrait-multimethod matrixes (à la Campbell and Fiske), nomological nets (Cronbach and Meehl again), or any of the other theoretical provisos, addenda, riders, or doo-dads with which psychologists have been burdening their concepts of validity since the 1950s. Instead, all we need—and all we must have—for valid measurement is the fulfillment of two conditions: (1) a real force or trait or substance (2) whose presence exerts a causal influence on the physical state of a device. Once those conditions are fulfilled, a scientist can read off the physical changes to the device as measurements of T. And voila: We’ve got valid measurement.

Boorsboom and colleagues’ position is such a departure from 20th century notions of validity precisely because they are committed to scientific realism—a stance to which many mid-20th-century philosophers of science were quite allergic. But most philosophers of science have gotten over their aversion to scientific realism now. In general, they’re mostly comfortable with the idea that there could be hidden realities that are responsible for observable experience. Realism seemed like a lot to swallow in 1950. It doesn’t in 2017.

As soon as you commit to scientific realism, there is a kind of data you will prize more highly than any other for assessing validity, and that’s causal evidence. What a realist wants more than anything else on earth or in the heavens is evidence that the hypothesized invisible reality (the trait, or substance, or whatever) is causally responsible for the measurements the device produces. Every other productive branch of science is already working from this definition of validity. Why aren’t the social sciences?

For some of the research areas I’ve messed around with over the past few years, the implications of embracing the concept of validity in The Concept of Validity are profound, and potentially nettlesome: If we follow Borsboom and colleagues’ advice, we can discover that some scientific devices do indeed provide valid measurement, precisely because the trait or substance T they supposedly measure actually seems to exist (fulfilling Condition #1) and because there is good evidence that T is causally responsible for physical features of the device that can be read off as measurements of T (fulfilling Condition #2). In other areas, the validity of certain devices as measures looks less certain because even though we can be reasonably confident that the trait or substance T exists, we cannot be sure that changes in T are responsible for the physical changes in the device. In still other areas, it’s not clear that T exists at all, in which case there’s no way that the device can be a measure of T.

I will look at some of these scenarios more closely in an upcoming post.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.

*Weirdly, The Concept of Validity does not come up in Google Scholar. I’ve seen this before, actually. Why does this happen?

35 Reasons to Join Our Lab, Updated, with New Reasons (Thanks, Lab!)

  1. Our lab has 17 18 Papers in Nature (four 3 others in press).
  2. Our lab is currently funded by two NIH grants, an angel investor, and three crappy little NSF grants.
  3. Our lab partners with the private sector.
  4. Our lab has alumni organizations in Los Angeles, New York City, Boston, and London, and Paris (Sorry, Paris!).
  5. Our lab had 11 post-docs last year. 14 of them got tenure-track jobs.
  6. Our lab does strategic planning retreats.
  7. Our lab has a growler refilling station.
  8. All the students in our lab do New York Times op-eds as first-year projects.
  9. Our lab maintains a database of job offers we’ve turned down.
    Our lab is seeking an angel investor.
  10. Our lab can tell you every professor of every good cognitive science program from 1956 to 1978.
  11. Our lab already tested that.
  12. Our lab was there in 1994 at the first Science of Consciousness conference in Tucson.
  13. Our lab has a Michelin star.
  14. Did you ever slow down and really think about how such a thing could ever evolve? Our lab did. We actually checked it out empirically. It can’t.
  15. Our lab installed whiteboards on the outside of the building.
  16. Two years ago, our lab director got one MacArthur genius grant for his research and another to fix the nonprofit sector.
  17. Our lab was featured on Anthony Bordain: No Reservations.
  18. Our lab is a member of the National Academy.
  19. Our lab director has an amazing TED talk titled “Can attractive CVs change the world?”
  20. Our lab has its own Battle of the Bands.
  21. Tom Wolfe is writing a hit piece about our lab.
  22. Our lab has a condiments station.
  23. When our lab goes to scientific conferences, somebody stays behind as a designated survivor.
  24. I have a Science Review paper coming out on our lab’s Nature papers (embargoed)
  25. Our lab has a Tesla recharging station.
  26. This second-year dude in our lab used data from Google Scholar and Spotify to prove we’re in the same cultural clade as Charles Darwin, Marvin Minsky, and Bowie.
  27. Our lab director has no qualms about dialing it in.
  28. What did you think of Ethan Hawke’s performance in the biopic about our lab?
  29. Our lab knows a shoe can be both stylish and comfortable.
  30. Our lab has gone back to Blackberrys, fax machines, and AOL. They are more real.
  31. September through May, our lab hosts a “Second Saturdays” arts and culture event.
  32. As soon as our lab gets back from winter break, we’re going to disrupt six different research areas.
  33. This American Life’s is doing their next live event from our lab’s journal club.
  34. Our lab is a hub for Spirit Airlines.
  35. Our lab didn’t even bother to test that: It’s axiomatic.

~

33 Reasons to Join Our Lab

  1. Our lab has 17 Papers in Nature (four others in press).
  2. Our lab is currently funded by two NIH grants and three crappy little NSF grants.
  3. Our lab partners with the private sector.
  4. Our lab has alumni organizations in Los Angeles, New York City, London, and Boston.
  5. Our lab had 11 post-docs last year. 14 of them got tenure-track jobs.
  6. Our lab does strategic planning retreats.
  7. Our lab has a growler refilling station.
  8. All the students in our lab do New York Times op-eds as first-year projects.
  9. Our lab maintains a database of job offers we’ve turned down.
  10. Our lab is seeking an angel investor.
  11. Our lab already tested that.
  12. Our lab has a Michelin star.
  13. Did you ever slow down and really think about how such a thing could ever evolve? Our lab did.
  14. Our lab installed whiteboards on the outside of the building.
  15. Two years ago, our lab director got one MacArthur genius grant for his research and another to fix the nonprofit sector.
  16. Our lab was featured on Anthony Bordain: No Reservations.
  17. Our lab is a member of the National Academy.
  18. Our lab director has an amazing TED talk titled “Can attractive CVs change the world?”
  19. Our lab has its own Battle of the Bands.
  20. Tom Wolfe is writing a hit piece about our lab.
  21. Our lab has a condiments station.
  22. When our lab goes to scientific conferences, somebody stays behind as a designated survivor.
  23. I have a Science Review paper coming out on our lab’s Nature papers.
  24. Our lab has a Tesla recharging station.
  25. This second-year dude in our lab used data from Google Scholar and Spotify to prove we’re in the same cultural clade as Charles Darwin, Marvin Minsky, and Bowie.
  26. Our lab director has no qualms about dialing it in.
  27. What did you think of Ethan Hawke’s performance in the biopic about our lab?
  28. Our lab knows a shoe can be both stylish and comfortable.
  29. September through May, our lab hosts a “Second Saturdays” arts and culture event.
  30. As soon as our lab gets back from winter break, we’re going to disrupt six different research areas.
  31. This American Life’s is doing their next live event from our lab’s journal club.
  32. Our lab is a hub for Spirit Airlines.
  33. Our lab didn’t even bother to test that: It’s axiomatic.

~

Are We Just Going to Burn Illegal Ivory? How About Flooding the Market with Synthetic Ivory?

ON Saturday, Kenyan president Uhuru Kenyatta oversaw the destruction of over $100 Million in illegal ivory as a protest against the international ivory trade, which is slowly killing the African Elephant. As a heavily publicized international event (it was covered extensively by many news agencies, including in this piece in Sunday’s New York Times), it cannot be a bad thing to use this opportunity to raise awareness of the existential threat that poaching poses to Africa’s elephants, and I’m all for consciousness raising, but this is a half-measure at best. Can’t we do better than just grandstanding? How about driving down the value of poached ivory?

Attribution: Muhammad Mahdi Karim

Attribution: Muhammad Mahdi Karim

I think I read somewhere that the prices of goods tend to go down as the supply goes up. Is anybody talking seriously about just swamping the illegal ivory market with ivory we can make in the laboratory? I’m not just spinning a sci-fi scenario here. This company has figured out how to synthesize rhinoceros horn in the laboratory by taking the genetic code that built rhinoceros horn in actual living rhinos and using it as a set of instructions for 3-D printing. Is there some reason why the same approach could not be used to synthesize elephant ivory in the laboratory and then flood the market with it? Stiff penalties for poachers and ivory traders is certainly worth pursuing as a strategy, as is placing bans on ivory imports and exports, but why not deter poachers and traders by just flat-out removing their economic incentives?

Evidently, conservation experts are far from convinced that laboratory-synthesized rhino horn will displace the poaching industry by causing the prices to crash, and with so few rhinos left I suppose there isn’t much time for chasing options that won’t help.  Perhaps you can’t do everything at once when you’re facing a crisis as dire as the one that is facing the rhino. But honestly, it’s hard for me to see how a technological solution based on the laws of supply and demand don’t deserve some honest consideration as well as we think about saving Africa’s elephants.

The Generosity of Nations Before the Welfare State

Lange-MigrantMother02FOR the past four years or so, I have been working on a book about the evolutionary and cultural basis for humans’ generosity toward strangers. As I’ve worked to understand the major transitions in human generosity over the past few millennia, I’ve regularly lamented the fact that the quantitative data on formalized efforts to meet the needs of the poor and destitute are very poor in the historical record until around 1880. Nevertheless, I have continued to look on an almost weekly basis for new results that could help make sense of the history of human generosity before the welfare state.

This past week, I finally hit paydirt. In this new paper, Bas van Bavel and Auke Rijpma used data from a variety of historical documents to estimate the proportions of GDP devoted to formalized efforts to meet the needs of the poor in three European countries (Italy, the Netherlands, and England) going back as far as 1430. As far as I am aware, these are the oldest data on formalized poor relief that have ever been assembled. Between 1430 and 1850, it looks like these three nations had the will (and the ways) to devote about 1.5% of GDP to poor relief.

The numbers from Van Bavel and Rijpma are more or less directly comparable to the percentages of GDP that the modern OECD nations devote to so-called “social transfers.” The OECD numbers incorporate everything from health insurance to unemployment assistance to school lunches, and everything in between. Across all of the OECD nations, about 22% of GDP goes to social spending, but these numbers exclude private social spending (for example by corporations in the forms of health insurance and retirement pensions for workers, which are particularly important sources of social spending in the UK, the US, and Canada). Thus the OECD figures actually underestimate how much of GDP is going to social spending in the world’s richest nations today.

Compared to the 20-30% of GDP that most OECD nations currently devote to health, education, social security, and the like, the 1 or 2% expenditures in Italy, the Netherlands, and England from the Renaissance through the 19th century look paltry indeed. But it’s important to remember that most people were living at subsistence levels until the end of the 19th century anyway. You can’t expect people, or their nations, to take an abiding interest in the welfare of strangers in need until they have enough surpluses of their own to meet their own needs and the needs of their loved ones.

My Kid’s Middle School Teacher Teams Up with Richard Dawkins to Start The Teacher Institute for Evolutionary Science (TIES)

Last night, I got a text message from my kid’s middle school science teacher, Bertha Vazquez. She’s a hell of a teacher, and she’s also a huge fan of Richard Dawkins–particularly because of how effective he is at making evolution make sense. Some time ago Bertha told me that she was working on a project with Richard to create a project designed to provide teachers with better tools for teaching evolution in the middle-school curriculum.

The project has now been launched! The mission of Teacher Institute for Evolutionary Science (TIES), a project of the Richard Dawkins Foundation for Reason and Science, is to teach “middle school science teachers the most-up-to-date concepts of evolutionary science.” They have plans for workshops, web-based tools, and the quantitative measurement of learning outcomes. It’s a brand-new endeavor, and you can read more here.

You can watch a video introduction of TIES here.

If you want to make a donation (which the Louis J. Appignani Foundation will match dollar-for-dollar through the rest of 2015), visit the TIES fund-raising site on RocketHub.

Is Kim Davis Really Wrestling with a Violated Religious Conscience?

AP_kim_davis_mm_150901_4x3_992-640x402

Kim Davis, you’ll surely recall, is the Kentucky Clerk of Courts who ended up spending five days in a jail cell after refusing to comply with a federal judge’s orders to sign marriage licenses for gay couples. She’s out now, and back at work, and so far has been allowing her deputy clerks to sign the licenses in her stead. So most of us were hoping we’d never hear about Kim Davis again.

But she’s back in the news. Upon returning to work, according to Brian Mason, one of her deputies, Davis confiscated all copies of the County’s standard marriage certificate and replaced them with the Kim Davis Custom Edition, which “deletes all mentions of the County, fills in one of the blanks that would otherwise be the County with the Court’s styling, deletes her name, deletes all of the deputy clerk references, and in place of deputy clerk types in the name of Brian Mason, and has him initial rather than sign.”

So the struggle continues. In an interview with ABC News that came out today, Davis tried to explain why signing a marriage license for a gay couple violates her religious conscience, and some of these statements are helpful in helping us to get into Davis’s mind:

“I can’t put my name on a license that doesn’t represent what God ordained marriage to be,” she told a reporter. “They’re not valid in God’s eyes.”

Sounds like a classic claim of a violated religious conscience, doesn’t it? Davis can’t participate in the duties that the State of Kentucky requires of her office, she says, because of a religious objection. Most bloggers are focusing on whether the law entitles Davis to accommodations, and whether such accommodations are possible without placing an undue burden on a class of people, but I have been thinking about a different issue:

Is it possible that Davis only thinks that signing a gay couple’s marriage certificate would violate her religious conscience? Is it possible that even Kim Davis doesn’t understand Kim Davis’s motivations?

If we’re not dealing with a violation of religious conscience, what else might we be dealing with? I see three possible alternatives.

Open-Hearted Ignorance? Why are Clerks of Courts required to sign marriage licenses in the first place? Best I can tell, the Clerk’s function in this setting is merely to aver that all relevant documents are in order (i.e., “I looked for everything; it’s all there.”). Now, I’ve never seen a Kentucky marriage license, so I may be wrong here, but I have seen a California marriage license (my own), and as far as I can tell, the clerk is merely certifying that all of the documents that are required to execute a legal marriage contract are on file, that there are no valid legal objections to the marriage on file, and that the credentials of the person officiating (“solemnizing”) the marriage are approved by the State. In other words, the Clerk’s signature is merely an assurance that the paperwork is right. This fits with my intuitions about what a Clerk of Court’s job is more generally: Opening envelopes, cashing the money order you sent in to pay for your traffic ticket, checking boxes, and not losing things.

I wonder if perhaps Davis simply hasn’t thought carefully enough about what her signature on a marriage license—any marriage license–actually means. She’s in pretty deep now, and has dug in her heels, so she probably cannot be reached with reason alone, but it seems like someone ought to try. Who knows? Perhaps with time, perspective, and some discussion with people who love her and whom she trusts, Davis could be convinced that her signature is really just a bit of truth-telling (“Yep!  Everything’s on file!), and thus is in fact extremely consistent with her Christian values rather than an affront to them.

Disgust Masquerading as a Violated Religious Conscience? Second, I wonder whether Davis might be confusing disgust with a violation of religious conscience. Legal scholars such as Martha Nussbaum, as well as many social scientists, have written on the centrality of disgust to people’s anti-gay prejudices, and the Kim Davis situation may provide an interesting and never-before-seen variation on this theme. If Davis is disgusted by gay people or by gay marriage, then she might be hesitant to associate her name with a gay couple’s marriage license because disgust follows the laws of sympathetic magic: The things associated with you carry some of your traits along with them. In the same way that people don’t want to drink from a glass that once held a cockroach, those who are extremely disgusted by homosexuality or homosexual marriage might be unwilling to associate themselves—or their signatures—with a marriage license—a piece of paper—that has been filed by two gay people. After all, the signature has been a highly cherished representation of the self, one’s will, and one’s intentions, for 3,500 years.

If Davis’s problem is in fact not that her religious conscience has been violated, but instead is that she is disgusted by the idea of gay marriage, then the religious language can be removed from the debate entirely and we can start discussing instead what society should do with an elected public official who finds part of her work disgusting.

Weaponized Conscience? Finally, there is the possibility that Davis isn’t experiencing a genuine violation of her religious conscience at all, but instead is simply using that plea as a cudgel that enables her to impose burdens on a class of people whose behavior she (or, perhaps, she believes, God) dislikes. By Davis’s own account, she believes God has been “using” her through this debacle (“For God to use somebody like me, it’s so humbling,” she has said). I’d certainly like for her to explain that statement. It could mean a lot of things, but one thing it could mean is that this entire dust-up, in her heart of hearts, isn’t actually about “U.S. versus God & Davis:” Instead, in Davis’s heart of hearts, it could be about “God versus the U.S. via Davis.”

Take a moment to let that one sink in.

No Truth Serum

None of us, save perhaps those people in Davis’s most intimate inner circle, are ever going to know what’s really going on in Davis’s heart. Is her conscience really, truly, legitimately violated? Is she absolutely sure? Could she be disabused of that intuition by taking a little time to think about what a Clerk of the Courts actually does (remember, their job is merely to open envelopes, cash checks, check boxes, and not lose things) and what the Clerk’s signature on a marriage license actually means? Is her perceived violation of religious conscience just a case of projected disgust dressed in its Sunday best? Or is her plea of violated religious conscience a disingenuous one—a rhetorical weapon that Davis is using to impose a burden on a class of people whose behavior she (or, she thinks, God) dislikes?

The last possibility is the most interesting—and the most troubling, actually, for it raises an ominous but delightfully “meta” sort of question about religious conscience. And it’s a question that defenders of religious liberty—both liberal and conservative—need to take seriously. Should one’s religious (or non-religious) conscience be bothered if someone has debased the very notion of religious conscience by using it as moral cover for a less conscionable goal? I’m not saying she has, and I’m not saying she hasn’t, because I do not know.

But I’d certainly like to know, because the liberal compact is something like this: “You’re free to believe what you want, and I’m going to defend that freedom on your behalf. But it seems to me that there’s a caveat: “If you’re just messing around and don’t actually believe what you claim to believe, then don’t count on me to make good on my side of the bargain either.”

h/t Deb Lieberman.