Marginal Utility

by Tom Bozzo

Sunday's NY Times Magazine Freakonomics column mostly drops the "onomics" to discuss Seth Roberts's career of conducting psychological experiments on himself (described at much greater length in this postprint). I was less than impressed a few posts ago with Roberts's ability to recognize complex versus pseudo-complex political discourse, and I wonder how the wacky solutions to weight loss and an array of other personal problems generalize, if at all. For instance:

Stranger yet was the fix he discovered for lifting his mood: at least one hour each morning of TV viewing, specifically life-size talking heads - but never such TV at night. Once he stumbled upon this solution, Roberts, like many scientists, looked back to the Stone Age for explication. Anthropological research suggests that early humans had lots of face-to-face contact every morning but precious little after dark, a pattern that Roberts's TV viewing now mimicked.

Perhaps my best social psychology pal can let me know if I should give the after-the-fact evolutionary explanation any more credence than I'm instinctively inclined to give it, which is nearly none (*).

What grabbed me more from the column was the Dubner and Levitt gloss on the self-experimentation methodology. Curiously, they don't mention the first stated purpose of Roberts's research in the postprint, which is using self-experimentation "to generate plausible new scientific ideas." This actually does not strike me as very freaky at all, since I'd guess that navel-gazing of one form or another inspires a lot of real research (**). Roberts's research program would look more coherent if he carried out conventional experiments related to his "discoveries;" if he does so, he's quiet about it in the postprint. It doesn't especially bother me if it's not the most efficient means of idea-generation, though it would have been freakonomically interesting if Dubner and Levitt had shown that it was.

Rather, after noting the absence of experimental controls and the inability to conduct double-blind trials (***), Dubner and Levitt wonder if

the not-quite-scientific nature of self-experimentation [might] also be a good thing? A great many laboratory-based scientific experiments, especially those in the medical field (****), are later revealed to have been marred by poor methodology or blatant self-interest.

This is a big leap. Why not just promote better experimental methodology, or scrutinize research funded by interested parties more closely? It also doesn't obviously follow that the "extreme" self-interest inherent in self-experimentation necessarily increases the "transparency" of self-experimenters' methods. Would you trust the safety of a "safe" cigarette "discovered" through a "simple" method such as a brute force search because of the naked self-interest of the tobacco company?

Moreover, as self-experimentation is elevated beyond its not unreasonable role as a means for generating testable ideas, it risks running into the buzzsaw of the counterfactualist critique a la Robert William Fogel. Discovering that treatment X is associated with outcome Y in an uncontrolled or eccentrically self-directed self-experiment says bloody little about whether there exists an alternative Z that's no more (or maybe not much more) costly than X and that can give you at least Y. For his own part, Roberts doesn't seem to be positing any "axioms of indispensability" for his discoveries, but by traditional standards they aren't even shown to be effective let alone indispensable.

I'm also more than a little astonished by this bit of association:

In some ways, self-experimentation has more in common with economics than with the hard sciences. Without the ability to run randomized experiments, economists are often left to exploit whatever data they can get hold of.

It is true enough that few economists have the luxury of using data from randominzed experiments. But in applied econometrics, clever exploitation of data sources means finding data, and employing appropriate statistical methods, that under some economic model can be shown to estimate (*****) the quantities of interest as if one had data from nicely controlled experiments; Dubner and Levitt call this relying on "creative proxies" for well-controlled experimental data. The unavoidable leaps from econometric theory to most applied implementations don't change this basic scheme.

At least Levitt knows why he got the John Bates Clark medal. But surely he knows that it's not sufficient for self-experimentation just to produce variation within its "samples." Self-experimentation must produce data that are amenable to analysis equivalent to controlled experimental data — which it seemingly is not well suited to do by the column's own account. As a result, there's little demonstrated about Roberts's solutions beyond the freakiness. Stick to the physics diet in the meanwhile.

--------------------------

(*) How many generations after the invention of the campfire would it take to make this effect go away?

(**) My dissertation was inspired in part by a desire to predict whether I'd could expect to be able to buy Macs indefinitely in an increasing-returns-to-technology-adoption world.

(***) In some areas of study, this might actually not be much of a practical impediment because of the difficulty of developing "placebo" controls — what's a placebo diet? — but Dubner and Levitt don't distinguish areas where this is the case from experimental research in general.

(****) Maybe they're still smarting over criticism of their column on the efficacy of car restraints for children, which I've resisted discussing pending Levitt making the July 2005 working paper upon which the column was based available. (Otherwise, the statistical claims made in the column are unintelligible.) I suppose I could write him if curiosity gets the better of me.

(*****) E.g., identify and estimate consistently and/or "efficiently" with respect to a specified loss criterion.