Friday, December 31, 2004
The Island of Lost Posts II: Education Research
by Tom Bozzo
"Revealed Preference Ranking" is one of those papers whose copyright notice forbids quotation without written permission from the copyright holders. (I wonder, does this mean I'd have to contact all four?) I guess I'd like to see how the notice fared against a "fair use" defense, but IANACL so I'll just describe its contents as needed.
The paper purports to provide a hard-to-manipulate alternative to popular rankings such as those published by U.S. News. The general deficiencies of the U.S. News methodology are described here by Brian Leiter, specifically w.r.t. law school rankings, though my understanding is that much of Prof. Leiter's critique carries over to the undergraduate rankings. Foremost among them is the reliance of popular rankings on criteria easily manipulated (or even just plain made up) by the schools themselves.
"Revealed Preference Ranking" is an annoying paper, partly for relying heavily on straw man arguments (*), but mainly for setting the reader up for results it fails to deliver. The former manifests itself in the supsiciously extensive verbiage the authors expend comparing their rankings to the much different rankings one gets looking at admission and matriculation rates (see, e.g., p. 34) as opposed to the actual ranking indexes published by U.S. News or other sources.
The obvious question is how does their ranking compare to U.S. News, which they do mention by name? If the defects of the U.S. News ranking matter, then the true preference ranking ought to be materially different. You can see the U.S. News top 50 reported by Prof. Leiter here.
It just so happens that the tournament model and the accompanying (thinly described) statistical analysis yields a top 10 that shares eight schools with the U.S. News ranking, and the other two in the U.S. News top 10 are both in the Hoxby et al. top 20. The only "surprise" from the "revealed preference" ranking is Amherst, not in the U.S. News top 50.
Amherst was a surprise to me, anyway, as I attended a relatively high powered mid-Atlantic private high school and didn't know anyone who would have preferred it (#9) over Duke (#19), let alone Swarthmore (#14).
------------
(*) Also present to some extent in the charter school papers, along with a bunch of other economist sins. Moreover, one of the charter school papers uses one of my least favorite methods of persuasion: criticizing a sampling-based study for using a sample, which would set my BS-O-Meter on high sensitivity if I were to give that paper a serious going over.
Following posts early this past fall at MaxSpeak and Marginal Revolution on charter school research by Harvard economics professor Caroline Hoxby, I came across a working paper by Hoxby and three coauthors, "A Revealed Preference Ranking of U.S. Colleges and Universities." I just lost the will to write about it after some time had passed.
"Revealed Preference Ranking" is one of those papers whose copyright notice forbids quotation without written permission from the copyright holders. (I wonder, does this mean I'd have to contact all four?) I guess I'd like to see how the notice fared against a "fair use" defense, but IANACL so I'll just describe its contents as needed.
The paper purports to provide a hard-to-manipulate alternative to popular rankings such as those published by U.S. News. The general deficiencies of the U.S. News methodology are described here by Brian Leiter, specifically w.r.t. law school rankings, though my understanding is that much of Prof. Leiter's critique carries over to the undergraduate rankings. Foremost among them is the reliance of popular rankings on criteria easily manipulated (or even just plain made up) by the schools themselves.
"Revealed Preference Ranking" is an annoying paper, partly for relying heavily on straw man arguments (*), but mainly for setting the reader up for results it fails to deliver. The former manifests itself in the supsiciously extensive verbiage the authors expend comparing their rankings to the much different rankings one gets looking at admission and matriculation rates (see, e.g., p. 34) as opposed to the actual ranking indexes published by U.S. News or other sources.
The obvious question is how does their ranking compare to U.S. News, which they do mention by name? If the defects of the U.S. News ranking matter, then the true preference ranking ought to be materially different. You can see the U.S. News top 50 reported by Prof. Leiter here.
It just so happens that the tournament model and the accompanying (thinly described) statistical analysis yields a top 10 that shares eight schools with the U.S. News ranking, and the other two in the U.S. News top 10 are both in the Hoxby et al. top 20. The only "surprise" from the "revealed preference" ranking is Amherst, not in the U.S. News top 50.
Amherst was a surprise to me, anyway, as I attended a relatively high powered mid-Atlantic private high school and didn't know anyone who would have preferred it (#9) over Duke (#19), let alone Swarthmore (#14).
------------
(*) Also present to some extent in the charter school papers, along with a bunch of other economist sins. Moreover, one of the charter school papers uses one of my least favorite methods of persuasion: criticizing a sampling-based study for using a sample, which would set my BS-O-Meter on high sensitivity if I were to give that paper a serious going over.
