Momentary Language

Results of Science Panel's Hwang Investigation Released

Nicholas Wade reports in today's NY Times on the release of a panel report detailing ways in which the journal Science can improve its measures for preventing fraudulent research reporting on the level of the now-infamous case of the 2004 article in which Dr. Hwang Woo-suk claimed to have grown stem-cells from an adult human cell. The text of the panel report and a response from the editor of Science, as well as other documents involving the Hwang case, are here.

After only a cursory reading of the documents, I couldn't help but note this passage, the first of the panel's suggestions:

There should be a formal, required 'risk assessment' for papers that have been selected for publication. This assessment would be a new procedure, and would explicitly ask questions about the probability that the work might be intentionally deceptive, or just wrong, and the consequences for the reputation of Science and science, and for other issues (public policy, intellectual property, academic credit). Papers that are likely to have high visibility, for example in climate, energy, human health, etc., should get special scrutiny.

I was pleased to see how well this suggestion comports with my claim that scientific practice is self-regulating to the extent that those purported discoveries with greater practical import will receive correspondingly greater scrutiny, as argued in this blog here and here.

Making visible the collective work of science

In an essay in today's Sunday Book Review, "Twilight of the Idols," Peter Dizikes notes that the "great-man" view of history implied by the traditional science biography is being made obsolete by the current, highly collaborative structures of contemporary scientific practice:

Contemporary science is ... far more collaborative than non-scientists may imagine. A giant new particle collider soon to open in Switzerland, for example, will have more than 7,000 physicists participating in its experiments. “If you try to understand that kind of work through a biography,” Peter Galison, a historian of science at Harvard, said in a recent interview, “you’re really cutting the wood against the grain, and you can easily end up making invisible the collective work that’s so crucial for that kind of accomplishment.”

Nor, Dizikes suggests, is the phenomenon of widespread collaboration observable only among experimentalists:

This trend applies even to theorists, who, in the mode of Newton or Einstein, still tackle problems in relative isolation. “The main difference today is there are so many people working on these deep problems,” the physicist Brian Greene said recently. In one 15-page span of his best-selling book “The Elegant Universe” (1999), Greene mentions 24 scientists whose work prefigured a rich period in the development of string theory, the dominant idea in theoretical physics for the past 20 years. Faithfully untangling these various threads makes it harder for any writer to create a linear narrative of discovery, in which a few key influences on one scientist lead to a singular insight.

These observations seem, perhaps, unsurprising to anyone who has surveyed contemporary scientific practice. What I found most interesting, however, were historians' recent reappraisals of the role of collaboration in some of the earlier revolutionary periods in the history of science:

Indeed, science biography has sometimes distorted its subject by ignoring the communal aspects of scientific thought. Geology once had a great-man history in which 18th- and 19th-century British iconoclasts like James Hutton and William Smith discerned the antiquity of the earth, an idea that was absorbed in turn by the geologist Charles Lyell, and then set the stage for Darwin’s theory of evolution. But as the historian of science Martin Rudwick has shown, a whole network of European scientists was advancing geological research in this period. Daniel Kevles, a historian of science at Yale, said recently, “What’s happened in the last 15 or 20 years is that we’ve learned how even the very greatest scientists — Newton, Darwin, Einstein — were always engaged in collaboration of a very important, fundamental nature with their contemporaries.”

Oddly enough, Dizikes's presentation of the example of the development of 18th and 19th century geologic science as culminating in Darwin's theory of evolution seems at least to imply that -- despite Rudwick's research on the geologists' practices -- Darwin himself was still a lone revolutionary. This implication, however, would be false, as we now know. Recent studies of Darwin's life -- such as Janet Browne's Charles Darwin: The Power of Place, to cite one prominent, and excellent, example -- have demonstrated that Darwin himself was at the center of a massive network, spanning the globe, of researchers, explorers, amateur naturalists, all of whose investigations were instrumental for Darwin in the completion of his own work.

Unscientific Study Suggests Wikipedia is Reliably Self-Correcting

A recent article in the Chronicle, "Can Wikipedia Ever Make the Grade," reports on the very slow process by which academics are beginning to write articles for the on-line encyclopedia Wikipedia.

One piece of information in the article that I found interesting involved an -- admittedly unscientific -- study of the speed with which Wikipedia is capable of self-correction, conducted by Communications professor Alexander Halavais of Quinnipiac University, then of SUNY Buffalo. Halavais, using a pseudonym, scattered 13 errors in various, often obscure articles. Here were the results:

Mr. Halavais expected some of his fabrications to languish online for some time. Like many academics, he was skeptical about a mob-edited publication that called itself an authoritative encyclopedia. But less than three hours after he posted them, all of his false facts had been deleted, thanks to the vigilance of Wikipedia editors who regularly check a page on the Web site that displays recently updated entries. On Dr. al-Halawi's "user talk" page, one Wikipedian pleaded with him to "refrain from writing nonsense articles and falsifying information." ...

Writing about the experiment on his blog (http://alex .halavais.net), Mr. Halavais argued that a more determined "troll" — in Web-forum parlance, a poster who contributes only inflammatory or disruptive content — could have done a better job of slipping mistakes into the encyclopedia. But he said he was "impressed" by Wikipedia participants' ability to root out his fabrications. Since then several other high-profile studies have confirmed that the site does a fairly good job at getting its facts straight — particularly in articles on science, an area where Wikipedia excels.

More on Fraud in Science: the Poehlman Case

There is a fascinating article by Jeneen Interlandi ("An Unwelcome Discovery") on the case of Eric Poehlman, only the second researcher in United States history to be prosecuted for falsifying research data. While the case is disturbing -- and suggests that there are other cases of fraud in science that remain undiscovered -- it would be more damaging to overstate its significance.

First, as Interlandi herself notes, relationships based on trust are central within the scientific enterprise:

The principal investigator is not just a boss; he is also a teacher with knowledge and experience. “Trust is an essential component in any relationship, but especially between a student and mentor, especially in a research environment,” Tchernof [a former Poehlman postdoc who left to head a lab in Quebec City] told [Interlandi] in a telephone conversation last spring, before Poehlman’s sentencing hearing. “If you didn’t trust the person you were working with, you’d have to check every single raw data point. It simply would not work. But then it takes a substantial amount of doubt to overcome that established trust.”

The last point of that quote from Tchernof, that there is a default assumption of trust among scientific researchers -- a default assumption that is very difficult to overcome, is also important. If Tchernof is correct that trust is essential to the working of science, it would be disastrous if trusting relationships could too easily be destroyed; collaborators must be able to assume that they need not adopt a paranoid stance in their dealings with each other.

How does this fact jibe with Interlandi's attempt to suggest that the Poehlman case raises questions about the ability of the scientific process to be self-correcting? Here's Interlandi:

The scientific process is meant to be self-correcting. Peer review of scientific journals and the ability of scientists to replicate one another’s results are supposed to weed out erroneous conclusions and preserve the integrity of the scientific record over time. But the Poehlman case shows how a committed cheater can elude detection for years by playing on the trust — and the self-interest — of his or her junior colleagues.

My sense is that the Poehlman case is less alarming than Interlandi suggests, and that she perhaps failed to appreciate this because she failed adequately to consider one salient feature of the Poehlman case. The strategy that Poehlman employed in perpetrating his fraud is one that confirms a hypothesis that appeared in an earlier post on this site, "When Is Absence of Evidence Evidence of Absence? The Case of Fraud in Science ..." There I suggested that the most successful cases of fraud would also be the least revolutionary, because they would involve claims that would little repay other researchers' attempts at disconfirmation. This, however, is exactly what Interlandi's article suggests.

Thus, Poehlman's fatal faux pas that caused his lab technician, Walter DeNino, first to suspect that Poehlman was falsifying data involved the attempt by Poehlman to alter entries so that the data confirmed a widely-held hypothesis:

DeNino’s task was to compare the levels of lipids, or fats, in two sets of blood samples taken several years apart from a large group of patients. As the patients aged, Poehlman expected, the data would show an increase in low-density lipoprotein (LDL), which deposits cholesterol in arteries, and a decrease in high-density lipoprotein (HDL), which carries it to the liver, where it can be broken down. Poehlman’s hypothesis was not controversial; the idea that lipid levels worsen with age was supported by decades of circumstantial evidence. ... But when DeNino ran his first analysis, the data did not support the premise.

When Poehlman saw the unexpected results, he took the electronic file home with him. The following week, Poehlman returned the database to DeNino, explained that he had corrected some mistaken entries and asked DeNino to re-run the statistical analysis. Now the trend was clear: HDL appeared to decrease markedly over time, while LDL increased, exactly as they had hypothesized.

Indeed, the research that made Poehlman's reputation was, to use Kuhn's term, normal science, as opposed to revolutionary science:

Because most studies that examine the physiology of aging look at only one point in time, researchers can’t tell whether the differences measured are because of age, menopause or individual variation. Poehlman’s longitudinal study on menopause collected the same measurements from each person twice over a six-year period. This enabled him to show, for the first time, that some metabolic changes were from menopause, not aging. Published in 1995 in the Annals of Internal Medicine, the study confirmed a long-held assumption and helped establish Poehlman’s reputation. [Emphasis mine.]

This assessment is borne out by those Interlandi interviewed:

Some scientists believe that his ability to beat the system for so long had as much to do with the research topics he chose as with his aggressive tactics. His work was prominent, but none of his studies broke new scientific ground. (This may also be why no other scientists working in the field have retracted papers as a result of Poehlman’s fraud.) By testing undisputed assumptions on popular topics, Poehlman attracted enough attention to maintain his status but not enough to invite suspicion.

Now of course Poehlman's fraud was serious; the hundreds of thousands of dollars that Poehlman received could have funded genuine research. Notice, however, the amazing fact buried within the parentheses: no other scientists needed to retract papers as a result of Poehlman's fraud.

If anything, it is this fact that is perhaps most disturbing, particularly for those of us who work in the academy -- namely, how little of the research that is done, research receiving hundreds of thousands of dollars of support, in fact finds an audience outside of an often very tightly inscribed circle of one's closest associates. Although the lack of wider implications of the ten-year-long fraud unearthed by the investigation of the Poehlman case for the edifice of science should be reassuring to those who worry about the integrity of the scientific process, it should nevertheless be sobering for those who worry about the impact of much of the business of the academy on the world at large.

How radical was Locke on testimony?

Those familiar with the literature on testimony will recognize the following quote, from Locke's Essay Concerning Human Understanding, Bk. I, Chap. IV, Sec. 24:

we may as rationally hope to see with other men's eyes as to know by other men's understanding ... The floating of other men's opinions in our brains makes us not one jot the more knowing, though they happen to be true. What in them was science is in us but opiniatrety. (Quoted in C.A.J. Coady, Testimony, 14)

I was reminded of this passage from Locke while reading Steven Shapin's "The House of Experiment in Seventeenth-Century England." Shapin quotes Sprat, who, in his History of the Royal Society, noted that there was a division of labor among the members of the Society, with those particularly skilled in experimentation responsible for carrying out experiments and reporting back on the results to the other members. Sprat writes:

Those, to whom the conduct of the Experiment is committed ... do (as it were) carry the eyes, and the imaginations of the whole company into the Laboratory with them." (Quoted in Shapin.)

What was initially striking for me about this passage from Sprat should be obvious: Locke's own contemporaries in fact did not find it irrational to hope to "see with other men's eyes"!
The underlying position that Locke took with respect to testimony, however, is in some sense borne out by Sprat. Locke's main target in the passage quoted above is opinion -- i.e., the conclusions drawn as the result of experimental evidence. And here the traditions of the Royal Society seem to be more in line with Lockean prejudices. Again, quoting Sprat:

though the Experiment was but the private task of one or two, or some such small number; yet the conjecturing, and debating on its consequences, was still the employment of their full, and solemn Assemblies. (Quoted in Shapin.)

Of course, this partial defense of Locke relies on the distinction between observation statements and conjectures, etc., made on the basis of those statements (a distinction that Coady, in his discussions of historical figures in Testimony, criticizes a number of philosophers for making), but this version of Locke seems less immediately vulnerable to criticism, at the very least.

When is Absence of Evidence Evidence of Absence? The Case of Fraud in Science ...

According to Horace Freeland Judson, author of The Great Betrayal,

grandees of the scientific establishment regularly proclaim that scientific fraud is vanishingly rare and that perpetrators are isolated individuals who act out of a twisted psychopathology. As a corrolary, they insist that science is self-correcting. ... They could not be so dogmatic if they had considered what evidence there is that might back up general conclusions, positive or negative, about the nature and incidence of scientific fraud. Their claims about fraud are unscientific." (26-7)

The implication here is that Judson himself is in a position to provide evidence that backs up negative general conclusions about the nature and incidence of scientific fraud. Let's see.

First modify the claim to the following:

[Modified Self-Correction] Science is structured in such a way that fraudulent research either has relatively trivial results (both in terms of its detrimental effects for the edifice of science and of its positive effects for the perpetrator of the fraud) or is doomed to be revealed as fraudulent.

That is, either the fraud concerns a result so trivial that it will not impinge upon central result in any area of science, nor bring great credit upon the perpetrator of the fraud, or the fraud concerns a result so central that it is bound to be discovered in relatively short order. I'll call this the MSC thesis. My suggestion is that none of Judson's evidence provides better support for another, more negative claim, than it does for MSC.

Judson himself recognizes that "the brute fact is that many published papers, tens of thousands every year, are ignored. They are not woven into the fabric [of science]; they are cited rarely or not at all except perhaps by their authors. Such vanishing papers are not part of live science." (37) If this is the case, however, then certainly fraudulent research leading to such "vanishing papers" would hardly pose a significant challenge to the edifice of science. Let's call these the insignificant results; they would seem to be covered by the first disjunct of MSC. So any support for a more negative finding by Judson would seem to rest on the significant laboratory results.

Judson suggests that "significant laboratory results don't normally get verified ... for three reasons ... . Two of them are institutional: repeating other scientists' work is not an enterprise that attracts funding, and journals rarely publish negative results. The third reason lies in the practical problems of laboratory research: some experiments can't be rerun." (39)

Let's grant that significant results are never verified -- in the sense of being performed again -- and that this is the case for the reasons Judson cites. Nevertheless, if the results are significant, as we are assuming, then the results will be indirectly verified. This will occur in at least two ways.
First,

when you publish an interesting new result, your closest competitor on reading your paper slaps his forehead and exclaims, "Why didn't I think of that!" But then he realizes, "If that's true, why then the next step is X, and I'd better get back to the lab pronto." Confirmation is indirect: if the new finding works, it can be built into the growing edifice. (36)

Indirect confirmation, however, is still -- at least provisionally -- confirmation. Indeed, there is no reason why such indirect confirmation shouldn't provide at least as much evidential confirmation as a successful repetition of an initial experiment.

Second,

In much science, confirmation comes about by the use of independent but neighboring and convergent approaches and results -- hence, triangulation. ... [T]he mid-nineteenth century philosopher and historian of science William Whewell called this by the satisfying phrase "the consilience of inductions." ... Triangulation can yield new findings. It allows the scientist building on another's claimed result to receive credit. It provides cross-checks that will undermine a claim or make it more robust. (36)

The example that Judson cites of triangulation (or consilience), the determination of Avogadro's number "from a variety of unrelated methods" at the beginning of the 20th century, is a stark example of the powerful evidence that such triangulation can provide.

Where does this leave the prospects for a negative finding concerning the significance of fraud in science? It seems that the only potentially damaging case would be a significant finding that wasn't potentially testable in terms of the first sort of indirect confirmation or of triangulation. Such a finding, however, could hardly be termed significant. Thus, it would seem impossible to give a case supporting a more negative thesis than MSC.

Initial Thoughts on Testimony and Defeat

In a recent post on "Testimony and Defeat" at Certain Doubts, Jon Kvanvig has suggested difficulties with a thesis that he attributes to Gary Gutting, namely,

(GT) One's judgment that one disagrees with an intellectual peer is a defeater for one's justification.

As an alternative, Kvanvig proposes what he terms the "irrelevance of competence" position. In the post, he suggests that this position amounts to the claim that "information about the intellectual competence of a disagree-er is never an underminer of present opinion." The example he gives to support this involves a case where I think someone is smarter than I am and has thought more about the matter in question, and thereby come to doubt my own belief. According to Kvanvig, "what is doing the defeating here is my view that you know more about this issue."

I'm confused -- I would've thought that knowing more about the issue just is competence. I take it that part of what is bothering Kvanvig is the distinction between general and specific competence. If I merely believe you to be an intellectual peer -- I have had conversations with you in which you picked up on all of my allusions to books and journals that I've read, and in which you've responded with interesting and thought-provoking allusions of your own, you've presented interesting arguments and picked up on potential weaknesses in arguments of mine, etc. -- without having any reason to suppose that you possess competence regarding a particular subject matter of interest, then that's general competence. If you have evidenced great interest and sustained thought about a particular subject matter, then that's specific competence. Perhaps we can then reformulate what Kvanvig means using something he writes in a comment to the initial post:

(IC) Knowledge that someone equally competent disagrees is not by itself a defeater.

My difficulty now in weighing (GT) against (IC) is that, as Kvanvig notes, (GT), the view that disagreement with an intellectual peer provides one with a defeater, is only plausible if the defeater in question is easily overridden. Let us so understand (GT). Given this understanding, however, Kvanvig's own examples in support of his position are not terribly helpful, as they all involve cases in which we are assuming that one has greater experience in the specific are than one's supposed intellectual peer -- and thus would presumably have grounds to override the defeater.

Couple this fact with the fact that specificity and generality come in a wide variety of flavors -- I could consider someone who also has a Ph.D., regardless of discipline, to be my intellectual peer, or only philosophy Ph.D.'s, or only philosophy Ph.D.'s from certain schools, or in certain subdisciplines, etc. This would also mean that whether one ought to embrace (GT) or (IC) will depend on the level of specificity of intellectual peerage involved in the case under consideration. My sense is that the more specific the intellectual peerage, the more plausible (GT) appears.

Philosophy's a tough field to figure out how disagreement with one's peers leads to defeaters. For the sake of the advancement of knowledge, I'm hoping it's something of an outlier in that respect. Let's use a different example. Suppose two scholars in 17th century German history have a dispute about whether Otto von Guericke performed his famous experiment in which a team of horses attempt to pull apart two metal half-spheres held together by a vacuum. Let's assume neither one is an expert on Saxony-Anhalt, Magdeburg, von Guericke, or the history of science in that period. One believes the experiment was performed with 6 horses, the other with 16. Let's suppose that, prior to their disagreement, each of the scholars would've confidently held their respective beliefs on the basis of what they would've picked up in the course of their extensive readings and conference experiences regarding that period in Germany. What should each do regarding their own belief?

I take it that (IC) would dictate that they may each still hold onto their respective beliefs, while (GT) would require them to suspend judgment until they receive further corroboration for their positions. Ignoring additional difficulties posed by the fact that we aren't taking into consideration what's riding on the accuracy of their beliefs, I have some preference for the outcome required by (GT).

The Affair Foster

Read the NYT editorial from July 13th on the Health and Human Services report released last week that absolved the former top Medicare official of the Bush Administration, Thomas Scully, of any legal wrongdoing in threatening to fire Richard Foster, the agency's top actuary, were Foster to release accurate estimates of the Administration's prescription drug plan to Congress. As with so many of the rationalizations advanced by the White House to whitewash their utter lack of an ethical compass, the foundation of the Administration's conclusion that no laws were broken is a Justice Department legal analysis that the estimates fall under executive privilege.

This use of privilege exposes the utter bankruptcy of this blatantly self-serving tactic. (No one would've heard any of the Administration's supporters applauding a similar appeal to privilege on the part of the Clinton Administration.) As the Times notes, this policy would "deprive legislators of the information they need to make informed choices."

Although it's becoming increasingly hard to avoid concluding that this is in fact the point of Administration policy, it is still worthwhile to go through the motions of attempting to provoke the White House to appeal to some principled reason for their action. Thus, the House Ways and Means Committee should, as the Times suggested, subpoena Scully and Doug Badger, the White House health policy advisor, to explain themselves and, by extension, their boss, if for no other reason than to avoid the precedent of a complete capitulation by Congress to such a bare-faced presidential violation of legislative independence.

A Modest Social-Epistemic Proposal

Last week’s Economist reports (here) that two Spanish researchers, Emili García-Berthou and Carles Alcaraz, "found that 38% of a sample of papers in Nature, and a quarter of those sampled in the British Medical Journal (BMJ) … contained one or more statistical errors", of which "4% … may have caused non-significant findings to be misrepresented as being significant". The researchers assumed that the errors were inadvertant, caused by the researchers' relative lack of skill in statistical analysis.

Inadvertant or not, the errors represent a significant cause for concern. Given this, the suggestion put forward by the Economist, that the researchers make their raw data available to the scientific community, is certainly worth adopting. However, if the problem is that researchers in other fields often lack the statistical skills necessary to analyze the raw data properly, merely making the raw data available will not always ensure that some other researcher will take it upon herself to check another researcher’s statistics, based on the raw data.

As the deputy editor of the BMJ noted, journals receive too many submissions to be able to subject all of them to a detailed review of the statistics involved, in addition to the normal peer review process. Why not additionally require that all research relying heavily on statistical analysis be submitted by the authors to an independent review by a statistician, whose seal of approval would then be included with the paper when it is initially sent to the journal? Since most researchers work in larger educational institutions or dedicated research facilities, it shouldn’t be too much of a problem for those institutions to have independent statisticians on staff, as part of the growing amount of research support staff (such as grant writers, etc.) such institutions already provide for the assistance of their research personnel.

The Implementation of a New Federal Clinical Trial Registry

A recent NYT article reports on the New York AG Eliot Spitzer’s recent attempt to sue Glaxo Smith-Kline for fraud for failing to inform doctor’s of a series of studies suggesting that its multi-million dollar drug Paxil was either ineffective in the treatment of depression in adolescents, or perhaps even harmful, increasing suicidal tendencies in those taking the drug.

The Times goes on to suggest that many see this lawsuit as helpful in pushing for the adoption of a federal registry of all clinical trials. The only criticism the Times raises of such a registry is as follows:

some say that doctors are unlikely to consult such a registry and will continue to be influenced by trial results published in leading journals.

That is to say, if the positive clinical trials are all published in prestige journals, while the negative trials are all buried in the data deluge that would be such a federal registry, merely instituting such a registry will not be enough. This is a point well taken, and should encourage those advocating the institution of such a registry to ensure that its implementation will involve a user interface conducive to its being regularly consulted by practicing physicians.

I want to raise a different problem with respect to the implementation of a federal registry. The drugs at issue are those that are being promoted for off-label prescription, i.e., for uses for which the FDA did not approve the drug in question. In fact, the very promotion of these drugs by drug companies for off-label uses is itself illegal. Thus, David Franklin, a former medical liaison for Parke-Davis, now part of Pfizer, brought a suit claiming that Parke-Davis violated FDA regulations by marketing Neurontin, a drug approved for the treatment of epilepsy, for a variety of other, unrelated, illnesses. (See story here.)

The difficulty I see with respect to the institution of a federal registry is that, if it is implemented without concurrent efforts on the part of the FDA and others to continue to crack down on the promotion of off-label uses of FDA approved drugs, it could have a chilling effect on clinic research trials, many of which would not be performed without the drug industry’s backing. That is, as long as the FDA continues to ensure that drug companies will be punished for promoting off-label prescriptions, this will ensure that the only way for the companies to provide positive word of mouth for off-label uses of their drugs is for those drugs to receive positive reviews in the research literature for those off-label uses. However, if the FDA and others fail to crack down on the promotion of off-label uses, and trust merely to the institution of the federal clinical trial registry, this will run the risk of reducing the incentive for drug companies to invest in clinical trials of drugs already approved by the FDA for other purposes. For, once all results are publicized in a national registry, although a positive clinical trial would be a boost for the drug manufacturer, the risk of having a negative result might well outweigh those potential benefits.

To be clear, my point is NOT to argue against the federal registry of clinical trials. Indeed, I am strongly in favor of the registry. The point is rather that, absent other strong regulation of the drug industry, the implementation of the registry on its own will likely not be the panacea some might have thought it to be.