Momentary Language

What was he thinking?

I always thought Crick was the smart one. This proves it. (Here is a link to the Times of London, to which Watson first made his remarks.)

Results of Science Panel's Hwang Investigation Released

Nicholas Wade reports in today's NY Times on the release of a panel report detailing ways in which the journal Science can improve its measures for preventing fraudulent research reporting on the level of the now-infamous case of the 2004 article in which Dr. Hwang Woo-suk claimed to have grown stem-cells from an adult human cell. The text of the panel report and a response from the editor of Science, as well as other documents involving the Hwang case, are here.

After only a cursory reading of the documents, I couldn't help but note this passage, the first of the panel's suggestions:

There should be a formal, required 'risk assessment' for papers that have been selected for publication. This assessment would be a new procedure, and would explicitly ask questions about the probability that the work might be intentionally deceptive, or just wrong, and the consequences for the reputation of Science and science, and for other issues (public policy, intellectual property, academic credit). Papers that are likely to have high visibility, for example in climate, energy, human health, etc., should get special scrutiny.

I was pleased to see how well this suggestion comports with my claim that scientific practice is self-regulating to the extent that those purported discoveries with greater practical import will receive correspondingly greater scrutiny, as argued in this blog here and here.

More on Fraud in Science: the Poehlman Case

There is a fascinating article by Jeneen Interlandi ("An Unwelcome Discovery") on the case of Eric Poehlman, only the second researcher in United States history to be prosecuted for falsifying research data. While the case is disturbing -- and suggests that there are other cases of fraud in science that remain undiscovered -- it would be more damaging to overstate its significance.

First, as Interlandi herself notes, relationships based on trust are central within the scientific enterprise:

The principal investigator is not just a boss; he is also a teacher with knowledge and experience. “Trust is an essential component in any relationship, but especially between a student and mentor, especially in a research environment,” Tchernof [a former Poehlman postdoc who left to head a lab in Quebec City] told [Interlandi] in a telephone conversation last spring, before Poehlman’s sentencing hearing. “If you didn’t trust the person you were working with, you’d have to check every single raw data point. It simply would not work. But then it takes a substantial amount of doubt to overcome that established trust.”

The last point of that quote from Tchernof, that there is a default assumption of trust among scientific researchers -- a default assumption that is very difficult to overcome, is also important. If Tchernof is correct that trust is essential to the working of science, it would be disastrous if trusting relationships could too easily be destroyed; collaborators must be able to assume that they need not adopt a paranoid stance in their dealings with each other.

How does this fact jibe with Interlandi's attempt to suggest that the Poehlman case raises questions about the ability of the scientific process to be self-correcting? Here's Interlandi:

The scientific process is meant to be self-correcting. Peer review of scientific journals and the ability of scientists to replicate one another’s results are supposed to weed out erroneous conclusions and preserve the integrity of the scientific record over time. But the Poehlman case shows how a committed cheater can elude detection for years by playing on the trust — and the self-interest — of his or her junior colleagues.

My sense is that the Poehlman case is less alarming than Interlandi suggests, and that she perhaps failed to appreciate this because she failed adequately to consider one salient feature of the Poehlman case. The strategy that Poehlman employed in perpetrating his fraud is one that confirms a hypothesis that appeared in an earlier post on this site, "When Is Absence of Evidence Evidence of Absence? The Case of Fraud in Science ..." There I suggested that the most successful cases of fraud would also be the least revolutionary, because they would involve claims that would little repay other researchers' attempts at disconfirmation. This, however, is exactly what Interlandi's article suggests.

Thus, Poehlman's fatal faux pas that caused his lab technician, Walter DeNino, first to suspect that Poehlman was falsifying data involved the attempt by Poehlman to alter entries so that the data confirmed a widely-held hypothesis:

DeNino’s task was to compare the levels of lipids, or fats, in two sets of blood samples taken several years apart from a large group of patients. As the patients aged, Poehlman expected, the data would show an increase in low-density lipoprotein (LDL), which deposits cholesterol in arteries, and a decrease in high-density lipoprotein (HDL), which carries it to the liver, where it can be broken down. Poehlman’s hypothesis was not controversial; the idea that lipid levels worsen with age was supported by decades of circumstantial evidence. ... But when DeNino ran his first analysis, the data did not support the premise.

When Poehlman saw the unexpected results, he took the electronic file home with him. The following week, Poehlman returned the database to DeNino, explained that he had corrected some mistaken entries and asked DeNino to re-run the statistical analysis. Now the trend was clear: HDL appeared to decrease markedly over time, while LDL increased, exactly as they had hypothesized.

Indeed, the research that made Poehlman's reputation was, to use Kuhn's term, normal science, as opposed to revolutionary science:

Because most studies that examine the physiology of aging look at only one point in time, researchers can’t tell whether the differences measured are because of age, menopause or individual variation. Poehlman’s longitudinal study on menopause collected the same measurements from each person twice over a six-year period. This enabled him to show, for the first time, that some metabolic changes were from menopause, not aging. Published in 1995 in the Annals of Internal Medicine, the study confirmed a long-held assumption and helped establish Poehlman’s reputation. [Emphasis mine.]

This assessment is borne out by those Interlandi interviewed:

Some scientists believe that his ability to beat the system for so long had as much to do with the research topics he chose as with his aggressive tactics. His work was prominent, but none of his studies broke new scientific ground. (This may also be why no other scientists working in the field have retracted papers as a result of Poehlman’s fraud.) By testing undisputed assumptions on popular topics, Poehlman attracted enough attention to maintain his status but not enough to invite suspicion.

Now of course Poehlman's fraud was serious; the hundreds of thousands of dollars that Poehlman received could have funded genuine research. Notice, however, the amazing fact buried within the parentheses: no other scientists needed to retract papers as a result of Poehlman's fraud.

If anything, it is this fact that is perhaps most disturbing, particularly for those of us who work in the academy -- namely, how little of the research that is done, research receiving hundreds of thousands of dollars of support, in fact finds an audience outside of an often very tightly inscribed circle of one's closest associates. Although the lack of wider implications of the ten-year-long fraud unearthed by the investigation of the Poehlman case for the edifice of science should be reassuring to those who worry about the integrity of the scientific process, it should nevertheless be sobering for those who worry about the impact of much of the business of the academy on the world at large.

Bush Administration Again Ignores Science

In one of its editorials from October 14, "Science Ignored, Again," the NY Times decries the latest instance of the Bush Administration's failure to heed the best advice given by its scientific advisors. In this latest case, the E.P.A. ignored a 20-2 vote by its own Clean Air Scientific Advisory Council in favor of tightening the annual standard for exposure levels to fine particles, microscopic specks of soot in the air that are believed to lead to coronary and respiratory disease.

How radical was Locke on testimony?

Those familiar with the literature on testimony will recognize the following quote, from Locke's Essay Concerning Human Understanding, Bk. I, Chap. IV, Sec. 24:

we may as rationally hope to see with other men's eyes as to know by other men's understanding ... The floating of other men's opinions in our brains makes us not one jot the more knowing, though they happen to be true. What in them was science is in us but opiniatrety. (Quoted in C.A.J. Coady, Testimony, 14)

I was reminded of this passage from Locke while reading Steven Shapin's "The House of Experiment in Seventeenth-Century England." Shapin quotes Sprat, who, in his History of the Royal Society, noted that there was a division of labor among the members of the Society, with those particularly skilled in experimentation responsible for carrying out experiments and reporting back on the results to the other members. Sprat writes:

Those, to whom the conduct of the Experiment is committed ... do (as it were) carry the eyes, and the imaginations of the whole company into the Laboratory with them." (Quoted in Shapin.)

What was initially striking for me about this passage from Sprat should be obvious: Locke's own contemporaries in fact did not find it irrational to hope to "see with other men's eyes"!
The underlying position that Locke took with respect to testimony, however, is in some sense borne out by Sprat. Locke's main target in the passage quoted above is opinion -- i.e., the conclusions drawn as the result of experimental evidence. And here the traditions of the Royal Society seem to be more in line with Lockean prejudices. Again, quoting Sprat:

though the Experiment was but the private task of one or two, or some such small number; yet the conjecturing, and debating on its consequences, was still the employment of their full, and solemn Assemblies. (Quoted in Shapin.)

Of course, this partial defense of Locke relies on the distinction between observation statements and conjectures, etc., made on the basis of those statements (a distinction that Coady, in his discussions of historical figures in Testimony, criticizes a number of philosophers for making), but this version of Locke seems less immediately vulnerable to criticism, at the very least.

When is Absence of Evidence Evidence of Absence? The Case of Fraud in Science ...

According to Horace Freeland Judson, author of The Great Betrayal,

grandees of the scientific establishment regularly proclaim that scientific fraud is vanishingly rare and that perpetrators are isolated individuals who act out of a twisted psychopathology. As a corrolary, they insist that science is self-correcting. ... They could not be so dogmatic if they had considered what evidence there is that might back up general conclusions, positive or negative, about the nature and incidence of scientific fraud. Their claims about fraud are unscientific." (26-7)

The implication here is that Judson himself is in a position to provide evidence that backs up negative general conclusions about the nature and incidence of scientific fraud. Let's see.

First modify the claim to the following:

[Modified Self-Correction] Science is structured in such a way that fraudulent research either has relatively trivial results (both in terms of its detrimental effects for the edifice of science and of its positive effects for the perpetrator of the fraud) or is doomed to be revealed as fraudulent.

That is, either the fraud concerns a result so trivial that it will not impinge upon central result in any area of science, nor bring great credit upon the perpetrator of the fraud, or the fraud concerns a result so central that it is bound to be discovered in relatively short order. I'll call this the MSC thesis. My suggestion is that none of Judson's evidence provides better support for another, more negative claim, than it does for MSC.

Judson himself recognizes that "the brute fact is that many published papers, tens of thousands every year, are ignored. They are not woven into the fabric [of science]; they are cited rarely or not at all except perhaps by their authors. Such vanishing papers are not part of live science." (37) If this is the case, however, then certainly fraudulent research leading to such "vanishing papers" would hardly pose a significant challenge to the edifice of science. Let's call these the insignificant results; they would seem to be covered by the first disjunct of MSC. So any support for a more negative finding by Judson would seem to rest on the significant laboratory results.

Judson suggests that "significant laboratory results don't normally get verified ... for three reasons ... . Two of them are institutional: repeating other scientists' work is not an enterprise that attracts funding, and journals rarely publish negative results. The third reason lies in the practical problems of laboratory research: some experiments can't be rerun." (39)

Let's grant that significant results are never verified -- in the sense of being performed again -- and that this is the case for the reasons Judson cites. Nevertheless, if the results are significant, as we are assuming, then the results will be indirectly verified. This will occur in at least two ways.
First,

when you publish an interesting new result, your closest competitor on reading your paper slaps his forehead and exclaims, "Why didn't I think of that!" But then he realizes, "If that's true, why then the next step is X, and I'd better get back to the lab pronto." Confirmation is indirect: if the new finding works, it can be built into the growing edifice. (36)

Indirect confirmation, however, is still -- at least provisionally -- confirmation. Indeed, there is no reason why such indirect confirmation shouldn't provide at least as much evidential confirmation as a successful repetition of an initial experiment.

Second,

In much science, confirmation comes about by the use of independent but neighboring and convergent approaches and results -- hence, triangulation. ... [T]he mid-nineteenth century philosopher and historian of science William Whewell called this by the satisfying phrase "the consilience of inductions." ... Triangulation can yield new findings. It allows the scientist building on another's claimed result to receive credit. It provides cross-checks that will undermine a claim or make it more robust. (36)

The example that Judson cites of triangulation (or consilience), the determination of Avogadro's number "from a variety of unrelated methods" at the beginning of the 20th century, is a stark example of the powerful evidence that such triangulation can provide.

Where does this leave the prospects for a negative finding concerning the significance of fraud in science? It seems that the only potentially damaging case would be a significant finding that wasn't potentially testable in terms of the first sort of indirect confirmation or of triangulation. Such a finding, however, could hardly be termed significant. Thus, it would seem impossible to give a case supporting a more negative thesis than MSC.

Trustworthiness and Academic Lineage in Collaborative Science

One of the earliest papers arguing for the importance of the sociology of science for the epistemology of testimony was John Hardwig’s “The Role of Trust in Knowledge.”  There, Hardwig argues, among other things, that (1) merely prudential reasons for trusting others—i.e., reasons stemming from one’s knowledge of the fact that scientific fraud would be imprudent on the part of the person attempting to defraud the scientific community—are insufficient, given the demonstrative prevalence of fraud in science. (702-5)  Furthermore, Hardwig argues (2) that if prudential reasons are insufficient to guarantee good science, then ethical researchers are required. (705)

Let’s leave aside the plausibility of whether, despite the much-hyped Darsee, Imanishi-Kari, and Schoen cases, among others, serious fraud really is so widespread within the scientific community.  The issue I wish to consider briefly here is whether prudential reasons are exhausted by the fear of exposure of fraudulent research practices and of the attendant punishment for those transgressions.

The point I want to raise here is that the prevalence of the use of familial terminology to describe the relationship between mentors and their graduate students and research assistants might point to an extremely powerful mechanism for promoting honest communications among collaborators—if the collaborators are linked by such “familial” bonds.

First, note that many of the most successful scientists are linked by “familial” bonds.  There are numerous references in the literature of the history and sociology of science (most famously, perhaps, Merton’s seminal “Matthew Effect”) to the power of one’s academic genealogy as a predictor of success in a given scientific field. 

Second, note that many invoke those “familial” relationships to explain their communications with other, often otherwise unknown, researchers.  Just to cite one example, consider H.M. Collins’s quote from a Scottish scientist, explaining why some of their American colleagues singled their group out as one with whom to share the then cutting edge information as to how to construct a Transversely Excited Atmospheric Pressure C02 (or, simply, TEA) laser, in Collins’s “The TEA Set”:

these people of course had got a Ph.D. working in the same group under the same supervisor, albeit eight or ten years ago.  But they’re still one of our family.

This quote suggests the power of familial relationships in research science to forge collaborative links.  However, it doesn’t explain that power.

My suggestion is that “familial” links of the sort under consideration here provide a further prudential motivation for relying on other researchers.  In the same way that one might expect the successes of members of one’s biological family to redound to one’s own advantage, one may also expect the successes of researchers with whom one shares a common academic lineage to increase one’s own prestige.  If this is the case, however, it would not be strategic for one to act in such a way to jeopardize one’s status within one’s academic “family”—and it would be strategic for one to act in such a way to make the “family” proud.  Fraudulent research would put one, thus, in jeopardy—even in cases in which the academic community failed sufficiently to exact formal penalties for wrongdoing.  In this way, then, bonds of academic lineage provide shorthand evidence of trustworthiness, and it is little wonder that they therefore prove of such importance to working researchers.

Of course, this discussion leaves still open the question of whether such “familial” relationships provide strong evidence of the importance of trust in science, as Hardwig, for one, might claim, or whether the predominance of such relationships points to the importance of obviating the need for truly trusting relationships in science—i.e., trusting relationships in which the trustor is forced to rely on the trustee despite uncertainty as to the trustee’s motives, competence, etc.  The resolution of this question, however, awaits a more detailed discussion of the nature of trust itself.