Stories tagged peer review

Mar
04
2010

Missing Link - Not

Darwinius masillae
Darwinius masillaeCourtesy University of Oslo
Perhaps taking advantage of the Darwin publicity last year (200th birthday), a scientific paper was published revealing Ida, a 47 million year old fossil classified Darwinius masillae.
The study's lead author, Jørn Hurum of the University of Oslo, variously called the fossil the holy grail of paleontology and the lost ark of archeology. A two-hour documentary called "The Link" was on the History Channel and a book with the same title hit bookstores.

One million dollars

How big money became mixed with science is described in the Guardian post titled Deal in Hamburg bar led scientist to Ida fossil, the 'eighth wonder of the world'.
Now that money has been made, it is time for the scientific process (peer review).

John Fleagle, a professor at Stony Brook University, in New York state, who reviewed the paper for the journal, agrees that the fossil is not a lemur. But Ida's full significance would not be known until other scientists had seen the paper. "That will be sorted out, or at least debated extensively, in the coming years."

Confirmed: Fossil Ida is not a human ancestor

In a paper in the Journal of Human Evolution, Chris Kirk strongly argue(d) that Darwinius is not one of our ancestors. Science blogger, Brian Switek, also explains why ... That "Ida" is Not Our Great-Great-Great-Great-Etc. Grandmother. Dissenting scientists are awaiting a response from Jørn Hurum.

How science should be done

I am reminded of another case where the media was used to hype a story before it was properly reviewed by others. I wrote about it here: Jesus and family found in tomb? What moral is to be learned here?

Don't announce discoveries through the media, but through the tried and tested peer-review process.

Nov
25
2009

A controversy is brewing in the world of climate science. On Thursday, November 19, a Russian website posted over 1,000 e-mails and almost 3,000 data files from the Climate Research Unit (CRU) at the University of East Anglia in Great Britain. The CRU is one of the major centers of climate research in the world, and provided much of the data for the United Nation’s Intergovernmental Panel on Climate Change (IPCC) report.

The e-mails, written by some of the leading climate scientists in Britain and America, seem to suggest some very disturbing behavior:

* manipulating climate data to fit pre-existing theory
* refusing to share data with peers to check for accuracy
* circumventing legal requirements to release information, and even deleting some of it
* pressuring journals to reject papers that don’t fit the theory, and even pushing editors out of their posts

The story has been covered by the New York Times, the Washington Post, and the Wall Street Journal. You can find a good summary of how the story broke on Pajamas Media. Blogger Bishop Hill is keeping a running list of the most controversial e-mails. And, if you just want a quick summary, there’s “Three Things You Absolutely Must Know About Climategate.”

The University has acknowledged that its system was illegally hacked, but cannot vouch for the authenticity of every item. (There is also some suggestion that the information may have been leaked by an insider.) Several authors and recipients have verified some of the e-mails as genuine; as of this writing, none of the messages have been refuted. The sheer amount of data – over 170 megabytes – suggests this is not a hoax, though many authors have cautioned that it would be easy for a prankster to slip a few bogus e-mails in with all the legitimate ones.

But, assuming the e-mails are genuine, what do they tell us?

The alleged non-compliance with the Freedom of Information Act is a legal matter. We can say nothing about it, other than no charges have been filed, and everyone is presumed innocent until proven guilty.

The e-mails which seem to describe fudging the facts to fit the theory have received the most attention. It would be disturbing indeed if scientists at a major research institute were falsifying data. Though only a handful of papers have so far been implicated, if the allegations are borne out it would cast a pall over these scientists’ other work, their collaborations, and even work done by other scientists which was based on the disputed data.

These particular e-mails have also received the strongest defense. The authors, and even some third-party observers, maintain that the messages are being quoted out of context and misinterpreted, and that some phrases which appear damning actually have innocent explanations. (To date, there has been little reporting on the much larger, much more complex data files, which may shed light on this issue.)

Perhaps most disturbing, from a science standpoint, are the withholding of data from outside researchers, and the pressure put on journals to not publish dissenting views. Science absolutely relies on vigorous, evidence-based debate. If the evidence is not made available, the debate cannot take place. Furthermore, proponents of human-caused global warming have long criticized dissenters for not publishing their papers in peer-reviewed scientific journals. However, if it turns out that those journals were controlled by proponents who actively kept dissenters out, then the argument loses merit.

On this last point, global warming proponents and dissenters agree. Writers such as Megan McArdle and George Monbiot argue that the case for human-caused global warming remains strong, but that subverting the peer-review process blocks scientific progress and is a major blow to credibility.

So, what next? Politicians in Britain, Australia and America are calling for investigations. Climate studies are funded with taxpayer dollars, and lawmakers pass legislation based on the information the studies provide. Governments have an obligation to make sure it is accurate. And, as noted earlier, the easy work of reading the e-mails has largely been done. The more difficult task of sifting through the data files will take longer. Already, some programmers are questioning the computer models CRU developed to predict climate. If there are more updates, we’ll be sure to post them here.

Oct
13
2008

A display on adult stem cells, here at the SMM: In fact, this exhibit features Catherine Verfaillie herself.    (Good looking out, BK)
A display on adult stem cells, here at the SMM: In fact, this exhibit features Catherine Verfaillie herself. (Good looking out, BK)Courtesy bryankennedy
Following the results of an evaluation by a panel of experts at the University of Minnesota, the magazine New Scientist published an article last week announcing that some of the data used in a groundbreaking study on adult stem cells had been falsified.

The study, performed at the University of Minnesota under the supervision of Catherine Verfaillie, is part of a line of research that seemed to indicate that adult stem cells, taken from bone marrow, are pluripotent—that is that they have the potential to develop into any type of cell. Previously, only embryonic stem cells were thought to be pluripotent, and Verfaillie’s research looked like it could eventually offer an alternative to the ethically complicated use of embryonic cells for research (which requires the destruction of an embryo).

Unfortunately, other scientists had trouble replicating Verfaillie’s results, which were published in the journal Nature. New Scientist began examining the research done by Verfaillie and her team, and found that key images in the research appeared several times in papers for different experiments, and, in the case of a related study in the publication Blood, were used twice in the same paper, but had been visually altered slightly, and flipped 180 degrees. New Scientist reported their findings to the University, which began a formal investigation of the matter.

The University just recently completed the investigation, and found that data in the blood article had indeed been falsified (the images in particular), by a former PhD student of Verfaillies’, Morayma Reyes. The University and Catherine Verfaillie have asked Blood to redact the study.

Verfaillie has stated that she was unaware of the problems with the published study, and while she didn’t believe that the data was deliberately falsified, she takes ultimate responsibility for the errors.

Reyes, who now works as an assistant professor at the University of Washington, denies that the images represent deliberately altered data, and blames the errors on inadequate supervision and training. She claims that she had neither the equipment (photo editing software) nor knowledge required to alter the images. The differences in the reoccurring images were likely the result of the inadvertent use of the image adjusting tools built into lab equipment, she says, and the duplication of a figure within the Blood paper was accidental. Reyes also feels that she has been treated unfairly by the University, and that the expert panel in the investigation demonstrated a clear “lack of expertise” in the field of stem cell biology.

Reyes’ full position can be read here. The University’s response can be found here.

The altered images, Reyes asserts, shouldn’t change the results of the paper, but the whole incident brings up some interesting issues on the process of vetting science. While the errors in the paper never should have made it past Verfaillie and the rest of her team, the process of peer review should have caught them anyway. Generally, before research is published in a scientific journal, the editors select several scientists in the particular field of the paper to evaluate and comment (often anonymously) on the paper. The review panel is meant to confirm that the methodology of the experiments and the interpretation of the results are sound. Research can then be recommended (or not) for publication.

Publishing research essentially formally submits it to the scientific community, and it’s common for other scientists to attempt to replicate experiments, especially if a study makes particularly striking claims (like adult stem cells being pluripotent). The work of other scientists in replicating results is, obviously, essential to the scientific method—in this case is was what finally drew attention to some of the irregularities in Verfaillie’s team’s work.

Reproducibility can be a tricky thing, though—difficulty in repeating results doesn’t necessarily mean that they aren’t reproducible. (Here’s a good article on repeating and reproducing results.) But the problems in reproducing these results drew attention to the questionable data, which brought up another aspect of scientific vetting: the University’s investigation into academic misconduct. If the problems with reproducibility seem to come from data being changed, added, or omitted to strengthen a conclusion, then there could be a serious problem. This sort of misconduct undermines scientific progress, and can call into question the reputation of the institution it came out of and the validity of other research performed there. And if Morayma Reyes seems a little extra defensive in her letter, it’s understandable, because being accused of academic misconduct is a big deal, and no good for your career and future work.

The subject of the research further complicates the situation—this isn’t the first time issues of academic dishonesty have come up with regards to stem cell research. In 2006, a Korean scientist’s claims that he had cloned human embryos (thereby eliminating the need to destroy new embryos for stem cells) turned out to be based on lies. There’s a fear that the potentially huge medical payoff of stem cell research, as well as the ethical debate surrounding the use of human embryonic stem cells, could lead to science that is less than completely thorough, or even situations like the Korean controversy. And that’s bad for science in general. There’s also the thought that errors that are unintentional (as may be the case with Reyes’ images) could be the result of “pathological science,” where results are steered in a particular direction by scientists because of “subjective effects, wishful thinking, or threshold interactions.” It doesn’t have the same ethical problems, but pathological results aren’t a whole lot better for science than straight-out misconduct, and it’s a serious potential pitfall with the benefits of stem cell research waiting out there as temptations.

So there you go. It looks like things are, for the most part, being handled appropriately in this situation, but it’s an interesting window into scientific process.

Any thoughts? Does it seem like the vetting process of science is lacking in some way? Or is it maybe too thorough? Professor Reyes, I imagine, would argue that too much has been made of this situation, and there are many who argue that the process of peer review limits the communication and dissemination of scientific ideas.

Or, even better, does it seem like I got something wrong here?

Let’s have it, Buzzketeers.

Oct
19
2006

Directory of Open Access Journals (DOAJ)

Have you ever wanted to learn about something on the internet only to hit a paywall. A paywall only allows you to read the article if you pay money first. When I look for information on the internet, I depend upon free, full text, and quality controlled scientific and scholarly journals. The Directory of Open Access Journals provides this service.

The Directory of Open Access Journals covers all subjects and languages. There are now 2423 journals in the directory. Currently 709 journals are searchable at the article level. As of today 118178 articles are included in the DOAJ service.

Junk science not allowed

Open access means I can "read, download, copy, distribute, print, search, or link to the full texts of these articles" Quality control means the journal must exercise peer-review or editorial quality control.

Check out their Frequently asked questions and the search articles features.