Climate Science Glossary

Term Lookup

Enter a term in the search box to find its definition.

Settings

Use the controls in the far right panel to increase or decrease the number of terms automatically displayed (or to completely turn that feature off).

Term Lookup

Settings


All IPCC definitions taken from Climate Change 2007: The Physical Science Basis. Working Group I Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Annex I, Glossary, pp. 941-954. Cambridge University Press.

Home Arguments Software Resources Comments The Consensus Project Translations About Support

Twitter Facebook YouTube Mastodon MeWe

RSS Posts RSS Comments Email Subscribe


Climate's changed before
It's the sun
It's not bad
There is no consensus
It's cooling
Models are unreliable
Temp record is unreliable
Animals and plants can adapt
It hasn't warmed since 1998
Antarctica is gaining ice
View All Arguments...



Username
Password
New? Register here
Forgot your password?

Latest Posts

Archives

Peer review vs commercials and spam

Posted on 16 June 2010 by Stephan Lewandowsky

Guest post by Stephan Lewandowsky

Suppose a guy drifts up to you in a cafe and says that, like Julius Caesar, he’s a universal genius and that he’s willing to give you some brilliant investment advice. Would you entrust him with your retirement savings?

Probably not.

Now suppose your dentist tells you that you need a tooth capped soon in order to avoid a much more painful and extensive root canal job in a few years. Would you get the tooth capped?

Probably.

We all know that there are differences in credibility between opinions, with some carrying much more weight than others. Most of the time, we have an intuitive understanding of whom to trust and whom to dismiss. But is there a better way to find out whether a piece of information should be trusted?

When it comes to scientific issues of any kind, including climate change, there is no better source of information than the famous “peer-reviewed” literature.

Why is that? And what exactly is peer review?

The concept of peer review is about 300 years old and is the cornerstone of modern science. The best analogy for peer review is that it acts as a spam filter: rubbish ideas are kept from being published so that other scientists don’t waste their time reading spam. Only ideas that are not obviously rubbish make it into the literature, and once in the literature, the scientific marketplace of ideas determines their ultimate fate.

Let’s examine how different this process is from just posting one’s ideas on some web page.

Suppose I am a scientist and I believe that I have made a remarkable discovery; for example, I may believe that I have developed a new vaccine against swine-flu in my laboratory. I then write a paper that reports those results and submit it for publication to a peer-reviewed journal.

Writing a scientific paper is no trivial task: For example, in most cases, I must report my method in sufficient detail for other scientists to be able to replicate my experiment. I cannot keep my method secret; I must explain precisely what I did and how and with what equipment and for how long and so on. I typically must state who is funding my research and I must declare conflicts of interest, if any—for example, if I own the company that produces the equipment for my experiment, then I must typically disclose that. I also have to specify exactly how I analysed my data so that some other scientist could, in principle, repeat not only my experiment but also repeat the data analysis.

Now I submit my paper and a long waiting period begins.

The editor of the journal sends the paper to several other scientists. Their job is to independently, and usually anonymously, evaluate my work and to determine whether my experiment was sound and is worthy of being published.

This is the heart of peer review: Other scientists examine whether something is good enough to survive scrutiny and to be published.

Oh, and don’t be fooled by the name “peer” review; those scientists might be my peers, but they are definitely also my competitors! They have every intention of critiquing my work because they have no reason to give me a competitive edge—so there is nothing chummy about peer review; on the contrary, it’s as competitive as the Olympics.

After a few weeks, the reviews have been completed, and the editor now makes a decision on their basis. I was an editor for three years during which I made 300 editorial decisions. Sometimes a decision would take days because so much hinges on it: If an editor rejects a paper, someone’s career may suffer a set-back. If an editor accepts a rubbish paper, then the reputation of the journal is compromised; so it is important to get it right.

The best journals in the field pride themselves in the high proportion of papers they reject: It is not unusual for a top journal to reject 90% or more of all submissions, and the very best journals in the world, Science and Nature, reject even more than that.

For a paper to be rejected, all that is required is for one of the reviewers to find a flaw: Science is not a democracy, so if 2 reviewers like a paper and only 1 reviewer recommends against publication, that’s irrelevant—if the 1 critical reviewer has found a fatal flaw, then the paper is history.

Science is based on merit, not on democracy.

This sounds tough, and it is.

But it is that ruthless focus on merit and quality that has enabled science to deliver the accomplishments we all benefit from: It was peer-reviewed science that developed the anti-retroviral drugs that can now control HIV. It was peer-reviewed science that discovered the physics that got us to the moon, and it was peer-reviewed science that identified the potential threat posed by climate change and that also delivered the knowledge necessary to deal with the problem.

Of course, like any other human endeavour, peer review is not without its problems: It is possible that some papers that were rejected later on turn out to be noteworthy after all. Or, far more likely, bad papers may make it into press that should have been culled. Luckily, those bad papers don’t last very long because other scientists publish rebuttals or commentaries that reveal the flaws—thus putting egg on a lot of faces, including those of the reviewers and editors.

Here are a few more things about peer review that most people don’t know or tend to forget: First of all, there is no commercial consideration involved in the publication decision—none, zero, zippo, zilch. This is because the people who make editorial decisions stand nothing to gain or lose from a publication decision. And the journals are published either by professional organizations that don’t need to make a profit, or if they are private publishers, they make their profit by selling the journal to university libraries. And those libraries will subscribe to journals regardless of whether or not Dr. Smith or Professor Jones gets to publish his favourite theory.

In other words, making or losing money does not enter into a peer-reviewed publication decision.

Compare that to the decision about whether or not a book will get published: Harry Potter was published because the publisher thought they could make money. If they had thought that the book was a loser, they wouldn’t have published it—as indeed many of them did before J K Rowling found a publisher. So even when a scientist publishes a book, it is a commercial venture from the publisher’s point of view, and the quality of the author’s science remains to be ascertained, for example by reviews of the book by other scholars.

Compare peer reviewed science to the tabloids, which will print virtually anything to drive up their sales! If a quick lurid lie can sell another few thousand extra copies, sure thing, it’ll be in the headlines. If misrepresenting climate science is currently fashionable, sure enough, the tabloids will go out of their way to turn things upside down to make a buck, never mind what the science actually says.

And finally, compare the process of peer review to the people who run websites that sprout nonsense about conspiracy theories ranging from “MI6 killed Princess Diana” to “9/11 was an inside job” and “Climate change is a hoax.” Well, the only hoax is played on those people gullible enough to put any credence in that nonsense.

So now you know.

Peer review is a spam filter. Peer review is quality control. And peer review is independent of commercial interests. No wonder peer review gets us closer to the truth, even if that truth is inconvenient.

NOTE: this post is also being "climatecast" by Stephan Lewandowsky on RTR -FM 92.1 at 11.30 AM WAST today. It should air a few minutes after this post goes live so if you're reading this immediately (eg - you've subscribed to the SkS mailing list and just got this email), you can listen online via http://www.rtrfm.com.au/listen.

0 0

Printable Version  |  Link to this page

Comments

Prev  1  2  

Comments 51 to 58 out of 58:

  1. Oh dear, more assertions that peer review reinforces the 'consensus', that reviewers abuse it to keep rival interpretations out to corner grant money, or that they bow to the opinion of 'powerful personalities'. FYI, there is no group that love a proper controversy more than scientists. And by 'controversy' I don't mean fake ones like Creationism vs Evolution. They are inherently anti-authoritarian, and while most students probably start out thinking their supervisor's interpretations are perfect, by doing their own research they soon figure out that they're not.
    0 0
  2. Unless PLoS Medicine behaves much differently from the PLoS journals I know about, the Ionnadis article itself was peer-reviewed. And therefore...most likely false? And no, chriscanaris, peer review isn't 'as good a system *as any*'. And CoalGeologist, no one has ever claimed that peer review has a nonzero failure rate; there is no denial among scientists that peer review isn't perfect. If AGW skeptics require something impossible, then it's their problem, not science's. richard.hockey, the EMBO J model is still peer-review. It's just more *transparent* peer-review. Why should peer review's 'days be numbered'? To the chagrin of internet warriors, the fact is that in matters scientific, all opinions are not equally valid.
    0 0
  3. Svensmark: There was no substantive criticism of our work Translation: We didn't like the criticism of our work. Svensmark: We sent it to 4 different letters, but each time met with refusal. This shows that if you keep submitting sub-par material to one of the many dozens of relevant journals, you'll eventually find a lax, incompetent, or non-independent reviewer. In some unfortunate cases, the editor will just roll with the individuals you've "recommended" as reviewers, which removes the independence entirely. This happened with Lindzen and Choi. L&C, GRL, comments on peer review and peer-reviewed comments
    0 0
  4. Steven Sullivan @ 52 Unless PLoS Medicine behaves much differently from the PLoS journals I know about, the Ioannidis article itself was peer-reviewed. And therefore...most likely false? The old chestnut: The statement on the other side of the paper is true & The statement on the other side of the paper is false. Actually, the PLoS Medicine article simply highlights the pitfalls of even rigorous statistical approaches and good scientific method conducted in good faith. Publication bias is also a problem - few authors ever strive to publish negative findings which languish between the dusty covers of someone's long forgotten thesis (I ought to know - I sent a torpedo in the sides of my erstwhile supervisor's then pet theory way back in 1979). However, I don't think the world of science has been impoverished by the non-publication of my work. This can be corrosive, however, when looking at effectiveness of treatment interventions and drug trials. One attempt to address this bias is the Cochrane set up for registration of clinical trials to prevent negative findings from being buried. I don't know if any equivalent exists in the physical sciences. However, I recall a suggestion that we ought to have 'A Journal of Negative Findings' which could be quite illuminating and reminiscent of the IgNobel Prizes for Research That Ought Not To Have Been Done Or Repeated.
    0 0
  5. JMurphy @v 50: This applies to the sceptical side as much as (in some cases more)to the AWG side. Should have read: This applies to the sceptical side as much as (in many cases more than)to the AWG side. Otherwise, I refer you to Argus' posts.
    0 0
  6. I wasn't sure where to put this comment but I had the argument thrown at me: "Keeping the grant/research $$ flowing is a strong motivation. Scientists will write grant proposals and research questions they know will get them money and approval. Science is just another business, and global warming is just the current hot product. " Can you help refute this one please?
    0 0
  7. You could do worse that have a look at (meteorology) Professor Scott Mandia's blog, where he has looked into that fallacy : Part I Part II Also, have a look at the Industry, Energy, Biofuel, etc. groups that are engaged in lobbying and spending millions of dollars (also shown in the previous link) : The Center for Public Integrity.
    0 0
  8. Sorry, previous comment directed towards hucmht
    0 0

Prev  1  2  

You need to be logged in to post a comment. Login via the left margin or if you're new, register here.



The Consensus Project Website

THE ESCALATOR

(free to republish)


© Copyright 2024 John Cook
Home | Translations | About Us | Privacy | Contact Us