Climate Science Glossary

Term Lookup

Enter a term in the search box to find its definition.

Settings

Use the controls in the far right panel to increase or decrease the number of terms automatically displayed (or to completely turn that feature off).

Term Lookup

Settings


All IPCC definitions taken from Climate Change 2007: The Physical Science Basis. Working Group I Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Annex I, Glossary, pp. 941-954. Cambridge University Press.

Home Arguments Software Resources Comments The Consensus Project Translations About Donate

Twitter Facebook YouTube Pinterest

RSS Posts RSS Comments Email Subscribe


Climate's changed before
It's the sun
It's not bad
There is no consensus
It's cooling
Models are unreliable
Temp record is unreliable
Animals and plants can adapt
It hasn't warmed since 1998
Antarctica is gaining ice
View All Arguments...



Username
Password
Keep me logged in
New? Register here
Forgot your password?

Latest Posts

Archives

Peer review vs commercials and spam

Posted on 16 June 2010 by Stephan Lewandowsky

Guest post by Stephan Lewandowsky

Suppose a guy drifts up to you in a cafe and says that, like Julius Caesar, he’s a universal genius and that he’s willing to give you some brilliant investment advice. Would you entrust him with your retirement savings?

Probably not.

Now suppose your dentist tells you that you need a tooth capped soon in order to avoid a much more painful and extensive root canal job in a few years. Would you get the tooth capped?

Probably.

We all know that there are differences in credibility between opinions, with some carrying much more weight than others. Most of the time, we have an intuitive understanding of whom to trust and whom to dismiss. But is there a better way to find out whether a piece of information should be trusted?

When it comes to scientific issues of any kind, including climate change, there is no better source of information than the famous “peer-reviewed” literature.

Why is that? And what exactly is peer review?

The concept of peer review is about 300 years old and is the cornerstone of modern science. The best analogy for peer review is that it acts as a spam filter: rubbish ideas are kept from being published so that other scientists don’t waste their time reading spam. Only ideas that are not obviously rubbish make it into the literature, and once in the literature, the scientific marketplace of ideas determines their ultimate fate.

Let’s examine how different this process is from just posting one’s ideas on some web page.

Suppose I am a scientist and I believe that I have made a remarkable discovery; for example, I may believe that I have developed a new vaccine against swine-flu in my laboratory. I then write a paper that reports those results and submit it for publication to a peer-reviewed journal.

Writing a scientific paper is no trivial task: For example, in most cases, I must report my method in sufficient detail for other scientists to be able to replicate my experiment. I cannot keep my method secret; I must explain precisely what I did and how and with what equipment and for how long and so on. I typically must state who is funding my research and I must declare conflicts of interest, if any—for example, if I own the company that produces the equipment for my experiment, then I must typically disclose that. I also have to specify exactly how I analysed my data so that some other scientist could, in principle, repeat not only my experiment but also repeat the data analysis.

Now I submit my paper and a long waiting period begins.

The editor of the journal sends the paper to several other scientists. Their job is to independently, and usually anonymously, evaluate my work and to determine whether my experiment was sound and is worthy of being published.

This is the heart of peer review: Other scientists examine whether something is good enough to survive scrutiny and to be published.

Oh, and don’t be fooled by the name “peer” review; those scientists might be my peers, but they are definitely also my competitors! They have every intention of critiquing my work because they have no reason to give me a competitive edge—so there is nothing chummy about peer review; on the contrary, it’s as competitive as the Olympics.

After a few weeks, the reviews have been completed, and the editor now makes a decision on their basis. I was an editor for three years during which I made 300 editorial decisions. Sometimes a decision would take days because so much hinges on it: If an editor rejects a paper, someone’s career may suffer a set-back. If an editor accepts a rubbish paper, then the reputation of the journal is compromised; so it is important to get it right.

The best journals in the field pride themselves in the high proportion of papers they reject: It is not unusual for a top journal to reject 90% or more of all submissions, and the very best journals in the world, Science and Nature, reject even more than that.

For a paper to be rejected, all that is required is for one of the reviewers to find a flaw: Science is not a democracy, so if 2 reviewers like a paper and only 1 reviewer recommends against publication, that’s irrelevant—if the 1 critical reviewer has found a fatal flaw, then the paper is history.

Science is based on merit, not on democracy.

This sounds tough, and it is.

But it is that ruthless focus on merit and quality that has enabled science to deliver the accomplishments we all benefit from: It was peer-reviewed science that developed the anti-retroviral drugs that can now control HIV. It was peer-reviewed science that discovered the physics that got us to the moon, and it was peer-reviewed science that identified the potential threat posed by climate change and that also delivered the knowledge necessary to deal with the problem.

Of course, like any other human endeavour, peer review is not without its problems: It is possible that some papers that were rejected later on turn out to be noteworthy after all. Or, far more likely, bad papers may make it into press that should have been culled. Luckily, those bad papers don’t last very long because other scientists publish rebuttals or commentaries that reveal the flaws—thus putting egg on a lot of faces, including those of the reviewers and editors.

Here are a few more things about peer review that most people don’t know or tend to forget: First of all, there is no commercial consideration involved in the publication decision—none, zero, zippo, zilch. This is because the people who make editorial decisions stand nothing to gain or lose from a publication decision. And the journals are published either by professional organizations that don’t need to make a profit, or if they are private publishers, they make their profit by selling the journal to university libraries. And those libraries will subscribe to journals regardless of whether or not Dr. Smith or Professor Jones gets to publish his favourite theory.

In other words, making or losing money does not enter into a peer-reviewed publication decision.

Compare that to the decision about whether or not a book will get published: Harry Potter was published because the publisher thought they could make money. If they had thought that the book was a loser, they wouldn’t have published it—as indeed many of them did before J K Rowling found a publisher. So even when a scientist publishes a book, it is a commercial venture from the publisher’s point of view, and the quality of the author’s science remains to be ascertained, for example by reviews of the book by other scholars.

Compare peer reviewed science to the tabloids, which will print virtually anything to drive up their sales! If a quick lurid lie can sell another few thousand extra copies, sure thing, it’ll be in the headlines. If misrepresenting climate science is currently fashionable, sure enough, the tabloids will go out of their way to turn things upside down to make a buck, never mind what the science actually says.

And finally, compare the process of peer review to the people who run websites that sprout nonsense about conspiracy theories ranging from “MI6 killed Princess Diana” to “9/11 was an inside job” and “Climate change is a hoax.” Well, the only hoax is played on those people gullible enough to put any credence in that nonsense.

So now you know.

Peer review is a spam filter. Peer review is quality control. And peer review is independent of commercial interests. No wonder peer review gets us closer to the truth, even if that truth is inconvenient.

NOTE: this post is also being "climatecast" by Stephan Lewandowsky on RTR -FM 92.1 at 11.30 AM WAST today. It should air a few minutes after this post goes live so if you're reading this immediately (eg - you've subscribed to the SkS mailing list and just got this email), you can listen online via http://www.rtrfm.com.au/listen.

0 0

Bookmark and Share Printable Version  |  Link to this page

Comments

1  2  Next

Comments 1 to 50 out of 58:

  1. A very starry eyed perspective on peer review. It often doesn't work that way (though that's the way it's supposed to work).
    0 0
    Moderator Response: chriscanaris it actually usually does work precisely like that. If anything the article underplays how nasty and tough peer review is. Reviewers are ruthless and sciences is a highly competitive profession. (JB)
  2. Yes Chris. And often psychoactive drugs do not work exactly the way they're supposed to. Same for therapy. Does that mean we should buy into the propaganda piece against psychiatry that scientologists circulate?
    0 0
  3. A very naive view of peer review. But I guess its the best we have. Have a look at this recent article on PLOS.
    "Why Most Published Research Findings Are False" by John P. A. Ioannidis.
    http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124
    0 0
  4. Phillipe @ 2

    And often psychoactive drugs do not work exactly the way they're supposed to. Same for therapy.

    We don't pretend otherwise. But we don't howl down psychiatrists who admit this. But some (by no means all or even most) proponents of AGW do behave more like Scientologists than scientists.
    0 0
  5. Richard.Hockey

    What an odd paper! It's assessment of what constitutes "True" vs "False" is based solely on an uncorrected statistical p-value of 0.05. Of course, p-values involving multiple comparisons are often corrected in a number of ways to make them more conservative, and effect sizes are often taken into account. The author doesn't seem to acknowledge this.

    Every test that yields a successful p-value is not set in stone and accepted as fact. Scientists are human so personal biases are involved too. The point of peer review is to average out such biases as much as possible. Also every positive result is taken as provisional by anyone with a brain, which generally includes most scientists.

    This paper ignores the second stage of peer review...the judgement by the larger peer community after publication. Scientists account for the amount of data and the skill of the researchers when making judgements about which papers to focus on. It helps if those papers echo others (which is a less probable event being a conjoined probability function).

    I wonder if the idea that specific findings meeting the p<0.05 standard are right or wrong is unique to applied/regulatory specific fields such as clinical medicine.
    0 0
  6. Actually, I think analogising peer review to a spam filter is quite apt.

    My spam filter gets rid of a lot of dross (one account gets ~40+ messages per day advertising mail-order degrees, fake rolexes, anatomical enhancements, and assorted pharmaceuticals). I check the spam box regularly, looking for false positives. I've found, on average, about one every 6 or 7 months (a false-positive rate of about 1 in about 7000, compared to a negligible false-negative rate - I've flagged messages as spam three times in the 6-year life of that account).

    On the other hand, the "filtered" email also contains a lot of stuff that, while not actually spam, is still kind of pointless and a waste of my time (do I really need to see another bunch of lolcats?).

    So while the 'filter' gets rid of the obvious rubbish, it doesn't necessarily only admit pearls of electronic wisdom. Similarly for peer review. Nobody claims that it's perfect (as pointed out in the article, it gets the occasional 'false positive', and also lets some questionable stuff slip through), but it's by far the best system we have for large-scale review of scientific publications.

    Feel free to suggest a better system - I'm sure you'd receive appropriate accolades if you came up with one that did a better job!

    @richard.hockey at #3: that paper seems to be referring to problems in medical research, specifically looking at papers publishing results of research that has not been replicated. I'm not sure it's applicable to science in general, but I'm also not at all qualified to make that call! ;-)
    0 0
  7. Stephen Baines's reply that the paper richard Hockey cited is "odd" is more polite than I would have replied. Perhaps we can avoid repeating the details of critiques that have been published elsewhere, such as the one by Goodman and Greenland in PLOS (2007). Stephan Lewandowsky certainly is well qualified to reply.

    (Hi, Stephan! I don't remember if we ever met, or if Frank's frequent mention of you is why I remember you. I left OU in the Fall of 1989.)
    0 0
  8. To me, the peer review system, as described here, seems to be a system that assures that every new article that gets published, agrees with the present consensus. In other words, new or controversial ideas will have a hard time to get past a band of reviewers who prefer their own ideas.
    0 0
    Moderator Response: This is a fair comment. It can be difficult to publish something that goes against the grain, challenges a bit shot or a major paradigm, etc. But it isn't impossible. (JB)
  9. Stephen Baines @5

    Your comment about the 2 stage nature of peer review is very apt The first stage of peer review is like getting a pass to enter the exam room. Then you have to pass the exam which is the peer review of ALL your scientific colleagues. The entire readership of the journal. Therefore perhaps the best test of the merit of a piece of work is not simply its publication, but the extent to which has subsequently been cited by others, and not in rebuttal.

    Citations analysis might be a useful tool for examining the importance of published works for & against AGW. And the extent to which AGW Sceptical papers actually gain any traction.
    0 0
  10. 'The editor of the journal sends the paper to several other scientists. Their job is to independently, and usually anonymously, evaluate my work and to determine whether my experiment was sound and is worthy of being published.'

    Now you would think reading this that the peer reviewer does not know who the author of the article under review is. This is not so.

    Moreover:

    'First of all, there is no commercial consideration involved in the publication decision—none, zero, zippo, zilch.'

    Really? Doesn't keeping a rival's work out of the public eye give you a competitive edge when you go hunting for grants?

    BTW, I didn't think the Ioannides paper was 'strange.'
    0 0
  11. PS: Citation analysis is the latest fad - indeed, there are league tables for journals and the frequency with which their articles are cited.

    So 'no commercial interest?' Don't journal editors lose their jobs when their journals lose money?

    Mind you, peer review is probably as good a system as any. Just don't turn it into an article of faith
    0 0
  12. @Argus

    Wrong.

    Otherwise, Einstein's "Jewish Science" would never have got published, and we never would have heard of Relativity. In fact, I would not be reading your post, because it relies on technologies developed from quantum mechanics - one of the weirdest, new and most controversial areas of science ever developed. Even Einstein railed against it ("God does not play dice!").

    However, if you do try to publish a paper about how the moon really is made of green cheese, then you had better have some pretty sound evidence why the previous 10,000 or so peer-reviewed papers all got it wrong. Same for climate change.
    0 0
  13. Rebuttals and commentaries that reveal flaws in earlier papers can drive up citation counts (rather than the original authors getting egg on their face, they can receive a benefit of including a flaw in their analysis). It seems that oftentimes authors intend to leave something incomplete, and assumption unchecked, anything that will give them a chance to re-analyze the same data with a little tweak to show that they're making progress. I hardly ever produce publishable work, so maybe I'm just ignorant, but it seems to me that peer review isn't all that good a spam filter. (Note: my opinion on this is probably coloured by the fact that my field isn't very competitive.)
    0 0
  14. Argus, you presume that people cannot look past their "own ideas" to see the data for what it says. My experience says otherwise. Generally, the reviewers I have dealt with are more than fair. There have been notable exceptions, but the fact that they stick out in my mind is telling as well-- they are relatively few and far between. The social pressure on scientists to discharge their duty appropriately in peer review (or in editing) is actually quite strong. Nobody wants to be taken for a fool. And vindictivenes also has serious costs.

    I think Stephan acknowledged that the system isn't perfect, but in the end the proof is in the pudding and there is little doubt it has generated some very useful knowledge. It's certainly less biased than decision making based on the maximization of sales.
    0 0
  15. Chris, many journals in fact do attempt to keep the authors of papers under review unknown to reviewers but it's a tricky thing because of self-cites and the like. Lots of journals don't bother with the attempt.

    Meanwhile, your unfounded speculations about misconduct by reviewers do not remotely resemble a case that commercial considerations drive reviewer behavior.
    0 0
  16. Don't journal editors lose their jobs when their journals lose money?

    No. They lose their jobs if they harm the reputation of the journal, or fail to enlist reviewers or the like. By "journal" we're not speaking of Newsweek.

    Chris, you should probably stop now.
    0 0
  17. While the author concedes that peer-review is not perfect, I feel it's important to acknowledge a bit more candidly that the system has the potential to fail, both by generating "false positives" (approving non-deserving papers for publication) as well as "false negatives" (rejecting valid papers for unfair or invalid reasons). The number of false positives and false negatives is not nearly as high as Argus's eyes (all of which seem to be both myopic and jaundiced) might perceive (@#8). But whatever the failure rate might be, it's non-zero. It is these very flaws that are most troubling to many AGW skeptics.

    Most branches of science can provide their own examples of die-hards clinging to old ideas, while stubbornly resisting new ones. A famous example in geology is the debate over the theory of plate tectonics during the 1960's & 70s. The key point is that the peer review system does not need to be successful 100% of the time in order to yield valid results in the long run. Despite any flaws on a case-by-case basis, 'science' as a whole will eventually reject non-viable hypotheses and faulty data, in favor of better hypotheses and better quality data. It's simply Darwinian selection applied to scientific hypotheses.
    0 0
  18. Doug @ 17

    Chris, you should probably stop now.

    What a good idea :-)
    0 0
  19. @CoalGeologist

    Good points. However, I wonder if, with the expansion of scientific ranks since the 60s, the ability of single strong voices to drive the terms of the debate throuh peer review has declined somewhat. It may be that the much larger number of scientists adds another kind of inertia to the system (more people to convince, slower dispersal of ideas), but it is a different one to that typified by the plate tectonics debate....maybe.
    0 0
  20. Probably the only 'peer' review that actually wins is the proof over time that the science works, or works in the context of it's use.

    The story of Harrisons clocks to solve the longitude problem is a classic example. Not exactly science, but shows that eventually the correct solution is found.

    Physics in particular is littered with ideas that were initially rejected then accepted. But the key point is that in Harrisons case the Naval Board accepted his solution as did the scientific community accept the various Physics ideas.

    What didn't happen:

    The Naval Board was not disbanded because they were to conservative.
    The Physics community didn't get their funding cut because they initially rejected an idea.

    What is clear is that if an idea in science works, then it will be around for hundreds of years, probably thousands. Not because of democracy or because of a political campaign in favour of one theory or another.

    Peer review in a journal is probably the starting point, but it isn't the final outcome.
    0 0
  21. A couple of people have pointed out that peer review doesn't stop once a paper is published. I would go further and say that peer review starts long before most papers are submitted, and that there is an additional crucial element to publishing scientific papers:

    (i) The work submitted in a manuscript doesn't flow linearly from bench to paper. In pretty much all cases the work has undergone major elements of peer review before it is submitted to a journal. The work will have likely gone through the hierarchy of peer-review involving presentation at lab meetings, presentation at departmental or faculty seminer series, presentation in poster or platform form at scientific meetings and perhaps also presented to a grant awarding committee. All of these constitute tests of the inherent validity of the work prior to submission, and contribute to polishing of the presentation, identifying errors, inconsistencies or alternative interpretations that might lead to additional experiments and so on, before a manuscript is submitted.

    So in general most serious scientific work is submitted with the expectation that it will be accepted (even if it may have to undergo revisions acording to referees and editors critiques).

    (ii) A fundamental element of science and scientific publishing relates to the basic integrity of the scientist and this consitutes a major element of quality control. Basically scientists want to find out stuff and have strong desire to get to the truth (the "truth" often being a rather proximal "truth" that relates to a particular sub-element of a scientific field). When I referee a paper I do this with the expectation that the authors have made a genuine attempt to do careful experiments and to interpret their data faithfully. I might not agree with their methodologies and interpretations, but I never consider that the authors are trying to sneak a paper into press under false pretences [we know from examples of major scientific frauds that this does happen and the (front-line referee-based) peer-review process isn't very good at picking this up].

    The latter is interesting, since it's clear that in climate-related science and other areas of political contentiousness (e.g. intelligent design) people do try (and can suceed) to sneak bad science into the scientific literature.
    0 0
  22. CoalGeologist at 16:43 PM on 16 June, 2010

    "While the author concedes that peer-review is not perfect, I feel it's important to acknowledge a bit more candidly that the system has the potential to fail, both by generating "false positives" (approving non-deserving papers for publication) as well as "false negatives" (rejecting valid papers for unfair or invalid reasons)."

    That's an interesting one CoalGeologist: My feeling is that false positives usually don't have a particularily deleterious effect, although in rare cases (outright fraud that leads to a huge amount of wasted time, and badly done, but over-publicised research like the one underlying the MMR jab-autism scare in the UK) they clearly can.

    The smattering of false positives that dribble into climate-related journals or genetics/evolution journals (to pursue "intelligent design" politcs) and so on that serve political agendas are annoying, but only becasue they tend to promote a huge amount of hot air in non-science outlets like the blogosphere. They don't really affect the scientific process. It's very difficult to pick up outright fraud, but I do think some efforts could be made to intercept the occasional paper that clearly shouldn't have been published and is obviously (even if editors and reviewers don't notice) only submitted to promote non-scientific agenda positions (we could name some of these papers specifically!).


    I'm not sure that "false negatives" really exist. Of course papers are continuously being rejected from journals, in some cases unfairly, but that's not the end of the road. One simply takes the reviewers comments on board if there's anything useful in them, and submits the paper elsewhere. A paper that is sound will always be publishable somewhere.

    In fact with the modern emphasis on metrics (impact factors, citation counts etc.) there is a tendency for papers to go down a route which might involve something like:

    (i) hopeful submission to Nature (quick rejection)
    (ii) let's try Science (quick rejection)
    (iii) perhaps we can get it into PNAS (nope)
    (iv) O.K. we'll send it to the normal "house journal" of our field where it probably should have been submitted in the first place.

    There is certainly a smattering of very good papers that are undesrvedly rejected from good journals and end up in run of the mill journals. However if these papers are truly important then they will be noticed and their impact will be recognised.
    0 0
  23. "Wrong", says Chemware (#12) about my reflection on peer reviews (#8), whereas the moderator of this site seems to partly agree with me: ''This is a fair comment. It can be difficult to publish something that goes against the grain ...''.

    I think climate science may have problems that other sciences do not have - everybody likes a new possible cure for cancer or a new dwarf planet, but the atmosphere within climate research seems infected.

    Here is a quote from The Washington Post:

    In one e-mail, the center's director, Phil Jones, writes Pennsylvania State University's Michael E. Mann and questions whether the work of academics that question the link between human activities and global warming deserve to make it into the prestigious IPCC report, which represents the global consensus view on climate science.

    "I can't see either of these papers being in the next IPCC report," Jones writes. "Kevin and I will keep them out somehow -- even if we have to redefine what the peer-review literature is!"

    In another, Jones and Mann discuss how they can pressure an academic journal not to accept the work of climate skeptics with whom they disagree. "Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal," Mann writes.

    "I will be emailing the journal to tell them I'm having nothing more to do with it until they rid themselves of this troublesome editor," Jones replies.

    link to source (bolds by me)
    0 0
  24. Argus at 20:22 PM on 16 June, 2010

    Argus, which specific papers (""I can't see either of these papers being in the next IPCC report,") are being referred to here? I don't think it's unacceptable to choose to leave out obviously flawed work from summaries of a scientific field. That happens all the time (every time someone writes a review). So we'd really need to know which particular papers are being referred to here. Of course the language of the email is a little spicy, but's that's emails for you...

    As for the second point, I think this refers to the Baliunas Soon paper that was "shepherded through" the Climate science review process by a sub-editor. Is that correct? if so again I don't think Jones point is particularly problematic. One of the roles of scientists (which is inherent in the editorial and peer-review processes) is to maintain standards of scientific integrity. If an editorial process is being abused (as was clearly the case in the Climate Research instance) then it's appropriate for scientists to highlight this robustly and to take steps to address the problem. You might remember that most of the editorial board resigned over this bit of chicanery and the Publisher took the rare step of issuing a statement that the paper shouldn't have been published in the form it was.

    We're taking about one specific and dismal example of an abuse of the peer-review system. It was met with a suitably robust response. I don't find that problematic at all.

    As you say there is some elements of "infection" within climate science. A very small number of individuals attempt to sneak flawed work into the scientific literature. This happens in all fields in which science has implications that abut the political sphere (see e.g. efforts to publish "Intelligent Design" papers in the scientific literature). These instances should be highlighted for what they are (ultimately these are attempts to cheat Joe Public of his democratic right to the information required to make informed decisions), and opposed robustly by those that have the knowledge to recognise efforts to subvert acceptable scientific practice.
    0 0
  25. I think that this is a very good synopsis of the process. Another very good post related to this topic is by Dr. Steve Easterbrook. He allowed me to repost his comments on my blog: How Scientists Think.

    FYI:

    Nature only published 6.8% of the submissions in 2009. See the stats for the previous 22 years.

    Science averages less than 8%.

    Scott A. Mandia, Professor of Physical Sciences
    Selden, NY
    Global Warming: Man or Myth?
    My Global Warming Blog
    Twitter: AGW_Prof
    "Global Warming Fact of the Day" Facebook Group
    0 0
  26. I review about one paper a week. I find it an invaluable learning experience as you have to scrutinize and understand the science at a different level than just simply reading the paper. It is also a joy to be able to make good suggestions for improving a worthy paper. From an authors viewpoint there are two things that stand out about peer review for me. First of all how long it can take to get a paper published, it usually takes me at least three submitted drafts spanning about a year. Second my papers emerge much better than the original draft, so we will all gain, if the authors and the reviewers put in the time and effort.
    0 0
  27. Thank you for removing the ad hominem (feel free to delete this line from the post if you wish - and the rest of the post if you must).

    Couple of points. I do a little bit of peer reviewing myself (not much but just enough to have a sense of what it's about). It is interesting and challenging and a thought provoking paper can send you hunting references - it is an excellent learning exercise.

    A journal like Nature or Science naturally turns away 90 - 95% of submissions. Smaller journals in more specialised fields face the opposite problem - attracting submissions. I've no idea what the 'reject' rate would be in the major climatology journals.

    One measure of the quality of a journal is the number of articles I would actually read. Thus, staying within the bounds of my field, I'd read 75% of articles in, for example, The American Journal of Psychiatry. I guess this is a measure of traction. Other journals which shall remain nameless fare rather poorly.

    The more technical the journal & the 'harder' the science,the more rigorous the peer review process &the more arduous the publication process. I have no doubt that most peer reviewers try to do their job conscientiously. However, in some areas with major policy implications, processes such as those described by Argus @ 23 and rebutted by the other Chris @ 24 may gain salience. Finally, lets not forget banal human motivations such as professional rivalry, personal dislikes, and all the rest.

    Doug @ 16. As Argus has pointed out, journal editors may indeed be out of a job if key members of a scientific community rightly or wrongly decline to submit papers. A journal which gets no submissions goes broke. Academic publishers are not charities. Please note, I am in no way entering into the rights and wrongs of the Soon Baliunas debate or the email exchange between Jones & Mann purportedly related to the paper.

    Finally,I did write somewhat provocatively: 'But some (by no means all or even most) proponents of AGW do behave more like Scientologists than scientists.' I think I have made it clear in posts elsewhere that I have grave reservations about the MO of much of the sceptical commentariat.
    0 0
  28. Need I point out that Anthropogenic Global Warming, about 30-40 years ago, was the *new* paradigm on the block & took many years, & a *lot* of evidence, before it got accepted as the *new* paradigm for recent warming. So this helps to disprove the idea that only stuff which fits the existing paradigm will get accepted via peer-review. Yes some articles get through that have no business getting through, & others don't get through that should have but, with a little hard work & persistence, these *errors* in the system usually get corrected eventually.
    0 0
  29. "As Argus has pointed out, journal editors may indeed be out of a job if key members of a scientific community rightly or wrongly decline to submit papers. A journal which gets no submissions goes broke. Academic publishers are not charities."

    So in other words, you're saying that journal editors are under pressure to not publish junk science.

    I would hope you don't consider this a bad thing.

    In biology, there's the example of an editor of a relatively obscure little journal, whose term was expiring, allowing a paper "disproving evolution" to be published. Absolute crap. Got a similar "WTF???" response from the evolutionary biology community as the one Argus thinks is so heinous:

    In another, Jones and Mann discuss how they can pressure an academic journal not to accept the work of climate skeptics with whom they disagree. "Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal," Mann writes.

    "I will be emailing the journal to tell them I'm having nothing more to do with it until they rid themselves of this troublesome editor," Jones replied.


    What I can't wrap my head around is why anyone thinks such a response to crap being published is a bad thing.
    0 0
  30. And this is just bad journalism:

    In another, Jones and Mann discuss how they can pressure an academic journal not to accept the work of climate skeptics with whom they disagree.


    It's not disagreement that's the point, it's the fact that the paper under discussion was absolute crap. It was so bad that later, one half of the editorial board *resigned*.
    0 0

  31. "Wrong", says Chemware (#12) about my reflection on peer reviews (#8), whereas the moderator of this site seems to partly agree with me: ''This is a fair comment. It can be difficult to publish something that goes against the grain ...''.

    I think climate science may have problems that other sciences do not have - everybody likes a new possible cure for cancer or a new dwarf planet, but the atmosphere within climate research seems infected.

    Here is a quote from The Washington Post:


    One must make the distinction between innovative papers that truly "go against the grain" and papers containing errors that a professor would flunk a freshman college student for. In the email messages cited by the Washington Post above, Mann et al. were discussing the latter.

    If a journal editor demonstrates a pattern of approving "freshman f*&@up" papers for publication, it should not surprise anyone that scientists would complain about said editor in private email messages.

    Denialists need to learn the difference between censorship and professionalism.
    0 0
  32. Argus,

    Lindzen gets published, even if his evidence against the known climate sensitivity is fragile.

    Pat Michaels gets published, even if it´s rubbish. No grounds for suggesting group thinking prevents "mavericks" to have their space.

    Papers that confirm AGW, on the other hand, not only survive peer scrutiny, but also get confirmed by independent research.
    0 0
  33. As was pointed out in the Nature editorial on the topic of those two mysterious papers..

    "A fair reading of the e-mails reveals nothing to support the denialists' conspiracy theories. In one of the more controversial exchanges, UEA scientists sharply criticized the quality of two papers that question the uniqueness of recent global warming (S. McIntyre and R. McKitrick Energy Environ. 14, 751–771; 2003 and W. Soon and S. Baliunas Clim. Res. 23, 89–110; 2003) and vowed to keep at least the first paper out of the upcoming Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC). Whatever the e-mail authors may have said to one another in (supposed) privacy, however, what matters is how they acted. And the fact is that, in the end, neither they nor the IPCC suppressed anything: when the assessment report was published in 2007 it referenced and discussed both papers."
    0 0
  34. Well then I don't want to sound like a little bitch, but there was some contreversy as Spencer tried to publish hi's own latest paper that got rejected from Journal of Climate and Geophysical Research Letters without no reason and is finally getting published in the Journal of Geophysical Research.

    I would call that a bitt odd.
    0 0
  35. dhogaza 2 29:

    So in other words, you're saying that journal editors are under pressure to not publish junk science.

    I would hope you don't consider this a bad thing.

    Agreed. However, science has its share of powerful personalities who dominate the scene by their presence (and not always by their integrity). This applies to the sceptical side as much as (in some cases more)to the AWG side.

    Moreover, the metric of success for any academic is mainly their publication record - publish or get no grant.

    Somewhat off topic, but Henry Kissinger disingenuously liked to pass himself off as a naive academic and newcomer to politics when recruited by Richard Nixon. In fact, he already had a giant footprint - how else do you get to be a professor at Harvard?
    0 0
  36. Chriscanaris @ 4. Yes, some behave like that. Can you cite precisely climate scientists who try to pretend that uncertainties do not exist? And then there are the Beck and Limbaugh. So, really who's doing the worst howling out there?
    0 0
  37. cloneof, it is extremely unlikely that Spencer was given no reason for his paper's rejection. Most likely is that he didn't like the reason that was given. That's not odd. Rarely is an author happy with the reasons for rejection. Nor is it odd that his paper eventually was accepted in a different journal. It happens most of the time.
    0 0
  38. Argus quoted... "I can't see either of these papers being in the next IPCC report," Jones writes. "Kevin and I will keep them out somehow -- even if we have to redefine what the peer-review literature is!"

    If I remember correctly, the papers in question here actually ended up being mentioned in the IPCC report. It's a perfect example of what this article is discussing. Peer review can be a very competitive sport. Sometimes it's bare knuckle and back biting but what comes out as a result is that better science generally prevails. I seem to also remember that there were equally bitter battles waged in science when it was proposed that some dinosaurs and modern birds were related. So, I don't believe this is confined to climate science.
    0 0
  39. Well then I don't want to sound like a little bitch, but there was some contreversy as Spencer tried to publish hi's own latest paper that got rejected from Journal of Climate and Geophysical Research Letters without no reason and is finally getting published in the Journal of Geophysical Research.

    I would call that a bitt odd.


    Cloneof, as Tom Dayton says, it's not odd at all. It's not at all unusual for a scientist to have to trot around a paper before it gets accepted. Think of journals like Science and Nature, where 90-95% of the submissions get rejected. Do you think the authors of those papers just round-file them, or do they shop them around looking for a journal that will accept it?

    What's odd is Spencer trumping this up in the denialsphere, when he knows perfectly well that there's nothing unusual in not getting accepted into the journal one picks as one's first choice.
    0 0
  40. Agreed. However, science has its share of powerful personalities who dominate the scene by their presence (and not always by their integrity). This applies to the sceptical side as much as (in some cases more)to the AWG side.


    In this case, the six editors who resigned included von Storch, who leans towards the skeptical side of the argument. In other words, the paper (Soon and Ballunis) really was crap.
    0 0
  41. Many journal submissions are rejected by editors without the editors sending the submissions to reviewers. That is normal, expected, and necessary. See also this comment.
    0 0
  42. Cloneof, it is possible that this Spencer paper is actually quite bad. "Skeptics" have a tremendous variety in their rationales as to why peer-review does not work, or can't be trusted, yet strangely enough they trumpet vigorously every skeptic paper that gets published, regardless of the potential value.

    From the reading I've done on skeptic sources, it appears that all the objections raised about peer-review do not seem to apply the same to skeptic papers.
    0 0
  43. Interesting article in The Scientist:
    "Is Peer Review Broken?"
    http://www.the-scientist.com/article/display/23061/

    I would also contend that if the internet existed 50 years ago peer review wouldn't exist. However seeing that it does exist, although I think its days are numbered, I support the idea of full disclosure. see:
    http://www.nature.com/emboj/about/emboj_review_process.html
    0 0
  44. Richard.hockey - I'm intrigued. On the internet we have all these complete rubbish "papers" being put out by the clueless and the mendacious in a vast no. of fields. Peer review is a starting point for sorting the chaff. If the internet was around 90 years ago, what process do you would have replaced peer review for providing some basic gate-keeping?
    0 0
  45. Philippe @ 36

    Can you cite precisely climate scientists who try to pretend that uncertainties do not exist?

    None would be so naive or silly. I don't even suggest it in my post @ 4 or anywhere else.
    0 0
  46. Brain, Volume 123, Number 9, Pp. 1964-1969
    September 2000
    Reproducibility of peer review in clinical neuroscience
    Is agreement between reviewers any greater than would be expected by chance alone?
    Peter M. Rothwell and Christopher N. Martyn

    Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance
    0 0
  47. Svensmark: "But for some unknown reason I never could publish our work. We sent it to 4 different letters, but each time met with refusal. No one accused is not anything - do not reproached errors[...]. They said: We are not interested, either: the text is too long. There was no substantive criticism of our work - it gave rise to even greater disappointment. [...]"

    Eugene Parker, a professor of astronomy and astrophysics at the University of Chicago, commented on this:
    "Publishers tend to be very naive [!?]. A negative opinion of the reviewer, if you did not put it in her compelling reasons against the publication, should lead to reflection. In such a situation should appoint another reviewer. [...]"
    "Global warming has become a "hot" and political topic. I have evidence that, for example in the United States, blocked the publication of serious scientific research on warming. People who are convinced that they know the truth, deny others the right to vote. It harms science, the United States and globally. [...] Tying the discussion does not help in solving the problem."
    0 0
  48. Arkadiusz Semczyszak, could you give links for those quotes, please ? I don't think anyone should be able to quote others without a link, but maybe that's just me.
    0 0
  49. Funny, Arkadiusz, but many US scientists complained that the Bush Administration was cutting off Federal Funding to institutions pushing the pro-AGW line. Here in Australia, the former Howard Government did much the same thing. However, wheras politicians, & the press, are happy to act in a partisan fashion-I doubt that Reviewers could long get away with similar behaviour.
    0 0
  50. chriscanaris wrote :

    However, science has its share of powerful personalities who dominate the scene by their presence (and not always by their integrity). This applies to the sceptical side as much as (in some cases more)to the AWG side.


    Could you give some names of those on the "AWG side" who are "more" likely (according to you) to "dominate" by their lack of integrity - "not always by their integrity" ?
    0 0

1  2  Next

You need to be logged in to post a comment. Login via the left margin or if you're new, register here.



The Consensus Project Website

TEXTBOOK

THE ESCALATOR

(free to republish)

THE DEBUNKING HANDBOOK

BOOK NOW AVAILABLE

The Scientific Guide to
Global Warming Skepticism

Smartphone Apps

iPhone
Android
Nokia

© Copyright 2014 John Cook
Home | Links | Translations | About Us | Contact Us