Tweet

August 13, 2015

Our Article for Cracked.com (The Director's Cut!)

A couple of months ago, we wrote an article for Cracked.com highlighting how academic publishing companies have hurt the progress of Science. They published the article today on their website, and we're very happy with how the final article turned out (except for their choice of title...).

Our original submission was much longer than what was published, so we thought we would post the unedited version here on our website. While we did also submit photos and captions with our version, we will not post the photos here, as we do not own their copyright. So, please enjoy!

**************************************************************

7 Crazy Realities of Scientific Publishing

The scientific method is one of the cornerstones of science (the others being lab coats and the use of Greek letters in equations). First you make an observation (“why is my car making a funny noise?”), then you form a hypothesis (“it sounds like my tire is flat”), make a testable prediction (“if my tire is flat, then it should be deflated when I pull my car over and look at it”), and finally obtain data by performing your test (“Yes, it’s flat! I’m a Science God!...Oh, shit.”).

Modern science has added another step to the scientific method that you didn’t learn in third grade: publish your results. Publications in peer-reviewed scientific journals are arguably the most important avenue for scientists to follow the research of others and share their own research with other scientists or the public. However, the rise of academic publishing companies as insanely profitable businesses and the intense pressure on scientists to publish has poked so many holes in the scientific method that it now resembles an undergrad’s lab coat after experiments with sulfuric acid.

7. Scientists that don’t find a positive result probably won’t get published

If you do the scientific method correctly, you will end up with one of two possible outcomes: a positive result, meaning that your data supported your hypothesis (ex: my car did have a flat tire) or a negative result, meaning that your data did not support your hypothesis (ex: my tire’s fine—maybe I ran over the cat?). While either result can arise from an experimental test, it’s estimated that only about 0.1 – 10% of hypotheses generated by scientists should be supported. So, it’s a bit odd to find out that experiments touting positive results make up anywhere from 70 – 90% of all published papers. Are scientists just really mutants with the power to consistently generate true scientific hypotheses?

Well, no. The majority of scientific papers contain positive results because that’s what scientific publishing companies want. Nobody wants to read about all those weight-loss drugs that failed to work; they want to read about the one that worked. This practice among publishers is so common that it has its own name: publication bias.

And like trained mice, some scientists have adapted to publication bias by undergoing a process called ‘p-hacking’ (take it easy men, the ‘p’ refers to something called a ‘probability value’). Essentially, p-hacking is running one hypothesis through the scientific method over and over again until it gives you a positive result. This temptation is so strong because the more papers a scientist publishes, the more likely they are to get grants, employment, and tenure.

How it affects you
Publication bias is especially harmful in the medical community. Because journals favor the positive results, only the papers stating that a drug works are published and, thus, visible to doctors. Meanwhile studies that find a negative result remain locked away forever and are probably told that they should feel bad.

This leads to doctors falsely believing a drug is fantastic because all they are able to read about it are the rave reviews, when in reality the drug works no better than a can of Pringles. Sometimes, doctors go so far as to assume that something must be wrong with patient if the drugs don’t seem to have an effect.

Well, at least publication bias can’t kill you, right? Well, it can if you were unlucky enough to get prescribed Locrainide. This drug was supposed to help against heart arrhythmia, but trials in 1980 found the troubling side effect of death. Because of this ‘negative’ result, the study was never published. Several years later, pharmaceutical companies were itching to push out an anti-arrhythmic drug and settled on Locrainide, unaware of the unpublished previous study. The result was over 100,000 deaths before the unpublished study was brought to light and the prescriptions stopped. Perhaps even more shocking is that even after all of the reported deaths, the authors of the initial study were still rejected by multiple journals before finally getting it accepted for publication.

6. Scientists are seldom required to show their data

When your friend tells you that they had drinks with Bill Murray last night after you wussed out and went home, your first reaction is to get proof—pics or it didn’t happen. Congratulations! You have higher standards than most scientific journals!

When scientists submit a paper to a journal, they send the final product containing all the text, tables/figures, and analyses of the data. But seldom are scientists required to show any of the raw data. In fact, only about 15% of journals even have so much as a statement on data sharing, and the journals that do require deposition of data are pretty lax at enforcing that policy.

For scientists, restricting access to data stalls scientific progress and hinders the ability of other scientists to fully replicate a study. It’s hard to stand on the shoulders of giants when you’re not even allowed to see them.

For you non-scientists however…

How this affects you
When scientists don’t publish raw data with their papers, it makes it very difficult for others to double-check their work. And scientists, being human (with apologies to Mr. Peabody), make mistakes. One particularly damaging mistake occurred in 2010 with the publication of an influential economics paper, which concluded that countries with large debts experience lower economic growth. This single study was consistently used by deficit hawks in the United States, like Paul Ryan, to justify large cuts in government spending. Even famed economist Paul Krugman said that this paper “may have had more immediate influence on public debate than any previous paper in the history of economics.”

However, the paper’s authors made a rather intricate data analysis error. And by rather intricate data analysis error, we mean that they didn’t drag a formula in MS Excel to include all the rows of data.

The correct analysis actually shows that countries with debt experienced small increases in economic growth. This error was only found after three other economists had to personally ask the paper’s original authors for their Excel Spreadsheet.

The above example was an unintentional error. But letting scientists keep their data a secret allows them to make intentional errors with the data or, even worse, just make up a completely fake dataset. Once this fake data is published, it’s accepted by the scientific community—meaning that your doctor might be basing their conclusions off the data figments of some unethical scientist’s imagination.

Just because we know you have a doctor’s appointment next week, we’ll give two such disheartening examples. The research of ex-doctor Don Poldermans showed that beta-blockers (a type of drug) given during surgery did not lead to an increase in deaths. The research of Don Poldermans also seems to have consisted of finding the best random number generator to produce his fake dataset.

When Poldermans’ set of random numbers (sorry, we can’t call this data) is removed from a review of all studies on beta-blockers, it turns out that the drugs cause a 27% increase in deaths. Based on the 2.5 million patients (in the UK alone) that were treated with beta-blockers, it’s estimated that at least 10,000 deaths were caused by his faked data, and one estimate put it at 800,000.

The case of Joachim Boldt, one-time record holder for number of retracted papers with 90 (what’s more depressing, that number or the fact that he was the “one-time” record holder?) may be even worse. Boldt faked data claiming that a particular type of IV drip caused no increase in patient death, whereas most other researchers were finding the opposite—about 4 deaths in 100 patients treated with the drip. Boldt’s ‘data’, however, swayed many doctors to continue using that IV drip for several years, and it’s been estimated that the continued use of that IV has caused at least 20,000 deaths.

Now, if data were required to be made available with the publication of papers, the risks associated with faking data would be much higher (faked data can be easily spotted by data scientists). And there is a small movement among some journals and government agencies to require data deposition. But until scientists adopt large-scale data sharing policies, papers retraced due to fraud will continue to rise.

5. Peer review is a joke

Peer review—the practice of having a group of unbiased expert scientists determine if a paper will get accepted for publication—is often seen as the Guardian of the Scientific Method. If a study is able to pass rigorous peer review and get published, then you can be assured that what you’re reading is 100% good science.

So, what is peer review in reality? To use the words of Michael Eisen, Professor at UC Berkeley, “peer review is a joke." Peer review consists of, on average, just two people, who may or may not be an expert on the paper’s subject matter. They’re often biased against lesser-known scientists (author’s names are often visible to peer reviewers) and unconventional ideas. And they’re terrible at spotting weaknesses and errors in papers. In a sting operation, Fiona Godlee, a former editor of the British Medical Journal, found that, on average, peer reviewers could only find 2 of 8 mistakes deliberately inserted into a paper. More recently, Harvard professor John Bohannon submitted an obviously fake paper claiming that lichens cure cancer to over 300 journals, and 70% of the journals that used peer review accepted the paper.

Scientists can also rig peer review in their favor. Some have reviewed their own papers by setting up fake email accounts and posing as eager reviewers. There are even companies that offer to favorably review papers for a fee. Yes, peer review is now a profitable business model.

How this affects you
So, it seems like peer review is less like getting into Harvard and more like getting into your fifth safety-net school. What that means is that some really junk science can find its way past reviewers. The most infamous is arguably Andrew Wakefield’s paper linking autism to vaccinations. The study (now retracted) was published in The Lancet, a high-profile peer-reviewed journal, and is thought to have contributed to thousands of preventable deaths.

Actually, it’s said that 4 of the 6 reviewers rejected Wakefield’s paper. But if it got accepted anyways, what was the point of the peer review?

For journalists, the fact that Wakefield’s study was peer-reviewed lent it an aura of legitimacy, kind of like putting a candle on a piece of dog shit and calling it a birthday cake. Peer-review carries this connotation of accuracy and infallibility and some corporations are starting to wise up. For example, the tobacco industry helped start a journal so that they could have their people ‘peer-review’ studies claiming that secondary smoke wasn’t hazardous to your health.

For all the shitty science peer review lets in, it also does a good job of keeping legitimately good and life-saving science out. In 1797, after developing a vaccine against smallpox, Edward Jenner submitted his study to a journal. Apparently, destroying smallpox wasn’t good enough for the journal and they promptly rejected his study (either that, or there is a serious conspiracy between peer review and vaccinations). In the end, Jenner had to self-publish his study. Who knows how many people would have been saved from smallpox had the journal not rejected his study?

4. Scientists don’t get paid for contributions and often have to pay to get published

An artist creates a painting with the hopes of selling it. A baker invents a stupid cronut with the hopes of reaping in the profits. Even a writer for Cracked can expect to make up to $10,000 per article (right, guys?). So, how much money does a scientist get for each paper they produce? About $0 (for the Europeans, that converts to €0).

Actually, it should be negative monies if you consider that publishers often charge scientists an article processing charge (APC). That’s right, many times scientists have to pay in order to publish their papers. Remember, scientists are judged and ranked almost solely by the number of publications they have. Asking them to pay a journal to publish a paper is like asking Tom Hanks to pay the movie studio to film a scene.

Well, maybe this seems reasonable. How much are these APCs anyways, like $50? No, more like $2,300 per article with some journals charging as much as $5,000. And God forbid you need color to show your sciencey results, as some journals charge $1,000 for each color figure.

Obviously, some scientists cannot afford APCs, which restricts the impacts their research can have. In fact, APCs have gotten so out of hand that one scientist had to open a Kickstarter to raise funds for publication.

How this affects you
Well, if you’re a scientist, please see the above section. If not, you might be thinking that this probably doesn’t affect you at all. Well, consider that most scientists pay APCs using their grant money, which they usually get from the government, which gets that money from you and your taxes. In Britain, it’s estimated that about 1 billion pounds a year is spent just on these APCs. That’s 1 billion pounds that could have been spent on whatever it is the British are spending their public funds on.

If scientists don’t have the grant money, they get their University to pay up. One researcher has calculated how much British Universities spend on APCs every year, and the estimates range into the hundreds of thousands. For instance, the University of Manchester pays £580,000 (~$900,000) in APCs every year. That’s enough money to buy textbooks for about 1,400 of its students.

The less fortunate scientists, such as those from developing countries in which external funding for APCs is severely limited, are pretty much screwed (their only option is to apply for an APC waiver from the journal). For example, scientists in India typically cannot afford the APCs of high-profile journals and must settle for journals based in India (publications in India do not charge APCs). Unfortunately, these journals are not widely shared outside of India. I say unfortunately not for the scientist but for the public, because India is a gold mine for research on many public health issues like disease epidemics, but most of this data is effectively locked away in Indian journals. For example, Indian Libraries contain over 100 years of tuberculosis research that would prove invaluable to any TB researcher.

3. Scientific publishing companies have higher profit margins than almost all other businesses

Historically, scientific societies would self-publish their own journals (which is why many journals start with Proceedings of the [insert name of society]). Today, large publishing companies manage most of the new and existing journals, with three such companies (Reed Elsevier, Wiley-Blackwell, and Springer) accounting for 42% of all published articles. This oligopoly over scientific literature might be bad enough, but what makes these publishing companies rise to supervillian-levels is their obscenely high profit margins, which regularly clock in between 30 and 40%. That’s higher than every other company with the possible exception of Pfizer.

They achieve such high margins by charging scientists twice: once for preparing your paper for publication (the aforementioned APC) and again for the privilege of reading your article (reading an article can cost about $35). Yes, you read that correctly, scientists need to pay to read scientific papers, including their own. Don’t believe me? Here’s an article that I wrote that would now cost me $40 to read.

People have compared the business model of a scientific publisher to the world’s worst restaurant: you bring in your own ingredients, pay the restaurant to assemble and cook your food, and then pay them again for the right to eat it. With a business model like this, academic publishers make Rupert Murdoch look like a socialist. Oh, I should have probably put quotes around that last clause in that sentence, because that was totally the title of a recent article.

How this affects you
We already saw how charging an APC can affect the sharing of science. Charging a fee to read articles can restrict scientific progress even more, especially to scientists and doctors in developing countries, and this denied access can have drastic consequences. For example, a paper published in 1982 warned that the Ebola virus was actually present in western Africa, when everyone at that time assumed that it wasn’t. However, this information was unknown to Liberian doctors because they couldn’t afford to pay the $32 fee to read the paper, about half a week’s salary. If the companies didn’t charge a fee, or possibly a fee not as outrageous as $32, Liberian doctors could have been aware that an outbreak was imminent and could have taken better precautions.

Like most large corporations, publishing companies can act downright evil at times. Reed Elsevier once accepted money from Merck, a pharmaceutical company, to launch 6 new journals that published studies favorable to the company.

Perhaps most egregiously, Elsevier was also caught offering $25 vouchers to any scientist that would give a book published by Elsevier a 5 star rating on Amazon. You can hasten Ebola outbreaks and publish fake studies promoting questionable drugs, but you do NOT fuck with Amazon reviews, Elsevier!!!

2. Some publishing companies actively prohibit scientists from sharing their own papers

Congratulations! You got your first scientific paper published and, naturally, you want to send a copy to your parents to show them that mortgaging their house to pay for your tuition wasn’t a total waste of money. Well, remember those forms that the publishing company had you sign, and that you didn’t read because you were so excited that you might actually make it as a productive scientist? You just signed over your copyright to the article, giving the publisher the only exclusive rights to distribute it. Meaning that, legally, you can’t give a copy to anyone, not even your own mother.

In the past, publishers would look the other way when scientists would share papers or post them on their personal website. Now, however, like that cop that nabs you for going 1 mph over the limit, publishers are actively sending take-down notices to Universities, including Harvard and John Hopkins, requesting that scientists stop this sharing. At least one library in Switzerland got hit with a lawsuit by publishers for distributing scientific papers.

How this affects you
An important part of the scientific method is to share your results, so scientists need access to these papers. When they can’t get them from their peers, they have to rely on subscriptions purchased by themselves (rare) or by the library or institution that they work for (more common). And because now we know that publishing companies are supervillians, the fees for these subscriptions are astronomically high. How high? Try in the millions of dollars per year. Harvard University alone pays $3.75 million a year in subscription fees, and the situation got so ridiculous that Harvard Library sent out a memo to its faculty urging them to publish in journals that don’t charge subscriptions. By the way, that $3.75 million is enough money to cover the cost of textbooks for half of Harvard’s undergraduate class.

Things aren’t much better on the other side of the pond, as one researcher calculated that British Universities pay a combined total of £79.4 million (or $123.8 million) per year.

Keep in mind that these companies are essentially charging scientists for access to papers that the scientists themselves wrote. Also keep in mind that most research is taxpayer funded meaning that you are ultimately footing the bill. This is one reason why the US National Institute of Health (NIH) and other government agencies wanted to pass legislation to make government-funded studies freely available after one year. Surprise, surprise, publishing companies tried to push their own bill, The Research Works Act, which would have basically prevented free access to these papers. Fortunately for you and public health, this bill failed to pass.

1. There are fake publishing companies that scam scientists and publish junk science

With profit margins that make CEO’s of investment banking firms orgasm, it was only a matter of time before fake publishing companies appeared and began straight-out scamming scientists by pretending to be legitimate journals. After all, it only takes a website to open an online publishing business. These companies, referred to as Predatory Publishers offer to publish any paper, regardless of quality, for a hefty processing fee that ranges in the thousands of dollars. And many times this fee is first mentioned after the paper has been accepted and the scientist has signed away their copyright. One librarian has counted over several thousands of these journals.

Scientists aren’t completely innocent, however. Because these companies publish anything (and we mean anything) for money, some scientists have figured out that they can build their resumes by simply paying for publications, no matter how crappy their science is. So, kind of like that friend of yours who bought 1,000 followers on Twitter. With these fake journals in play, scientists can completely bypass the scientific method just by spending some cash.

How this affects you
We derived an equation to illustrate the problem that we intend to publish in the Journal of Totally Legitimate Math Equations:

(junk science + publisher that publishes anything^{aura of peer review}) * media exposure = public acceptance of bad science.

This formula was illustrated recently when a Harvard researcher wrote up a sham study about how chocolate helps you lose weight and then published it in a journal run by a predatory publisher. The media totally bought the study (and, why not? it is published in a peer-reviewed journal), and you probably saw it posted on your aunt’s facebook page.

Publishing fake science can have greater implications than temporary diabetes. Several scientists have used predatory publishers to push their anti-nuclear power agenda. While nuclear power is not without its hazards, these scientists have greatly exaggerated its risk by claiming that somehow the Fukushima disaster in Japan caused thousands of deaths in California (it didn’t) or that just living next to a nuclear power plant increases risk of cancer (it doesn’t).

The crappiness of that science hasn’t stopped many media outlets from reporting these stories, and needlessly scaring the shit out of Californians in the process. Upping the scare has been Alec Baldwin who has completely bought into this junk science and has spoken about this research in presentations and even wrote the foreword to the book of one of these scientists. And when Alec Baldwin gets fooled by bad science, you know that the scientific method is broken.