What most media doesn't tell you on reporting science

In the twenty-first century society, science is the building block that helps us construct and understand the world we live in. However, every so often misreported scientific studies take a toll on science’s credibility. Science is constantly producing new studies and as more intricate segments of nature are being investigated, they are becoming increasingly complex to interpret by non-scientists. Consequently, journalist and television producers often rely on press releases. They shorten those even further, stretching and torturing academic papers until their original meaning of the study has been completely lost. The lack of first-hand interpretation of the study, coupled with sensationalism, exaggeration and hype results in the indoctrination of common people with misleading information.

“Scientists say smelling farts prevents cancer”

This is one among the many glaring examples of a scientific study being blown out of proportion. The article was published by Time, that was later retracted and heavily corrected. In fact, the scientific study behind the report mentioned nothing about cancer or fart. Misinterpreted stories such as this lead to contradictions and ultimately less faith in science, which is unfortunate because science by its nature is imperfect but it deserves better than to be twisted out of proportion.

The news media, like all forms of entertainment, is constantly trying to gain a larger audience and that combined with our love for reading fun poppy science, science news articles have become precarious. To add fuel to fire, we now have social media by which we consume and share information faster than ever before, fueling the spread of misinformation.

Sharing misinformation is like polluting the environment and is irresponsible & dangerous.

So what can we do to minimize the spread of misinformation and consequent indoctrination? The general public can’t be blamed for science being too technical or risk waiting for a change among science writers or the newspapers that are constantly in need of new, sexy studies to report. Fortunately, many scientists can instantly tell apart a news piece that has been misrepresented or hyped and if we can reproduce their thought process, maybe we can make a difference.

So I am going to prep you up to read and share Science like a Scientist.

Based on my scientific training and personal experience, I hope to give you an understanding of how science works, highlighting the most common limitations that the media does not report. Hopefully, by knowing this, you can make a more informed judgment when reading your next science-related news article.

Design matters

Astrophysicist Neil deGrasse once said, “The good thing about science is that it’s true whether or not you believe in it.” As ironic as it may sound, one of the first thing that we get trained in PhD is to not believe. Scientists call this ‘critical reasoning’. Within academia and the scientific community at large, research output is documented in peer-reviewed journals. When we read a discovery published in a peer-reviewed journal, we spend some time to contemplate on the experimental design and judge its strengths and weakness. No study is one hundred percent perfect and this exercise helps us to understand the extent to which the results can be generalized.

However, a non-scientist will likely not go looking for information in peer-reviewed journals (where the experimental design is available) for a number of reasons. Firstly, because many of these journals are behind a pay-wall and secondly, it might be too technical to interpret without a reasonable level of domain knowledge. For example, reading the ‘methods section’ of a quantum physics paper and making a decision on its reliability is beyond the realms of my skills.

Hence, one can look out for a few basic pointers that may be mentioned in the news articles to gauge the credibility or the level of extrapolation. If this basic information is not mentioned in the article, I would suggest you be more cynical about the points made.

Research with what?

First and foremost are the species of experimentation. Throughout the era of modern medicine, animals have been used extensively to develop and test therapies before they are tested in humans. Virtually every medical therapy in use today owes its existence, at some level, to animal experiments. However, animal trial results don’t always translate to humans. A 2006 review looked at studies where medical interventions were tested on animals and whether the results were replicated in human trials. It showed that among animal studies published in prestigious scientific journals, only 37% were replicated in humans and 18% were contradicted in human trials. So always remember that animal trial results don’t always translate to humans.

Research on how many?

In 2015, various versions of ‘Women may be more primed for romance if their stomachs are full’, made headlines across major newspapers. CNN, in fact, went on to suggest- “If you really want to get lucky in love, forget about taking her to a movie or a bar; you may be better off taking her out for dinner first”. It became an internet sensation almost instantly. But the devil lies in the details, no one noticed that the study included only 20 women. Although there is nothing wrong with conducting well-designed small studies; they just need to be interpreted carefully. I think we can all agree that twenty women cannot speak for all women.

While small studies can provide results quickly, they do not normally yield reliable or precise estimates. Therefore, it is important not to make strong interpretations, whether the results are positive or not. So always keep an eye on sample size to make sure that the conclusions are fairly generalizable. A scientist will always tell you to not to attach too much significance to any individual study until they are placed in a larger context.

What else affects the research?

Research is never black and white and so whenever you read research news, your brain should automatically start thinking of the possible confounding factors that the author may not have accounted for. For example, there have been studies suggesting, coffee is good for you one day, bad the next. If we knew no better, coffee could be a God, it will either kill you or save you depending on how much you believe in its magical power. The reason why there is so much variation among different studies is that it’s hard to measure the impact of diet on health. There are simply too many variables that can affect the outcome (like age, environment, smoking and drinking habits, genetics, country, microbiome etc). As you can imagine this is one of the more complicated things to account for when reading science news. Most likely these details will not be reported by the media and one might need to go back to the original article. Even if you don’t, a simple way of gauging is to examine how broad the field is (e.g. effect of chocolates on pregnancy is a super broad field). The broader the study, the more confounding factors could affect the results and consequently don’t attach too much significance to the results unless multiple studies conclude the same.

Cherry picking results?

Researchers themselves are under constant pressure to publish, which in turn determines their career and research funding. Current scientific practices create strong incentives to publish statistically significant results. So researchers may indulge in something referred to as p-hacking. It occurs when researchers try out several statistical analyses and then selectively report those that produce significant results. Although the results may be statistically significant they probably could also be meaningless. To highlight an exaggerated example, a high correlation was found between ‘US spending on science, space and technology’ and ‘suicides by hanging, strangulation or suffocation’. The attained mathematical correlation by p-hacking is true but its real-world association is bogus. In reality, the associations are more subtle and harder to spot. Some more famous example can be found on this website tylervigen. The misuse of p-value can drive bad science. Fortunately, scientists are aware of this issue and are trying to adopt more sophisticated statistical tests in peer-review to make sure that such bad science does not see the light of day.

Misc.

Firstly, if you ever start reading a news article that has not referenced the original study, close it and leave. To me, if a journalist couldn’t take a moment to link back to the original article, it is not worth reading. It is either they are trying to hide something or do not want you to leave their website, both of which are inadmissible. Secondly, as I mentioned earlier research outputs are documented in peer-reviewed journals. However, there are less than legitimate journals that publish pseudoscience. The media should do a background check before rushing into reporting an article published in such predatory journals, however, if they do not, a certain level of googling could help. Here are the top ten ways to identify a predatory journal published by PLoS blogs. Finally, always place more trust in a meta-analysis. Conceptually, a meta-analysis uses a statistical approach to combine the results from multiple studies in an effort to increase power (over individual studies), improve estimates of the size of the effect and to resolve uncertainty when reports disagree.

Science is continuously evolving

Previously, I suggested a few pointers that you could go looking for in a news article to determine its legitimacy. However, as such is not enough. You need to always remember that science is a field of continuous evolution.

During the time of Aristotle and Ptolem, the earth was believed to be at the centre of the universe. Two observations actively supported this geocentric model. First, from the view of earth, the sun appeared to revolve around the earth once per day. Second, the earth did not seem to move from the perspective of an earth-bound observer; it appeared to be solid, stable, and unmoving. It was only during the late 16th century when the model was revised by the works of renowned mathematicians and astronomers such as Nicolaus Copernicus, Johannes Kepler and Galileo Galilei.

Scientific truths exist in context. They exist relative to past discoveries and are constantly subject to review and revision. There are uncertainties associated with most discoveries, but they are accepted as working theories, and as such, they can be used as stepping stones to a new level of understanding. Many scientific discoveries have practical applications that help validate them, but some indicate new directions for research without any immediate application.

I urge you to bear in mind the imperfection of our current knowledge. Science is never finished. It proceeds by successive approximations, edging closer and closer to a complete and accurate understanding of nature, but it is never fully there. — Carl Sagan

New technologies often revise our understanding of older discoveries. For example, as many of us can relate to being taught that atom is made up of electrons, neutrons and protons. With the advancement in technology, we have been able to discover at least another 10+ subatomic particles. So when you are reading a media report, if the result is summarized to be black and white, clear-cut or simple, remember- it most probably isn’t and is likely to be revised in the future.

Let not uncertainty alarm you though. Science is great, but it’s slow. Most experiments fail. That doesn’t mean the challenge isn’t worth it. We need to remind ourselves that in the last 200 years our efforts in the research and development of health and medicine has resulted in doubling our average life expectancy (from 40 to 80 years). It may not be even an overly ambitious target to attain a life expectancy of 150 years with another 50 years of research.

Change the way you think of science

While the norm is to ask a question, do some research and find an answer; there lies a more realistic approach. It is a bumpy path wherein you ask a question, do some study, get a partial or ambiguous answer and then you keep repeating until you get a better picture. Yes, it is slow, daunting and difficult and this is precisely why science deserves the highest order of respect. The inherent scientific uncertainty doesn’t mean that we cant’t use it to make significant and distinctive policy decisions or apply to everyday practice. It simply means that we should be cautious and adopt a mindset that’s open to change, as ideas and discoveries evolve. We should be able to make the best judgment we can with the existent evidences and take precautionary measures to not lose sight of its strength and degree of certainty.

Before I end, let me introduce what I call, “mental virginity”. It is a state of mental certainty, where one has not been influenced by someone else’s perception.

Not just science but every news article that you read is most likely a perspective of someone else. My advice to you is first to deconstruct it, find the few sentences that actually matter, analyze it, critically challenge it and then form an opinion for yourself. Don’t let someone else’s perspective manipulate your life. Stay a ‘mental virgin’ and be a wise consumer of information in an increasingly artificial reality.