How To Read Science Articles Without Being Led Astray By Bad Headlines

Do not be taken in by those attractive science headlines. Anyone can write a headline that will entice people to load a page, but how many times has the article turned out to be a 400 word summary of another worthless article? How many times do we fail to find any support for the claims that were made in the headline?

How many times was "a study" or "an experiment" referred to without any link to the actual study report? How many times was the title false, misleading or an absolute cheat disguised as news?

In some cases, people will read a science or survey headline, skim the article and rush to make business, investment and other decisions. Sometimes bad information and poor reading will cause embarrassment as a more savvy reader points out the errors.

It is time for the public to be armed with techniques that will lead us closer to the truth in the matter. Of course, it takes time. Sometimes we have to visit several sites and settle down for some patient reading in order to get those nuggets of truth.

It is fine if the whole endeavor was just another incremental step toward greater knowledge and understanding.

It is not fine if the headline makes false claims.

Here is a glaring example of a dirty headlining game: When a headline assured us that a supercomputer can predict revolutions, we paid attention. With a supercomputer that can predict revolutions, we could make safer vacation or business travel plans, or keep an eye on the parts of the world where revolution is going to break out!

But the truth is that a researcher borrowed an existing supercomputer, loaded in a massive number of media reports and ran those reports through a series of suppositions and decision points. The researcher found out that Bin Laden was probably in Pakistan and that revolution was going to happen in Egypt!

Here is the problem: The "predictions" were made after the fact!

When those "predictions" were made, the Egyptian revolution was already over and Bin Laden was dead. In a more alarming development, the lead scientist's published report was removed from the web and replaced weeks later, before many readers could see for themselves what was originally published.

But those facts did not stop the headline opportunists from misleading readers into believing that the experiment actually produced a computer that could predict revolutions.

These misleading and erroneous science and survey headlines come up every day, so it is time to arm ourselves with the power of critical and savvy online reading.

The savvy reader looks for information that is from a credible source, gives some background information, provides a link to the actual study report, and includes interviews with the primary and other scientists who know about the topic.

Sometimes the topic is part of evolving news and the reader must monitor the topic until the facts are corrected and more complete.

The problem with scientific claims is that today's scientists are forced to publish too early. Many scientists feel that the study report must be published and publicized before another scientist can go on the attack, to meet a grant deadline, or to be the first to get credit in the public forum.

This "rush to publish" allows reports to hit the headlines before they can be peer reviewed. Peer review is essential for verifying or debunking scientific claims or conclusions.

The news media creates the problem because commercial media thrives on being the first to get the scoop. An editor will construct the headline that is most likely to get readers to click on the page and bring in that lucrative ad revenue!

The first solution is to find several reputable and mainstream media sources. One good source will include information that another will leave out. Another good source will have an interview with scientists and go into more detail. Other articles will tell if anyone disagrees with the findings. Older articles will provide background information and more facts.

The second solution is to avoid the "news aggregator" trap. Many sites just send out web robots that snag the original article and republish it. The bottom line is that Google and the other search engines should be ashamed of themselves for putting this garbage at the top of the search results.

Find the actual scientific paper. It does not matter if the full study is too advanced for the reader to understand. Many abstracts are written in fairly plain language and the patient reader might be rewarded with a better understanding of the topic.

Another trick is to look for "news" instead of "web" search results. Also, try using "advanced search" to filter out the junk and get current results from specific web domains like .edu or .gov.

Many times, the "science" headlines are opinions, rumors, or interpretations that are written by bloggers, activists, or citizen journalists. This does not mean that the articles should be automatically ignored. Some of those bloggers and citizen journalists are well qualified to write about the issue. But if web results are the only sources for a story, then put the red flags up and read more carefully.

Since many science announcements are made before any peer reviews come out, the best strategy is to set up a Yahoo or Google alert for the topic and wait for more information to come in.

Last year, a "newly discovered life form" was proudly announced by the National Aeronautics and Space Administration (NASA). It turned out that the experiment was fatally flawed and that there was no "new" life form that could survive on a diet of pure arsenic. The exciting "discovery" turned out to be a major embarrassment for America's venerable Space Agency.

In that case, it was necessary to get information from multiple news sources even to find out that anything was wrong, then to find out what was wrong.

In summary, when the actual scientific report is unavailable, the reporter most likely used no other source than a press announcement or a brief interview with the scientist. If there is no peer review and actual report, the best approach is either to do more research or to wait for better information.

Anyone who has learned about "experiments" and social statistics understands that a survey is complicated and is one of the least controlled of the scientific experiments. Survey sample size, experiment design, and final interpretations are of the greatest importance when a misleading headline is involved.

Surveys are incredibly vulnerable to false and misleading headlines. In one case, a misleading series of headlines screamed that between 48 percent of men would dump their girlfriends for gaining weight.

Only 12% of the men would have definitely dumped their girlfriends for gaining too much weight! While 38% considered weight gain to be a serious turnoff, a whopping 41% of men would definitely not dump their girlfriends for gaining weight.

One source article did one thing right and made the actual survey available for the public to review. As a fun exercise, go to the search results above and try to find an article that has a link to the actual survey. (The successful reader will be rewarded with a very interesting and much more complete informal survey of men's attitudes and ideas.)

But the harm of such a misleading and controversial headline is that many people skim the headlines or do not know how to interpret survey result percentages.

The readers carry the terrible idea in their minds for days, but do not have the whole picture. This leads to arguments, emotional upset and hurt feelings. The poor soul who actually got the complete set of facts about men and their weight gaining girlfriends is a lost voice in a very upset and angry crowd.

Survey size is extremely problematic these days. Far too often, a survey organization will make phone calls to a few thousand people, then use the results to announce that hundreds of millions of people feel, think, or intend to behave just like the sample population. It is a pretty powerful expectation to think that a few thousand people can represent hundreds of millions of people, so the sample size becomes highly suspect.

The least controlled form of survey is to interview over the phone or to invite the public to take online surveys, then to trust that the answers are truthful.

The survey organizations tweak the numbers but do not reveal their tweaking methods for allowing a few thousand respondents to speak for several hundred million people. Even if the statistical manipulations are described, few people can understand the process.

The insidious outcome is that these small sample size surveys can lead vastly more people to act or think in the as the survey respondents act or think. This means that surveys can easily become a form of propaganda and mass manipulation.

The first solution is to track down the facts about the organization that commissioned or paid for the survey. If the survey is for political purposes and the results support the political leanings of the customer or the survey organization, send up the red flags.

The next step is to look for plain language explanations of the raw data, the survey design and the survey methods. Most reputable survey agencies will give some information about the methods that were used.

Finally, try find the actual survey questions and response percentages. In many cases, respondents give false or misleading answers or are forced to do so because the survey was designed to limit their options. These days, political and social issues are far too complex for "yes/no" answers, or limited choices between possible answers. Look at "dumbed down" or overly simple surveys with an extremely skeptical eye. If the reader wonders why a certain question was not asked or why more choices were not allowed, then there might have been some manipulation or design error going on.

Compare the interpretations to the facts. Stating that "a majority of", for example, is problematic. A majority of whom? What does this "majority" think about the other questions in the survey? What about questions that were not asked, but should have been asked?

In the men's survey, 38% felt that being overweight is a turnoff because it is a sign of lack of self discipline or self control. How many of the men who would not dump their overweight girlfriends felt that weight gain is a sign of lack of self discipline or self control?

This is just one example of how survey reports can skew or limit facts about people's beliefs, opinions and ideas. The final message becomes corrupted and inaccurate when the interpreters do not have enough facts.

Another political and social red flag is that many people do not put themselves into media imposed categories such as "liberal or conservative" or "Generation X and Generation Y." A well designed survey will only use ages or age groups, or will use carefully defined political groups. The reputable survey agency will not use popular or media imposed labels.

The news media will add the labels and misleading headlines after the fact. If a survey will have a big impact on the reader's business and life, it is important to find the actual survey results and to do more research before buying into the survey's or the news media's conclusions.

Science news buffs, whether they are interested in the latest social surveys or the latest discoveries, are better informed when they go beyond the sensational headlines and light articles. Take the time to read more thoroughly, to read multiple and reputable reports, to follow your own pathways of curiosity, and to read with a patient, critical and skeptical mind.