PLEASE NOTE: A more technical version of this article, intended for professionals in the field, can be found at this link.
In addition, these are both works in progress with additional text, videos and images to be added as appropriate.
From The New York Times
Is that sensational scientific study really so important that you have to change your life? Nope. More than likely, it’s the science that sucks.
Science can be confusing.
Sometimes for scientists, and especially for individuals in the general public who are trying to make informed and valid personal health decisions.
That confusion is further complicated by the proliferation of company and product-oriented websites whose purpose is to maximize the sales of their products. Paid internet influencers and PR folk just add to the problem. Those are significant reasons that reports of scientific studies are frequently inaccurate or sensationalized, uneven, many times inaccurate, or make sensational claims in pursuit of maximum clicks, readership and viewers.
To make matters worse, the journalist tasked with writing a study may not understand it any better than the average person.
This famous scientist has an opinion about that…
And, this doctor/scientist piles on…
You, however, have access to other methods to separate scientific hamburger from prime rib. Which is the point of this post.
Those issues have been addressed by a number of efforts to help improve the level of reporting, such as this one from The Science Media Center in the UK.
While those links are aimed at journalists, and may work to some extent, your better route is to familiarize your knowledge of what makes good science so you can separate the good, from the bad and the ugly.
Start by Asking Whether the Study Conclusions are Causal or Just links and Associations.
This is not an easy task because there are a lot of different study types. It’s important to realize that most studies are not causal. What does that mean? Causal means there is a definite, identifiable conclusion that “A causes B”.
A link, correlation, or association concludes that there may be some sort of relationship between the studied chemical, drug or lifestyle practice and some sort of outcome, but which is not causal because of known or unknown confounding factors or complications.
The Difference Between Causal and Associations: How Ice Cream Kills
Scientific studies can be categorized as either demonstrating causality or simply showing some sort of connection, link or association. The difference is based on the study design and methodology, and the strength of their evidence.
Regardless of the type of study, informed consent must be obtained from all participants. The study must adhere to ethical standards, ensuring that the benefits outweigh the risks and that participants’ rights and well-being are protected.
Causal Studies:
Randomized controlled trials (RCTs) are the most causal of all study types, because they are designed to establish a cause-and-effect relationship between a specific treatment, chemical or other substance and a specific outcome.
Click image to start video which will begin with a different title page.
RCT protocols are developed to control for confounding variables and biases. As a result, RCTs have the strongest Strength of Evidence of all study types and provide strongest evidence for a cause-and-effect relationship.
Randomization means that test subjects/participants are randomly assigned to either the treatment group or the control group.
This helps ensure that the test and control groups are as similar to each other as possible. That minimizes selection bias, and tries to balance both known and unknown confounding factors across the groups.
Control Group: The control group serves as a benchmark. It may receive a placebo (an inactive substance) or a standard treatment. This allows for a comparison with the group receiving the experimental treatment.
Blinding: Good Random Controlled Trials (RCTs) are blinded. The best are are “double-blind”, meaning neither the participants nor the researchers know who is receiving the treatment and who is receiving the placebo. The video, below, offers a view at the difference.
is this video better or worse?
This minimizes bias in interactions with test subjects, and the administration of treatment and assessment of outcomes.
Clear Definition of Variables: The primary outcome to be measured is clearly defined, as are the treatments/interventions and the population being studied. This includes precise dosages, duration of treatment, and specific criteria for participant selection.
Follow-up: When practical, participants are followed over a period of time to allow the treatment to show effects.
Statistical Analysis: The data from a trial are analyzed using statistical methods to determine whether there are significant differences in outcomes between the treatment and control groups.
Replicability: The methodology of the study must be clear, and the conduct precisely followed, to allow for replication, which is essential for verifying results.
Registration and Protocol: In the United States, high-quality RCTs should be registered with ClinicalTrials.Gov, or, if in other countries, in a public trials registry before the commencement of the study.
A Closer Look At Confounding Factors
Take, for example, an observational study finding that people who exercise regularly have lower rates of heart disease. This shows an association but does not prove that the exercise directly causes the heart health improvement.
For example, people who run may have higher income, be better educated, live in areas with better air and water, have better medical care, more time to run, and neighbors who also run.
Many Factors Exist To Confound Observational Studies
In addition to the simple sample above, below is a list of common types of confounding factors that could lead to erroneous conclusions.
Lifestyle Factors: These include habits or behaviors like diet, exercise, smoking, and alcohol consumption. For instance, in a study examining the relationship between a dietary supplement and heart health, a person’s overall diet and exercise regimen could be confounding factors.
Socioeconomic Status: Factors such as income, education level, and occupation can impact health outcomes, and may confound studies examining medical or social interventions.
Genetic Predispositions: In health studies, genetic backgrounds can be a significant confounder, as some populations may have a genetic predisposition to certain conditions or responses to treatments.
Environmental Exposures: Exposure to various environmental factors, like pollution, climate, or geographic location, can impact health and other outcomes, potentially confounding studies.
Age and Gender: These are common confounders in many types of research. Age and gender can influence susceptibility to diseases, response to treatments, and behaviors.
Psychological Factors: Mental health status, stress levels, and other psychological factors can influence various health outcomes, and may confound certain studies.
Cultural Factors: Cultural practices and beliefs can influence behavior and health outcomes, thus potentially confounding studies.
Access to Healthcare: Variability in access to and quality of healthcare can influence health outcomes, and may confound studies examining medical treatments or health interventions.
Pre-existing Conditions: In health studies, pre-existing medical conditions can influence the outcomes independently of the variables being studied.
Biases in Data Collection: This includes measurement bias, selection bias, or recall bias, where the way data are collected or reported can introduce confounding.
Researchers often use statistical methods, such as multivariable regression analysis, or design strategies, like matching or stratification, to minimize the impact of confounders on their study outcomes. Regardless of those efforts, none of them rise to a causal strength of evidence.
Study Subjects Other Than Humans
Photo: Shutterstock
A large number of studies are conducted in species other than humans — rats, mice, zebra fish, to name a few common ones. These may be suggestive of a possible effect in humans, but are a far leap from projecting a given outcome in people, given that 92% of drugs deemed safe and effective in animals fail when tested in humans.
Published and Peer-Reviewed
Finally, the credibility of a study is affected by whether it has been published and peer-reviewed.
Strength of Evidence
To make good decisions about whether the results of a study should influence your personal health decisions, it’s helpful to try and figure out a study’s strength of evidence.
While consideration should eventually be given to the study authors’ reputations and their institutions, and experience, the quality of a study can be first assessed on its type. Fundamental to this is the quality of the data, and how that data was generated.
The measure of this is called the Strength of Evidence, which was first developed to assist physicians and other health care personnel in developing the best treatment for a given ailment. In fact, the book How to Read a Paper: The Basics of Evidence-Based Medicine by Trisha Greenhalgh has long been the fundamental work on assessing the applied strength of evidence for the purpose of developing clinical treatments for patients.
While Greenhalgh’s book has been required reading for decades of medical school students, its principles for assessing the strength of evidence apply to individual choices in personal health care. Below is our effort to guide individuals in their critical assessment of personal health care , including supplements, non-prescription options and treatments. This pyramid or hierarchy is not a prescription, but simply a guide to understanding an establish way to sort through the confusing multitude of study types
It is always best to consult your healthcare physician before embarking on personal healthcare decisions
Graphical Exampls of Assessing the Strength of Evidence: The Pyramid
The3se are some of the very best of the many efforts that have been made to use pyramids to represent levels of scientific evidence among the various study designs. It’s important to realize that pyramids are always oversimplified, but convey the overall concept.
Below, read on to see our effort to be complete at ranking the strength of evidence of a relatively complete survey of study types and explaining what they are.
To Dig Deeper into evidence-based strengths and how to determine if a study is a jewel or junk, please see: Not All Science Is Equal. Here’s How To Assess The Value Of A Study .
Many thanks to David Morrison for his invaluable judgment and editing assistance with this post.