Black Box: “A complex system or device whose internal workings are hidden or not readily understood.” (From Oxford.)
Dietary intervention black boxes are those in which catering companies, institutional food preparers or even test subjects create meals whose sourcing, preparation, cooking, and serving details are mostly unknown and unguided by investigators.
At best, food preparers in the studies from our literature search were given only vague and general instructions for sourcing and preparation. None of those studies included details necessary for a valid replication study. Those missing details also invalidate conclusions drawn by study investigators.
This gap in study detail ignores the needed scientific scrutiny in which a kitchen must be treated as a laboratory, food ingredients as reagents, and recipes as formula protocols to be faithfully and accurately followed. And capable of being reproduced by other investigators.
For more 0n this, please see Basic scientific lab standards — Best practices for a replicable dietary intervention.
Lessons from the Non-Replicability of Previous Studies
Because the authors of this article conducted a small proof-of-concept investigation focused upon the clinical effects of Bisphenol A and phthalates, the study cohort for examining replicability and methodology focused, on published papers that involved the same or similar chemical contaminants.
The dietary intervention studies in this cohort differed widely in terms of methodology. Some lacked basic information and scientific methods. Significantly, none of them made available any specific details about food sourcing, recipes, food preparation, or facility conditions.
A search of the literature revealed five relevant dietary intervention studies regarding BPA, phthalates, or both.
Three are not relevant because of fundamental method0logical flaws. These flaws included (among other issues):
- test subject diet did not represent the general population,
- a lack of measurement,
- no strict subject/food intake controls,
- lack of scientific supervision,
- self-reported consumption, and
- lack of supplemental details.
Of the three badly flawed studies:
- One involved members of a single family, all of whom already had very low baseline BPA levels (some below LOD) because of an existing habit of avoiding plastics: Hutter, K.P., Kundi, M., Hohenblum, P., Scharf, S., Shelton, J.F., Piegler, K., and Wallner, P. 2016. Life without plastic: A family experiment and biomonitoring study. Environmental Research 150: 639-644.
- A second recruited a cohort of guests at a spiritual retreat: Ji, K., Lim Kho, Y., Park, Y., and Choi, K. 2010. Influence of a five-day vegetarian diet on urinary levels of antibiotics and phthalate metabolites: a pilot study with “Temple Stay” participants. Environmental Research 110(4): 375-382.
- The third was self-administered by high-school students given a list of foods to avoid: Galloway, T.S., Baglin, N., Lee, B.P., Kocur, A.L., Shepherd, M.H., Steele, A.M., BPA Schools Study Consortium, and Harries, L.W. 2018. An engaged research study to assess the effect of a “real-world” dietary intervention on urinary bisphenol A (BPA) levels in teenagers. BMJ Open 8: e018742.
The two remaining structured and better-designed studies treated meals as a “Black Box” because they relied upon outside food preparation operations, and did not make available any specific details about recipes, food preparation, or facility conditions.
- The first broke new ground as the first of its kind with solid scientific underpinnings: Rudel, R.A., Gray, J.M., Engel, C.L., Rawsthorne, T.W., Dodson, R.E., Ackerman, J.M., Rizzo, J., Nudelman, J.L., and Brody, J.G. 2011. Food packaging and Bisphenol A and Bis(2-Ethyhexyl) Phthalate exposure: Findings from a dietary intervention. Environmental Health Perspectives 119(7): 914-920.
- The second attempted to replicate the Rudel study. It presented conflicting results and found that some of the participants had higher levels of BPA and phthalates after the intervention: Sathyanarayana, S., Alcedo, G., Saelens, B.E., Zhou, C., Dills, R.L., Yu, J., and Lanphear, B. 2013. Unexpected results in a randomized dietary trial to reduce phthalate and bisphenol exposures. Journal of Exposure Science & Environmental Epidemiology 23: 378-384.
According to Sathyanarayana et al. (2013, above), the investigators “tested food ingredients from the same suppliers but were unable to test the exact same food ingredients used in the intervention because testing was performed 3 weeks after the study was completed.”
This illustrates the need to source foods that are readily and constantly available.
In the absence of data, the investigators reasoned that the unexpected increases came from far more than food contact material and reasoned that “the food supply is systematically contaminated….”
Finally, they concluded “Federal or industry-wide regulation aimed at reducing phthalate and BPA concentrations in foods may be the only effective mechanism to ensure the food supply is safe from contamination.”
Therefore, the lessons of Sathyanarayana et al. (2013) comprise the first of several novel recommendations/innovations to improve replicability for food intervention studies.
Sathyanarayana et al. (2013) had worked “to develop a controlled dietary replacement focused on fresh, organic, and local foods without the use of plastics:
- “We conclude that currently accepted methods to reduce phthalate and BPA exposures (both dietary replacement and written recommendations) may not lead to anticipated changes in urinary phthalate and BPA concentrations.”
- “Our study team undertook several measures to ensure that our dietary replacement would consist of fresh, local, organic food prepared, stored, and transported without plastics. For example, the caterer called local farms and asked that fresh foods be delivered in wood crates instead of plastic cartons.
- “All dairy was delivered in glass (milk/cream) or paper except for one delivery of butter in plastic. In the kitchen, the cooks prepared dishes without the use of plastic utensils, appliances, or storage containers.
- “Families were instructed to eat using ceramic dishes and metal utensils. We provided glass containers for food storage and transport. Despite these measures, DEHP metabolite concentrations increased significantly ….”
The application of a set of best practices for a replicable dietary intervention that incorporate standard bench science would improve replicability and reduce confounding factors. However, while incorporating those types of standards will increase replicability, the conclusions of such studies will be complicated by the ubiquitous — and unknowable — contamination of the food chain by plastic and other environmental chemicals.
In addition to the black box lack of precise controls on the meals, and need to drastically reduce chemical contamination in easily sourced foods, investigators must work to control non-food exposures that can obscure a study’s monitored outcomes.
Banishing those complications and working toward to credible assessments of causality will most likely require a dormitory environment, and the need for a more structured design involving precise dosing of the relevant substance.
Notably, a 2019, well-monitored NIH-funded dietary intervention on Ultra-Processed Foods that :
- employed a rigorously selected cohort,
- carefully monitored them for a month in a dormitory environment made and
- precisely measured the subjects’ metabolic and physiological conditions
still fell victim to irreplicability and flawed causality conclusions caused by:
- Black Box Syndrome, and
- Failure to recognize and account for co-confounding factors and micro-nutrients.