Unbreaking Science

Rarely, if ever, do scientists train outside of their field of specialization. In fact, that is part of why science seems so stagnant in many areas of research. The pressures for scientists to "distinguish themselves" and also to conform to a set of norms in terms of shared and agreed upon consensus of what constitutes background knowledge means that many PhDs amount to adding leaves to the tips of branches of the tree of knowledge.

I have always felt that if we are to learn anything profound, we need to grab the tree of knowledge and shake it by its roots.

Because creativity plays a role in imagining the possible, chance and happenstance have an unpredictable effect on progress in understanding in any discipline that uses the scientific method. What if, for example, Darwin fell ill and had been unable to make his voyage to the Galapagos on the HMS Beagle? What if Stephen Hawking had died when he caught pneumonia at the age of 14? What if Einstein had decided that running a bakery was more to his liking? And even if these events had not taken place, there are millions of other possibilities that could have happened that could have changed key cerebral events in the lives of scientists whose contributions might have been.... different.

It turns out that the history of science is obviously dependent on chance discoveries, eureka moments (imagine if Kary Mullis never tested LCD - no PCR, no human genome project - no microarrays for functional genomics - no real tsunami of data that demanded bioinformatics). This not only makes the particular set of discoveries in Science available in a given year, or decade arbitrary - it also makes the cognitive dialog - and therefore the exact nature of understanding of nature and the world around us - entry-order sensitive. Good trial lawyers are keenly aware that jurists' minds and decisions are susceptible not only to the evidence they see - but also very plastic to the order in which evidence is presented to them. It takes a certain feature of cognition - meta-cognition - to rise above the problem, see oneself as a participant in a grand experiment of understanding - to begin to understand how to transcend that foibles of human illogic. A regular divorce of one's ego from one's hypothesis is helpful, too.

In the inaugural edition of the Unbreaking Science podcast, I introduce the idea that there appears to be a general Ontogeny of Science. So far, these seem to be capured by Five Ontological Stages of Science. It is no doubt, and hopefully so, incomplete. But I stand by it as a useful guide, and optimistic prescription, for precisely how Science can emerge from the current epistemological crisis (lack of replicability, publication bias and the resulting abundance of double false positive 'validations') to create a superbly objective and honest playing field - a proving ground - upon which we can test our knowledge claims. So here is a first draft of the Five Ontological Stages of Science, which contains the general Ontogeny, which, as developmental biology has taught us, not only fits each and every field of inquiry that uses the scientific method as a way of knowing about ourselves, our world, and the Universe around us - but also very likely recapitulates the actual progress of Science writ large. It is highly meta-cognitive, not prescriptive, and reflective (descriptive). In fact, because chance plays and role in bringing entropy to science, and reason (hopefully) works to offset entropy in science, any individual discovery process can visit these Stages of Ontology in any order. It would be an interesting and perhaps important exercise to use AI to go through scientific discovery to comprehensively and openly pass through these phases, or to plan and fund future areas of inquiry according to these phases as a plan, but that is not my specific intent.

If you must, consider it speculative, but I will offer that the Stages come from a lifetime of experience of endeavor in many fields of inquiry. I estimate its subjectivity to be less than 10%, but it is, hopefully, worthy of fruitful discussion.

Stage 1. Taking Stock: The "Whats"

In this stage, we are concerned with cataloging what is present. Which species are present in the world? What are the parts of the human brain? Who are the members of a terrorist organization? What are the fundamental particles of matter? This first stage involves data collection to survey/canvass/inventory the system we seek to understand. Here true positives mean confirmation by assessment, and quantification is mostly limited to counting, measuring. Comparisons may be made to test if there are significantly more "X" in A than in B, but processes, functionality and mechanisms are not approached. Think systematic surveys, case studies and case series in clinical research.

Stage 2. The Relationships

In Stage 2, possible relationships among the Whats are explored. These, like the Whats, are inventoried. We do not necessarily understand the full nature of those relationships, in terms of processes, functions, and mechanisms, but we mark them down as existing, and interesting (worthy of further exploration). Which species interact? Which parts of the brain communicate and share information? Who in the terrorist organization communicates with whom? Which fundamental particles of matter interact? At this stage, hypotheses of “Hows” and Whys can begin to be formulated, but they cannot be tested with observational data. Criteria like "Consistency" and "Best Explanation of the Evidence" put a domain of scientific inquiry into a confirmationist mode and the Domain must move beyond those inductive types of reasoning to perform critical tests of hypotheses to move beyond Stage 2. Think case series and observational studies, and occasional case studies that serve as counterexamples to the norm.

Stage 3. The “Hows”

In Stage 3, we begin to assess the nature of the relationships among the entities. We begin to test hypotheses to distinguish direct vs indirect relationships among the variables or entities we are studying; between correlational and causal relations among the variables or entities being studied. Which species are competing with others for resources? Precisely how do various parts of the brain interact? Who play what roles in the terrorist organization? How do the fundamental particles of matter interact? These are fundamentally different types of hypotheses than are considered in Stage 2. We begin to ask Why questions - but classically, most scientists find it difficult to state accurate at this stage whether they are asking "How" questions and overtly causal "Why" questions.

Many factors can cause investigators to miss relationships that actually exist; they include low statistical power, and publication bias. Confusion exists over the extent to which evidence can tests hypotheses of causality at this level; specifically, observational studies cannot be used to reliably test hypotheses of causality. It is an unfortunate that most of epidemiology falls short of actually testing hypotheses of causality, especially studies that only ever measure the significance of statistical association. Bradford Hill criteria of causality are invoked on causality in studies that employs epidemiological approaches, but they are wanting for any empirical demonstration of reliable utility and may suffer from underdetermination.

Stage 4. The "Why's"

In Stage 4, Hypotheses of Causality and Direct and Indirect relationships of causality are being actively - and objectively - and specifically - sought and tested. There are a number of interesting ways to do this - and note thus far the Stages are described independent of Levels of Evidence. This Stage is where underpowered studies - studies that are too small to reliably test the hypothesis of interest - can damage a field of inquiry. For mechanisms of pathophysiology, and mechanisms of drug action, animal studies and cell line studies may be performed after, or alongside clinical trials. It is still difficult many times to distinguish processes and "reasons", in large part because there is interplay of causality at different scales, and Science is expected to be a tool useful for studying the world and the Universe around us at every scale - simultaneously. Models therefore fit nicely as tools In the move from and between Stages 3 and 4. For example, "when we pass electricity through it, photons are emitted" is a Stage 1 observation at the meta-scale, a Stage 2 observation is we are testing a How question about the material, or electricity, and it is ripe for Stage 2, 3 and 4 questions at the atomic and subatomic level if we are focused on fundamental questions of processes.

Similarly, "this chemotherapy agent reduces the tumor size in xx% of patients" is a finding that does not tell us how, but it can lead to How and Why questions. The overall effect of patient survivorship is a question that exists at the population level might point to benefits and risks (chemotherapy-associated mortality) of the treatment... but none of those studies specific can test Why questions on the mechanisms of action at the cellular level or at the level of systems biology.

How and Why questions often involve tighter experimental designs than the Observational Studies use at the initial What and Relationship questions.

The Stage at which an inquiry is being conducted can change based on what the investigator knows (their background knowledge), on the experimental unit or unit of observation (drug molecule, tumor cell, individual patient, patient population). Confusion about which Stage of Inquiry one is conducting an Inquiry is a source of entropy and can lead to public health decisions that can have disastrous consequences.

Stage 5. Prediction Science

It is not until we reach Stage 5 that we begin to realize that we have secured reliable knowledge. In Stage 5, we understand whatever system we are studying, whether that be a cell, an ecological community, a society, or the Universe itself - when we can predict with the effects of our perturbations of that systems with very high accuracy. For domains of scientific inquiry that are struggling with a replication crisis, my advice is to hire Machine Learning experts to train Domain experts in Machine Learning, and to plan to move away from mere significance testing. Significance testing in Machine Learning is just another example of Feature Selection - and realize that in small data sets, significance testing can rule out predictive features of your system too early simply due to high measurement variability. The focus at Stage 5 is not mere significance, but rather, generalizability. Can we predict who will develop ADHD, or cancer? Oddly, when a domain of Science reaches Prediction Science, the stunning accuracy of prediction models are often false positive because even experts can forget to focus on generalizability - i.e., testing the performance of the prediction models on new data, not used in the generation of the model parameters or model structures. It does no good to study weather patterns and create new weather prediction models to predict yesterday's weather. If one wants their domain of Scientific Inquiry to advance beyond the conundrum of Broken Science, study and learn the utility of performance evaluation measures and tools like ROC curves, accuracy, sensitivity, specificity, positive predictive value and generalizability - with the dual goals of determining if and demonstrating that the knowledge represented by the prediction model is, in fact, useful to society.

You can support Unbreaking Science at Patreon!

The five ontological stages of science