Scientific experiments for businesses

Estimated time to read: 8 minutes

Published on December 20, 2017 and updated on November 05, 2021.

Experiments aren’t done simply by setting up tools, creating hypotheses and performing tests. It is necessary to “lay the terrain” before starting even the first experiment, which is done by creating an experimentation culture that spread across the entire organization. Without the required preparation, experiments can result in findings that do not reflect reality and may lead to bad decisions.

Experimentation culture

Culture is materializing the set of values and beliefs that a company have on everything that it does. In the context of experiments, culture is the adoption of the scientific method, organized set of steps to obtain knowledge. This concept is important for the following reasons:

Experiments that run without scientific precision have increased chance to be affected by random noise;
There is not a single expected outcome for an experiment;
Findings may not be properly handled, registered or spread across the organization.

In a nutshell, experiments lose their reason to be.

The so called scientific method is a method of learning that is based on following a set of process with a specific goal. Methodical learning have known context, is traceable and processes can be repeated with high precision. Performing experiments under the scientific method minimize risks and maximize returns, even if the end results are not the ones favorable for the business. When this is the case, learn that a given treatment doesn’t yield gains means that the organization didn’t make the wrong decision to bring changes to the business that could end up in losses of any kind.

Adopting the culture of experimentation within organizations

Organizations that are driven by tests and experiments are comprised of people that base their decisions on verifiable learning. Nothing is done before validation under processes that generate results capable of supporting decision making. On the other hand, organizations that were and are still successful making decisions with few or no data may not see why they should embrace the scientific approach for testing, since the business has been running like this for years. Both types of organizations base their decisions on data, the difference being is the amount of weight that is given to a specific kind of data: personal experience from decision makers. Take the following image, extracted from Marketoonist:

Decision being made based on what the person with the highest job title believe is correct. Source: Marketoonist.

The HiPPO, or Highest Paid Person’s Opinion, is the opinion of the one with the highest authority inside the organization. It can be the CEO, a customer or the sponsor of the project. Since in many occasions this person possibly have a lot of experience, it is common that this person trust her instincts and beliefs about the correct way to go. Then comes the question: if traditional businesses still work after so many years with decisions being made the same way, why then convince them that the scientific approach to experimentation is more effective than trusting on experience alone? Two reasons are that decisions have higher assertiveness and return on investment may be higher. It is about investing on things that are desired and that bring healthy returns for the organization.

Experiments start under the premise that people don’t know what will work for a business. The advent of the internet and how it shaped the way societies function brought new ways to live and consume. Past experiences cannot fully explain what consumers from 21st century want. It is necessary to know this customer in order to satisfy it. We can interview people and register their customer journeys, but this knowledge will not be enough to tell if the would buy a given product or service. Selling is what will tell, and this is experimenting. Adopting a culture of experimentation across the entire organization start by understanding the importance of experimenting in the first place. It is also important to know how to correctly perform experiments.

Adopting the scientific method for experimentation

Experimenting by itself is not a guaranteed path to success. It isn’t any better than to randomly make decisions or trust solely on personal experiences. Experimentation must be based on an experimentation model that provide well defined processes, with expected inputs and outputs. Enters the science behind the experimentation. This method of learning is borrow from academic, financial and medical areas, used by scientists and statistical mathematicians. What changes between the method adopted on these places and business environments is that the precision is lower. This happens for at least two reasons:

Information should be quickly available;
It is impossible to fully sanitize the laboratory in which experiments will happen (digital platforms).

There is an exchange between data collection and quality. Think that testing in the field of medicine, for instance, is about dealing with lives. In this context, a single error could result in death, and this is why data quality weights more than the time that is required to obtain it. This exchange makes room for errors at the results of experiments, even if all procedures have been done correctly. A scientific method followed with precision will allow for experiments to be redone under the same conditions that produced the data from previous experiments, which in turn will allow for comparisons between experiments. Since there will always be an error margin with each study, it is important to be able to replicate it in order to eliminate any doubts.

A well followed scientific method will have at least the following steps:

Define hypotheses that determine both an independent variable and a dependent variable;
Define the size of the study group and its divisions;
Setup the experiment and define data quality standards (or statistical rigor);
Execute and monitor the experiment;
Perform exploratory analysis on results;
Document and spread what was learned.

Dependent and independent variables are related to the idea of cause and effect. A variable called independent, or cause, is a observed phenomena, like the yearly divorce rate in the state of Maine or yearly per capita consumption of margarine, for instance. The dependent variable, or effect quantify a cause. Following previous examples, it is the amount of divorces in Maine or the amount of margarine eaten. Experimenting is about adjusting effects to study its impact on the cause. Isolate variables is important to ensure that cause and effect are correlated, to then understand if there is a causality relationship between them.

Correlation is not causation

Performing an experiment is only part of the methodical learning process. Learning with experiments is not done only when all tests are finished, but it is also done when studying the reasons behind results. Possible opportunities may be lost and wrong interpretations may be done if experiments are finished and the resulting data is not further analyzed. One of the problems with finishing experiments without further analyzing its results it accepting a possible causality relationship that may not exist. To illustrate the problem, consider the chart produced by technology consultant Tyler Vigen:

Divorce rate in Maine correlates with per capita margarine consumption. Source: Tyler Vigen.

The previous examples were not chosen by accident: when plotted on the same chart, the effects that relate both causes correlate. When reading the chart, one might conclude that margarine is bad for marriages. Although they correlate, these events just coincide, with no causality relationship. Understand causality is important to know if observed differences within an experiment are due to induced treatments or some other cause. A common example on spurious correlations is about shark attacks and ice cream consumption. Eating more ice cream correlates with increasing shark attacks. Although may be easy to conclude that ice cream are evil, a more plausible explanation could be that these variables correlate due to the weather: people eat more ice cream on summer days and people go to the beach more often on summer days, increasing the probability of being attacked by sharks. Causality in this case is given by the weather. A variable that can link two other variables is known as a proxy.

Learning management and spread

Experiments are about learning. Learning fast and cheap. Experiments produce information that can be used to drive decisions on large scale. What makes that information valuable is its trustworthiness: not only the information is representative of whole populations, when correctly produced, but it is possible to know its level of certainty. By the scientific method approach, knowledge is spread and stored on a common knowledge database, as well as how it was obtained. If knowledge is lost or restricted, knowledge about the business and its customers is lost or restricted. Some known issues with this is to spend more time to acquire the same knowledge or lose potential business opportunities. Document and spread knowledge democratizes it and stimulate people to build solutions for the problems faced in the organization.

In the end, science and business hang out together

Experiments without a higher purpose and with no scientific approach are up for failure and aren’t better than guessing or luck. For them to work, experiments must be embraced by every person in the organization, and executed with scientific rigor that justify its results and allow for verification. There are infinite reasons why experiments can fail, even under rigorous execution. The challenge is to mitigate risks associated with experiments and lower error margins to collect trustworthy information on time. Decision making will still be made, possibly by HiPPOs. The difference is that decisions will be better explained, and options with weak sustenance on data will be discarded. It is important to notice that personal experiences and business instincts are valuable information sources and have their own use cases, differentiating the data driven approach from the data oriented one. In the context of experimentation, what happens on practice is that the weight that is given to personal experiences and business instincts is lower than the data collected throughout a study.