In this series of three we’ll go into Experiment Driven Development and A/B testing. The opposite of developing features based on anecdotes heard in stories from the CEO’s next-door neighbor, EDD seeks proof and is iterative. EDD can be defined as fact-based development: development based on evidence gathered from the field, not intuition.
In EDD every new feature or process implemented is validated through a formal experiment design process, which looks to test a hypothesis that describes the status quo of the feature. Example hypotheses could range from "Making a button bigger does not impact clicks" all the way to "Making a web app responsive does not increase visitation" compared. In statistical terms, the base statement is referred to as the null hypothesis (H0), the status quo, and then an alternative hypothesis(H1) is proposed. The null hypothesis will usually state that the change introduced by the experiment will not affect the current behavior while the alternative supports that there is in fact a change.
The alternative hypothesis is a prediction of what is expected to happen before running the experiment. It can be a bold statement, not an open question and it should have three parts:
The rationale behind the prediction (because we have seen that...): prior knowledge that has led you to come up with the current hypothesis (from prior observation).
For example, one can define the pair of hypotheses for a new registration form in a website as:
EDD is based on A/B Testing, which is a randomized experiment method to compare two variants of a single variable. In this case, a baseline metric is compared thanks to the definition control (status quo) and treatment (new feature) groups in order to determine if the variation has a significant impact or not. Ideally, most decisions to release a feature would be based on the results given by A/B Tests. At Sama, we want to find viable ideas or fail fast. Instead of developing a monolithic solution and pushing a release, we iterate through experiments, evaluating how features perform and, most importantly, if and how customers use them.
Next up: A/B Testing and A/B Testing with Python.
Rafael and Juan are both senior members of Sama R&D where they focus on Data Engineering and Data Science initiatives that help boost operational efficiency and achieve the highest quality.