Online experimentation platforms abstract away many of the details of experimental design, and their users do not have to worry about sampling, randomisation, subject tracking, data collection, metric definition and interpretation of results. The rapid adoption of these platforms in the industry might in part be attributed to the ease-of-use these abstractions provide. However, there are common pitfalls to avoid when running controlled experiments on the web, and one needs experts familiar with the entire software stack to be involved in the process.

Read more: https://booking.ai/leaky-abstractions-in-online-experimentation-platforms-ae4cf05013f9