Due to the substantial traffic volume on Square’s pages, any modifications to them can have a significant impact on downstream metrics and multiple cross-functional teams that depend on these pages for seller acquisition and lead generation. Thus experimentation is one of key focus areas in our team, to provide visibility on the impact of changes to stakeholders, assess risk and enable data driven decision making. Here are some of the key learnings from the dozens of experiments our team has run on the website traffic:

1. Create a metric hierarchy and trade-off matrix to simplify rollout decisions
2. Ensure that the bucketing is done correctly
3. Lower the risk by ramping up A/B test traffic in phases
4. Learn the best ways to collaborate with internal teams and stakeholders
5. Invest in experiment automation
6. Document best practices and share learnings
7. Don’t take A/B test results personally

Read more: https://developer.squareup.com/blog/lessons-learned-from-running-web-experiments/