Continuous experimentation for software organizations with low control of roadmap and a large distance to users - A case study

Abstract

Continuous experimentation is a recently popular subject in the field of Software Engineering. There are many resources on how to conduct experimentation with techniques such as A/B tests and canary releases, how to assess organizations in how well they use experimentation and what the benefits and challenges are of using more experimentation. However, there is little differentiation in research regarding the two concepts of control of roadmap and distance to users. The first indicating how much control the company has over the product roadmap and the planning/prioritization of product changes, and the second indicating how easy it is to access the users of the products for data-collection purposes. Not all companies have high control of roadmap and a short distance to users, which is something that needs to be addressed when it comes to continuous experimentation. There exists a clear research gap in this area. This exploratory case study aims to be a starting point in filling this research gap. The thesis work is done together with a single case study company, which is a small-scale software consultancy company. Additionally, four other companies are also involved in the process as part of a static validation process. There are four main contributions resulting from this thesis. First, it presents a deterministic way of deciding how much/little control of roadmap and distance to users a company has. Secondly, it shows that there is a relationship between control of roadmap, distance to users and continuous experimentation. Thirdly, it shows how to assess a software company with low control of roadmap and a large distance to users regarding how well they use continuous experimentation. Finally, the thesis identifies what the perceived advantages, disadvantages and blocking issues are for such a company to use more continuous experimentation. The thesis also makes suggestions for future work, such as if the control of roadmap and distance to users are fixed or if they can be changed, as well as understanding if the control of roadmap and distance to users acts as barriers to evolving the use of experimentation.