
 
Surview 
 
 
Improving the survey industry by using chatbot technology 

 
Bachelor Thesis in Industrial Engineering and Management 
 

Filip Hallqvist 
David Helldén 
Mats Högberg 
Emil Lundgren 
Henrik Nilson 
Johanna Sigvardsson 
 
 
Department of  Technology Management and Economic  
Division of Entrepreneurship and Strategy 
CHALMERS UNIVERSITY OF TECHNOLOGY 
Gothenburg, Sweden 2017 
Bachelor Thesis TEKX04-17-02 


Preface

This Bachelor’s thesis was written during the spring of 2017 at the Department of Technology Man-
agement and Economics Organisation at Chalmers University of Technology.

Many thanks go out to all the companies that were involved in the process of developing Surview,
either through being interviewed or agreeing to conducting tests. We would also like to thank our
supervisors Yashar Mansoori and Mats Lundqvist for helping us adapting the work of the project to
fit the style of an academic thesis, and Carl Josefsson at Chalmers Ventures for his support on the
business side of things during the entire project.

Finally, we would like to thank our friends and family for always supporting us.

Filip Hallqvist David Helldén

Mats Högberg Emil Lundgren

Henrik Nilson Johanna Sigvardsson

Chalmers University of Technology
Gothenburg
2017-05-12


Sammandrag

Syftet med detta kandidatarbete är att undersöka om aktuella problem inom undersökningsbranschen
kan lösas med hjälp av chatbot-teknologi, och huruvida en h̊allbar a↵ärsverksamhet kan byggas kring
detta koncept genom att applicera Lean Startup-metodologin. Teorin bakom projektet är uppdelad i
fem delar: Svarsfrekvenser för undersökningar, Net Promoter Score, Chatbot-teknologi, Att använda
SMS som ett medium för undersökningar, och H̊allbar utveckling. Metodologin som används i pro-
jektet best̊ar av Lean Startup-metodologin, Business Model Generation, Intervjuer och A/B-testning.
Projektets resultat var att det finns ett värde i att använda chatbot-teknologi för att genomföra un-
dersökningar tillsammans med en lämplig undersökningsmetodologi. Resultatet visade ocks̊a att ett
möjligt kundsegment, för ett verktyg som till̊ater användaren att genomföra chatbot-undersökningar, är
undersökningskonsulter och webbshoppar. För att verifiera detta behövs dock ytterligare efterforskn-
ing genomföras. Projektet visar att vidare undersökningar kring en möjlig a↵ärsmodell, tillsammans
med utformningen av en produkt, är nödvändig för att validera att en framg̊angsrik a↵ärsverksamhet
kan byggas runt idén att använda chatbot-teknologi för att utföra undersökningar.

Nyckelord: Undersökningar, kund̊aterkoppling, marknadsundersökningar, chatbot-teknologi, Lean Startup,
Business Model Generation, SMS.


Abstract

The purpose of this thesis is to research if existing problems within the survey industry can be solved by
applying chatbot technology, and if a sustainable business can be formed around this idea by applying
the Lean Startup Methodology. The theoretical framework of the project is divided into five areas:
Response rates in surveys, Net Promoter Score, Chatbot technology, SMS as a medium for surveys, and
Sustainability. The methodologies applied in the project are the Lean Startup Methodology, Business
Model Generation, Interviews and A/B Testing. The result of the project was that there lies value in
applying chatbot technology when performing surveys with an appropriate surveying methodology. A
potential customer segment for a chatbot surveying tool is survey consultants and webshops. However,
in order to validate this, further research needs to be conducted. The implications of this project is
that further research on a potential business model, along with packaging of a product, is required in
order to validate if a successful business can be built around the idea of using chatbots to conduct
surveys.

Keywords: Surveys, customer feedback, market research, chatbot technology, Lean Startup, Business
Model Generation, SMS.


Contents

Glossary

List of Figures

List of Tables

1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Theoretical framework 3
2.1 Response rates in surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Techniques to increase response rates . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Net Promoter Score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Chatbot technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.3.1 Intelligent agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.2 Natural language processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4 SMS as a medium for surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4.1 Previous work on SMS surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4.2 How to send SMS programmatically . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.5 Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5.1 Economic Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5.2 Environmental Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5.3 Social Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Methodology 11
3.1 The Lean Startup Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Business Model Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2.1 Business Model Canvas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.2 Value Proposition Canvas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 A/B Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Process 17
4.1 Previous work on Surview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Coming up with a potential business model . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.2.1 Designing a Value Proposition Canvas . . . . . . . . . . . . . . . . . . . . . . . . 21


4.2.2 Designing a Business Model Canvas . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.3 Initial hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.3 Talking with survey consultants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.1 Setting up Learning and Measurement . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.2 Building: Formulating questions . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.3 Measuring and Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3.4 Updated Business Model Canvas and Value Proposition Canvas . . . . . . . . . . 26

4.4 Testing the existing product with survey consultants . . . . . . . . . . . . . . . . . . . . 27
4.4.1 Setting up Learning and Measurement . . . . . . . . . . . . . . . . . . . . . . . . 28
4.4.2 Building: Adding features to make testing easier . . . . . . . . . . . . . . . . . . 29
4.4.3 Test with random people . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.4.3.1 Measuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4.3.2 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.4.4 Test with Company Z . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4.4.1 Measuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4.4.2 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.4.5 Test with Company T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.4.5.1 Measuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4.5.2 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.4.6 Test with Company N . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4.6.1 Measuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.4.6.2 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.4.7 Summary of performed tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.5 Customer segment pivot: Talking to webshops . . . . . . . . . . . . . . . . . . . . . . . . 37

4.5.1 Setting up Learning and Measurement . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5.2 Measuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.5.3 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.6 Expanding customer segment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.7 Summary of process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5 Discussion 41
5.1 Using chatbot technology for surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 The Lean Startup Methodology and Value Proposition and Business Model Canvases . . 42
5.3 Test Card and Learning Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.4 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.5 A/B testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6 Conclusion 45

7 Bibliography 46

Appendix A The product 49

Appendix B Test Cards 58

Appendix C Learning Cards 63

Appendix D Surveys and questionnaires 67


Glossary

Startup A Startup is a human institution designed to create a new product or service under conditions
of extreme uncertainty (Ries, 2011).

MVP (Minimum Viable Product) A product with just enough features to test an hypothesis or
to be released to customers.

Product/market fit The degree to which a product satisfies a strong market demand. Prod-
uct/market fit means being in a good market with a product that can satisfy that market (Andreesen,
2007).

Pivot Changing parts of a business plan while keeping other parts unchanged. A pivot can for example
be a change in the product or a change of customer segment.

Corpora Plural of corpus, which is a large and structured set of texts.

Parsing The process of analyzing a string of symbols in natural language.

Stemming The process of reducing inflected or derived words to their base word form.

Session An interaction with one of the respondents through a survey.

Engagement rate The amount of respondents, in percent, that answered any of the questions in a
survey.

Conversion rate The amount of respondents, in percent, that completed the survey either by declining
to participate or completing the entire survey.

Completion rate The amount of respondents, in percent, that completed an entire survey.

Semantic level A group of tasks related to the meaning of words in their respective context.

Syntax level A group of tasks pertaining to the rules of language in terms of sentence construction,
use of prefixes and su�xes, and punctuation.

Financial capital Any economic resource measured in terms of money.

Human capital Fundamental features of humans such as knowledge, intelligence and wisdom which
can produce economic value to a nation or a company.

Societal capital An economic idea that focus on that the social networks between people in a group
can be economically valuable.

Tangible capital All types of physical assets, for example buildings.

Intangible capital All types of assets that are not physical, for example intellectual assets.

API (Application Programming Interface) Code that allows software programs to communicate
with each other.


HTTP (Hypertext Transfer Protocol) The application protocol used for theWorldWideWeb.


List of Figures

2.1 The evolution of the dropout rate for the Labor Force survey between 1970 and 1995
(Japek et al., 2001). The thick red line depicts the total dropout. The smaller red line
represents respondents who were unavailable, and the black line represents respondents
who refused to answer. The line at the bottom of the figure represents respondents who
dropped out due to other reasons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 The NPS scale (Satmetrix Systems, Inc, 2017). . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 The general main components of an intelligent agent (Russell and Norvig, 2010). . . . . 6

3.1 The build-measure-learn feedback loop (Ries, 2011). . . . . . . . . . . . . . . . . . . . . 12
3.2 Test Card (Strategyzer, 2017). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3 Learning Card (Strategyzer, 2017). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.4 Business Model Canvas (Strategyzer, 2017). . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.5 Value Proposition Canvas (Strategyzer, 2017). . . . . . . . . . . . . . . . . . . . . . . . . 15

4.1 The landing page built for Gothenburg Startup Hack. . . . . . . . . . . . . . . . . . . . 18
4.2 A survey conducted with the first MVP. . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3 The landing page for the dashboard in the first MVP. . . . . . . . . . . . . . . . . . . . 19
4.4 Flowchart showing an example of how a survey can be built. . . . . . . . . . . . . . . . . 20
4.5 Initial Value Proposition Canvas. Notes in red indicate parts of the canvas that have

not yet been validated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.6 Initial Business Model Canvas. Notes in red indicate parts of the canvas that have not

yet been validated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.7 Updated Business Model Canvas after talking to survey consultants. Notes in red indi-

cate parts of the canvas that have not yet been validated, and notes in green indicate
parts of the canvas that have been validated. . . . . . . . . . . . . . . . . . . . . . . . . 27

4.8 Updated Value Proposition Canvas after talking to survey consultants. Notes in red
indicate parts of the canvas that have not yet been validated, and notes in green indicate
parts of the canvas that have been validated. . . . . . . . . . . . . . . . . . . . . . . . . 28

4.9 The landing page for the dashboard after implementing features for making testing easier. 30
4.10 PDF file sent to Company T. Illustrating how a dashboard view could look, with an

overview of data for an entire company. . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.11 PDF file sent to Company T. Illustrating how a dashboard view could look, with data

for a specific individual. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.12 Value Proposition Canvas for the webshop customer segment. . . . . . . . . . . . . . . . 38

A.1 The landing page of the dashboard built for Gothenburg Startup Hack. . . . . . . . . . 49
A.2 The page for creating a new survey in the dashboard built for Gothenburg Startup Hack. 50
A.3 The page for modifying a survey in the dashboard built for Gothenburg Startup Hack. . 50
A.4 The overview of a survey in the dashboard in the first MVP. . . . . . . . . . . . . . . . . 51
A.5 The options for a survey in the dashboard in the first MVP. . . . . . . . . . . . . . . . . 51


A.6 Creating and managing sessions in the dashboard in the first MVP. . . . . . . . . . . . . 52
A.7 The overview of a survey in the dashboard after implementing features for making

testing easier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
A.8 The interface for adding questions to a survey in the dashboard after implementing

features for making testing easier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
A.9 Viewing received answers in the dashboard after implementing features to make testing

easier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
A.10 An overview of created sessions for a survey in the dashboard after implementing features

to make testing easier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
A.11 The flowchart view of a survey in the dashboard. . . . . . . . . . . . . . . . . . . . . . . 54
A.12 The view for creating new surveys in the dashboard. . . . . . . . . . . . . . . . . . . . . 55
A.13 Dropdown showing all available question types in the dashboard. . . . . . . . . . . . . . 55
A.14 The upper part of the landing page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A.15 The lower part of the landing page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

B.1 Test Card describing testing whether low response rate is a problem survey consultants
want to solve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

B.2 Test Card describing testing whether survey consultants can sell potential gain creators
and pain relievers to their customers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

B.3 Test Card describing testing whether slow feedback is a problem survey consultants
want to solve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

B.4 Test Card describing testing whether reaching certain customers segments is a problem
survey consultants want to solve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

B.5 Test Card describing testing whether survey consultants can and want to use tools
developed externally. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

B.6 Test Card describing testing whether consultans can use a external tool independently. . 60
B.7 Test Card describing the test with random people. . . . . . . . . . . . . . . . . . . . . . 60
B.8 Test Card describing the test with Company T. . . . . . . . . . . . . . . . . . . . . . . . 60
B.9 Test Card describing the test with Company Z. . . . . . . . . . . . . . . . . . . . . . . . 61
B.10 Test Card describing testing whether webshops have problems with, and want to pay

for more, feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
B.11 Test Card describing testing whether webshops want to, and are willing to pay for,

verifying that deliveries are followed up properly. . . . . . . . . . . . . . . . . . . . . . 61
B.12 Test Card describing testing whether webshops want, and are willing to pay for, a more

personal relationship with their customers. . . . . . . . . . . . . . . . . . . . . . . . . . . 61
B.13 Test Card describing testing whether webshops benefit from, and are willing to pay for,

deeper analysis of their customers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

C.1 Learning Card describing learning from testing if low response rates are a problem survey
consultants want to solve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

C.2 Learning Card describing learning from testing if survey consultants can sell the poten-
tial gain creators and pain relievers to their customers. . . . . . . . . . . . . . . . . . . . 64

C.3 Learning Card describing learning from testing if slow feedback is a problem survey
consultants want to solve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

C.4 Learning Card describing learning from testing if reaching certain customer segments is
a problem survey consultants want to solve. . . . . . . . . . . . . . . . . . . . . . . . . . 64

C.5 Learning Card describing learning from testing if survey consultants can, and want to,
use external tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

C.6 Learning Car describing learning from testing if survey consultants can, and want to,
use external tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

C.7 Learning Card describing learning from the test with random people. . . . . . . . . . . . 65
C.8 Learning Card describing learning from the test with Company T. . . . . . . . . . . . . 65


C.9 Learning Card describing learning from the test with Company Z. . . . . . . . . . . . . 66

D.1 Flowchart showing how the survey sent to random people about their Internet habits
was designed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

D.2 Flowchart showing how the survey sent to Z’s survey participants was designed. . . . . . 69
D.3 Flowchart showing how the survey sent to T’s customers technicians was designed. . . . 70
D.4 Flowchart showing how the survey sent to N’s co-workers was designed. . . . . . . . . . 70
D.5 Questionnaire used as a framework when interviewing consultancy companies. . . . . . . 71
D.6 Questionnaire used as a framework when interviewing webshops. . . . . . . . . . . . . . 72


List of Tables

4.1 Table presenting the results from interviews with survey consultants. Green cells indi-
cate that the company validated the hypothesis, red cells indicate that a hypothesis was
invalidated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2 Results from test with random people . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.3 Results from test with Company Z. The engagement rate for the email survey could not

be measured, since the tool used to send out the email survey only saved answers for
respondents that completed the entire survey. . . . . . . . . . . . . . . . . . . . . . . . . 32

4.4 Results from test with Company T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.5 Results from test with Company N . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.6 Table presenting the results from interviews with webshops. Green cells indicate that the

webshop validated the hypothesis, red cells indicate that a hypothesis was invalidated
and yellow cells indicate that more data is needed in order to validate, or invalidate, the
hypothesis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39


Chapter 1

Introduction

1.1 Background

It is important for companies to make sure that their customers are happy. According to Hoyer
and MacInnis, customer satisfaction is the foundation of a successful business as it leads to repeat
purchases, brand loyalty and positive word of mouth (Hoyer and MacInnis, 2001). The same study
also showed that a dissatisfied customer will discontinue purchasing the product or service, complain
to the company, to a third party or engage in negative word of mouth communication. According to
Coldwell, who performed a statistical analysis of customer satisfaction, a totally dissatisfied customer
decreases revenue at a rate equal to 1.8 times what a totally satisfied customer contributes to a business
(Coldwell, 2001). The analysis also concluded that a totally satisfied customer contributes 2.6 times
as much revenue to a company as a somewhat satisfied customer, and 17 times as much as a somewhat
dissatisfied customer. In order to keep customers happy, it is of utmost importance for companies to
stay in touch with their customers, in order to identify and solve problems with their services as fast
as possible.

One way to stay in touch with customers is through surveying, where two common alternatives to-
day are web forms distributed by email (Liu and Inchaust, 2017) and phone interviews. A common
problem for email surveys and phone interviews is low response rates (Bladon, 2010). In a study from
2001, where the results of several other studies on surveys were combined and analyzed, it was found
that response rates for surveys done with email had been steadily decreasing between 1986 and 2000
(Sheehan, 2001). During this time the mean response rate for surveys had decreased from 61.5 percent
to 24 percent. Although the number of companies that used such surveys increased, response rates
still decreased. Because of this the industry seems to be in need of a change.

An interesting area that may have potential to solve the problems in the survey industry is chatbot
technology. Chatbot technology refers to a computer generated response where the purpose is to
simulate human-like conversation either through auditory or textual response (Lokman and Zain,
2010). Focusing on this kind of technology is believed to give a closer relation with the customer which
could generate a more genuine response and a higher customer satisfaction, compared to existing
surveying methods. According to a study from 2015, computer generated conversations are not yet as
good as human-to-human (Hill et al., 2015), but the study still suggests that humans want and can
have extensive conversations with a computer and that it adds a certain factor of satisfaction.

1


1.2 Purpose

The purpose of this thesis is to research if existing problems within the survey industry can be solved
by applying chatbot technology, and if a sustainable business can be formed around this idea. This is
done by applying the Lean Startup Methodology.

1.3 Problem Statement

From the purpose of the thesis the following research questions have been formulated:

• How can chatbot technology be applied to surveys?

• Is there a market need for a new product in the survey industry?

• Does applying chatbot technology to surveys lead to better response rates?

1.4 Scope

The project has a fixed time limit, which influences the chances of drawing feasible answers to the
full set of research questions. This also means that the project may not produce a finished, validated
product at its deadline. As the Lean Startup Methodology is an iterative process, it is important to
realize that the product proposal can change during all of the steps in the process.

1.5 Outline

The thesis is structured with Introduction, Theoretical framework, Methodology, Process, Discussion
and Conclusion. The Introduction presents the main reasons for why this project is relevant and
what it aims to achieve. The Theoretical framework chapter serves the reader the essential theoretical
knowledge for the project and explains the di↵erent concepts used. The Methodology chapter explains
how the project was performed and the necessities in using certain methods. The Process chapter
uses the Methodology chapter as its foundation and explains in a concrete way what was done during
the project. The Process chapter also includes discussions on the di↵erent phases of the project. The
Discussion chapter aims to review the findings from the project and give the reader an insight in areas
still in need of research or improvement.

2


Chapter 2

Theoretical framework

In this chapter, the theoretical framework used throughout the project is explained. In order to answer
the question of whether the customer feedback industry can be improved by using chatbot technology,
it is important to get a clear understanding of the underlying technical concepts and ideas of chatbot
technology. Section 2.1 goes into more depth on the problem of low response rates in surveys today,
and common techniques used to increase response rates. Section 2.2 describes the feedback method
known as Net Promotor Score, commonly used by market researchers to measure customer satisfaction.
Section 2.3 goes into more depth of chatbot technology, how it works, and how it can be applied to
surveys. Section 2.4 discusses the use of SMS as a medium for surveys, and lastly, Section 2.5 describes
di↵erent aspects of sustainability and how sustainability relates to this project.

2.1 Response rates in surveys

Since the early 70’s the dropout rates in surveys have increased significantly according to the Central
Bureau of Statistics in Sweden (Japek et al., 2001). Dropouts refer to all respondents who choose not
to participate in a survey. It can be that respondents drop out on a specific question in the survey, that
respondents are not accessible or that respondents choose not to participate at all, so called refusers.
The main problem with high dropout rates is that if significant groups of people do not respond to
surveys, the results of the surveys are not reliable. There are groups which are more di�cult to reach
than others. A few examples are young people, people living in larger cities, and foreigners (Strandell
and S., 2015). It is believed that using chatbot technology to conduct surveys will reduce dropout
rates, and make it easier to reach groups that are hard to reach with existing surveying methods.

Telephone interviews generally have a higher response rate than paper forms distributed by mail, but
the amount of dropouts through telephone interviews have increased remarkably the last couple of
years (Strandell and S., 2015). One example is the Labour Force survey as seen in Figure 1. The
Labour Force survey is a survey conducted through telephone interviews once a year, and dropout
rates for this survey has risen steadily from just a couple of percent in 1970 to almost 15 percent in
1995 (Japek et al., 2001).

The use of web surveys distributed by email has grown rapidly since the beginning of the 21st century
(Dahmström, 2004). The reason for the increased usage is probably that email surveys are easy to
conduct at a large scale and at a low cost. Conducting such surveys often result in respondents feeling
overwhelmed by constantly receiving surveys and thus stop responding to them. This phenomenon is
defined as survey fatigue, and may be one of the reasons for increasing dropout rates (Strandell and
S., 2015).

3


Figure 2.1: The evolution of the dropout rate for the Labor Force survey between 1970 and 1995
(Japek et al., 2001). The thick red line depicts the total dropout. The smaller red line represents
respondents who were unavailable, and the black line represents respondents who refused to answer.
The line at the bottom of the figure represents respondents who dropped out due to other reasons.

2.1.1 Techniques to increase response rates

The Central Bureau of Statistics has two strategies on how to reduce dropouts (Japek et al., 2001).
The first is to work on reducing dropouts by sending reminders and making sure that the questions
are adequate and relevant for the respondent. The second strategy is to estimate the results of the
dropouts and adjusting the results of the survey. The Central Bureau of Statistics use their indexes
of the Swedish population and compare it to the result of the survey (Strandell and S., 2015). For
example, if only people with a high income has answered the survey, this is taken into account when
estimating the results. Both of these strategies can potentially consume a lot of time and resources
(Japek et al., 2001).

As described above, one way of preventing respondents from dropping out is to write adequate ques-
tions. According to Fowler there are five challenges in writing adequate questions in surveys (Fowler,
1995):

• Defining the goal of the survey, specifying what kind of answers are needed to meet the goals
and after that constructing questions to achieve these answers.

• Ensuring that all respondents have a shared, common understanding of the meaning of the
question. Specifically, all respondents should have the same understanding of the key terms of
the questions, and their understanding of those terms should be the same as that intended by
the person writing the questions

• Ensuring that people are asked questions to which they know the answers. There should not be
any barriers to knowing the answers.

• Asking questions that respondents are able to answer in the terms required by the question

• Asking questions respondents are willing to answer accurately

How to write adequate questions is important to have in mind when designing a chatbot to be used
for conducting surveys. One of the main challenges in this project is to use questions that fits the
conversational medium of chatbots.

4


Mentioned earlier, a commonly used strategy to make people respond to websurveys is to send re-
minders. In telephone interviews, dropouts can be reduced by contacting the respondents that were
unavailable or did not respond the first several times (Strandell and S., 2015).

Another way of reducing dropouts is to make sure that the respondents have an interest in responding
to the survey (Galesic, 2006). Respondents who are more interested in the survey topic may be less
prone to drop out. The interest in the topic in general and the topic in each question in the survey
might determine if the respondent will complete the survey or not.

2.2 Net Promoter Score

Net Promoter Score (NPS) was first introduced by Frederick Reichheld in 2003 (Grisa↵e, 2007). The
idea is that a summary number from one customer survey question can be used as a basis to measure
customer loyalty (Grisa↵e, 2007). Customers answer the question: ”On a 0-10 scale, how likely is
it that you would recommend this business to a friend or a colleague?”. Customers that rate the
company between 0 and 6 are called detractors, customers rating 7 or 8 are called passively satisfied
and customers rating 9 or 10 are called promoters. The scale is shown in Figure 2.2. The NPS is
calculated by taking the percent of promoters minus the percent of detractors. For example, if 50
percent of the respondents have rated the company 9 or 10 and 30 percent of the respondents have
rated the company 6 or less, the NPS is 20.

Figure 2.2: The NPS scale (Satmetrix Systems, Inc, 2017).

Many companies have adopted the NPS concept, including American Express and Microsoft. One
of the reasons for its success is its simplicity and its claims of links to profitability (Grisa↵e, 2007).
Net Promoter Score is an interesting measurement of customer loyalty that could be used in chatbot
technology.

2.3 Chatbot technology

The use of intelligent agents, such as chatbots, for commercial purposes has soared in recent years
(Tsvetkova et al., 2016), and the deployment of new chatbots is at a steady increase (Radziwill and
Benton, 2017). One of the more popular examples is Siri – the personal assistant developed by Apple
(Lemon, 2012). The benefits of utilizing robotic intelligent agents compared to human personnel are
many, but some in particular are reduced time-to-response, enhanced customer service and increased
customer engagement (Radziwill and Benton, 2017).

In order to provide a service for running automated conversations, such as chat-like surveys, one
must achieve a deep understanding on the development of communicating agents. In the following
chapter the technicalities of intelligent agents such as chatbots, as well as their practical usage, will be
discussed.

5


2.3.1 Intelligent agents

The formal definition of an intelligent agent is an elaborate one. It is generally agreed upon that it
is a system that thinks and acts, but the many definitions vary in terms of whether it thinks at acts
”rationally” or ”like humans”. In practice, an intelligent agent uses sensors to receive information
about the environment it is in, chooses a best action to take based on the information received, and
then uses its so called e↵ectors to perform the action (Russell and Norvig, 2010). An illustration of
how the intelligent agent works can be seen in Figure 2.3

Figure 2.3: The general main components of an intelligent agent (Russell and Norvig, 2010).

There are many classes of intelligent agents, but this project is primarily focused on a specific type
called conversational agents, or chatbots for short (Radziwill and Benton, 2017). Where a human agent
has ears for listening and a brain for thinking, a chatbot has digital inputs which receive information
and a programmed decision engine to act upon it. The decision engine could be deterministic as a series
of conditional statements, or probabilistic as a chain of numerical calculations (Russell and Norvig,
2010).

An early example of a chatbot with a deterministic decision engine is Eliza, developed in the 1960’s.
Based on keyword matching, the agent searched for certain words in the given text input and responded
accordingly (Russell and Norvig, 2010). More recent chatbots, such as Apple’s Siri, often employ a
mixture of deterministic and probabilistic models (Aron, 2011). One reason being that this induces
more flexibility in terms of handling spelling errors or ambiguity in language.

In this sense, the development of a surveying chatbot is done best by using a combination of di↵erent
models. Despite the fact that a personal assistant chatbot and a surveying chatbot has considerably
dissimilar purposes, similar decision engines can still be used in both agents, with only minor changes in
the way the chatbot produces output. The technical details of how these models operate is a common
theme in natural language processing, and will be further outlined in the next section.

2.3.2 Natural language processing

The field of natural language processing (NLP) is concerned with the interaction between computers
and human language, or on the more technical side, parsing and analyzing large language corpora
(Pascal, 2007). The technicalities of a natural language processing model di↵er a lot depending on
what task it is trying to solve. This chapter however, will mainly be focused around components
involved in the development of a surveying chatbot. First, a few common NLP tasks are listed, then
an explanation of how a probabilistic NLP model might be constructed, is given.

Natural language processing can be applied in many di↵erent tasks, on the syntax level in the form
of parsing and stemming, and on the semantic level in the form of classification or entity recognition

6


(Nadeau and Sekine, 2007). Parsing and stemming are tasks related to cleaning input and normalizing
word inflection, respectively. In the context of a chatbot, these tasks are most frequent when trans-
forming the data received as input so that the decision engine can read it. Models aimed at solving
these tasks range in complexity from advanced neural networks to simple rule-based models, one such
example being the Porter Stemming Algorithm (Porter, 1980).

In contrast to parsing and stemming, classification and entity recognition are tasks rather related to
the semantics of language. One example of a classification task is sentiment analysis, in which the
NLP model is supposed to correctly annotate text with a numerical score describing its sentiment on a
scale between negative and positive. Classification models are especially interesting in the development
of surveying chatbots since they allow the chatbot to choose the next question to ask based on how
answers to previous questions are classified. NLP tasks pertaining to analyzing the semantics of text
are oftentimes more di�cult to solve due to the many dimensions of semantics compared to syntax.
When analyzing syntax, a word is always a word, but when analyzing semantics, a word can have
di↵erent interpretations depending on context.

Many NLP models are based on mathematical equations. In the sentiment analysis task, one could
for example make use of Bayes’ theorem to calculate the probability that a word appears in a positive
context, and thus decide on the sentiment of the word (Bishop, 2006). The task could also be solved
using a neural network approach, in which the text is sent through layers of mathematical calcula-
tions which are consolidated into a final sentiment score (Bishop, 2006). In spite of the apparent
dissimilarities between the two approaches, they both require training and testing before they can be
used in production. Below follows a more technical description of how such a model is developed and
used.

In the case of a surveying chatbot, one valuable feature is for the bot to be able to interpret di↵erent
types of ”yes” and ”no”-answers. A somewhat simple solution is to declare two functions p1(x)
and p0(x) defining the probability that a vector x of words is a ”yes” or ”no”-answer, respectively.
The goal is to define these functions so that all ”yes”-answers, such as x = [”yes”, ”you”, ”can”]>,
gives a very high probability when applied to the function p1, and on the other hand a very low
probability when applied to p0. The opposite should hold for word vectors of a ”no”-answer such
as x = [”no”, ”definitely”, ”not”]>. This kind of natural language processing model is denoted as a
probabilistic classifier, and there are a variety of such models to choose from. A few examples are
naive Bayes classifiers, support vector machines, or neural networks (Bishop, 2006).

After a model is chosen, its parameters must be tuned for the specific task which the model is supposed
to solve. This is called training or learning. In supervised learning, which is a branch of training often
used with classification models, the model is fed a large set of samples, and the produced output is
compared to expected output. If there is a di↵erence between produced output and expected output,
the parameters are tuned proportionally. In practice, the training phase is fully automated, and the
parameters of the model are tuned using mathematical calculations. A classifier designed to decide
whether a response is a ”yes” or ”no”-answer might perform well in that specific task, while still having
a disastrous performance if it is used in trying to classify whether a statement is positive or negative.
This is because the parameters of the model are often exclusively tuned for one specific task, which in
this case is classifying ”yes”- and ”no”-answers. This means that for each task, a new model needs to
be developed, and new data to feed it with needs to be found (Russell and Norvig, 2010).

2.4 SMS as a medium for surveys

Short Message Service (SMS) is a standardized way of communicating text messages over a GSM
network and was first introduced in the 1980’s (ETSI, n.d.). This paper will treat the terms SMS and

7


text messaging as synonyms, therefore when ”texting” or ”text-message” is mentioned the technology
known as SMS is referred to.

The subject of SMS is of interest to this project in particular for two of its traits. First, the number
of mobile cellular subscriptions per 100 people is over 98 (The World Bank, 2015), and since SMS is
device independent, it works on almost all mobile phones (Hillebrand et al., 2010). Second, SMS has an
inherent cost to distribute, encouraging usage with restriction and avoiding spam when compared to,
for example, email. These traits could thus be attractive, from a business standpoint, when distributing
conversational surveys to respondents.

2.4.1 Previous work on SMS surveys

A study named ”Precision and Disclosure in Text and Voice Interviews on Smartphones” performed
in 2015 showed that text-message based surveying resulted in more qualitative answers, more di↵eren-
tiated answers and more disclosure of socially undesirable information, when compared to traditional
phone-interviews (Schober et al., 2015). This was shown by comparing text-messaging both to auto-
mated phone surveys as well as manually performed ones, with quality of answers defined mainly as
precision of numerical answers (how many numbers were rounded). The authors argue that this could
be attributed to the fact that responding to a text-message is much more convenient than answering
the phone and allows responding while multi-tasking.

The report mentioned above is of interest because it gives indication that there is clear value in using
text-message based surveying rather than traditional phone surveys. However, the study presented
here di↵ers from this thesis in that it does not leverage the use of intelligent agents to create a more
natural texting experience.

2.4.2 How to send SMS programmatically

When using SMS as a medium for automated surveys, sending the SMS messages manually from a
mobile phone is not a scalable solution. One solution for sending SMS messages programmatically
using a computer is to use an SMS gateway, which is a device allowing for transmission or receipt of
SMS messages without the use of a mobile phone (K.Katankar and Dr.V.M.Thakare, 2010). Modern
API:s, such as Twilio or 46elks, expose the functionality of an SMS gateway through HTTP, which
allows for easy integration by developers.

When using an API to send SMS, it is possible to use alphanumeric sender ID:s. When using this, the
person receiving the SMS will see a word, such as a company name, as the sender of the SMS, instead of
a regular phone number. This is often used by companies when sending out promotional SMS. When
using alphanumeric sender ID:s it is not possible for the respondent to reply to the SMS (infobip,
2017), making alphanumeric sender ID:s unsuitable for two-way SMS as required when conducting
surveys in a conversational manner.

2.5 Sustainability

The definition of corporate sustainability is in the book ”Beyond the Business Case for Corporate
Sustainability” described as: ”Meeting the needs of a firm’s direct and indirect stakeholders without
compromising its ability to meet the needs of future stakeholders as well” (Dyllick et al., 2002). A
common way to describe sustainability is in the three defined dimensions of corporate sustainability,
economic, environmental and social sustainability (Adams, 2006). In the following subsections the

8


three dimensions of corporate sustainability are described in more detail, and it is discussed how
di↵erent areas of this project relates to the three dimensions.

2.5.1 Economic Sustainability

Dyllick et al. defines economic corporate sustainability as ”Economically sustainable companies guar-
antee at any time cashflow su�cient to ensure liquidity while producing a persistent above average
return to their shareholders” (Dyllick et al., 2002). The reasoning behind this definition is that a com-
pany ceases to exists when there is no economic capital left. However, in reality a company will become
economically unsustainable long before that. In order for a company to be economically sustainable,
it has to be able to handle multiple types of economic capital, for example, financial capital, tangible
capital and intangible capital.

This goal of this project relates to economic sustainability by proposing a solution that eliminates waste
of financial, tangible and intangible capital in companies. If data can be collected about customer
opinions in a more su�cient way, misinvestments caused by decision making based on bad or non-
existent data could be minimized. In addition to minimizing misinvestments, high quality customer
data can also help improve existing products and processes within a company. Thus adding value
to both the company and society and by ensuring that the company focuses resources on producing
tangible and intangible capital that customers actually demand.

2.5.2 Environmental Sustainability

Environmental sustainability is perhaps most commonly what people refer to when mentioning sus-
tainability in their everyday lives. Environmentally sustainable companies can be defined as companies
that only use natural resources below their natural rate of reproduction, or below the development of
substitutes (Dyllick et al., 2002). Along with this the companies neither cause emissions that accu-
mulate in the environment at a higher rate than the natural capacity of the ecosystem, or engage in
activities that degrades eco-system services.

If this project is successful, it will relate to environmental sustainability by reducing the use of paper
surveys in exchange for a chatbot solution. This is done by developing a product that gives the
respondents a better result and experience than paper surveys. In doing this the project will reduce
the use of natural resources by companies, in this case trees, possibly bringing their usage under the
natural rate of reproduction.

2.5.3 Social Sustainability

Social sustainability can be understood by dividing it into two types of capital, human capital and
societal capital (Dyllick et al., 2002). A socially sustainable company can then be defined as a com-
pany which adds value to the communities in which it operates by improving human capital of the
individuals within that community, as well as furthering its societal capital. Managing social capital in
a sustainable way means to make sure that all stakeholders in a company understands why a company
is doing something, as well as broadly agree that action is justified. Human capital on the other hand
primarily relates to aspects such as skills, motivation and loyalty of employees.

This project is focused on developing a product that aims to be more user friendly in terms of answering
surveys. By creating a more personalized experience the survey will appear more as a conversation and
could possibly provide a more positive experience where respondents feel appreciated. If the project is
successful it could increase response rates from groups of society that are not reached with surveying
technology today, helping boost equality and diversity of both societal and human capital.

9


This section has described the theory behind surveying techniques, intelligent agents, text messaging
and sustainability. These theoretical subjects, combined with the methodology described in Chapter
3, provide a basis required when trying to deliver a solution to some of the problems within the survey
industry.

10


Chapter 3

Methodology

In this chapter the methodology that was applied in the project is explained in more detail, and the
reason for using this particular methodology is discussed. In Section 3.1, the Lean Startup Methodology
is described and discussed. In Section 3.2, Business Model Generation is described along with the tools
used when applying this methodology in the project. In Section 3.3, how interviews were performed in
order to collect data for the project is described. In Section 3.4, A/B testing and how it was applied
in the project is described.

3.1 The Lean Startup Methodology

This project is focused around developing a product where there are extremely high uncertainties,
both about whether a problem actually exists, as well as whether a product that solves these problems
can be built. Because of this, the Lean Startup Methodology introduced in The Lean Startup by Eric
Ries, will be applied. This methodology provides a framework for startups, defined as a company
operating under conditions of extreme uncertainty, in order to eliminate waste related to developing
products and businesses (Ries, 2011). The theory takes inspiration from the Toyota production system
described, for example, in the book The Toyota Way (Liker et al., 2009).

Figure 3.1 illustrates one of the core concepts introduced in the Lean Startup Methodology, the build-
measure-learn feedback loop. It describes how uncertainties can gradually be eliminated with minimum
waste, by breaking a business model into smaller parts where hypothesis about the di↵erent parts can
be formulated and validated. This starts by working the feedback loop in reverse order.

The loop is started at the learn step, where hypotheses that express the learning that needs to be done
are formulated. These hypotheses should be designed in such a way that they can gradually reduce
uncertainties about the startups business model as they are validated or invalidated. The hypotheses
can relate to anything regarding the startup, such as what problems customers are facing, if a proposed
product can solve identified problems, how a service should be priced, and so on.

After one or several hypotheses have been formulated the next step is the measure step in which various
variables are determined that supports measurement. The idea is that these variables can determine
whether or not a hypothesis can be validated. To test these measurements various tests are set up in
order to validate or invalidate a previously formulated hypothesis.

To support the measure step The Test Card will be utilized, seen in Figure 3.2, a tool used to state
hypothesis, assumptions, how to test the hypothesis, how it should be measured and what is required
to validate or invalidate hypothesis. As well as The Learning Card, seen in Figure 3.3, used in the

11


Figure 3.1: The build-measure-learn feedback loop (Ries, 2011).

learn step to state which hypothesis were tested, what was observed, what was learned and what the
next steps are.

Figure 3.2: Test Card (Strategyzer,
2017).

Figure 3.3: Learning Card (Strategyzer,
2017).

In the build step it is already known what is needed to be learnt and how to validate this learning.
The purpose of the build step is to build an MVP that can be used achieve this validated learning.

12


What is important here is that no more than exactly was is required to perform the testing is built,
thus reducing waste from building features that later might prove unnecessary.

When the loop has been set up it can be worked through in the order build, measure, learn. First an
MVP is built, which is then used to measure some metric to validate learning, and then the results are
analyzed in order to validate or invalidate hypotheses. The learnings usually result in new hypotheses
that in turn need to be validated, which starts the loop all over again. If a business critical hypothesis
is invalidated and the new hypotheses change the business model to a large extent, it needs to be
decided whether to pivot away from the business model or persevere. (Ries, 2011).

By continuously testing and validating hypothesis, this methodology allows for spending as little time
as possible developing product features that ultimately prove useless for the customers. As using
chatbots for surveys is something that has not been done before, uncertainties about both market and
product are high, making the traits of the methodology a good fit to this particular project.

3.2 Business Model Generation

To compliment the Lean Startup Methodology, tools from Business Model Generation, a business
strategy and economics handbook by Alexander Osterwalder and Yves Pigneur (Osterwalder and
Pigneur, 2010), will be used in this project. The book provides a strategy framework for businesses.
In this particular project the theories and tools presented in the book provide a way to structure the
strategic work and help in identifying and formulating hypothesis.

3.2.1 Business Model Canvas

One of the concepts introduced by Osterwalder and Pigneur is the Business Model Canvas, seen in
Figure 3.4. The purpose of the canvas is to provide a tool that allows the creation of a perceptible
model of a business idea. In this project the Business Model Canvas was used as a foundation for
the learn step in the build-measure-learn feedback loop explained in Section 3.1. The canvas was
used to structure and visualize a potential business model for Surview, as well as to identify di↵erent
hypothesis about specific parts of the business model.

The canvas is split in nine parts that are tackled in the following order:

1. Customer Segments. This part of the canvas defines who the customers are and why they would
buy the product. In order to create a successful business, it is important to understand who the
customers are and how the the business model satisfies the customers needs. This is explained
in more detail in Section 3.2.2 where the Value Proposition Canvas is explained.

2. Value Propositions. Here the problems or needs that the business want to solve are outlined. The
value proposition should describe what pains the product is solving or what gains the product is
giving the customer. This too is explained in more detail in Section 3.2.2.

3. Channels. In this section the channels that are most suitable for delivering the value proposition
to the chosen customer segments are defined.

4. Customer Relationships. Defines the type of relationship between the company and the cus-
tomers.

5. Revenue Streams. Defines how a company generates income from the chosen customer segments.

6. Key Resources. This is the resources a company is going to use to create value.

7. Key Activities. This is the tasks that are viable to perform the business model.

13


.H\�3DUWQHUV .H\�$FWLYLWLHV

.H\�5HVRXUFHV

9DOXH�3URSRVLWLRQV &XVWRPHU�5HODWLRQVKLSV &XVWRPHU�6HJPHQWV

&KDQQHOV

&RVW�6WUXFWXUH 5HYHQXH�6WUHDPV

7KH�%XVLQHVV�0RGHO�&DQYDV
'HVLJQHG�IRU� 'HVLJQHG�E\� 2Q��������GG�PP�\\\\

,WHUDWLRQ��

Figure 3.4: Business Model Canvas (Strategyzer, 2017).

8. Key Partnerships. The key partners are the suppliers and partners of the company.

9. Cost Structure. This is the key costs that the company is going to have.

The first 5 parts define how the value from the business idea is delivered to the customer, and the last
4 parts define how the business works internally.

3.2.2 Value Proposition Canvas

The Value Proposition Canvas, as shown in Figure 3.5, can be viewed as a zoomed in canvas on the
customer segments and value proposition in the Business Model Canvas. The Value Proposition Canvas
allows a more in depth analysis on what the customer actually wants and how the product fits these
problems or needs. The canvas is split into two parts, the Value Map to the left and the Customer
Profile to the right. The Customer Profile describes the customer, and the Value Map describes the
product. The di↵erent parts of the Value Proposition Canvas are here described by using a chef at a
restaurant as an example.

• Customer Jobs. Describes what jobs the customer is trying to get done. This would for a chef
be to deliver tasty food to his guests.

• Pains and Gains. A pain is something that the customer perceives or could perceive as trouble
before, during or after a job, and a gain is something that would benefit the customer before,
during or after a job. For a chef a pain could be that he often burns food he forgot in the oven,
and a gain could be more tasty food to serve to guests.

• Products and Services. Describes what products and services will be used and served to the
customers. This could for a chef be a new spice or an automated timer to fix on an oven.

14


Figure 3.5: Value Proposition Canvas (Strategyzer, 2017).

• Gain Creators and Pain Relievers. Describe how the products and services actually eliminate
pains or create new gains. The new spice will create tastier food and the automated timer will
eliminate the pain of forgetting food in the oven.

3.3 Interviews

Talking to customers is a central part of the Lean Startup Methodology (Ries, 2011). By talking to
customers, a startup can validate or invalidate hypotheses about its business model without having to
develop a finished product or service. This allows the startup to iterate through the build-measure-learn
feedback loop at a faster rate, by minimizing time and money spent on technical development.

During this project interviews with potential customers were performed in order to validate hypotheses
about di↵erent business models for Surview. In order to get the maximum possible value out of these
meetings, techniques from The Mom Test (Fitzpatrick, 2013) were used.

The Mom Test is a handbook that contains practical advice on how to do customer interviews. As
previously discussed in Section 2.1, one thing that can largely influence the answers given in an
interview is the way questions are asked. According to Fitzpatrick, the three most important things
for a startup to have in mind when asking questions during a customer interview are:

1. Talk about the customer’s life instead of the idea

2. Ask about specifics in the past instead of generics or opinions about the future

3. Talk less and listen more

Fitzpatrick also emphasises the importance of pushing for commitment at the end of a meeting, in
order to make sure that the customer that is being interviewed actually has an interest in the product
or service that the startup is developing. A commitment can come in several di↵erent form, but the
most common are:

1. Risking reputation, for example by sharing contact information to a valuable contact

15


2. Spending more time, for example by scheduling another meeting for further discussion

3. Giving up money, for example by paying to test the startup’s product or service

If the outcome of a interview has one of the three commitments listed above, the meeting was successful
which outlines the process going forward.

3.4 A/B Testing

A/B testing is a method of testing that can be used to compare two di↵erent versions of a product to
determine which of them performs the best. This method of testing can be used to iteratively develop
a product in a data-driven way, where it is ensured that each new version performs better than the
previous.

When doing A/B testing, the first step is to define the di↵erent versions that should be tested. This
could for example be two di↵erent ways to ask the first question of a survey. When the di↵erent
versions have been defined and implemented, they are distributed to di↵erent subsets of customers,
and after some amount of time the results of the di↵erent versions are measured using one or more pre-
defined metrics. In the example of testing di↵erent introduction questions of a survey, one such metric
could be the number of answers to the survey. When the test is concluded, the version that performed
the best in regards to the pre-defined metrics is selected as the winner, and all the other versions are
discarded. The winning version can then be compared with new versions using another round of A/B
testing, and the whole process is repeated all over again. This can be repeated indefinitely or until a
su�ciently good result has been observed during the tests.

When doing A/B testing, it is important that the di↵erent subsets of customers that are exposed to
di↵erent versions of the product are of su�cient size. If the subsets are too small, the results from
tests can not be trusted from a statistical standpoint, which makes it di�cult to draw conclusions and
learn from tests.

A/B testing was used in this project to compare the proposed product with existing ways of conducting
surveys, and also to compare di↵erent ways of asking questions in a conversational format. This will
be explained in more detail in Section 4.4.

16


Chapter 4

Process

This chapter explains how the project was carried out. Section 4.1 starts by describing the starting
point of the project, and the process is then divided into four major phases. First, Section 4.2 describes
how an initial business model was hypothesized. Second, Section 4.3 describes the phase of contacting
the customer segment defined as survey consultants. Third, Section 4.4 describes how testing with the
consultants were performed. Finally, Sections 4.5 and 4.6 describe how a pivot of customer segment
was carried out. Each section ends with a discussion, where the results and learnings are discussed.
Since the work was completed in an iterative manner, each phase uses the results and learnings of the
earlier phases as a starting point.

4.1 Previous work on Surview

Surview started as a project at Gothenburg Startup Hack in April 2016. Gothenburg Startup Hack
is an event where developers and entrepreneurs get together for a day to build prototypes for their
ideas (Gothenburg Startup Hack, 2017), and during this event a prototype for conducting surveys as
conversations over SMS was built.

This prototype included functionality for asking questions and receiving answers via SMS, a landing
page where sessions could be started for a pre-defined survey, as shown in Figure 4.1, and a simple
dashboard where received answers could be viewed, as shown in Appendix Figure A.1 to Figure A.3.
The AlchemyLanguage API (AlchemyAPI, 2017) was used to make a survey feel more like a conver-
sation, by routing answers to di↵erent questions depending on the sentiment of the answer. This only
worked for the English language, so only surveys done in the English language were supported.

17


Figure 4.1: The landing page built for Gothenburg Startup Hack.

In the fall of 2016 the product was further developed into a MVP. One example survey conducted
with this MVP can be seen in Figure 4.2. AlchemyLanguage was replaced by an internally developed
algorithm that could route answers to di↵erent questions depending on polarity, sentiment, and NPS-
score. This algorithm also supported the Swedish language. In addition, the dashboard was developed
with more functionality for creating and modifying surveys, creating sessions for surveys and viewing
received answers. The updated dashboard can be seen in Figure 4.3.

Internally, surveys built using this tool had the shape of a decision tree. Every survey had a first
question that was sent to the respondent when a session was started. When an answer arrived, the
next question was sent out, and the same process was repeated until the survey was over, or the
respondent stopped answering. To make this feel more like a regular conversation, the next question
was chosen depending on the incoming answer, and this is where the decision tree comes in. Each
question had a type, and each question type had di↵erent rules for which question to ask next. In
Appendix Figure 4.4 a flowchart can be seen that demonstrates how an example survey works. The
rest of the dashboard can be seen in Appendix Figure A.4 to Figure A.6.

In the MVP there were five di↵erent types of questions – sentiment, yes/no, NPS, static and final.
For a sentiment question, di↵erent next questions could be asked depending on whether the answer
was positive, negative or neutral. For the yes/no question, the polarity of the answer was analyzed
to determine whether the respondent was answering yes or no, and the next question was chosen
depending on the result. This was very useful for asking questions like ”Is it okay if we ask you some
questions?”. For a NPS question, the answer was scanned for a number from 0-10, and the next
question was chosen depending on whether the respondent was passive, a detractor, or a promoter.
Static questions always lead to the same next question, regardless of the answer, and final questions
indicated that a survey had been completed and there were no more questions to ask.

If an answer did not correspond to any of the rules for a question, for example when answering
”Maybe” to a yes/no-question, or when the routing algorithm was not sure of how to classify an
answer, it resulted in the session being paused. This meant that no next question would be sent to the
respondent until the session was unpaused, which was only done when a new answer that corresponded
to one of the rules was received. Answers could be added to a session through the dashboard, which

18


Figure 4.2: A survey conducted with the first MVP.

Figure 4.3: The landing page for the dashboard in the first MVP.

19


Figure 4.4: Flowchart showing an example of how a survey can be built.

allowed for manual unpausing of sessions by an administrator by adding an answer for which it was
known that the algorithm would give a correct classification.

This way of designing surveys allows for great customization, but it requires that the creator has
some expertise in how to ask questions in this particular format. Asking just a single question the
wrong way could throw o↵ the entire flow in a survey, which in turns could cause a respondent to stop
answering.

Creating sessions for a survey in the MVP could be done either through the dashboard or through an
API. The idea was that sessions for trialing purposes could be created through the dashboard, and
that the API would be used by external companies to create sessions for their customers when the
service was used in production.

Meetings were held with a handful of companies in parallel to developing the product in order to learn
more about how companies work with surveys and customer feedback today. During these meetings
it was discovered that most companies are not satisfied with just receiving collected data, but rather
need extensive analysis of the data in order to make educated decisions. It was also discovered that
there are a lot of consultant companies that help other companies to set up and maintain solutions for
gathering, analyzing and taking actions from market research and customer feedback.

This section described the work that had been done on the product prior to starting the project. The
following sections will describe the work that was done during the project, and explain all the learnings
that were achieved during the process.

4.2 Coming up with a potential business model

The previous work on Surview had mainly been focused on developing a great product, and not much
e↵ort had gone towards thinking about how to design a sustainable business around it. This was why,
before any new work was started, a Value Proposition Canvas and a Business Model Canvas were

20


set up in order to visualize a potential business model for Surview. These canvases were then used
to identify hypotheses about the business model, which were later prioritized and tested using the
build-measure-learn feedback loop as described in Section 3.1.

4.2.1 Designing a Value Proposition Canvas

In this section the thought process behind creating an initial Value Proposition Canvas is described.
The resulting canvas is shown in Figure 4.5.

7KH�9DOXH�3URSRVLWLRQ�&DQYDV

9DOXH�3URSRVLWLRQ� &XVWRPHU�6HJPHQW�

3URGXFWV�	�6HUYLFHV

*DLQ�&UHDWRUV

3DLQ�5HOLHYHUV

*DLQV

3DLQV

&XVWRPHU�-REV

&RQYHUVDWLRQDO�VXUYH\�YLD�606 6XUYH\�FRQVXOWDQWV

606&RQYHUVDWLRQ�
IRUPDW

0RUH�
VDWLVILHG�
FXVWRPHUV

)DVWHU�
UHVSRQVHV

3URYLGLQJ�
FXVWRPHUV�
ZLWK�JRRG�
GDWD�IRU�
GHFLVLRQV

5HDFKLQJ�
FHUWDLQ�
FXVWRPHU�
VHJPHQWV

/RZ�
UHVSRQVH�

UDWH

'DVKERDUG

$3,

&RQYHUVDWLRQ�
IRUPDW 606

Figure 4.5: Initial Value Proposition Canvas. Notes in red indicate parts of the canvas that have not
yet been validated.

21


It was hypothesized that the most likely customer segment for the product that had been developed
would be survey consultants. There were two main reasons for this. The first reason was that a
lot companies that had been been interviewed during the previous work required extensive analysis
of collected data. Developing systems for this kind of analysis takes a long time, and the idea was
that Surview could provide only a tool for collecting data, and that the collected data could then
be inserted into the survey consultants existing tools for further analysis. This would decrease the
amount of time needed to develop a product that could be used by real customers, and also allow
companies to use tools that they already know for working with the data, making the e↵ort needed
to switch from another system as little as possible. The second reason was that setting up surveys
in a conversational format required a lot of knowledge of how to ask adequate questions. This was
not necessarily knowledge that regular companies had, but it was believed that the survey consultants
would have enough expertise in the area in order to set up adequate surveys without needing too much
assistance. This was why Customer Segment in the Value Proposition Canvas was defined as survey
consultants.

Survey consultants provide other companies with solutions for market research and customer feedback.
There can be many reasons for hiring a survey consultant, but in the end it usually boils down to
wanting data as a foundation for making qualified decisions in order to improve the business. This
was why Customer Jobs in the Value Proposition Canvas was defined as providing customers with good
data for decisions.

Research about problems in the surveying industry, as described in Section 2.1, together with previous
interviews with companies, gave indications that low response rates and inability to reach certain
customer segments are two problems that exist in the survey industry today. It was therefore believed
that these two problems were pains that survey consultants want to solve. It was also believed that if
customers feel like their opinions matter, for example when filling out a survey and seeing that their
answers lead to actions, the customers will get a more positive image of the company that sent out the
survey, leading to more satisfied customers. Increasing customer satisfaction should be a good selling
point for the consultants, which was why more satisfied customers was identified as a potential gain.
Faster responses was also identified as a potential gain, with the reasoning that it would be valuable for
companies to receive data more quickly, for example by making it easier to turn dissatisfied customers
into satisfied customers before they have time to switch to a competitor or complain to friends and
family.

With the customer part of the canvas in place, the products and services part was defined as Dashboard
and API. The dashboard would be used to create and manage conversational surveys, and also to view
and make exports of received answers. The API would be used to create sessions for surveys created
through the dashboard, and the idea was that the survey consultants could integrate their existing
systems with the API in order to start sessions whenever they deemed fit. This was in line with the
product that had already been developed, as described in Section 4.1.

To complete the work with the Value Proposition Canvas, the conversational format for surveys to-
gether with SMS as a medium were identified as both gain creators and pain relievers. Asking questions
via SMS and allowing the respondent to answer directly by replying with a new SMS, instead of hav-
ing to click a link and opening the survey in a web browser, was believed to lead to faster responses
compared to existing methods. The conversational format was expected to yield more satisfied cus-
tomers, since when using this format it is possible to instantly acknowledge the respondent’s opinion
by modifying the next question sent. The pain relief was expected to come from the fact that the SMS
format along with the conversation would invite respondents to fulfill the survey and thus increase
response rates. It was also hypothesized that the format would be suitable for reaching young people,
as were identified as a challenging to reach customer segment in Section 2.1.

22


4.2.2 Designing a Business Model Canvas

With an initial Value Proposition Canvas in place, an entire possible business model could visualised
with a Business Model Canvas, using the Value Proposition Canvas as base. The resulting Business
Model Canvas can be seen in Figure 4.6.

Figure 4.6: Initial Business Model Canvas. Notes in red indicate parts of the canvas that have not yet
been validated.

After an initial Value Proposition Canvas had been designed, focus was on how to formulate hypotheses
about the entire Business Model Canvas. Firstly translating the hypothesis from the Value Proposition
Canvas into the value propositions and customer segments into the Business Model Canvas. The actual
value for the customers were formulated as Find Dissatisfied Customers, Reach out to new audiences,
Reliable data, Better analysis, More feedback as seen in Figure 4.6. The customer was, as formulated
in the Value Proposition Canvas, Customer Survey Consultants. Here it was decided to also add
Continuous surveys as an additional constraint on the customer segment. This was done in order
to put a stricter specification on the customer segment, and because it was believed that conducting
surveys as conversations via SMS would be more suitable for continuous surveys.

The rest of the hypotheses on the canvas were formulated, though they were very likely to change as
the project progressed. Thus only a shallow analysis was performed on every hypothesis that was not
related to customers segments or value proposition.

4.2.3 Initial hypotheses

From the Value Proposition Canvas and the Business Model Canvas initial hypotheses about the
business model were set up. These were the hypotheses that were identified as necessary for the
business:

23


• Surveys conducted in a conversational format over SMS give higher response rates than existing
surveying methods

• Surveys conducted over SMS give faster feedback than existing surveying methods

• Surveys conducted over SMS reach certain customer segments that are di�cult to reach today

• Surveys conducted in a conversational format over SMS result in useful data

• Respondents do not have a negative attitude towards receiving surveys via SMS

• Surveys conducted in a conversational format over SMS work best for shorter surveys, ongoing
surveys and to get feedback from existing customers

• Low response rate is a problem that survey consultants want to solve

• Slow feedback is a problem that survey consultants want to solve

• Not getting answers from young people, busy people and ”neutrals”, is a problem that survey
consultants want to solve

• Survey consultants can and want to buy and work with external tools that are not developed
in-house

• Survey consultants can and want to work independently with external tools

• Survey consultants can sell better answer rates, ability to reach new customer segments and
faster feedback to their customers

• License fee and a fee per survey is a good way to pay for the product

The first six hypotheses in the list above, related to the traits of SMS, were identified as important
because it was critical for them to be true for the product to work. If they proved to be false, the
proposed product would not work at all, and a new product had to be designed.

The hypothesis about consultants being willing to pay for higher response rates, faster feedback and
reaching certain customer segments was deemed important because if this hypothesis proved to be
false, it would be impossible to sell a product that addressed these problems to the survey consultants.
If the hypotheses was true it would be a good indication that the problems were big enough for the
customer segment to pay for a solution. The hypothesis that consultants can sell better answer rates,
ability to reach new customer segments and faster feedback to their customers was also important,
since if this was not the case the consultants would have no incentive to sell the product to their
customers.

The hypothesis that consultants can and want to buy and work with external tools that are not
developed in-house was important, since if this was not the case, it would be impossible to sell a new
product to the consultants.

It was also important that the consultants could and wanted to work independently with external
tools, since it was envisioned that the consultants would work independently with the product. If this
was not the case, a lot of time and e↵ort would need to be dedicated towards support and helping
the consultants with using the tool, instead of focusing on just developing the tool. This was not the
business model had in mind when designing the initial Business Model Canvas, so if this hypotheses
proved to be false the business model had to be changed.

There are a lot of di↵erent survey consultants, and some work more with customer feedback, while some
work more with market research. If it could be validated that surveys conducted in a conversational
format over SMS worked better for shorter surveys, ongoing surveys and contacting existing customers,
it would help in further segmenting the customer segment.

24


License fee and fee per survey was identified as a potential way to charge for the product. However,
this was deemed as less important to validate at this point, since the pricing would not matter if the
customer segment and value propositions proved to be inaccurate.

With canvases and hypotheses in place, the next step was to test the most critical hypotheses to
see if they could be validated or invalidated. The process of doing this is explained in the following
sections.

4.3 Talking with survey consultants

At this point, it was important to confirm the most critical hypotheses for the business, which were the
hypotheses related to the chosen customer segment. If these hypothesis were proven to be invalidated,
there would be no reason to start developing a product at all and a customer segment pivot would be
necessary. The testing of the hypotheses related to the chosen customer segment was done by talking
to potential customers, which in this case was survey consultants as defined in the Business Model
Canvas and Value Proposition Canvas.

4.3.1 Setting up Learning and Measurement

Before customer interviews could be performed, hypotheses that could be validated by talking to
customers were selected from the full set of hypotheses previously defined in Section 4.2.3. The
selected hypotheses were:

• Low response rate is a problem that survey consultants want to solve

• Slow feedback is a problem that survey consultants want to solve

• Not getting answers from young people, busy people and ”neutrals”, is a problem that survey
consultants want to solve

• Survey consultants can and want to buy and work with external tools that are not developed
in-house

• Survey consultants can and want to work independently with external tools

• Survey consultants can sell better response rates, ability to reach new customer segments and
faster feedback to their customers

To allow for reliable measurement, it was decided to perform meetings with at least ten consultancy
companies. A hypothesis would be deemed validated if confirmed in at least five of these meetings.
Doing this it was estimated that the collected data would be enough to validate or invalidate the
formulated hypotheses. This was documented in test cards, see Appendix Figures: B.1, B.2, B.3, B.4,
B.5 and B.6.

4.3.2 Building: Formulating questions

To verify the hypotheses, a necessary step is to talk to potential customers. A starting point was to
contact potential customers in the set customer segment, mid sized customer feedback consultants.
Moreover, a company landing page, as depicted in Appendix Figure A.14 and A.15, was built in order
to improve the appearance of the business from an outside perspective. It was believed that this would
improve the chances of getting in contact with the consultants.

25


The focus in the meetings was to ask questions in a way that would not a↵ect the answers in any way,
using the theory from The Mom Test as described in Section 3.3. Some of the questions that were
asked were: ”What are your problems today?”, ”How do you work today?”, ”Do you take in externally
developed products?”. For the entire questionnaire see Appendix Figure D.5.

4.3.3 Measuring and Learning

After meetings with ten companies, learning cards were used and five out of six hypotheses were
validated. An overview of the results, and what companies validated which hypothesis, can be seen in
Table 4.1.

Table 4.1: Table presenting the results from interviews with survey consultants. Green cells indicate
that the company validated the hypothesis, red cells indicate that a hypothesis was invalidated.

For the learning cards, see Appendix Figures C.1, C.2 C.3, C.4, C.5 and C.6. The hypothesis about
not reaching certain customer segments was not validated since it proved to vary a lot from type of
survey, company performing the survey and medium. Learnings from the meetings was that there
are some customer segments that are di�cult to reach but its not a general problem for the entire
customer segment. By identifying which customer segment this way of surveying reaches, data can be
derived from that for further segmenting of the market.

The other five hypotheses were validated because the problems actually existed and the consultants
actively tried and invested a lot of time and resources to solve these. For example some companies
described how they would pay respondents in order for them to answer surveys and thus increase
response rates. This was a strong indicator that the problems were important and that the survey
consultants would pay to get them solved. After the meetings, a couple of the companies were interested
in testing the product. According to the theory describes in The Mom Test, this is a great indicator of a
”next step” and that the meeting was successful (Fitzpatrick, 2013). It was validated that companies
are not reluctant to purchase external tools if it helps their business, and that they work with the
technical tools on their own.

4.3.4 Updated Business Model Canvas and Value Proposition Canvas

From the interviews with survey consultants some hypotheses were validated, and the Business Model
Canvas and Value Proposition Canvas were updated. The updated canvases can be seen in Figure 4.7
and Figure 4.8.

The updated Business Model Canvas includes some validated hypotheses. It had been validated
that:

• Survey consultants is a good customer segment

26


• Survey consultants can work independently with the product after start-up.

Figure 4.7: Updated Business Model Canvas after talking to survey consultants. Notes in red indicate
parts of the canvas that have not yet been validated, and notes in green indicate parts of the canvas
that have been validated.

The updated Value Proposition Canvas included a lot of validated hypotheses. It had been validated
that:

• Customer gains include faster responses

• Customer pains include low response rate and reaching certain customer segments

• The product will provide customers with good data for decisions

• The dashboard and API will work as the delivered products and services

4.4 Testing the existing product with survey consultants

When talking to survey consultants some of the pains and gains in the Value Proposition Canvas
had been validated, and there were no indications that conducting surveys in a conversational format
via SMS would not work. The next step was to test the product, to validate or invalidate the Gain
Creators and Pain Relievers in the Value Proposition Canvas.

Five of the interviewed survey consultants were interested in testing the product, and tests were
conducted with three of them. Due to the time constraints of the project, the other two tests were
not performed. This section describes how the tests were conducted and what the results of the tests
were, and discusses the learnings that came from testing the product.

27


Figure 4.8: Updated Value Proposition Canvas after talking to survey consultants. Notes in red
indicate parts of the canvas that have not yet been validated, and notes in green indicate parts of the
canvas that have been validated.

4.4.1 Setting up Learning and Measurement

Prior to setting up any tests, the initial hypotheses were revisited to see if they were still relevant, and
the hypotheses that could be validated or invalidated by testing the product with survey consultants
were selected. These hypotheses were:

• Surveys conducted in a conversational format over SMS give higher response rates than existing
surveying methods

• Surveys conducted over SMS give faster feedback than existing surveying methods

• Surveys conducted over SMS reaches certain customer segments that are di�cult to reach today

• Surveys conducted in a conversational format over SMS result in useful data

• Respondents do not have a negative attitude towards receiving surveys via SMS

• Surveys conducted in a conversational format over SMS works best for shorter surveys, ongoing
surveys and to get feedback from existing customers

In order to measure the result of a survey, three metrics were specified: engagement rate, conversion
rate and completion rate. Engagement rate was defined as the number of respondents that answered
any question divided by the total number of respondents. Conversion rate was defined as the number
of respondents that completed the entire questionnaire, including respondents that opted out of the
survey, divided by the total number of respondents. Completion rate was defined as the number of
respondents that completed the entire questionnaire, without opting out, divided by the total number
of respondents.

28


4.4.2 Building: Adding features to make testing easier

Since an MVP had already been built, the testing with survey consultants was initiated without doing
any further development on the product. During testing some features that could make testing easier
and less error prone, and also improve the quality of the surveys, were identified and implemented.
These features were:

• Statistics for conversion rate and engagement rate in the dashboard. Since the results of tests were
measured by comparing conversion and engagement rates, some way of viewing these rates for a
survey was needed. Prior to implementing this functionality in the dashboard the conversion and
engagement rates of a survey were calculated by exporting all answers to a spreadsheet and doing
the calculations there. This made it challenging to measure the results of A/B tests while the
tests were being conducted, so statistics for conversion and engagement rates for a survey were
added to the dashboard so that the results of tests could be tracked and compared in realtime
in an easy way.

• Ability to start sessions for multiple phone numbers at once. Previously the MVP only had
functionality for starting one session at a time. This became very time consuming for tests with
more than 100 respondents, and during one of the tests multiple sessions were started for a
single phone number by accident. It was thus deemed that adding the ability to start sessions
for multiple phone numbers at once was a feature worth implementing in the dashboard.

• Ability to send reminders to respondents who have not responded to the first question of a survey.
Sending reminders is one way to reduce dropout rates as described in Section 2.1. In order to
maximize the response rates of surveys, the ability to send reminders to all respondents who had
not responded to the first question of a survey was added to the dashboard.

• Functionality for manually tagging an answer with a correct analysis when a session has been
paused. As mentioned in the product description in Section 4.1, a session was paused when an
answer that did not correspond to any of the rules for the current question was received, or
when the routing algorithm was unsure of how to classify an answer, and paused sessions could
be unpaused by an administator through the dashboard by adding a new answer to the session.
This way of unpausing sessions added answers that were not given by the respondent, which
skewed the data from tests. Thus an easier way to unpause sessions was added to the dashboard,
where answers could be manually tagged with a correct analysis when the algorithm was unsure
of how to classify them.

• Automatically pausing a session when a counter-question is received and ability to manually send
a SMS to a respondent. It was noticed during tests that some respondents answered questions
with counter-questions, and that if the next question was sent without addressing this counter-
question the respondent would be likely to stop answering. Automatic pausing of such sessions,
and the ability to manually send a SMS to a respondent, was thus added in order to allow for a
human to answer the counter-question, so that the flow of the survey would not be interrupted.

• Sending a notification to Slack when a session has been paused. Previously, no notification was
sent when a session was paused. In order to find paused sessions, the dashboard needed to be
monitored manually, which was a very time consuming process, and it was easy to miss paused
sessions. In order to handle this more e�ciently, and not miss any paused sessions, functionality
was added for sending a notification to Surview’s Slack channel as soon as a session was paused.
This allowed for quick action as soon as a session was paused.

• Viewing the survey as a flowchart. The interface for creating surveys in the dashboard, as can be
seen in Appendix Figure A.5, did not give the survey designer the overview needed to determine
if a survey was set up correctly. To help with this, a feature that automatically generated a
flowchart for a survey was implemented. This made the process of setting up a new survey

29


less error prone, and helped ensure that surveys were set up in a correct way before they were
distributed to real respondents.

Some work on the styling of the dashboard was also done, so that the dashboard could be shown to
potential customers. The landing page of the dashboard after the changes can be seen in Figure 4.9.
The rest of the dashboard can be seen in Appendix Figure A.7 to A.13.

Figure 4.9: The landing page for the dashboard after implementing features for making testing easier.

4.4.3 Test with random people

Conducting tests with consultants was more time consuming than anticipated. When it had been
agreed upon that a test was to be conducted, the consultants needed to talk to their customers to
persuade them to try the product. This lead to long lead times, which made it hard to go through the
build-measure-learn feedback loop at a quick pace.

To mitigate this, a test with random people with a survey asking about Internet habits was conducted.
340 phone numbers to random people were collected, and the numbers were split up into two subgroups
of 170 phone numbers each. The first group got a conversational survey via SMS, and the other group
got an SMS with a link to a web form. The SMS for the conversational survey was sent from a regular
mobile phone number, and the SMS with a link to a web form was sent from an alphanumeric sender
ID that said ”Students”. The reason for not using an alphanumeric sender ID for the conversational
survey was that when using alphanumeric sender ID:s, it is not possible for the respondent to reply to
the SMS, as described in Section 2.4.2.

The questions in the conversational survey and the web form were the same, but the questions in the
conversational survey had been modified to fit the conversational format. A flowchart showing the
conversational survey can be seen in Appendix Figure D.1.

The hypotheses tested in this test were:

• Surveys conducted in a conversational format over SMS give higher response rates than existing
surveying methods

30


• Surveys conducted in a conversational format over SMS result in useful data

• Respondents do not have a negative attitude towards receiving surveys via SMS

It was also tested whether conversational surveys via SMS could be used for market research, where
no prior relationship to the respondent was in place. A test card that represents this test can be found
in Appendix Figure B.7.

4.4.3.1 Measuring

Test # respondents Engagement Rate Conversion Rate Completion Rate
Conversational SMS 170 st 18 % 13 % 10 %
SMS with link 170 st N/A 9 % 9 %

Table 4.2: Results from test with random people

The result of the test can be seen in Table 4.2. 16 out of 170 people responded to the survey with a link
to a web form and 23 out of 170 answered to the conversation based survey. Thus the completion rate
was 9.4 percent for the link survey and 10 percent for the conversational survey. The conversational
survey had a engagement rate of 18 percent and conversion rate of 13 percent. Many of the respondents
that got the conversational survey responded with counter questions, asking who the sender was and
why they had been chosen to participate in the survey, but other than that there were no noticeable
di↵erence in the quality of answers collected in the surveys.

4.4.3.2 Learning

Even if the results indicate that a survey done in a conversational format may lead to higher completion
rates, the results from this test were not significant enough to validate any hypotheses. The tests were
also not equal in their setup, since the conversational survey was sent from a regular mobile phone
number and the SMS with a link to a web form was sent from an alphanumeric sender ID. This may
have influenced the results of the survey, since it is possible that the alphanumeric sender ID instilled a
higher sense of professionalism, which may have made respondents more inclined to answer the survey.
For the same reason, not having an alphanumeric sender ID for the conversational survey may have
been what caused many respondents to be suspicious of the survey.

The answers received from the conversational survey during the test were of high quality, but since
only 17 respondents completed the entire survey, more testing needed to be done in order to validate
that surveys conducted in a conversational format over SMS result in useful data.

There were some indications that the respondents of the conversational survey had a negative attitude
towards receiving surveys via SMS. Many of the respondents of the conversational survey asked counter
questions, wanting to know who the sender was and why they had been selected for the survey. Though
it was hypothesised that the negativity was not caused by the survey itself, but rather had to do with
not having a pre-existing relationship with the entity conducting the survey, which in this case was
students at Chalmers. It was thus believed that conversation surveys most likely work better for
surveys where respondents have a pre-existing relationship with the entity surveying them, and thus
have a reason to be contacted.

The learning card for the test with random people can be found in Appendix Figure C.7.

31


4.4.4 Test with Company Z

After some waiting due to the long lead times, a test was conducted with Company Z. Company
Z is a market research consultancy that showed interest in the idea of using chatbot technology for
conducting surveys, and they wanted to try the concept on their customers. The aim of this test was
to compare surveys conducted in a conversational format over SMS with the tool that Company Z
usually used for surveys, which was a tool for sending emails with a link to a web form.

The questions in the survey were about Company Z’s new website that had recently been released. 92
phone numbers to customers of Company Z were received and, just as in the test with random people,
the numbers were split into two subgroups, this time with 46 phone numbers each. The first group
got a regular email survey, distributed with Company Z’s existing tool, and the second group got a
conversational survey via SMS. The questions in the conversational survey and the email survey were
the same, but the questions in the conversational survey had been modified to fit the conversational
format in the same way as in the questions in the test with random people. A flowchart showing the
conversational survey can be seen in Appendix Figure D.2.

The hypotheses tested in this test were:

• Surveys conducted in a conversational format over SMS give higher response rates than existing
surveying methods

• Surveys conducted over SMS give faster feedback than existing surveying methods

• Surveys conducted in a conversational format over SMS result in useful data

• Respondents do n