Developing an Evaluation Framework for Early Stage Technologies A Case Study in the Aerospace Industry Master’s thesis in Management and Economics of Innovation ALEXANDER SÄLL ISAK LARSSON DEPARTMENT OF TECHNOLOGY MANAGEMENT AND ECONOMICS DIVISION OF INNOVATION AND R&D MANAGEMENT CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2025 www.chalmers.se www.chalmers.se Master’s thesis 2025 Developing an Evaluation Framework for Early Stage Technologies A Case Study in the Aerospace Industry ALEXANDER SÄLL ISAK LARSSON Department of Technology Management and Economics Division of Innovation and R&D Management Chalmers University of Technology Gothenburg, Sweden 2025 Developing an Evaluation Framework for Early Stage Technologies A Case Study in the Aerospace Industry ALEXANDER SÄLL ISAK LARSSON © ALEXANDER SÄLL, 2025. © ISAK LARSSON, 2025. Supervisor: Lars Trygg, Department of Innovation and R&D Management Examiner: Lars Trygg, Department of Innovation and R&D Managemen Master’s Thesis 2025 Department of Innovation and R&D Managment Chalmers University of Technology SE-412 96 Gothenburg Telephone +46 31 772 1000 Typeset in LATEX, template by Kyriaki Antoniadou-Plytaria Printed by Chalmers Reproservice Gothenburg, Sweden 2025 iv Developing an Evalution Framework for Early Stage Technologies A Case Study in the Aerospace Industry ALEXANDER SÄLL ISAK LARSSON Department of Innovation and R&D Management Chalmers University of Technology Abstract In today’s rapidly evolving business landscape, innovation and R&D activities are crucial for competitive advantage. Many firms struggle with their innovation perfor- mance and especially at the early decisions, which often decide the outcome. This thesis delves into the evaluation of early stage technologies at a tier 1 supplier in the aerospace industry, focusing on the Research & Technology division. Today’s evaluation process is generally effective, but fragmented and lacks a structured ap- proach, clear tools, and a decision-making process. Through a case study approach, the thesis utilized a literature review, internal in- terviews, and workshops. First, with data obtained from the literature review and interviews, the thesis investigated what criteria are necessary when evaluating early stage aerospace engine technologies. Secondly, drawing from the fields of New Prod- uct Development, tools and methods were assessed and tested within Research & Technology through workshops. Lastly, based on the findings a systematic process was developed when evaluating early stage technologies. The research identified 22 criteria distributed within 4 overarching areas for a com- prehensive evaluation. A modified scoring model was deemed the most relevant utilizing rating scales and which stimulates discussion. Evaluators conduct indi- vidual evaluations of the technology before being merged with all evaluations. If large deviations between the evaluators occur, discussions are launched. Further, to enhance flexibility the ability to assign weights were introduced as well as visual representations of technologies currently in the portfolio to maintain a balanced and strategically aligned portfolio. By utilizing the framework proposed in this thesis the company can incorporate a systematic way to assess early stage technologies individually and across the portfolio. Keywords: Technology Evaluation, Portfolio Management, Scoring Model v Acknowledgements This master’s thesis was conducted during the spring of 2025 at the Department of Technology Management and Economics, under the division of Entrepreneurship and Strategy at Chalmers University of Technology. The thesis project was carried out in collaboration with a company within the Aerospace industry, hereby referred to as the Company for anonymity. We would like to express our heartfelt gratitude to our academic supervisor and examiner, Lars Trygg at Chalmers University of Technology. Your feedback, en- couragement, guidance, and insights throughout this process have been invaluable. We would also like to extend our sincere thanks to our supervisor at the Company, Gustav, for your support and expertise. Your insights and continuous engagement have been instrumental in shaping the direction and relevance of our research. The time and resources you provided were essential to the successful completion of this thesis. Additionally, we wish to thank all the interviewees who generously shared their time, experiences, and perspectives with us. Your valuable input and willingness to contribute enabled us to explore the subject matter in depth. This research would not have been possible without your participation. Alexander Säll & Isak Larsson, Gothenburg, May 2025 vii List of Acronyms Below is the list of acronyms that have been used throughout this thesis listed in alphabetical order: AHP Analytical Hierarchical Process ECV Expected Commercial Value NPD New Product Development RRSP Risk- and Revenue-Sharing Partnership TRL Technology Readiness Level ix Contents 1 Introduction 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Research Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Purpose and Research Questions . . . . . . . . . . . . . . . . . . . . . 4 1.4 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 Frame of Reference 7 2.1 Picking the Winners . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1 Benefit Measurement Techniques . . . . . . . . . . . . . . . . 8 2.1.2 Economic Models . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.3 Portfolio Selection and Management Methods . . . . . . . . . 11 2.1.3.1 Goal 1: Value Maximization . . . . . . . . . . . . . . 12 2.1.3.2 Goal 2: Balance . . . . . . . . . . . . . . . . . . . . 13 2.1.3.3 Goal 3: Strategic Alignment . . . . . . . . . . . . . . 14 2.2 Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2.1 Financial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2.2 Technical Feasibility . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.3 Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.4 Market Attractiveness . . . . . . . . . . . . . . . . . . . . . . 18 2.2.5 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.6 Intellectual Property (IP) . . . . . . . . . . . . . . . . . . . . 20 2.2.7 Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.8 Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.9 Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.10 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.2.11 Other Considerations . . . . . . . . . . . . . . . . . . . . . . . 22 2.3 Synthesizing the Findings . . . . . . . . . . . . . . . . . . . . . . . . 23 3 Method 25 3.1 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2 Research Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.1 Exploratory Phase . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.2 Data Collection Phase . . . . . . . . . . . . . . . . . . . . . . 27 3.3 Selection of Interviewees . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.4 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.4.1 Data Compilation and Derivation of Criteria . . . . . . . . . . 31 xi Contents 3.5 Research Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.6 Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4 Working with Technologies and Projects at the Company Today 35 4.1 Technological Maturity and Strategic Partnerships in Aerospace . . . 35 4.1.1 Technology Readiness Level . . . . . . . . . . . . . . . . . . . 35 4.1.2 Risk- and Revenue-Sharing Partnerships . . . . . . . . . . . . 37 4.2 Organizational Overview . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.2.1 Understanding Technologies at Company . . . . . . . . . . . . 42 4.2.2 Evaluating Technologies at the Company . . . . . . . . . . . . 43 5 Empirical Findings 45 5.1 Criteria to Consider When Evaluating Technologies . . . . . . . . . . 45 5.1.1 Financial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.1.2 Technical Feasibility . . . . . . . . . . . . . . . . . . . . . . . 47 5.1.3 Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.1.4 Intellectual Property (IP) . . . . . . . . . . . . . . . . . . . . 51 5.1.5 Support from Management and Stakeholders . . . . . . . . . . 51 5.1.6 Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.1.7 Funding and Research Collaborations (Network) . . . . . . . . 53 5.1.8 Alignment with Sustainability . . . . . . . . . . . . . . . . . . 54 5.2 Stakeholder Requirements for a Technology Evaluation Framework . . 55 6 Analysis 57 6.1 Analysis of Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . 57 6.1.1 Bang for the Buck . . . . . . . . . . . . . . . . . . . . . . . . 58 6.1.2 Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.1.3 Strategic Alignment . . . . . . . . . . . . . . . . . . . . . . . 61 6.1.4 Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 6.2 Identifying Tools for Technology Evaluation . . . . . . . . . . . . . . 65 6.2.1 Benefit Measurement Techniques . . . . . . . . . . . . . . . . 65 6.2.2 Economic Models . . . . . . . . . . . . . . . . . . . . . . . . . 66 6.2.3 Portfolio Selection and Management . . . . . . . . . . . . . . 67 6.3 The Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 6.3.1 Framework Design Considerations . . . . . . . . . . . . . . . . 69 6.3.2 Technology Evaluation . . . . . . . . . . . . . . . . . . . . . . 70 6.3.2.1 Scoring Model . . . . . . . . . . . . . . . . . . . . . 70 6.3.2.2 Basis for Discussion . . . . . . . . . . . . . . . . . . 70 6.3.3 Portfolio Management . . . . . . . . . . . . . . . . . . . . . . 72 6.3.3.1 Portfolio Input . . . . . . . . . . . . . . . . . . . . . 72 6.3.3.2 Portfolio Overview . . . . . . . . . . . . . . . . . . . 72 6.3.4 Additional Features . . . . . . . . . . . . . . . . . . . . . . . . 73 6.3.4.1 Weighting . . . . . . . . . . . . . . . . . . . . . . . . 73 6.3.4.2 Archiving Technology Scores . . . . . . . . . . . . . 74 7 Discussion 77 7.1 RQ1: Developing Evaluation Criteria for Early-Stage Technologies . . 77 xii Contents 7.2 RQ2: Risk- and Uncertainty Management in Aerospace Technologies . 79 7.3 RQ3: Designing a Framework for Technology Evaluation and Portfo- lio Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7.4 Limitations and Future Research . . . . . . . . . . . . . . . . . . . . 83 8 Conclusion 85 Bibliography 87 A Appendix: Interview Guide for Semi-Structured Interviews I xiii Contents xiv List of Figures 1.1 Aerospace Engine Industry Network (Prencipe, 2004) - Company in red box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1 Example of an Analytical Hierarchical Process . . . . . . . . . . . . . 9 2.2 The scores from each evaluator is displayed on a large screen at the decision meeting (Cooper & Sommer, 2023). . . . . . . . . . . . . . . 10 2.3 Two pie charts illustrated. The first pie chart (a) showcases allocation per market. The second pie chart (b) demonstrates allocation per technology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4 Illustration of a three-dimensional bubble diagram with X-values, Y- values, and size of the bubble. . . . . . . . . . . . . . . . . . . . . . . 14 2.5 Illustration of strategic buckets. . . . . . . . . . . . . . . . . . . . . . 15 3.1 Overview of the research process. . . . . . . . . . . . . . . . . . . . . 26 3.2 Grounded Theory model for data collection and data analysis . . . . 32 4.1 The incline wave principle for projects in the aerospace industry (“In- novair”, 2014). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.2 Cumulative cash flow per year for a partner in a RRSP within the aerospace industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.3 Engine RRSP lifespan . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.4 Organizational overview. . . . . . . . . . . . . . . . . . . . . . . . . . 40 4.5 Overview of the governance structure of portfolios and projects . . . . 41 4.6 Overview of the flow of decisions from R&T portfolio to Capability or Demonstrator portfolio . . . . . . . . . . . . . . . . . . . . . . . . 42 6.1 The Framework and its Constituing Components . . . . . . . . . . . 69 6.2 Portfolio Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.3 Weighting feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.4 Technology Archive . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 xv List of Figures xvi List of Tables 2.1 Synthesized findings on criteria mentions from the literature study . . 24 3.1 Keywords used during literature review . . . . . . . . . . . . . . . . . 27 3.2 Overview of interviews . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3 Overview of workshop attendees . . . . . . . . . . . . . . . . . . . . . 30 4.1 Technology readiness levels and their definitions(NASA TRA Best Practices Guide, n.d.) . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.1 Criteria that emerged from the empirical study . . . . . . . . . . . . 46 6.1 Bang for the Buck Criteria . . . . . . . . . . . . . . . . . . . . . . . . 59 6.2 Feasibility Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6.3 Strategic Alignment Criteria . . . . . . . . . . . . . . . . . . . . . . . 63 6.4 Risk Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.6 Basis for Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 6.7 Portfolio input table . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 6.5 Technology evaluation input table . . . . . . . . . . . . . . . . . . . . 76 7.1 A complete synthesis of the evaluation criteria . . . . . . . . . . . . . 79 xvii List of Tables xviii 1 Introduction This chapter provides an introduction to the thesis and its contextual foundation. It presents the background to the study, its overall purpose, and outlines the research questions guiding the investigation. Finally, the chapter describes the scope and delimitations of the research. 1.1 Background R&D is a crucial driver of innovation and competitive advantage in today’s rapidly evolving business landscape. New product development (NPD) and innovation for long-term success is critical but firms continue to struggle with finding the right strategy, achieving a balanced portfolio of projects, and the process model to deliver innovation performance (Knudsen et al., 2023). 75% of new products ultimately fail in the marketplace (Cooper, 2001) and in a study by Cooper and Kleinschmidt (1995) it was revealed that 55% of companies report not meeting sales or product performance objectives in their efforts. NPD resources are often limited and too valuable to be spent on the wrong projects (Cooper, 2001). With scarce resources, choosing the right NPD efforts involves con- centrating on the truly deserving products (Cooper, 2001). In adjacent fields of study, Savoia (2011) affirms that many are focused on doing it right, when they instead should focus on doing it right. Cooper (2001) argues that companies that succeed in optimizing their new product investments, including achieving an ideal balance of projects and building a portfolio that supports the business strategy, will win in the long run. Meanwhile, the early stages of the NPD process represent a critical point of vulnera- bility where initial choices lock in trajectories. Cooper (2001) finds that the greatest differences between winners and losers in NPD lies in these first few plays of the game that seem to decide the outcome. At many firms, these early decisions are the weakest link in their NPD process (Cooper, 2001). Common problems in NPD is the gap between required resources and available resources for the development (Cooper & Edgett, 2003). This gap frequently leads to longer than intended time-to-market (TTM), suboptimal performance, and imbalance in portfolios (Cooper & Edgett, 2003). 1 1. Introduction There exists different approaches to evaluating new products. Cooper (2001) states three main approaches that aim to enhance the NPD process. The first is benefit measurement techniques emphasizing methods that utilize subjective assessments of strategic factors. The second is traditional economic models such as calculating the expected payback period or the net present value (NPV). The last approach aims to attain a portfolio that maximizes the value, is balanced, and aligned to the firm’s strategy. 1.2 Research Setting The study is conducted in collaboration with a tier 1 supplier specializing in airplane structures and engine structures. The Company supplies products and services to a broad range of customers, including commercial and military aircraft manufactur- ers, engine contractors, and other Tier 1 suppliers. The Company operates across multiple nations and their components are featured on a majority of all flights that take off and land each day. As of 2025, the Company holds partnerships in nu- merous Risk and Revenue Sharing Programs (RRSPs), through which it maintains a minority share of ownership in its active programs. A significant portion of the Company’s cash flow is derived from aftermarket services, including repairs, part replacements, and maintenance operations. The Company provides advanced aerospace components and technical solutions to two different markets; Engines and Structure. The study will take place within the R&D-division of the Engine divsion at the Company. The R&D-division is facili- tated by a division called Research & Technology (R&T). R&T is divided into three sub-divisions. In this report, these sub-divisions will be referred to as R&T 1, R&T 2, and R&T 3 for confidentiality purposes. The role of aircraft engine OEMs can be likened with that of system integrators, coordinating complex networks of suppliers responsible for significant portions of engine design and development. Suppliers such as the Company have seen their roles evolve to include more advanced design responsibilities (Almeida, 2001), often taking full ownership of specific components or subsystems. This shift also entails greater involvement in the development of new and innovative technologies. How- ever, as Högman and Berglund (2007) point out, while global trends and industry expectations are typically well defined at a strategic level, the translation of these expectations into actionable technology development at the supplier level remains ambiguous. Unlike the conventional tiered buyer-seller model, the aircraft engine industry oper- ates within a network of six interdependent groups: airlines, airframers, certification agencies and professional bodies, government-funded laboratories and universities, risk and revenue-sharing partners, and suppliers as presented in Figure 1.1 (Prencipe, 2004). Much like in traditional buyer-supplier relationships though, the strategic di- rection of a supplier in partnership programs is dependent on the decisions made by 2 1. Introduction their customers. This in combination with long development timelines entails diffi- culties in deciding which strategic initiatives to prioritize. However, in stark contrast to traditional buyer-supplier relationships, the OEM rarely provides explicit spec- ifications for new products, but rather expects the supplier to bring forward new technology which results in a certain performance on the end product. This mix between technology-push and market-pull puts responsibility on the supplier to col- lect and compile information to align technology projects with market requirements. With new technologies being important means for seizing new business opportuni- ties and responding to the strategies of customers and partners, there is a need to systematically manage the uncertainty in new technology development. Figure 1.1: Aerospace Engine Industry Network (Prencipe, 2004) - Company in red box Financial constraints limit both the number of programs and how big of a share the company can pursue. Restricted access to skilled personnel and expertise poses a challenge, making it difficult to manage multiple new programs simultaneously. Meanwhile, all programs are not successful and the Company aims to participate in the programs that are deemed the most strategic, valuable, and with the highest probability of success. With both development timelines and payback periods being exceptionally long in the aerospace industry, evaluating whether to proceed with a new technology becomes a critical decision in the new product development phase. The current decision-making process at the Company is generally effective, but fragmented and lacks a structured approach. Further, there are no clear tools that are utilized when comparing and evaluating different technologies in the early stages of devel- opment.With no clear decision-making process, it is challenging to evaluate and prioritize technologies against each other. Given the industry’s long development cycles, systematically exploring tomorrow’s technologies is essential for securing fu- ture revenue streams and ensuring the long term sustainability for the Company. 3 1. Introduction 1.3 Purpose and Research Questions The purpose of this thesis is to develop a comprehensive and systematic approach for the early-stage evaluation of aerospace engine R&D technologies. To achieve this, the research integrates three key components into a coherent and structured evalu- ation framework. First, a set of criteria will be identified to enable a comprehensive assessment of relevant technologies. These criteria will then be embedded within an evaluation tool or method that serves as the operational platform for applying the framework. Finally, the tool and criteria will be supported by a structured process to ensure consistency, robustness, and practical applicability. The resulting frame- work aims to support decision-makers in analyzing and evaluating new technologies under conditions of uncertainty. In order to guide us in the fulfillment of this case study’s purpose, three research questions have been formulated which are intended to be resolved by the end of this study. These research questions are presented below. RQ1: What financial, technical, strategical, and other criteria are essential for evaluating early stage aerospace engine R&D technologies? RQ2: What tools or methods are the most suitable for managing risks and uncer- tainties in the evaluation of early-stage aerospace engine R&D technologies? RQ3: How can these insights be integrated into a systematic framework to guide the evaluation process in aerospace engine R&D technologies? 1.4 Delimitations The research contains a few delimitations which limit the scope and depth of the analysis. First of all, the research is exclusively applied to a specific company within the aerospace industry and the business area relevant to aero-engine systems and engine technologies. The purpose of delimiting the study to one business area is to reach a greater depth in the study. Additionally, the study does not aim for gener- alizability in the results, but rather an adaptation to the specific case company and its environment. The study will be conducted over a four month period. This leads to the research having to be adapted to these time-constraints, meaning that we will be forced to consider practical limitations with regards to the amount of data that could be collected and analyzed in the relatively short time-period. This might lead to a narrower than ideal scope in the data collected and could potentially mean that a theory will need to be formulated before the research has reached the point of theoretical saturation. 4 1. Introduction Lastly, as parts of the Company’s operations contain confidential information, some results are excluded from the final study. This could potentially compromise the transferability of the study. However, the intention is that no confidential informa- tion will be crucial for being able to fulfill the purpose of the study and answer the research questions. 5 1. Introduction 6 2 Frame of Reference This chapter represents the findings from established literature and presents a frame of reference in order to answer the research questions. The selected literature draws primarily from the fields of innovation management and NPD. These fields have been chosen because they provide a systematic and well-validated foundation for evaluating R&D initiatives, which are typically characterized by high uncertainty, long development timelines, and constrained resources. The literature used in the thesis has broad applicability and a practical orientation. When evaluat- ing a technology thoroughly, one must also consider the feasibility and execution of the associated projects needed to develop the technology. Thus, the evaluation must not only consider the technical and commercial potential of a technology but also the feasibility, resource requirements, and strategic fit of the associated development projects. Technology evaluation means one has to make a decision on what technology to invest in given limited resources. Three approaches to make sure to invest in the right initiatives will be introduced in this chapter; benefit measurement techniques, economic models, and portfolio selection and management. Lastly, several key technology evaluation criteria commonly cited in the literature are presented. These criteria, identified by multiple scholars, are regarded as essential considerations in the evaluation process. 2.1 Picking the Winners Cooper (2001), one of the most cited authors within the field of product portfolio management, mentions two fundamental ways to win at product innovation; doing projects right and doing the right projects. Within the context of this study, doing the right projects and choosing the right technologies to invest in is of utmost im- portance. New product development resources are often limited and too valuable to be spent on the wrong projects according to Cooper (2001). He also highlights that most businesses face more opportunities for new product development than they have re- sources available to bring them to market. Choosing projects is about concentrating scarce resources on the truly deserving products. Cooper (2001) argues that those 7 2. Frame of Reference companies which effectively optimize their new product investments — including achieving an ideal balance of projects and building a portfolio that supports the business strategy — will win in the long run. Cooper (2001) states that the three main approaches to project evaluation and selection include benefit measurement techniques, economic models, and portfolio selection and management methods. 2.1.1 Benefit Measurement Techniques As stated by Cooper (2001), benefit measurement techniques depend on a knowl- edgeable and well-informed management team to evaluate projects across a range of characteristics. Rather than relying on traditional economic metrics, these meth- ods emphasize subjective assessments of strategic factors, such as alignment with corporate objectives, potential for competitive advantage, and the attractiveness of the target market (Cooper, 2001). Benefit measurement techniques acknowledge the limited availability of concrete fi- nancial data during the early stages of project evaluation. According to Cooper (2001), these methods are widely used and are specifically designed to incorporate the subjective input of management, often through tools such as comparative ap- proaches, simple checklists, and scoring models. Comparative approaches, such as the Analytical Hierarchy Process (AHP) devel- oped by Saaty in 1980, provide structured support for multi-criteria decision-making. AHP integrates both qualitative and quantitative factors by breaking down complex problems into a hierarchy of criteria, enabling pairwise comparisons and priority weighting. As per Palcic (2009), AHP as a tool is especially applicable when a certain degree of expert judgement is required. In many situations, particularly in project selection, there is a need to evaluate attributes that cannot simply be quan- tified or put into monetary terms. These might include risk, technological feasibility, commercial value, and other factors that are subject to a high degree of uncertainty. In Figure 2.1, a simple AHP hierarchy is presented with an objective, underlying criteria at level 2 and the alternatives to choose from at level 3. Saaty (1987) outlines three key principles in the AHP method. The decomposition principle involves breaking down a complex decision problem into a hierarchical structure, starting from broad criteria and dividing them into more specific sub- criteria, meaning that in Figure 2.1, an extra level can be added between level 2 and 3. The comparative judgment principle allows decision-makers to evaluate ele- ments through pairwise comparisons to determine their relative importance. Finally, the synthesis of priorities combines these judgments by calculating weighted scores across the hierarchy to derive overall priorities for each alternative. This struc- tured approach helps decision-makers evaluate trade-offs and make more objective, transparent choices. Additionally the process enables the evaluation and analysis of multiple criteria simultaneously. 8 2. Frame of Reference Figure 2.1: Example of an Analytical Hierarchical Process The second benefit measurement technique is checklists and is described by Cooper (2001) as one of the simplest methods for evaluating new product proposals. Check- lists can be used to assess projects through a series of qualitative yes/no questions. A project may be required to meet all or a subset of these criteria to pass the eval- uation. Checklists are usually used as a tool to be able to make a go/no-go-decision for single projects but not for rating different projects against each other. In a study by Cooper et al. (2002) the most popular selection criteria to use in the checklist were strategic fit, financial reward, risk, probability of success, and the business’s technological and commercial capabilities to undertake the project. Cooper (2001) finds that checklists are effective for evaluating new products, especially in the early stages, and provide a practical approach to assessment. He further states that their implementation is simple, as they incorporate multiple criteria, ensuring that key factors are not overlooked. This method promotes a consistent and systematic eval- uation process while requiring minimal financial input and avoiding dependence on a single criterion. However, checklists have certain limitations. The selection of questions is subjective, relying on the developers’ best judgment of the most impor- tant factors to consider. Additionally, the method does not account for the varying significance of different elements within the list. Cooper (2001) also points out that the responses provided may be subjective and may not always reflect careful con- sideration. This leads to the third benefit measurement technique, the scoring model. A scoring model serves as an extension of the checklist by allowing projects to be evaluated on a range of criteria using rating scales rather than simple yes/no answers. The scales often range from 0-10 or 1-5. These ratings are then combined to produce an overall project score. This approach addresses several limitations of checklists, such as expanding the range of possible responses, acknowledging that some criteria carry greater importance than others, and generating a single project score, often referred to as the project attractiveness score which can be used to rank projects or products against each other or against a minimal acceptable score (Cooper, 2001). In the scoring model, stakeholders independently evaluate the project, and their ratings are consolidated into a single score (Cooper, 2001). 9 2. Frame of Reference In a recent article, Cooper and Sommer (2023) present a model that has proven to be both useful and predictive for evaluation of NPD. The model resembles the scoring model to a high degree but it’s also a decision-making system designed to stimulate critical thinking. Through empirical research, Cooper and Sommer (2023) found five factors that proved to be successful in evaluation; mission and strategy, customer energizer, synergy, technical feasibility, and reward versus risk. According to Cooper and Sommer (2023) users of the model highlighted that, while the overall project value score is helpful for prioritization, its true strength lies in the behavioral aspect. The process brings together senior stakeholders to collaboratively assess the project and through projecting each stakeholders rating on a large screen, are forced to discuss key questions (Cooper & Sommer, 2023). Figure 2.2 depicts how the projection can look. Each evaluator has scored the project on criteria within each factor and yields an overall score. If there exists a large enough deviation between the scores, the highest and lowest individual evaluator must engage in debate and align their perspectives. Ultimately the goal is to reach a well-informed decision (Cooper & Sommer, 2023). Figure 2.2: The scores from each evaluator is displayed on a large screen at the decision meeting (Cooper & Sommer, 2023). 2.1.2 Economic Models Economic models treat project evaluation much like a conventional investment de- cision. They are familiar to managers, and they are accepted for other types of in- vestment analysis in businesses, such as capital expenditure (Cooper, 2001). Cooper (2001) also states that the two most popular financial methods for new products are payback period and discounted cash flow (DCF), which includes NPV. According to Cooper et al. (2001), financial methods were found to be among the most popular methods when evaluating new products. The payback period of an investment is a simple financial metric that evaluates the time required to recoup the cost of investment (“Corporate Finance Institute”, n.d.). Cooper (2001) presents three different measures of time; cycle time as the time from project initiation to market launch, payback period as the time from 10 2. Frame of Reference launch date to the full recovery of initial expenditures, and break-even time as the time from project initiation to when all expenditures are recovered. Cooper (2001) acknowledges that these metrics — just as financial tools, have advantages such as capturing both risk and return, having a faster payback time which means a higher return on investment and a lower risk, and use a cash flow approach and hence avoid accounting method disputes. An NPV analysis involves projecting annual cash flows and discounting them us- ing the required rate of return, ultimately yielding the investment’s present value (Gallo, 2014). If the yielding NPV is positive, the project has cleared the discount rate. According to Cooper (2001), a DCF approach has certain advantages such as recognizing that money has time value and that it tends to place less emphasis on cash flow projects that are many years into the future. A third financial option mentioned by Cooper (2001) is the expected commercial value (ECV) of a project. The ECV utilizes an options pricing theory, meaning it realizes that new product projects are investments made in increments (Cooper, 2001). This stands in contrast to the NPV and payback period methods assuming an all-or-nothing decision situation (Cooper, 2001). A project can have different outcomes and each scenario is assigned a possibility with a subsequent project value. The ECV is calculated by multiplying each possibility with the situation’s value and adding the results (Cooper & Edgett, 2003). The most difficult project selection decisions occur early in the process when limited information is available, making traditional financial methods less effective due to their reliance on precise data Cooper (2001). Conducting financial analysis too soon tends to favor only low-risk projects, as probability-adjusted methods often under- value high-risk opportunities (Cooper, 2001). According to Cooper et al. (2001), firms that rely heavily on financial tools tend to perform worse, as the complex- ity of these methods often surpasses the quality of the available data. In contrast, top-performing companies place greater emphasis on non-financial approaches like strategic alignment and scoring models. However, Cooper (2001) states that eco- nomic models are still a powerful and useful tool in project evaluation, provided it is used at the right time and for the appropriate project type. 2.1.3 Portfolio Selection and Management Methods Patterson (2004) outlines portfolio management as a process involving portfolio as- sessment, resource management, and portfolio review, all aimed at enhancing a com- pany’s long-term performance. Souder (1994) emphasize five factors that need to be taken into consideration for effective project and portfolio management. Firstly, the organizational overarching strategic objectives and organizational mission must be reflected in the decision model. Cooper et al (2002) found in a study that busi- nesses that prioritize strategic alignment in their evaluation methods perform better than those that don’t. Secondly, Souder (1994) states that the model must allow a comparison between different types of projects. Thirdly, the model must be modifi- able and accommodate future potential adjustments. Fourthly, the model should be easy to use by people in all areas of the organization. And lastly, the model should not be overly time consuming and expensive (Souder, 1994). Cooper (2001) defines 11 2. Frame of Reference portfolio management as "A dynamic process, in which a business’s list of active new product (and development) projects is constantly updated and reviewed. In this process, new projects are evaluated, selected, and prioritized; existing projects may be accelerated, killed, or deprioritized; and resources are allocated and reallocated to active projects.” Chien (2002) notes that most studies on portfolio selection evaluate projects in iso- lation before aggregating them into an R&D portfolio. However, he argues that assembling a set of individually strong projects does not automatically lead to an optimal portfolio. Supporting this view, Schilling and Hill (1998) emphasizes that new product development should be managed as a balanced portfolio, comprising projects at various stages of development. Cooper et al. (2001) describe that portfolio management offers several key benefits, such as establishing a common foundation for decision-making, prioritizing major and breakthrough initiatives, enhancing the strategic alignment of the portfolio, balancing short- and long-term projects, fostering unified support and stakeholder buy-in, and strengthening overall strategic planning (Cooper et al., 2001). Further, Cooper et al. (2001) states that top performers in portfolio management distinguish themselves from weaker counterparts by employing a clearly defined and consistently implemented approach. Their methodology is explicitly structured, governed by clear rules and procedures that facilitate systematic decision-making. The method is supported by the management and it treats all projects as a portfolio — consid- ering them not in isolation but rather in aggregation (Cooper et al., 2001). In a study by Cooper et al. (2002), the authors analyze the usefulness and applica- bility of different portfolio management methods employed by businesses in a diverse set of industries. Furthermore, it was discovered that management support for an explicit portfolio method with clear and simple procedures leads to better overall portfolio performance. Chien (2002) second this notion by underscoring manager perception of the project selection method as a crucial factor for the practical adop- tion and application of the chosen methods. Cooper (2001) finds three goals of portfolio management; value maximization, bal- ance, and strategic alignment. Different tools tend to be best suited for achieving each goal. 2.1.3.1 Goal 1: Value Maximization The first goal of portfolio management is value maximization. Cooper et al. (2002) explain that the objective is to select new product projects in a way that maximizes the total value or commercial worth of all active projects in the firm’s pipeline, based on a specific business goal, such as profitability. Cooper (2001) explains that this is typically achieved using tools and methods described in benefit measurements and economic models such as scoring models, NPV, or ECV. 12 2. Frame of Reference 2.1.3.2 Goal 2: Balance Balance is mentioned by multiple authors as important in portfolio management. Cooper et al. (2001) finds a balance between factors such as time and risk as critical. Archer and Ghasemzadeh (1999) believe that a balance between different risk levels and project sizes are of importance when selecting projects. Also Schilling and Hill (1998) points that new product development must consist of a balanced portfolio that contains projects at different stages in development. Cooper and Edgett (2003) conducted a study that showcased that all companies have must-do projects such as responding to major customer requests, keeping the product line up-to-date, or to just fix a problem, but they often tend to take up the majority of the develop- ment budget. A majority of the studied firms indicated a poorly balanced portfolio, heavily skewed to the short-term rather than long-term (Cooper & Edgett, 2003). Achieving a desired level of portfolio balance requires considering several key di- mensions, such as long-term versus short-term investments, high-risk versus low- risk initiatives, and the distribution across markets, technologies, and project types (Cooper, 2001). Visual tools like portfolio maps, bubble diagrams, and pie charts can effectively illustrate this balance (Cooper, 2001). Pie charts visually represent spending by dividing the whole into slices, illustrating the allocation of resources or the number of projects across categories, product lines, or market segments (Cooper et al., 2001). Figure 2.3: Two pie charts illustrated. The first pie chart (a) showcases allocation per market. The second pie chart (b) demonstrates allocation per technology. Bubble diagrams present projects as bubbles on a two-dimensional grid, as shown in Figure 2.4. While the axes can vary, the most common version is the risk- reward diagram, where each project’s probability of success is plotted against the project’s NPV. The bubble’s sizes can depict their weighting, a common weight is each project’s budget. With the aid of this visualization, decision-makers can seek to identify an appropriate balance of spending and projects across different risk-reward levels (Cooper et al., 2001). Cooper et al. (2001) explain that unlike the maximization tools described earlier, 13 2. Frame of Reference Figure 2.4: Illustration of a three-dimensional bubble diagram with X-values, Y- values, and size of the bubble. bubble diagrams and pie charts are not decision-making models but visual tools for displaying information. They illustrate the current state of the portfolio and how resources are currently allocated, the “what is”. These charts serve as a valuable starting point for discussions around the “what should be”, or how resources should be distributed moving forward. 2.1.3.3 Goal 3: Strategic Alignment Nickols (2012) describes strategy as what bridges the gap between means and ends. Strategy is built up into a four-part structure. The ends to be obtained, the ways in which resources will be deployed, the ways in which resources that have been deployed are employed, and the resources or means at our disposal (Nickols, 2012). According to Cooper (2001), the third goal of portfolio management is that the portfolio of projects reflects the business’s strategy and that the spending within this portfolio mirrors the strategy. Cooper et al. (2002) mention forging a link between project selection and business strategy as critical in portfolio management — the portfolio is the expression of strat- egy and must therefore support the strategy. Cooper (2001) states that the objective of strategic alignment is to ensure that the final portfolio reflects the organization’s overarching strategy. This involves aligning the distribution of investments across projects, domains, and markets with strategic priorities, ensuring that each project supports the intended direction. Si et al. (2022) lists insufficient alignment with strategy as one of the greatest portfolio challenges. In an industry study, Cooper et al. (2002) found that the top performing firms at new product development utilize the business strategy to allocate resources and decide the portfolio to a much higher degree than the worst performing firms. 14 2. Frame of Reference Both Si et al. (2022) and Cooper (2001) identify the strategic buckets approach as an effective method for ensuring strategic alignment within an R&D portfolio. In this approach, the firm begins with its overarching business strategy and determines the product innovation strategy by identifying where investments should be made in order to support the strategy (Cooper et al., 2002). Based on this innovation strat- egy, distinct buckets are defined — such as project types, markets, technologies, or product lines (Cooper et al., 2002). Management then pre-allocates resources to each bucket, assigning a specific budget. Projects are subsequently evaluated and funded within these buckets according to their alignment and characteristics (Cooper et al., 2002). Cooper (2001) introduces a simplified version of the strategic bucket approach, as illustrated in Figure 2.5. The buckets are divided according to strategic priorities — namely, platform projects (major initiatives aimed at developing a new product platform that can serve as a base for a family of derivative products), new prod- uct projects (development of new products, often building on existing platforms or technologies), and others (non-innovation-focused projects necessary for business continuity such as cost reduction projects). Projects are first assigned to the appro- priate bucket and then ranked within each one, resulting in three distinct project portfolios. Figure 2.5: Illustration of strategic buckets. 15 2. Frame of Reference 2.2 Evaluation Criteria This section is organized around several central evaluation criteria frequently cited in the literature. Drawing on established literature in innovation management, new product development, and project selection, ten different overarching categories that have emerged from the study will be presented and synthesised to give an insight into the theoretical dataset that will be applied in the pursuit of an answer to re- search question 1. The criteria include traditional financial metrics such as NPV and ROI, but also extend to technical feasibility, strategic alignment, intellectual property considerations, stakeholder support, and sustainability. Furthermore, cri- teria such as risk, market attractiveness, team dynamics, and resource availability are examined for their impact on project potential and portfolio outcomes. Finally, other non-monetary considerations that can significantly influence long-term success such as synergies, corporate image, and the ability to leverage core competencies are presented. The findings from this part of the study are compiled in Table 2.1 for a more comprehensible overview. 2.2.1 Financial A recurring theme in the studied literature when it comes to both project selection and portfolio management is the central role of financial metrics in evaluating po- tential projects particularly within high-technology industries where investments are typically capital-intensive. Pinto (2019) groups financial criteria under the category of commercial factors, identifying ROI, payback period, and the project’s ability to generate future business as key indicators for decision-making. Cooper et al. (2001) introduces “project value” and the tools available to quantify it. Notably, they advocate for the use of NPV to prioritize projects based on the constraints of limited resources, aiming to maximize the overall value of the portfolio. Additionally, the ECV method is introduced as a means to account for uncertainty by weighing various future outcomes against their probabilities. This approach allows decision-makers to rank projects in terms of expected payoff relative to investment. The ultimate purpose of these financial criteria is to maximize the value of the portfolio. At the same time, it is important to consider the fact that inaccurate input will lead to skewed results and a model to determine things like NPV is only as good as the data put into the model. As Cooper (2001) states, “the sophistication of financial tools often far exceeds the quality of the data inputs,” which can render even the most theoretically sound methods ineffective in practice. 16 2. Frame of Reference Cooper and Sommer (2023) instead advocate for simpler criteria such as payback time over more complex but data-sensitive models like NPV, particularly in contexts where input assumptions are uncertain or difficult to verify. Cooper and Edgett (2003) argue that an exclusive focus on short-term financial returns can under- mine long-term strategic goals. They emphasize the importance of incorporating growth-oriented metrics alongside financial indicators to ensure a more balanced and forward-looking evaluation. 2.2.2 Technical Feasibility Technical feasibility is a recurring theme in the literature on project evaluation. Un- like financial metrics, which emphasize value realization, technical feasibility assesses whether a project or new product development can be successfully executed from a technological standpoint. According to Cooper and Sommer (2023), technical feasi- bility plays a central role in determining the likelihood of technical success in new projects. This probability can be evaluated through several dimensions, including the size of the technical gap, the level of technical complexity, and the degree of technical uncertainty (Cooper & Sommer, 2023). These factors help assess how far the proposed solution is from the organization’s current capabilities and how diffi- cult it may be to close that gap within existing constraints. A larger gap or greater complexity increases the risk of failure, while high uncertainty reflects the ambigu- ity surrounding whether the required technologies can be developed or integrated as intended. Similarly, Cooper et al. (2001) emphasize the importance of technical gap and com- plexity as key elements in assessing technical risk, which is frequently used in port- folio models to balance high-risk, high-reward initiatives against more incremental and technically secure investments. Cooper (2001) further argues that increased pre-development efforts in new product development should include early techni- cal assessments. Such assessments not only evaluate technical feasibility but also address implications for manufacturing and operations, and help identify critical technical risks and issues early in the process. Expanding on this, Silva et al. (2010) introduce the concept of technological maturity as a criterion for evaluating R&D projects, particularly within the aerospace industry. Projects working with technolo- gies at low maturity levels tend to be more feasible and predictable. Meanwhile, projects working with emerging and novel technologies often encounter obstacles in development. Silva et al. (2010) also recognize that a technology with strong fit to existing capabilities can reduce development time and enhance success. In addition, Silva et al. (2010) propose opportune attendance as a criterion, which evaluates whether the expected development timeline aligns with customer expec- tations — further refining the assessment of a project’s realization potential. 17 2. Frame of Reference 2.2.3 Strategy Strategic alignment is widely recognized in the literature as a foundational princi- ple for effective project selection and portfolio management. It refers to the extent to which a project or portfolio supports the broader mission, long-term goals, and competitive strategy of the business. Projects that align with strategic priorities are more likely to succeed, attract internal commitment, and result in more efficient resource allocation (Cooper, 2001). To enhance the effectiveness and efficiency of new product development processes, Schilling and Hill (1998) states that a selection and screening process must include strategic implications. Cooper and Edgett (2003) operationalize this principle through their concept of strategic buckets, which ensure resources are distributed according to predefined strategic priorities such as innovation type, market segment, or product class. This prevents the portfolio from becoming skewed toward either opportunistic or mis- aligned projects. Building on this, Cooper and Sommer (2023) emphasize that strategy should not only guide project selection but also be shaped by it — suggest- ing that innovation initiatives should help realize future strategic intent. Strategic fit, a related concept, is cited by Cooper (2001) as one of the most critical project evaluation criteria. It includes alignment with the firm’s long-term vision, product roadmap compatibility, and the ability to leverage existing competencies in areas such as R&D, marketing, or operations. Their research indicates that com- panies which consistently apply strategic criteria outperform those that rely solely on financial metrics. Similarly, Si et al. (2022) identify lack of strategic alignment as a major barrier in innovation portfolio management, while Cooper et al. (2001) associate such misalignment with ineffective portfolio performance. Within the aerospace sector, Silva et al. (2010) identify strategic alignment as one of the two most important overarching criteria for R&D project prioritization. They introduce several sub-criteria: potential to generate innovation (e.g., whether the project advances technological maturity), duality (whether the technology applies to both military and civilian contexts), and operational alignment (how well the project addresses actual operational needs). Lastly, both Cooper (2001) and Cooper and Sommer (2023) stress the importance of incorporating the voice of the customer into strategic alignment. A strong market orientation that is based on customer needs and preferences, improves the likeli- hood that R&D initiatives will generate successful outcomes and reduce the risk of misaligned efforts. 2.2.4 Market Attractiveness Market attractiveness appears as a central criterion in project selection, especially when commercial success is a prioritized outcome. Assessing the potential of a market allows organizations to prioritize projects that align not only with internal capabilities but also with external opportunities. 18 2. Frame of Reference Cooper (2001) emphasizes the importance of market need and product advantage by highlighting that successful projects often target markets where there is clear customer demand and where the offering delivers superior value. This includes the ability to provide unique customer benefits and a compelling value-for-money propo- sition. Projects targeting large or rapidly growing markets are more likely to deliver returns that justify the investment and higher risk. Relating to this, Silva et al. (2010) point to indicators such as market size, growth potential, and profit margins as essential dimensions in evaluating projects. According to them, a thorough assessment of market attractiveness ensures that selected projects are not only technically feasible and strategically aligned but also have a strong foundation for commercial success. Also, Cooper et al. (2001) highlight market size, market growth rate, and the level of competitive intensity as central factors in evaluating the attractiveness of a mar- ket. Similarly, Cooper and Kleinschmidt (1987) emphasize that larger market size, higher growth rates, and clearly identified market needs are positively associated with project success. Cooper (2001) reinforces this view by stating that products targeting attractive markets are more likely to succeed, making market attractive- ness a critical consideration in the selection and prioritization of innovation projects. 2.2.5 Resources The availability and quality of resources are critical factors influencing a project’s likelihood of success. In the context of project selection, assessing resource readi- ness goes beyond identifying what is available — it involves determining whether the organization can realistically support the execution and commercialization of selected initiatives. Cooper and Edgett (2003) identify inadequate resourcing as one of the most persistent weaknesses in new product development. When resources are stretched too thin, firms often default to smaller, lower-impact projects that are easier to fund but unlikely to yield substantial returns. This leads to suboptimal portfolio performance and missed opportunities for innovation. Cooper et al. (2001) further highlight that the availability of skilled personnel, in- frastructure, and technological capabilities directly contribute to the probability of technical success. Similarly, Silva et al. (2010) emphasize the importance of evalu- ating financial, human, and infrastructure resources in R&D project assessment — especially in industries where successful execution depends on substantial upfront investment and cross-functional collaboration. Reinforcing this, Cooper (2001) asserts that “there is no free lunch” in product in- novation, stressing the necessity of aligning development ambitions with resource capabilities. One practical approach he recommends is a resource capacity analysis, which helps organizations determine whether they have the right mix and volume of resources to handle the current pipeline and achieve their product development 19 2. Frame of Reference objectives. Such assessments play a vital role in ensuring that strategic priorities are not only well-defined but also realistically supported by the organization’s op- erational capacity. 2.2.6 Intellectual Property (IP) While often overlooked, IP plays a notable role in project selection and portfolio management. An effective protection of innovations can significantly influence the commercial viability of a project by creating barriers to entry and enabling firms to capture a greater share of the value they generate. Pinto (2019) identifies IP rights as one of the key factors influencing the project se- lection process. The ability to secure patents, trademarks, or trade secrets may give priority to one project over another, especially when uncertainties in other areas are high. Projects with stronger potential for IP protection can also be seen as more strategically valuable as they help safeguard returns on R&D investments. A strong proprietary position not only enhances the long-term potential of a project but may also reduce the risk of imitation by competitors. Cooper et al. (2001) include proprietary position as a formal evaluation criterion in their portfolio man- agement framework. This factor is assessed on a scale ranging from “easily copied” to “well protected,” allowing organizations to factor in how defensible an innovation is within the competitive landscape. 2.2.7 Stakeholders While not treated as a formal evaluation criterion in most of the literature, both internal and external stakeholders can play an important role in determining the success of project selection efforts. Without management backing, even well designed and executed projects may strug- gle to gain traction across the organization according to Cooper et al. (2001). The authors also concluded that successful portfolio management is often reinforced by strong and visible support from senior management and showed that it is not only the formal process, but also the commitment demonstrated through leadership ac- tions. Schilling and Hill (1998) extends this view by highlighting the importance of stake- holder engagement in new product development. The author points to the use of executive champions or senior figures who advocate for high-potential projects. Moreover, strategic partnerships and alliances with external stakeholders, such as suppliers or technology partners, can enhance access to capabilities and improve project feasibility. 20 2. Frame of Reference 2.2.8 Sustainability The relevance of sustainability in business is constantly growing although not widely emphasized in most project selection frameworks. Businesses face increasing pres- sure to align innovation efforts with broader environmental, social, and governance (ESG) goals. Causing sustainability to emerge as an additional theme through which projects can be evaluated. One way to assess a project’s value according to Cooper and Sommer (2023), is by examining how it contributes to the organization’s broader mission, which may include reducing environmental impact or advancing ESG commitments. Projects that actively support such goals can help future-proof the business by responding to regulatory developments, stakeholder expectations, and shifting market demands. Similarly, Cooper (2001) highlights the importance of ensuring that selected projects meet key environmental, health, and safety standards. This is considered to be an essential aspect of responsible innovation management. 2.2.9 Risk Theoretical frameworks within portfolio management and project selection empha- size the importance of understanding and managing risk not only to avoid having to kill projects that turn out less successful than predicted, but also to ensure that re- source allocation decisions align with the organization’s overarching strategic goals. Cooper (2001) states that total risk avoidance in new product development is im- possible, unless all innovation is avoided. Cooper (2001) proposes two components of risk in new product processes. The first component is the amounts at stake, this includes possible payoffs and possible down- side losses. The second component is the uncertainty regarding the development. Usually, the amounts at stake during early development is low and increases as the development progresses (Cooper, 2001). Silva et al. (2010) integrate risk response as a formal criterion in their evaluation model for R&D project selection. They propose to assess the perceived risks for the project and if there exists any measures to mitigate the risk. Cooper and Sommer (2023) highlight the utility of financial models to approximate a project’s risk level compared to the reward, such as calculating the reward/risk ratio by the expected NPV divided by the remaining financial resources to be spent. Cooper (2001) emphasizes the importance of evaluating projects based on the bal- ance between risk and return, highlighting factors such as the speed of investment recovery, the cost and duration of the project, and the certainty of expected returns. Additionally, Cooper et al. (2001) stress the need to maintain a balanced project portfolio, ensuring a mix of long- and short-term initiatives as well as high- and low- risk projects, to optimize both the potential for innovation and financial stability. These models, along with commonly used risk/reward ratios, help decision-makers 21 2. Frame of Reference evaluate whether the potential benefits of a project justify the risks involved. In essence, this involves assessing what the upside is if the project succeeds, and how significant the losses are if it fails. 2.2.10 Resources Effective portfolio management also involves navigating the organizational dynam- ics that shape how projects are delivered. Teams play a central role in this process, both as executors of projects and as stakeholders affected by portfolio decisions. Poor team morale is a recurring consequence of weak or misaligned portfolio man- agement according to Cooper and Edgett (2003). For instance, they demonstrate that when teams are assigned to too few or unimpactful projects, it can result in disengagement and lack of motivation illustrating how portfolio decisions have a downstream effect on workforce energy and innovation capacity. Cooper (2001) proposed that cross-functional teams that are empowered, resourced and held accountable have a positive effect on the profitability of development, how time efficient the development was, and if the launch was on time. Also Schilling and Hill (1998) notes that project teams are increasingly composed of cross-functional members, a structure which he identifies as beneficial for effective projects related to new product development. By bringing together diverse skill sets from across the organization, cross-functional teams enhance both creativity and executional strength (Schilling & Hill, 1998). Portfolio management decisions are also people decisions. Ensuring that teams are meaningfully engaged, properly supported, and organizationally aligned is essential not just for individual project success but for sustaining long-term innovation performance. 2.2.11 Other Considerations Additional non-monetary factors have been identified in the literature. These con- siderations may not translate directly into short-term profit, but they contribute significantly to long-term strategic positioning and organizational value. One such factor is the potential for synergies, whether between technologies, mar- keting channels, or across projects. Cooper (2001) highlights that projects offering opportunities to leverage core competencies, such as technical know-how or man- ufacturing capabilities, may hold strategic value beyond their immediate financial return. These synergies can lead to improved efficiency, accelerated development, or a stronger competitive edge (Cooper, 2001). Cooper and Sommer (2023) also highlight the importance of a compelling value proposition, referring to the extent to which a new product is perceived by users as unique and superior compared to existing alternatives. 22 2. Frame of Reference Another important non-monetary factor is the potential impact on company image. As Pinto (2019) notes, the types of projects a company chooses to pursue send signals to both internal and external stakeholders about its identity and direction. Projects that align with brand values, emphasize innovation, or result in unique and differentiated offerings may enhance the company’s reputation and perceived market leadership (Pinto, 2019). 2.3 Synthesizing the Findings Drawing from the literature, Cooper (2001) identified three principal approaches when evaluating projects. The first approach is the benefit measurement techniques which emphasize subjective assessments of strategic factors utilizing tools such as AHP, checklists, or scoring models. The second approach is applying economic models such as payback period, NPV, or ECV. Even though economic models are powerful, it is also noted by Cooper (2001) that firms who rely heavily on financial tools tend to perform worse than their counterparts who place greater emphasis on non-financial approaches. The third approach is managing the portfolio of projects which can be divided into three goals; maximizing the value of the portfolio, having a balance between e.g., risk and time, and lastly making sure that the portfolio of projects is strategically aligned. Through a review of the literature, ten key evaluation categories were identified as critical for assessing NPD projects and selecting R&D initiatives. These categories, alongside examples of specific criteria, are summarized in Table 2.1. Among these ten categories, five were most frequently highlighted in the reviewed sources: financial considerations, technical feasibility, strategic alignment, market attractiveness, and resource availability. • Financial considerations encompass traditional financial metrics such as ROI, payback period, and NPV. • Technical feasibility assesses the maturity of the technology to be developed, the technical gap between needed and current capabilities, and how technically complex the development is. • Strategic alignment evaluates how well a project supports the firm’s broader strategy, including considerations of portfolio balance (e.g., across different innovation types) and responsiveness to customer needs, wants, and expecta- tions. • Market attractiveness focuses on external market conditions such as market size, growth rate, profit margins, and competitive intensity. • Resource availability assesses whether the firm possesses, or can access, the necessary financial assets, skilled human resources, or infrastructure to support project development. 23 2. Frame of Reference Table 2.1: Synthesized findings on criteria mentions from the literature study Category Criterion Examples Sources Financial ROI, payback time, net present value, potential for rewards, ex- pected commercial value, costs vs benefits Pinto, J. K. (2020); Cooper et al. (2001); Cooper (2001); Cooper & Edgett (2003); Cooper & Sommer (2023) Technical fea- sibility Technological maturity, capabili- ties, assessments, technical gap, technical complexity, opportune attendance da Silva et al. (2010); Cooper & Sommer (2023); Cooper et al. (2001); Cooper (2001) Strategy Operational alignment, duality, alignment with firm’s strategy, strategic fit, portfolio balance, strategic alignment, compatibility with product roadmap, strategic intent, strategic leverage potential, voice of customer built in da Silva et al. (2010); Cooper & Sommer (2023); Cooper et al. (2001); Cooper (2001); Schilling (1998); Cooper & Edgett (2003); Si et al. (2022) IP Possibility to protect innovation Cooper et al. (2001); Pinto, J. K. (2020) Stakeholder Being part of strategic alliances, having executive champions Schilling (1998); Cooper et al. (2001) Sustainability Meet environmental, health, and safety standards Cooper (2001), Cooper & Sommer (2023) Risk Potential for risk response, reward vs risk, what is the perceived risk? da Silva et al. (2010); Cooper & Sommer (2023); Cooper et al. (2001); Cooper (2001) Market attrac- tiveness Market size, market growth, mar- ket margins, intensity of competi- tion, good value for money for cus- tomers da Silva et al. (2010); Cooper & Sommer (2023); Cooper et al. (2001); Cooper (2001); Cooper & Kleinschmidt (1987) Team Cross-functional teams, team is resourced and held accountable, clear leadership Schilling (1998); Cooper (2001); Cooper & Edgett (2003) Resources Means availability (financial re- sources, human capacitation, in- frastructure), resource limits, re- source capacity analysis da Silva et al. (2010); Cooper et al. (2001); Cooper (2001); Cooper & Edgett (2003) Other consid- erations Synergies between marketing, projects, and technologies, com- pany image impact, unique & differentiated product Cooper & Sommer (2023); Pinto, J. K. (2020); Cooper (2001) 24 3 Method This chapter outlines the methods that were employed in conducting the research, including the research design, data collection methods, selection of interviewees, and the ethical considerations. The aim is to offer transparency regarding the pro- cesses and principles guiding the research, facilitating for potential replications and ensuring credibility. 3.1 Research Design This study utilized a qualitative research methodology to gain a deep understanding of the context and complexity of R&D management in the aerospace industry and the decision-making processes. Both Bell et al. (2019) and Creswell and Poth (2018) emphasize the advantages of qualitative research and its ability to deliver rich and deep data, depth, and a contextual understanding. The thesis utilized a single-organization case study within a specific industry. The thesis highlights the characteristics of the Company. This approach aligns with what Bell et al. (2019) refer to as an idiographic response, focusing on the in-depth understanding of a unique case rather than seeking broad generalizations. The case can be coined as a representative case since it explored an everyday situation for the Company’s R&D department (Bell et al., 2019). The extensive and in-depth analysis and description of the Company made the choice of a case study relevant (Yin, 2014). The study started with an explorative phase where internal interviews were con- ducted and a review of existing literature to draw upon when formulating the pre- liminary theory. After the exploratory phase, an iterative process was conducted including developing the theoretical framework, conducting semi-structured inter- views, building a decision-making framework, and obtaining feedback from work- shops. An abductive approach was used in the study as often recommended for theory- driven innovative empirical research (Alvesson & Sköldberg, 2017). The abductive approach helps develop or refine existing theories by iteratively moving between data and theoretical explanations (Charmaz & Bryant, 2019). The inductive and deduc- tive approaches, even though valuable, the authors feared would not fully capture the dynamic interplay between the empirical findings and the theory. Induction is grounded in empirical findings, whereas deduction is rooted in established literature 25 3. Method (Alvesson & Sköldberg, 2017). By using an abductive approach, the research could adapt and refine theoretical frameworks based on the data gathered from the com- pany, leading to more nuanced and innovative insights. This approach allowed the authors to incorporate theoretical insights with real-time industry observations and support an adaptive research process. This enabled the refinement and expansion of theories based on emerging findings, ensuring the study effectively addresses uncertainties within the given context (Alvesson & Sköldberg, 2017) . 3.2 Research Process The research process was planned to consist of two different steps. Initially the fo- cus was on the exploratory phase, where it was deemed important to gain in-depth understanding of the Company, the environment in which it operates, and relevant theory. The second phase concentrated on data collection, engaging a broader range of internal interviewees, conducting workshops, and employing an iterative, sys- tematic approach to develop a framework intended to support the decision-making process. This approach is depicted in Figure 3.1. Figure 3.1: Overview of the research process. 3.2.1 Exploratory Phase The research started in the exploratory phase where internal interviews were con- ducted to gain knowledge about the aerospace industry and a literature review to build a preliminary theoretical framework. The phase was critical for gaining an in-depth understanding of the real-life problems experienced by the Company and paved the path for what the framework that was-to-be-built would transform into. In parallel to the initial interviews, a search for existing literature was conducted to build an initial frame of reference. This allowed the authors to situate the insights from the exploratory interviews within the wider academic and industrial contexts, ensuring the study was both theoretically grounded and practically relevant. As a rule, larger investigations or research tasks require an overview of the central liter- ature within the studied field (Eriksson & Wiedersheim-Paul, 2011). This search of the overview has been done through Chalmers Library utilizing literature databases 26 3. Method such as Scopus, ScienceDirect and WebScience. To find modern and relevant infor- mation, certain sources were occasionally excluded when their relevance could no longer be assured. According to Eriksson and Wiedersheim-Paul (2011), the require- ment for contemporaneity is one of the factors influencing the reliability of sources. Patel and Davidsson (2011) agree and conclude that books provide relevant theories and models but scientific articles provide a more up-to-date perspective. However, some books have still been utilized due to their substantial contribution in the the- oretical field and act as a foundation of the theory that modern literature draws upon. The search for relevant articles was guided by keywords refined during the exploratory phase, as shown in Table 3.1 below. Additionally, backward searching was conducted from relevant sources when deemed necessary. Further literature was incorporated based on recommendations from our Chalmers supervisor. Table 3.1: Keywords used during literature review Keywords Decision-making Tool, Scoring model, Analytical Hierarchical Process, R&D Aerospace, Portfolio Management, R&D management, Project management, Technology Evaluation, New product development, Tech- nology Management 3.2.2 Data Collection Phase In the data collection phase, building upon the exploratory phase, the construction of the decision-making framework was initiated and semi-structured interviews were conducted with different stakeholders in the R&D decision-making process. The goals were to gain an understanding of what criteria the interviewee believes are important when evaluating technologies at an early stage and how the evaluation process works at the moment. Based upon the discoveries in the semi-structured interviews, the framework was continuously updated through an iterative process with the insights gained from the interviews as the foundation. The interviews were of a semi-structured character due to its flexibility and depth allowing the interviewee a great deal of freedom to reply (Bell et al., 2019). In a semi-structured interview, the interviewer has a list of questions to ask, or an in- terview guide, but the questions may not follow exactly the process of the guide and new questions might arise based on the response of the interviewee (Bell et al., 2019). The semi-structured format aligned well with the exploratory character of the thesis as the study is qualitative in nature. The interviews served as a means to discover and identify aspects of the interviewee’s perception of the problem (Patel & Davidsson, 2011). The interview process began with general questions aimed at understanding the re- spondent’s background, their role in the decision-making process, and their perspec- tive on current shortcomings in the process. Following this, the discussion shifted toward a more in-depth exploration of the decision-making tool and the respondent’s specific area of expertise. In line with Patel and Davidsson (2019) recommendations, 27 3. Method participants were briefed beforehand on the thesis objectives to ensure they feel com- fortable and prepared to share their experiences and insights. An interview guide, see Appendix A, was created aiming to have a reasonable flow and formulate the questions such that it would help us answer the research questions (Bell et al., 2019). Once saturation was reached regarding the critical criteria for evaluating technolo- gies, three workshops were conducted. In preparation for the first two workshops, the identified criteria were integrated with the tools outlined in the frame of ref- erence. To ensure a productive discussion, participants received the criteria, tools, and accompanying user guides via email in advance. This preparatory step was intended to promote informed dialogue and ensure a shared understanding among participants during the sessions. In the final workshop, the framework deemed most suitable, conclusions derived from earlier workshops, was demonstrated to gain addi- tional feedback. The primary objective of all workshops was to collect comprehensive feedback on the framework as a whole. 3.3 Selection of Interviewees The interviews conducted were an essential part of the study. Through the inter- views, insights were gained which helped develop the framework and shed light on new areas to investigate in the literature. The diversity in the selection ensured that the study captured a wide range of perspectives. The selection criteria for interviewees were carefully designed to align with the study’s objectives, emphasizing stakeholders actively engaged in areas relevant to the research. Participants were chosen based on their involvement in key areas or their roles, ensuring that each interviewee could offer in-depth insights on the top- ics discussed. Additionally, a priority was given to individuals with decision-making authority, as their influence plays a crucial role in critical decision-making processes. Below can be found the different roles or areas of expertise that were targeted for the interviews: • Partnership office: Individuals working with and responsible for the part- nerships with OEMs. • Project Managers: Individuals responsible for the projects aimed at de- veloping technologies to give an understanding of the chance of technological success, competences needed, and capacity for the project to succeed or not. • End-users: Stakeholders from R&D at the Company and within the forums that new technologies are discussed. • Risk and Trends: Individuals highly involved with risk assessment within the aerospace industry and individuals with a great expertise in the regulatory environment. • Strategy: Individuals with deep knowledge in regards to the Company’s strat- egy and long-term vision. 28 3. Method The selection process was dynamic, incorporating both purposive sampling and snowball methods. Furthermore, the selection of interview participants was car- ried out with strategic intent, or defined as purposive sampling (Bell et al., 2019). Initially the Company provided a list of potential interviews primarily consisting of decision-makers and domain experts within the internal environment of the company. Through snowball sampling, the authors had the possibility to extend their network through existing interviewees recruiting through their own network. Through re- ferrals, hard-to-reach participants have been reached and has proven to be a time- efficient methodology. In total, 11 interviews were held with 9 interviewees. In Table 3.2 an overview of the interviewees can be found. Table 3.2: Overview of interviews ID Title of respondent Department Phase Time I1 Research Engineer External funding Explorative 60 min I2 Technology Manager External funding Explorative 70 min I3 Director of R&D GTC Explorative 60 min I2 Technology Manager External funding Data collecting 70 min I1 Research Engineer External funding Data collecting 45 min I4 Technology Project Man- ager External funding Data collecting 60 min I5 Project Manager Technology Data collecting 60 min I6 Customer Strategy Di- rector Partnership Program Office Data collecting 65 min I7 Capabilities Portfolio Di- rector Technology Data collecting 40 min I8 Program Director Partnership Program Office Data collecting 30 min I9 Head of Business Area 1 BA 1 Data collecting 50 min Table 3.3 presents the attendees of the workshops. It is important to note that some attendees had also participated in earlier interviews. Three employees participated in the first workshops, seven employees participated in the second workshop, and three attendees participated in the third workshop. Two attendees participated in all workshops. The selection of the participants in the workshops was orchestrated by the Company and involved personnel working within R&T. 29 3. Method Table 3.3: Overview of workshop attendees ID Title of attendee Department Workshop WA1 Research Engineer External Funding 1, 2 & 3 WA2 Technology Manager External Funding 1, 2 & 3 WA3 Director of R&D GTC 1 & 3 WA4 Technology Project Manager External Funding 2 WA5 Technology Project Support Officer External Funding 2 WA6 Technology Project Manager External Funding 2 WA7 Technology Project Support Officer External Funding 2 WA8 Senior Research Engineer External Funding 2 3.4 Data Analysis Qualitative research has a tendency to result in accumulation of large amounts of intense and unstructured sets of data (Bell et al., 2019). This entails challenges in managing and navigating through different types of material, being able to analyze the data in a structured manner is thus crucial to avoid being overwhelmed by it. As described in the research process, the data collection process primarily consisted of explorative interviews, a literature review, and semi-structured interviews. This iterative process of cycling between literature and empirical evidence resembles an abductive approach to qualitative data analysis as described by Bell et al. (2019). The main advantage of an iterative approach is to be able to use the empirical data collected in interviews to develop theory. Furthermore, it helps challenge precon- ceptions and biases generated by prior experiences or perspectives. In this case study, the purpose of the initial phase of the study was to gain an in- depth understanding of the topic and discover new insights through the exploratory interviews. As the authors entered the study with limited experience about the case company and the industry, the goal was to collect data with the intention of generating substantive theory directly grounded in the data. As such, the data collection and analysis strategy were deeply inspired by the method of Grounded theory (Glaser & Strauss, 1967). The main advantage of this method is that it allows the authors to use actual case-specific data to build theory from the ground up, avoiding assumptions and preventing the research from being contaminated by existing models. The grounded theory strategy offers tools and resources that can be used both in the collection and analysis of qualitative data, but they can also be employed in the research process as a whole (Bell et al., 2019). In this study, tools from grounded theory were used throughout the data collection and data analysis process to provide structure and consistency to the process. A key consideration when collecting qualitative data has to do with the organization 30 3. Method and management of large amounts of information. For this, the authors used a cod- ing approach where information from interviews were categorized from an early stage to identify early patterns and help break down the data into manageable units. The coding was initially done by labeling important segments and highlighting quotes from respondents. As codes started to emerge into themes, the authors made sure to keep a close connection with the data through constant comparison. As described by Glaser and Strauss (1967), constant comparison is an ongoing process where re- searchers revisit previously coded data to compare it to more recent findings. This approach facilitates a refinement of the theory by ensuring that it is closely connected to the real-world findings. In addition to the benefit of having the data organized and structured, the coding process entailed a better understanding and overview of the data which proved to be useful in the next step of the analysis, the theoretical sampling. Glaser and Strauss (1967) introduced the concept of theoretical sampling, describing it as “the process of data collection for generating theory whereby the analyst jointly collects, codes, and analyzes the data and decides what data to col- lect next and where to find them, in order to develop the theory as it emerges.” Consequently, the data collection process is controlled by the emerging theory. This implies a key characteristic of theoretical sampling, namely that it is an iterative and ongoing process rather than a single discrete stage in the data collection process. The coding process was conducted until a point of theoretical saturation was met. According to Bell et al. (2019) theoretical saturation occurs when the additional collection of data no longer provides new insights or when no further comparison is needed to fit the data into categories. Theoretical sampling was used to initiate a new iteration of interviews until theoretical saturation was met. The goal with this approach was to theorize on emerging insights rather than to collect as much data as possible, as such, the selection of participants was adapted to each stage of the study as the research progressed. In accordance with grounded theory strategy, it wasn’t until the point of which theoretical saturation was met that the theory development phase was fully initiated. 3.4.1 Data Compilation and Derivation of Criteria Following the point of theoretical saturation, empirical observations from the case study and insights from the literature review were compiled and merged into a com- mon data set. This compilation served as the foundation for constructing the set of evaluation criteria central to addressing Research Question 1. The data was systematically categorized and analyzed with the aim of identifying both alignments and discrepancies between the theoretical and empirical inputs. The process was guided by a search for recurring themes, and commonly referenced evaluation criteria across both data sources. Criteria that emerged frequently and consistently were considered to hold particular significance and were therefore pri- oritized in the development of the framework. 31 3. Method Figure 3.2: Grounded Theory model for data collection and data analysis In order to ensure that the resulting criteria were not only theoretically grounded but also practically applicable at the case company, a few case-specific considera- tions were incorporated into the evaluation. First, the applicability of each criterion within the organizational environment was assessed. Secondly, the criteria were reviewed through the lens of their ability to support the intended purpose of tech- nology investment evaluations. The formulation of criteria descriptions was also carefully reviewed to ensure both clarity but also alignment with the case company’s common terminology. This included adopting terms and phrasings already in use within the company’s internal documentation and communication channels, thereby increasing the degree of fit and likelihood of adoption. These layers of consideration were essential in adapting the theoretically-informed criteria into a contextually relevant and actionable framework for the case company. 3.5 Research Quality Discussing reliability and validity in this case study can be challenging, as these concepts do not hold the same meaning in a qualitative study as they do in a quantitative one. According to Patel and Davidsson (2011), validity in qualitative research is defined, among other things, as the ambition to identify phenomena and describe perceptions. Furthermore, they explain that reliability in qualitative studies often align closely with the concept of validity. Therefore, only the validity of this study will be discussed below. The interpretation of qualitative data from theory and interviews carried the risk of being influenced by the authors’ perspectives. To enhance the credibility of the study, interviewees were, when necessary, contacted via email or phone calls after- ward to clarify interpretations or provide additional explanations. This process, referred to as communicative validity by Patel and Davidsson (2011), helps ensure 32 3. Method accuracy. Additionally, the study’s results and analysis were shared with the par- ticipants, which reduces the risk of misinterpretation. 3.6 Ethical Considerations Each respondent was informed about the study’s purpose and methodology, and their participation was entirely voluntary, with the option to withdraw at any time. Participants were assured that the collected data would be used solely for academic purposes and handled with strict confidentiality. Additionally, participants remained anonymous which provided more open and candid responses, particularly on sen- sitive topics. This aligns with Bell et al. (2019) four key principles to help ensure that research is conducted in a responsible and ethical manner; avoiding harm to participants, informed consent, invasion of privacy, and deception. 33 3. Method 34 4 Working with Technologies and Projects at the Company Today 4.1 Technological Maturity and Strategic Part- nerships in Aerospace This chapter outlines the frameworks and structures that underpin technology devel- opment and investment in the aerospace industry. It first introduces the Technology Readiness Level (TRL) model as a tool for assessing technological maturity and guiding R&D decisions. It then explores Risk- and Revenue-Sharing Partnerships (RRSPs), a prevalent collaboration model in engine programs that enables firms to manage high development costs and long payback periods. Together, these con- cepts illustrate how technological progress and financial risk are jointly managed in long-term, capital-intensive aerospace projects. 4.1.1 Technology Readiness Level The TRL framework was developed by NASA in the 1980’s as an assessment tool to identify and evaluate both technical and commercial risk. The process includes assigning technologies a level on a scale of nine to determine the technological matu- rity of a project. The framework is meant to highlight the research and development progress of a certain technology from the very early stages of development to a fin- ished, manufactured and sold product (EARTO, 2014). As such, by verifying the criteria at each level, one can determine the readiness level of a technology at a cer- tain stage of development. Additionally, the TRL framework offers the possibility to verify when the research within a technical area is ready to be elevated to a higher level of maturity (“Innovair”, 2014). In order to determine the maturity of a technology, three main factors are con- sidered and measured against the use-case requirements of that technology. The factors include performance/function, the probability of the technology’s physical realization referred to by NASA as “form and fit”, and predicted resilience in a phys- ical and operational environment (NASA, n.d.). The lower TRLs are focused solely on function but in order for a technology to progress to a higher TRL, both form and function need to be proven. For the highest TRLs, technological fit with prac- tical applications must be recognized in addition to form and function (NASA, n.d.). 35 4. Working with Technologies and Projects at the Company Today At lower TRLs (level 1 and 2), practical applications may still be speculative as no proof or detailed analysis exists to support any real world applications. For TLR3 and TLR4, proof of concept and basic functionality must be achieved in a controlled environment while TRL5 and TRL6 requires validation and demonstration in an en- vironment that is relevant to the technology’s use case. At TRL7 and TRL8, the technology must be applied to a real-world operational environment and at TRL9 the technology is considered full-fledged and operational. A definition of each Tech- nology Readiness Level is presented in table 2. In addition to providing an aid in decision-making concerning R&D and technol- ogy management, the TRL framework offers a few concrete advantages. First of all, the framework provides a clearly communicated representation of a technology’s current status and supports decision making in the transitioning of technology. Fur- thermore, systematically assessing the stage of development for a certain system or technology can be an asset in risk management (Dawson, 2007). However, despite the framework being relatively widespread in aerospace and other technology-heavy and product-focused industries, the generalizability of the framework is debatable. For example, in software-based technologies, the readiness of a system seldom cor- relates with the technological maturity which diminishes the applicability of such a model (Smith II, 2005). Table 4.1: Technology readiness levels and their definitions(NASA TRA Best Practices Guide, n.d.) TRL Definition 1 Basic principles observed and reported 2 Technology concept and/or application formulated 3 Analytical and/or experimental proof-of-concept of critical function 4 Component and/or breadboard validated in laboratory environment 5 Component and/or brassboard validated in relevant environment 6 System/subsystem model or prototype demonstrated in a relevant environment 7 System prototype demonstration in an operational environment 8 Actual system completed and “flight qualified” through test and demonstration 9 Actual system flight proven through successful mission operations In the aerospace industry, R&D is characterized by development times stretching up to 15-20 years and new development cycles are generally initiated every four to five years (“Innovair”, 2014). Because of this, it is crucial to maintain activity and administer research in every phase of the technology development in order to retain a diverse project portfolio. This implies that, at any given time, research is conducted on multiple TRLs in multiple different areas. To demonstrate and communicate how this project portfolio might be constructed, manufacturers in the aerospace industry employ “the incline wave principle” as presented in Figure 4.1. The principle visualizes the project portfolio as a roadmap, providing an indication of the projects’ maturity levels in relation to each other as well as a prediction of the portfolio at some point in the future. Using the principle to backtrack future 36 4. Working with Technologies and Projects at the Company Today product applications allows for predictions about which underlying technologies will be required to facilitate that application in the future. As such, the framework can give an indication of what areas of research need to be prioritized. Furthermore, the principle facilitates a basis for efficient project portfolio planning by providing a structured and visual representation of projects at varying levels of maturity. Figure 4.1: The incline wave principle for projects in the aerospace industry (“In- novair”, 2014). 4.1.2 Risk- and Revenue-Sharing Partnerships The aerospace industry is a high-technology industry. For aircraft engines, the de- velopment is characterized by intensive collaborations among firms to divide the high risks and costs needed when realizing the complex products. The realization of the end-product is not conducted by only one firm but from networks of firms from many types of industries (Corallo et al., 2014). Due to the complex nature of the aircraft engine, recent aircraft programs have im- plemented RRSPs which focus on the collaboration between OEMs and Tier-1 or Tier-2 suppliers and are defined as collaborati