Benefits management in software development: A systematic review of empirical studies

Considerable resources are wasted on software projects delivering less than the planned benefits. Herein, the objective is to synthesize empirical evidence of the adoption and impact of benefits management (BM) in software development, and to suggest directions for future research. A systematic review of the literature is performed and identified 4836 scientific papers of which the authors found 47 to include relevant research. While most organizations identify and structure benefits at the outset of a project, fewer organizations report implementing BM as a continuous process throughout the project lifecycle. Empirical evidence gives support for positive impact on project outcome from the following BM practices: identifying and structuring benefits, planning benefits realization, BM during project execution, benefits evaluation and the practice of having people responsible for benefits realization. The authors suggest four research directions to understand (1) why BM practices sometimes not are adopted, (2) BM in relation to other management practices, (3) BM in agile software development and (4) BM in the context of organizations' value creation logics.


| INTRODUCTION
Software systems are supposed to benefit individuals, organizations and societies in solving problems or exploiting opportunities. The term benefit is defined by the UK Office of Government Commerce (OGC) as 'the measurable improvement resulting from an outcome perceived as an advantage by one or more stakeholders, which contributes towards one or more organizational objective(s)' [1, p. 75]. OGC suggests that projects deliver outputs which create capabilities. The capabilities are transitioned into outcomes that enable the realization of benefits. The achieved benefits contribute to the achievement of the organizational objectives [1]. A similar view of the relations between objectives, benefits and capabilities is offered by Peppard et al. [2] by linking objectives to the required benefits (the ends) with the necessary business changes (the ways) and the IT capabilities (the means) that enable these changes.
One of the early papers on benefits management (BM) states: 'The overall process of evaluation and realisation of IS/IT benefits has been termed benefits management and may be defined as: "The process of organizing and managing such that potential benefits arising from the use of IT are actually realised"' [3, p. 214]. As noted by Svejvig and Schlichter [4], the term BM is up for discussion; for example, the process originally termed BM also is termed benefits realization management [5]. Others use value management to emphasize the balance between benefits and costs [6]. As Svejvig and Schlichter, we use the term BM, but recognize that the literature on BM, benefits realization management, and value management relates to overlapping disciplines.
BM includes the processes of benefits identification and estimation, as well as benefits planning, BM during project execution, and evaluation and identification of additional benefits during the post-project period [3]. Although, benefits from software projects are focused, BM can be applied to a project, a programme, a portfolio of change initiatives or a whole organization [5]. The practice of managing benefits is not unique to software projects. General frameworks and industry standards exist with relevance across different industries (e.g. [7,8]).
Researchers have investigated various perspectives on IT project successes and failures (e.g. [9][10][11]). It seems that there is a consensus among researchers that given organizations' huge investments and potential benefits from IT projects, just a small percentage increase in the success rate can carry enormous benefits for the organizations. Even so, despite decades of research and practice in IT project management, IT projects often seem to disappoint stakeholders by wasting a lot of resources and failing to deliver the promised benefits (e.g. [12]).
Against this backdrop, one might expect to find a solid body of knowledge related to how the sought-after benefits are best realized. Interestingly, much research has focused on investigating the estimation of IT project costs and delivery of IT projects within the planned budget, the planned time and with the specified functionality, while the management of benefits has not received the same attention [13,14]. Increased knowledge regarding BM may help organizations to be evidence-based when selecting management practices in relation to their software projects [15,16]. Researchers have encouraged the use of empirical methods to advance our understanding of real-world problems [17,18]. Thus, unlike prior literature reviews of BM, we focus this review on empirical research. We will seek to add knowledge by answering the following research questions, synthesizing the results and providing directions for future research: � RQ1: What have previous empirical studies found about how, and to what extent, organizations implement BM? � RQ2: What is the impact of BM on project outcome according to previous empirical studies?
Herein, Section 2 introduces BM terminology and practices. Section 3 introduces existing reviews and presents how our review differs from and extends the results from previous reviews. Section 4 describes the research method. Section 5 presents the results. Section 6 discusses the findings and directions for future research. Section 7 reflects on the validity and limitations. Section 8 provides concluding remarks. Svejvig and Schlichter [4] note that BM is related to the creation of value in an organization, but there is no consensus on the differences between the concept of 'benefit' and 'value'. The terms are used interchangeably, herein.

| BM TERMINOLOGY AND FRAMEWORKS
When we discuss particular dimensions of benefits, such as monetary/non-monetary, this will be clearly stated. Our use of 'benefit' is not to be confused with the value concept in earned value management (EVM), which is a technique for managing project cost and time performance [19]. We use the term 'responsibility' for benefits realization, although the literature also uses 'accountability'. When we use the term BM 'practices', we refer to processes and activities related to BM. We use 'benefits realization' and 'benefits delivery' synonymously.
Ward et al. [3] propose a BM process model (Figure 1), which we use in our literature review as a framework for structuring the results and discussions. The model is called the Cranfield process model. This is the BM model that is most commonly referred to and is often employed as a benchmark against which to assess BM practices [20]. The process model consists of five elements: (1) identifying and structuring benefits, which focuses on the identification of potential benefits and on defining how each benefit will be measured; (2) planning benefits realization, which encompasses all activities needed to realize each benefit, including potential process and organizational changes; (3) executing the benefits realization plan, with an emphasis on the benefits realization plan being an integral part of the project management plan; (4) evaluating and reviewing results, which involves the evaluation of actual benefits delivered and (5) the potential for further benefits, which is about trying to capitalize further on the investments already made.
A central aspect of the process model is its project lifecycle emphasis-that is, BM being an integrated part of all phases of a software project (project lifecycle is defined as the series of phases that a project passes through from project initiation to its closure) [7]. In addition to pre-investment appraisal and post-investment evaluation, the model also includes the management of benefits during project execution. We refer to Doherty et al. [21] for an example of a city council with success in benefits realization by having a proactive focus on realizing benefits rather than delivery of IT solutions.
Ward et al. [22] put BM in a context surrounded by, and intersecting with, other processes and methods, such as programme and portfolio management, project management, investment appraisal, system development methods, change management methods and risk management techniques.
Recognizing that BM is not a stand-alone management discipline Svejvig and Schlichter [4] propose an integrative management model whereby BM and the other disciplines are orchestrated. Svejvig and Schlichter note that BM is often a sub-discipline within project and program management, where, for example, project management methods specify that benefits should be identified at the outset of a project and that benefits are verified during the life of the project. For practitioner-oriented guides for BM as part of project, program and portfolio management we refer to OGC [1,23,24].
Before the conception of BM, other research areas have been relevant in addressing shortcomings in benefits realization from IT projects. One such area is software economics, which started to gain traction in the 1960s, with substantial contributions in the 1980s and 1990s focusing on the application of microeconomic techniques in software engineering throughout the software lifecycle (e.g. [25,26]). Software economics seeks to understand relationships between economic objectives and technical software issues to improve value creation at multiple levels, including project, program and portfolio [26]. Software economics can, for example, provide better models for estimating benefits from software [26, p. 328], that can be used in the BM process element 'identifying and structuring benefits'. Another relevant area of research, developed partly in parallel with BM, is the valuebased software engineering approach, which, from the early 2000s, attracted significant attention from researchers seeking to link software engineering decisions further to economic value [27][28][29]. BM has several overlapping concerns with software economics and value-based software engineering, such as the estimation, planning and monitoring of benefits [27]. Software engineering and value-based software engineering differ from the process approach that comes with BM by having focus on value in software engineering disciplines such as requirements engineering, architecting, design, development, verification and validation of software (see e.g. [27]). In our review, we include relevant findings from these streams of research to the extent they are relevant to our RQs.

| EXISTING LITERATURE REVIEWS ON BM
We identified four previous literature reviews on BM (see Table 1). Braun et al. [30] give a high-level overview of BM research between 1990 and 2007. As in our review, Braun  Casey et al. [32] provide a literature review of BM and conclude that mechanistic approaches to benefits realization have never been adequate and that the social nature of benefits realization must be considered along with political intentions and behaviour. This has also been indicated by other researchers, such as Breese [6], who suggests that defining and measuring benefits is not a 'neutral' process. The rather low level of BM adoption beyond the establishment of the business case might be understood through the lens that Casey et al. [32] provide.
Breese et al. [20] provide a literature review covering the history of BM from 1990 to 2015; however, it is unclear if this was a systematic or exploratory literature review. Breese et al. look at the lack of BM adoption through the theoretical lens of translation, which focuses on the processes whereby management ideas influence management practice. Similar to our findings, Breese et al. suggest that few organizations seem to take a full lifecycle approach to BM. The authors suggest that certain factors can help explain this, such as the lack of a common understanding of the concepts of benefit and value. Breese et al. indicate that adoption will only happen when the benefits of BM are recognized and accompanied by short-term gains. Consequently, implementing BM practices at the project level is challenging.
Our literature review differs from previous reviews in a number of significant ways: (1) differences in research questions-in particular, none of the previous reviews has focused on our RQ2; (2) longer time span and inclusion of recently published studies and (3) use of a wider range of sources. With respect to papers included in previous reviews, we found a rather small overlap with our own review (see Table 1). This might be due to several factors, such as the fact that the reviews have different time spans, research questions and review protocols.

| RESEARCH METHOD
We have based this literature review on recommendations given in Brereton et al. [33] and Kitchenham [34]. We established a search strategy (Section 4.1) and developed a review protocol with inclusion and exclusion criteria (Section 4.2). The search strings (Section 4.3) were executed and the results extracted and synthesized (Section 4.4.). To distinguish between papers that are included in the review and other papers referenced, we have used 'LT' as a prefix for papers included in the review (Appendix A).

| Search strategy
We derived keywords to be included in our search strings starting with our research questions and continuing with an exploration of literature. We verified the quality of the search strings by assessing whether trial searches returned papers we HOLGEID ET AL. knew were relevant based on a combination of prior knowledge and findings from our exploratory review of literature. The search strategy combined the use of Google Scholar (GS) with more precise searches in SCOPUS, IEEE Explore and ACM Digital Library. Support for using GS can be found in the studies by Gehanno et al. and Halevi et al. [35,36], but we added the additional three databases to limit the risk of missing relevant publications and decrease the risk of including papers of low quality.

| Review protocol
Included in our review are primary empirical studies relevant to our research questions that are reported in peer-reviewed papers written in English. Only studies of the adoption and impact of BM in software development or other IT projects are included. Excluded are books and grey literature (such as discussion papers, technical reports, academic statements, lecture notes, presentations) and contributions that lack relevance and rigour. Relevance is considered in relation to our research questions. Rigour is assessed based on the dimensions suggested by Ivarsson and Gorschek [37]: description of context, study design and validity discussion. As presented in Figure 2, only four papers were excluded based on rigour. All of them also had weak relevance in relation to our research questions. No studies were excluded solely due to a lack of rigour.

| Literature review search strings
We used the following search string logic for this literature review: When using GS we did not include words in the search string to limit the identified studies to empirical studies, due to the limitation of the GS search string length. Instead we went through a manual identification process to remove nonempirical studies. When using the academic databases (SCO-PUS, IEEE Explore and ACM Digital Library), we applied a slightly more narrowing search string by only including papers that also contained some of the following words: 'empirical', 'case study', 'survey', 'action research', 'interview', 'delphi research', 'document study' and 'experiment'.
Precise search strings per database are presented in Appendix B. We filtered the results from each database, applying the inclusion and exclusion criteria, through four stages: (1) identify potentially relevant papers; (2) review titles and casual abstract review; (3) review abstract and (4) assess full papers.
We downloaded the result sets from the respective database in CVS format and imported the sets to MS Excel spreadsheets. We assessed the results from each database independently and did not remove duplicates before comparing the end set of papers coming out of Stage 4. This gave an extra verification as we were able to reassess papers that we had previously found in the other databases.
In total, 4836 papers were identified in stage 1 (see Figure 2). The filtering process in the subsequent stages eventually yielded 47 papers to be included in this literature review: GS resulted in 42 papers, SCOPUS five additional papers, and IEEE Explore and ACM Digital Library zero additional papers.

| Data extraction and synthesis of results
The method of thematic synthesis of results is one of the predominant methods used by software engineering researchers for synthesizing systematic review data, according to the study by Huang et al. [38]. We were inspired by the five steps for thematic synthesis suggested by Cruzes and Dybå [39]: (1) We extracted data from the selected papers in an iterative manner, focusing on the following items: context, type of publication (journal/conference paper), and research method; and, where relevant and available, number of respondents, response rate, place of study (country), time of study (year) (see Appendix C). We also extracted quantitative and qualitative empirical data relevant for our research questions (RQ1 and RQ2). All the extracted data is presented; in Section 5 and in the appendices. (2) We identified and coded interesting categories relating to the RQs in an iterative fashion, using an MS Excel spreadsheet. Among the categories that gradually emerged were BM adoption, BM adoption per process element in the BM framework presented in the study by Ward et al. [3] F I G U R E 2 Results from the filtering process HOLGEID ET AL.
(see Section 2), types of benefits, effectiveness of benefits identification, overstatement of benefits, responsibility for the realization of benefits, project lifecycle perspectives to BM, BM impact on the actual realization benefits. (3) We arrived at the themes that are presented in Section 5.
The extracted data on BM adoption (Appendices D and E) were analysed and papers categorized by various adoption rates (RQ1) and practices with impact on project outcome (RQ2). The results are aggregated in Section 5.3. (4) Based on the themes, we have included a higher-order discussion (Section 6), along with direction for further research. (5) We considered the validity and limitations of our study (Section 7). The first author executed the filtering and data extraction process. Quality checks were performed by the other authors, such as whether known and relevant empirical studies were included in the final set of papers. They also reviewed extracted data elements. This section is structured according to the RQs and the respective themes that emerged from this literature review. After a presentation of findings related to RQ1 (Section 5.1) and RQ1 (Section 5.2), we present an aggregation of the results (Section 5.3). Table 2 summarizes the themes with reference to the sections where the corresponding results are presented. For example, the first row of Table 2 shows that 11 empirical studies have relevant findings on BM adoption. This section presents studies with empirical data on BM adoption in general. Details are presented in Appendices D and E. For adoption rates of specific BM practices, see subsections referenced in Table 2.

|
Two of the studies report a very low adoption rate (<¼ 25%) [LT45, LT46], but we note that one of these is from the very early days of BM, so a low level of adoption would be expected in the 1994 survey of UK organizations reported by Ward et al. [LT45]. Very low adoption of BM has also been found in case studies, such as [LT27]. Of the remaining studies, five report a low (26-50%) [LT22, LT23, LT24, LT28, LT29], one a high (51-75%) [LT39] and two a very high adoption of BM (>75%) [LT33, LT37]. Of the studies with very high adoption, one is based on a sample of organizations from Switzerland with mandatory practices related to BM [LT37] and the other reports from studies of UK and South African organizations where elements of BM processes were present and mostly adopted in an ad hoc manner [LT33].
Comparisons of the adoption results of the studies must be done with caution. Although some studies use similar questionnaires, the studies sometimes use slightly different terminology. For example, some studies report whether the respondent's organization had (ever) used BM methods (e.g. an Australian study [LT23]). Other studies report whether the organization had such methods in place at the time of the study (e.g. a South African study [LT39]). A few studies report the actual level of usage (e.g. a Taiwanese study [LT24]).
Most surveys are based on small sample sizes in the range of 30-110 respondents, except for the Nordic survey reported in the study by Hallikainen et al.
[LT11] and the Australian survey reported in the study by Lin [LT22]. Survey response rates are between 4% and 34%, with the exception of [LT15] (71%) and [LT17] (85%). Furthermore, although most of the studies analyse convenience samples, few can claim complete random selection, as the studies have been done in certain geographies with organizations possessing certain characteristics ranging from a specific sector (e.g. in Australia [LT29] and United Kingdom and South Africa [LT33]) to a specific size (large-sized organizations in Australia [LT28] and SMEs in Taiwan [LT24]), and with the selection of potential respondents from social media groups in a study of organizations across the Arab World, Europe and US [LT3].
Most of the studies report a relatively low level of BM adoption and organizations that are in possession of BM methods report that they are not used to their full extent [LT22,LT24,LT28]. As can be seen in Table 3 and Appendix E, there is a tendency for higher adoption levels of process elements early in the project lifecycle (such as identifying and structuring benefits) than of process elements that typically come into focus later in the lifecycle (such as the potential for further benefits).

| Identifying and structuring benefits
Adoption Most of the papers (71%) report data that indicate high or very high adoption of identifying and structuring benefits (Table 3). Although the adoption level of this process element is higher than the other process elements, a UK survey indicates that few of the respondents were satisfied with the appraisal techniques [LT45]. A survey of Taiwanese organizations found the

Level of adoption a BM process elements suggested by [LT45]
Identifying and structuring benefits

Planning benefits realization
Executing benefits plan -7 methods to be present but not 'widely' adopted [LT24]. Business case creation has been found to be a common practice in organizations in the Nordics [LT11, LT15 report that non-monetized benefits were sufficiently documented in less than half of the 58 Norwegian public sector projects included in the study. One exception among the studies included in our review is Naidoo and Palk [LT33], who report that the agencies studied identified and monitored intangible benefits such as technological and organizational improvements (75.7%) and process improvements (73%). Ward et al.

Potential for further benefits
[LT46] found the types of benefit to be of relevance in determining the extent to which organizations practise identification and structuring benefits. The study found that organizations more regularly identify and structure benefits such as cost reduction and cost avoidance, compared with benefits related to, for example, societal benefits.  [LT15] who found 52% of the respondents to practice uncertainty assessment of costs while only 31% practiced quantitative uncertainty assessment of benefits estimates. Jørgensen [LT18] found that more realistic judgements of costs and benefits uncertainty were achieved when professionals looked back on previous estimation errors of similar projects instead of using traditional minimum-maximum methods (wider uncertainty intervals, more left-skewed benefits distributions and rightskewed costs distribution). Empirical studies have found that the benefits tend to be inflated. Over-optimism in general has been found to be a factor when planned benefits are not realized [LT15]. Two studies report high levels of overstatement of benefits (54%-70%) [LT29,LT39] and four studies report a lower level of overstatement (26-48%)

Estimation of benefits
The high level of overstating benefits seems not to drop over the years. The pre-project justification of a project seems to be focused on getting the project approved rather than establishing a realistic picture of the benefits [LT1, LT23]. To reduce risk of optimism in business case appraisals, external reviews have been found effective [LT44]. Studies report that the benefits identification processes in the early project phases do not uncover all benefits [LT28, LT39, LT45, LT46] and that there is a lack of emphasis on the identification of further benefits during, and after, the project (see Section 5.1.6).

| Planning benefits realization
Adoption Ten studies report quantitative measures of benefits planning adoption, of which six report a low level of adoption, one reports a high level and three a very high level (

| Executing the benefits realization plan
Four out of seven papers report low levels and three report high levels of adoption of BM during project execution (Table 3). Six of seven papers report activities related to benefits monitoring in the range of 29-67% of the surveyed organizations (Appendix E). Organizations have been found to adopt such practices in a sporadic manner and the practices have been found not to be in widespread use [LT1]. Naidoo and Palk [LT33] found 71 per cent of respondents to have ad hoc adoption of processes related to benefits monitoring.

| Evaluating and reviewing results
Organizations have been reported to perform postimplementation reviews. However, these reviews are not always concerned with assessing benefits delivery [LT28]. Such reviews are sometimes challenging to perform when circumstances have changed during project execution [LT9]. The level of adoption of this process element appears lower than that of 'identifying and structuring benefits' ( Table 3). The low level of adoption of benefits evaluation has been found in case studies as well, for example, in the studies by Ashurst et al. 5.1.6 | Potential for further benefits According to Table 3 and Appendix E, this process element is typically adopted less than all the other process elements presented in ref. [3]: six out of seven papers report low or very low levels of adoption. This is consistent with the case study findings reported by Ashurst et al.
[LT1], who found that project teams were typically disbanded immediately after the go-live date and further exploitations of benefits were not practised. 5.1.7 | Lifecycle perspective on BM Ward et al. [LT45] found that few organizations had a comprehensive process to ensure that suggested benefits were realized. Ten years later, Ward et al. [LT46] found that organizations tended to focus on benefits in the early stages of the project to build a sound business case and did not follow a BM method through the project lifecycle. A focus on BM in the early project stages were also reported in previous studies [LT4, LT12, LT37]. Nielsen and Persson [LT35,LT36] identified in their studied Danish organizations a potential to update the business cases as required throughout the lifecycle. A Swiss study [LT37] found only 19 per cent of the responding organizations to be tracing benefits over the whole project lifecycle. A rather sporadic adoption of BM was reported by Ashurst et al.
[LT1], who found a lack of on-going focus and commitment to the benefits, and Ashurst and Doherty [LT2] found some reluctance to apply benefits practices in a consistent and extensive manner. Hellang [LT13] report variations in BM adoption across selected public projects in Norway: some had a focus on actively managing benefits throughout the lifecycle, others had emphasis on providing good decision and prioritization support for IT investments. Variations in BM adoption across government entities were also found by Nielsen et al. [LT34], who present case studies of Danish municipalities with and without an on-going post-project benefits realization process. One of the case studies from United Kingdom presented by Doherty et al.
[LT7] shows how an on-going benefits review process was used to ensure that benefits were achieved. Doherty et al. [LT8] show how BM practices can help keep benefits firmly on the agenda and facilitate benefits-oriented communications between the stakeholders.

| Research question 2:
what is the impact of BM on project outcome, according to previous empirical studies?

| Adoption of BM
Ul Musawir et al. [LT43] report a survey with respondents from 47 countries investigating relationships between project governance, BM and project success. Project success was split into project management success (cost, time, quality/scope), project ownership success (project owner's success in realizing the business case) and project investment success (actual value generated from the investment), as suggested by Zwikael and Smyrk [40]. Ul Musawir et al. [LT43] report that BM had statistically significant positive correlations with all three types of project success (project management success, r ¼ 0.500; project ownership success, r ¼ 0.514; project investment success, r ¼ 0.533; all correlations were significant at the 0.01 level).
A positive relationship between BM and actual benefits realization was also reported in several Australian studies. Lin and Liu [LT26] reported case studies and surveys of Australian organizations and found the use of BM methods to be positively related to the organizations' confidence that the benefits would be delivered. Further evidence was provided by Lin et al. [LT25], who found a significant relationship between BM and benefits realization (path coefficient β ¼ 0.194, p < 0.01). In another Australian study, Standing and Lin [LT40] found that organizations using investment evaluation methodologies or BM methodologies achieve better benefits than other organizations.
In a survey of respondents from Brazil, the United Kingdom and the United States, Serra and Kunc [LT38] found BM to be associated with the creation of benefits for the business by measuring the degree to which the projects delivered products that helped the business to generate the expected outcome, avoid undesired outcomes, fulfil the business case and deliver the expected returns on the investments. In a Delphi research involving Belgian financial services professionals, De Haes and van Grembergen [LT6] found BM to be high in perceived effectiveness in helping to meet the demands of the business. Smith et al.
[LT39] reported on a survey from South Africa, finding that 56% of the respondents had BM methods in place and, of those, 70% reported the methods to be effective in achieving successful information systems to an extensive or frequent degree (scale: not at all, seldom, frequently, extensively). Further potential associations between the adoption of BM and good benefits are provided by Holgeid et al. [LT16].
Kopmann et al. [LT20] report on a German study of business case control (business case existence, business case monitoring, business case tracking) in relation to project portfolio success (defined as a multidimensional construct including average economic project success that focuses on project effectiveness in delivering, for example, increased revenues, customer satisfaction and profitability). The authors found a significant positive correlation between business case control and project portfolio success (r ¼ 0.30, p < 0.05).
BM has been found to reduce the likelihood of projects becoming out of control. Budzier and Flyvbjerg [LT5] studied a large sample of IT projects, investigating IT project cost and schedule overrun as deviations from expected benefits. The sample had a mean deviation from expected benefits of À 29.3% and a median shortfall of 0%, thereby indicating a left-skewed distribution. The authors found BM to contribute to fewer instances of cost and schedule overruns.
A survey of professionals from Germany, Austria and Switzerland found BM to be positively associated with effective project portfolio management, which in turn was found to be positively associated with strategic goal achievement (Mohan and Ahlemann [LT30]). Ward et al. [LT46] found that organizations that were more successful in delivering benefits than others performed better in identifying and structuring benefits, were less likely to overstate benefits to ensure approval, and typically included a wider set of types of benefit. In line with this finding, Ul Musawir et al. [LT43] found a significant positive effect on project investment success from the practice of having clearly measurable benefits (SEM 1 effect 0.146, p < 0.001). Mohan et al. [LT32] reported a positive relationship between benefits identification and benefits realization (path coefficient β ¼ 0.15, p < 0.01).

| Identification and structuring of benefits
Holgeid and Jørgensen [LT15] found practically no difference in client benefits between projects with or without a business case. Jørgensen [LT17] reported an increase (though not statistically significant) in successful client benefits delivery when organizations practised cost-benefit analysis of different alternatives before the start of a project (6% increase, p ¼ 0.8) and when the expected benefits were clearly communicated to stakeholders (22% increase, p ¼ 0.2). The weak link between cost-benefit analysis and actual benefits reported in the studies by Holgeid and Jørgensen [LT15] and Jørgensen [LT17] is supported by Badewi [LT3], who found that business cases alone were not sufficient to achieve investment success.

| Planning benefits realization
The studies reported by Lin and Pervan [LT28] and Ward et al.
[LT45] suggest that it is hard to envisage effective realization benefits without a plan. Empirical support for this is given by Mohan et al. [LT31], who found benefits planning to help in facilitating effectiveness in realizing planned benefits; Mohan et al. [LT32], who reported a significant influence on realized benefits from benefits planning (path coefficient β ¼ 0.17, p < 0.01); Holgeid and Jørgensen [LT15], reported a significant increase in perceived client benefits when benefit plans were present (mean rank present/not present: 37.1/23.3, p ¼ 0.01); and Jørgensen [LT17], who found a significant increase (31% increase, p ¼ 0.03) in successful client benefits delivery when plans for realizing benefits were present (processes for prioritizing and managing activities during a project for achieving benefits). Ward et al. [LT46] found that organizations successful in realizing benefits were more apt to assign responsibility for benefits realization to business managers. Similarly, Badewi [LT3] and Thomas et al. [LT42] found that the practice of assigning responsibility for benefits realization was the most important aspect of BM in achieving investment success. The practice of having clarified responsibility for the realization of benefits has been found to impact perceived client benefits [LT15]. Mohan et al. [LT31], based on a study of literature, a field study and a survey, proposed that incentives (e.g. bonuses and promotions) might be of importance to benefits realization. Mohan et al. [LT32] found incentive management to positively relate to the use of BM practices, which in turn positively influenced benefits realization success. Mohan et al. [LT32] also found positive relationships between good business-IT communication and business process knowledge and the use of BM practices. Kopmann et al. [LT20] found that the relationship between business case control and project portfolio success (introduced in Section 5.2.1) was strengthened when responsibility for business case realization was well defined. Thomas et al. [LT42] reported on a study of Australian organizations and found that when people were accountable for IT project results, the business cases were more accurate and that benefits were less frequently overstated to get approval than in organizations not holding people accountable for project results.

-
successful and very problematic if the project was cancelled or at least one dimension was very problematic. All other projects were categorized as problematic. Jørgensen et al.
[LT19] found that projects that did not practise BM during project execution were more likely to be problematic than other projects. Fifty percent of the projects that did not practise BM during project execution were problematic, while 18% of the projects implementing this practice were problematic (test of difference in proportions: p ¼ 0.027). Jørgensen et al.
[LT19] found additional factors that contributed to problematic projects. Projects that had fixed-price contracts, as opposed to time and material contracts, often did not practise BM during project execution. Sixty-two per cent of the projects with fixed-price contracts lacked BM during project execution, while 33% of the projects with time and material contracts lacked this practice (test of difference in proportions: p ¼ 0.059). Problematic projects were also found to be associated with weak client involvement (test of differences in proportions with/ without strong client involvement: p ¼ 0.055), a lack of frequent deliveries to production and a lack of flexible scope (test of differences in proportions with/without these practices: p ¼ 0.034).
Further support for the practice of BM during project execution is given by Holgeid and Jørgensen [LT15] and Mohan et al. [LT32], who reported that the ability to execute the benefits realization plans (measured by stakeholders' ability to manage the activities to realize benefits, benefits realization reporting, executing the benefits realization plan, and applying a methodology for benefits realization) was of significant importance in realizing benefits (path coefficient β ¼ 0.27, p < 0.001).

| Evaluating and reviewing realized benefits
The more successful organizations reported in Ward et al.
[LT46] reviewed and evaluated the results more extensively than the less successful ones. Jørgensen [LT17] found an increase in successful client benefits delivery when processes for evaluating benefits after project completion were present (19% increase, Several studies highlight positive effects on project outcomes from on-going review and evaluation of benefits. Mohan et al. [LT31] reported the practice of measuring benefits and reviewing benefits at any point in the project lifecycle to help in benefits realization. Mohan et al. [LT32] found a significant influence on benefits realization from on-going benefits reviews (path coefficient β ¼ 0.24, p < 0.001). They also found benefits reviews to have the greatest potential among the studied BM practices to increase the probability of benefits realization. Ul Musawir et al. [LT43] reported a significant positive effect (p < 0.001) on all three types of project success (introduced in Section 5.2.1; SEM effects: project management success 0.306, project ownership success 0.443, project investment success 0.456) of practising a process of continuously reviewing and realigning the expected benefits with the business. Ul Musawir et al. also found a significant positive effect on project investment success from the practice of having activities related to training, support, monitoring and outcomes evaluation to ensure the integration of project outputs into regular business routines (SEM effect 0.093, p < 0.05). Holgeid and Jørgensen [LT15] found perceived client benefits to be significantly higher when the practice of evaluating realized benefits was present (mean rank present/ not present: 33.4/16.4, p < 0.001). Thomas et al. [LT42] performed a multiple qualitative case study analysis of Australian organizations and found improved IT project outcomes from effective evaluation practices that help in selection of the right projects, consistent and timely decisionmaking, focused project delivery, corporate learning, timely stopping of projects, accurate estimation and reduced politics.

| Potential for further benefits
We found only one study to report findings on how the practice of doing post-project benefits identification can be associated with good benefits. This is not surprising, as the adoption level of this practice is rather low, as shown in Table 3. Holgeid and Jørgensen [LT15] found that perceived client benefits increased when this practice was present, however the increase was not significant (mean rank present/ not present: 33.0/25.1, p ¼ 0.058). Table 4 presents an aggregation of the main results, categorized by BM practices that have been found to be associated with project outcome. The table also shows to what extent the practices are adopted.

| Aggregation of results (RQ1 and RQ2)
Previous empirical studies have found many organizations to have a potential for further implementation of BM (RQ1). Only 30% of the studies with quantified levels of BM adoption found adoption levels above 50%. Organizations appear to use BM more often in the early phases of IT projects than in the later project phases and post-project period, as can be observed from Tables 3 and 4. Responsibility and incentives for realizing benefits are frequently reported with adoption rates above 50% but with large variations across the studies (Table 4 and Section 5.1.3).
We found evidence that organizations, in general, obtain positive effects on the realization of benefits from using BM practices (RQ2). Practices with such positive effects include all Cranfield process elements as well as the practice of assigning responsibility for realizing benefits (Table 4). However, limited evidence has been found regarding effect on benefits from usage of the process element 'potential for further benefits' (Section 5.2.7). Figure 3 corresponds to Table 4 and presents an aggregated view of the main findings from this literature review by plotting BM practices according to their adoption rates (X-axis) and frequency of papers with empirical evidence of positive project outcome when BM practices were adopted HOLGEID ET AL.
-11 (Y-axis). The practice of having a business case was only found to have neutral effect on project outcome. Therefore, this practice is not included in Figure 3. In Figure 3, the adoption rate of a practice is the proportion of the studies that are reporting adoption levels above 50%, that is, more than half of the respondents answering confirmative on the question on whether a practice has been adopted or not (or variations of this question as detailed in Appendix E).
As presented, three practices are in the bottom-left quadrant. This is indicating a low adoption level and scarce evidence on the effect of the practices on project outcome. This was true for the practices of identification of further benefits, benefits planning and BM during project execution. This does not mean that the practices have low effect on project outcome, but it appears that such effects have not frequently been studied. Another practice with low adoption rate is the practice of evaluating and reviewing results (upper-left quadrant). However, this practice is more frequently found to be associated with good benefits. The practice of having people responsible for benefits realization is frequently found in relation to projects with good benefits and is often adopted (upper-right quadrant). Identifying and structuring benefits, Size of effect and level of significance are presented in Section 5.2.

-
however, is frequently adopted but few studies have confirmed its effect on benefits realization. There is no surprise that practices with well documented positive effects are adopted, nor is it surprising that practices with scarce documented effects are less adopted. We find it more surprising that a practice with well-documented associations with good benefits has a low adoption level, which is true for the practice of evaluating and reviewing realized benefits.

| DISCUSSION AND FUTURE DIRECTIONS FOR RESEARCH
This section first discusses our main results and then presents four directions for future empirical research on BM. This literature review has documented empirical evidence of the impact of BM on project outcome, as summarized in Section 5.3, Table 4 and Figure 3. Ul Musawir et al. [LT43], for example, report that adoption of BM practices relates to higher levels of investment success. Researchers have found that the business case alone is not enough to achieve such success [LT3, LT15]; an on-going management of benefits throughout the project lifecycle is recommended.
As highlighted by Volden [LT44], referring to Ul Musawir et al. [LT43] and Serra and Kunc [LT38], the business case provides the rational for a project's preferred solution and is therefore important for future benefits and cost manage- who report that unexpected benefits were a major driver of value. In the digital era, unexpected change happen, as, for example, illustrated by Mandrella et al. [41] who found a fundamental change taking place in business value creation as it is shifting outside the direct control of a single organization and towards a diverse ecosystem of value chain partners. BM tries to capture unexpected and emerging benefits by the process element 'potential for further benefits', as well as the circular process as illustrated by the edges in the process model ( Figure 1). This literature review, however, found low adoption of the mentioned process element. As noted by Hellang et al.
[LT13], organizations can take other approaches to BM, such as the linear process of justification planning with the goal of providing good decision support for the investments, estimate benefits and ex post benefits realization.

F I G U R E 3
Benefits management (BM) practices with impact on project outcome: Proportion of papers with more than 50% BM adoption rate versus freq. of papers with positive project outcome and practice adoption. Abbreviations are references to Table 4 HOLGEID ET AL.
Several studies report that the benefits identification processes in the early project phases do not uncover all benefits (Section 5.1.2.3). However, it is not clear if this is just a lack of diligence or if this simply is because a more comprehensive list of benefits is not necessary to provide an attractive business case to ensure approval. Cantor [42] argues that innovative programs almost by definition begin with incomplete information, resulting in uncertainty in both expected project costs and benefits. The associated uncertainty might also be increased by having a long-term benefits realization horizon, as suggested by Sassone [43].
The body of empirical research seems not to provide much understanding of the purpose of the business case, to what extent it is established with an eye to help facilitate good benefits or mere a mechanism to secure project approval. Volden [LT44] put forward that there is limited research with attention paid to the quality and utility of cost-benefits analysis (CBA). Volden state 'This is surprising, as we would normally expect that the quality of an analysis affects the extent to which CBAs are used, their recommendations followed and social benefits realized' (p. 550).
While empirical research has found evidence to support the effectiveness of BM practices in contributing to good benefits, most organizations seem to have a potential to adopt such practices to a greater extent. Many organizations might have a potential to move from a summative approach to BM towards a formative one characterized by dynamically adaption to emerging benefits.

| Understanding why effective BM practices sometimes are not adopted
To further understand why many organizations have not made use of practices shown to contribute to good project outcomes, we might benefit from using other views of management practices than the ones used by most of the empirical studies, herein. Use of theories independent of the studied field of research is encouraged by Laursen and Svejvig [44]. Much information systems research leans on the assumption that managerial actions are rational and aimed at maximizing efficiency and effectiveness, as pointed out by Mignerat and Rivard [45]. Casey et al. [32] found mechanistic approaches to benefits realization not to be adequate and that the social nature of benefits realization should be considered alongside political behaviour and intentions.
We suggest the use of institutional theory to gain new insights into why BM practices sometimes are not adopted. As pointed out in Mignerat and Rivard [45], referencing [46]: 'The central underlying assumption of institutional theory is that organizations and organizational actors seek to gain legitimacy in their environments in order to be accepted and thus ensure their long-term survival'. Institutional effects are related to processes whereby institutions affect other institutions or organizations ( [45], referencing [47]). Institutions can be seen as the rules of the game and organizations the players [48].
New insight into BM adoption can potentially be provided by empirical studies leveraging the typology of strategic responses that organizations can implement when faced with institutional pressure towards conformity offered by Oliver [49]. Through the lens of institutional theory, we might further seek insight into why BM is perceived as challenging to implement and why benefits are often not quantified and lack measurability (Section 5.1.1.3). BM practices implemented according to Ward et al. [3] can provide transparency of benefits realization from the early identification of benefits throughout, and beyond, the project lifecycle, including transparency of who is responsible for the realization of benefits. Future studies might shed light on the pattern of practices adoption by studying how organizations are coping with this transparency-for example, through the lens of institutional theory, where actors can take on avoidance strategies, such as avoidance of responsibility of benefits realization, to cope with institutional pressures [49]. For an introduction to institutional theory applied to the management of projects, we refer to the study by Biesenthal et al. [50].

| Assessing the impact of BM in relation to, and integrated with, other management practices
As introduced in Section 2, Ward et al.
[LT46], referring to [51], suggest a high-level view of how BM relates to other processes and methods such as strategic planning, project portfolio management, investment appraisal, risk management, project management, systems development methods and change management methods. Ward et al. [LT46] state that BM 'links together decision making about which investments to make, based on the benefits that can be realized, with the selection of methodologies appropriate to the delivery of the benefits intended'. Some of the reviewed papers recognize that BM typically is not practiced in isolation but intersects with other management practices that together with BM can provide good effects on project outcome. Examples of such practices are project governance [LT43], project portfolio management [LT30] and investment evaluation practices [LT25]. Yet much work seems to be ahead to understand how to make the most of BM in combination with other processes and methods.
According to the study by Ward et al. [LT46], the Cranfield process model was originally designed to help realize benefits from a single project. The complexities following multiple related projects, or prioritization across a portfolio of projects, call for further study of how BM best fits with other processes and methods. In such complex settings, further studies are needed to advance our knowledge of how to manage uncertainty and risks associated with the realization of benefits; from the estimation of benefits through execution and post-project benefits realization. The estimation of benefits and how to account for uncertainty in benefits have not received much attention [52]. Sassone [43] states that 'benefits are frequently long term, uncertain and intangible'. The inherent uncertainty associated with benefits might motivate studies of connections between BM and risk management practices. As noted by Liu and Wang [53], risk management is recognized as effective to improve the performance of IT projects. Further studies of intersections between BM and risk management can be inspired, for example, by Liu and Wang [54], who investigate how managerial controls can impact project performance; by Haq et al. [55], who study the effectiveness of project government mechanisms in the presence of risks; and by Willumsen et al. [56], who provide results from a study of project risk management as a means to create value.

| BM research on projects using agile software development methods
While this literature review found 47 empirical studies on the adoption of BM practices and their effects on project outcome, they provide little insight into potential variations in realization of benefits between projects that use different software development methods. Apart from the previous studies reported [LT15, LT17, LT19], which studied two agile practices in relation to benefits realization, and [LT41], who found difficulty related to use of BM in agile projects, none of the reviewed studies investigated BM in agile projects. This may seem surprising given the first principle in the Agile Manifesto [57], 'Our highest priority is to satisfy the customer through early and continuous delivery of valuable software', and given the principles of SCRUM, 'deliver maximum business value, from beginning early in the project and continuing throughout' [58].
We suggest research to investigate the fit of BM in relation to agile software projects to establish whether the original cyclical nature of the BM model [3] is suitable or if adjustments are necessary to integrate with agile methods. For example, we have seen no empirical research investigating the edges in the Cranfield process model (linkages between process elements). In agile contexts, the original edges of the BM model might potentially be challenged-for example, by addressing why some process elements have no edges between them (e.g. 'evaluating and reviewing results' and 'identifying and structuring benefits'). The 'missing' linkages may make perfect sense in traditional software development, but interactions between process elements might be more intense in agile software development due to rapid cycles of development and continuous value delivery. We call for more research to better understand how BM can best complement agile practices.  [LT1,  LT3, LT5, LT11, LT17, LT19, LT20, LT24, LT28, LT40, LT43,  LT45, LT46]. Still, how adoption and effects of BM vary across industries or types of organization have not been investigated in depth. Research in the field of strategic management has long recognized that organizations can have unique ways of creating value [59]. However, we find no studies that seek to uncover whether organizations relate the way benefits are managed to the unique characteristics of how value is created in their organizations.

| BM studies in the context of organizations' value creation logics
We suggest future research on BM to include what types of benefit are being aimed for. Some studies distinguish between tangible and intangible benefits, but few specifics are provided about the benefits being aimed for. Furthermore, we suggest more studies on how organizations create value. Existing empirical research seems to take a generic view of benefits and seldom relates their realization to the organization's unique characteristics. In contrast, Zwikael et al. [60] suggest three goal-setting dimensions for effective benefits; (1) specificity: benefits targets should be specific rather than vague and nonquantifiable; (2) attainability: goals should be challenging, but possible to attain and (3) comprehensiveness: the extent to which the benefits fully reflect organizations' strategies and key stakeholders' objectives. Empirical research on software projects applying such dimensions can help advance our understanding of factors that can help improve project outcomes.

| VALIDITY AND LIMITATIONS
We have searched extensive parts of the literature; however, we have likely not covered all relevant sources. There is a risk that we have missed out on relevant publications as we only searched in four databases. However, we consider our inclusion of GS to some extent to mitigate this risk (see Section 4.1). The risk of missing relevant publications is related to search string limitations. For example, there is a possibility that relevant papers have made use of terms other than our search words 'benefit' and 'value' and thus not been included in our search results. We tried to mitigate this by carefully constructing the search strings presented in Section 4.3 and conducting several test runs to verify that relevant papers were included. In searching for additional empirical studies to include in our review, we carried out snowballing (exploratory, not systematic). This did not result in additional empirical studies being included in our review.
Paper selection consistency is another threat we considered when crafting the review protocol. Kitchenham [34] suggested that '[a] predefined protocol is necessary to reduce the possibility of researcher bias. For example, without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations' (p. 4). To some extent, the risk of researcher bias driving paper selection inconsistency is mitigated in our review by our research questions, which are not aimed at comparing practices. Nevertheless, there could be a risk of the researchers selecting papers that confirm an expectation of, for example, BM adoption rates and effects. Therefore, we have followed Kitchenham's advice to craft a review protocol and carefully adhered to this protocol during the execution of the review.
Publication bias, as defined in Kitchenham [34] and empirically assessed in the study by Zwikael et al. [61], refers to the problem that a positive research outcome is more likely to be published than a negative one. As noted by Kitchenham, special efforts are needed to address the problem of publication bias in systematic reviews. In our review, we consider this bias to some extent to be mitigated as the research questions are not designed to compare, for example, management practices. Our focus is rather on finding evidence of how, and to what extent, BM is used and what impact it has.
The body of empirical knowledge about BM is scarce, so we need to be careful when drawing conclusions from empirical findings based on a few studies. Furthermore, there is a risk of comparing results from studies that have taken place in different contexts (different countries, different times, different industries, etc.) and that sometimes also use slightly different terms and, for example, ask survey questions in a slightly different way. We have tried to mitigate this risk by introducing each study with a short context description (Appendix C) and by referencing carefully so the interested reader can trace back and get a better understanding of the context. Furthermore, most of the studies reported have been based on convenience samples (not random samples) with few respondents, and some with low response rates. This calls for careful interpretation and weakens the generalizability.
Eighty-four authors contributed to the included studies. While most of the authors were involved in only one study (65 authors), nine authors were involved in two publications, six authors in three publications, three authors in four publications and one author in nine publications (main author in six and co-author in three). The inclusion of several papers authored by the same researcher(s) might limit the variation in methodology and data sources used, and therefore represent a threat to the validity of our study.

| CONCLUSIONS
Relatively few empirical studies have investigated the adoption and impact of BM in the context of software development. We must thus be careful about drawing general conclusions. Bearing this in mind, we have found no systematic increase in the adoption of BM practices over the years, and most organizations seem to have the potential to a greater extent to adopt BM practices that are empirically found to be associated with good benefits.
Many organizations appear to have a potential to move from a summative approach to BM towards a formative one characterized by dynamically adaption to emerging benefits. Such move is especially relevant to consider given the current digital era characterized by rapid change where long-term planning of benefits can be found challenging and insufficient to drive more benefits from IT investments.
Based on our literature review and references to supporting papers, we identified four directions for future research on BM. Firstly, to provide a deeper understanding of BM adoption (and lack of adoption), we propose leveraging theories not frequently applied in the BM stream of research, such as institutional theory. Such research could potentially help us understand why some organizations do not implement BM practices and what is different about the organizations that practise successful BM and those that do not. Second, although researchers have long suggested that BM integrate with other management practices (e.g. [LT46]), much work seems to be ahead to understand how to make the most of BM in combination with other processes and methods. Third, although agile software development has for many become the default software development method, little research has been done of BM in agile projects. Finally, although research has been conducted in various industries and types of organization, little emphasis has been placed on the types of benefit being pursued, and few considerations have been made with regard to the unique characteristics of the studied organizations. We suggest that future research puts more emphasis on types of benefit and how they relate to organizations' specific value creation logics. Context of the studies (S ¼ survey, C ¼ case study, A ¼ Action research, D ¼ Document study)

Ref S C A D Context
[LT1] x Case study of 25 IT projects based on documents from a large IT consultancy's knowledge base and additional data collected from 15 project managers. The projects represent a wide range of industries (media, government, retail, IT services, etc.); however, limited context information about the projects is provided (such as country, size, year of completion etc.). [LT2] x Action research extended between 2008 and 2013 with the involvement of 25 key participants who were stakeholders or project members of five projects in a higher educational institution. The paper gives a longitudinal perspective on the adoption and impact of BM-related competences and practices. [LT3] x Analysis of 200 survey responses from the Arab world, Europe and the United States, focusing on the relationship between project and BM and project and investment success. The unit of analysis was the organization, not the project. The questionnaire was distributed to a range of groups on LinkedIn and Facebook. Response rate and year of data collection not available. [LT4] x A case study of eight major financial services organizations in The Netherlands, where the researchers performed interviews with management (CEO, CIO and line managers) investigating the management of costs and benefits.
Year of study not reported. [LT5] x Analysis of IT project outliers (failures). Analysis of 4227 projects; some data were given voluntarily, some were obtained from audit reports and academic studies, some through freedom-of-information request (American public sector) and from the US Office of Management and Budget.
Year of data collection not reported. [LT6] x Pilot case studies and Delphi research with a panel of 29 consultants, senior IT and senior business professionals with knowledge of the Belgian financial services sector.
Year of data collection not reported. [LT7] x Three case studies on factors affecting the realization of benefits from system development (health authority in United Kingdom, university, city council), through interviews, document reviews, observations and follow-up meetings.
Year of data collection not presented. [LT8] x A UK case study of the development of a clinical trial support system, collecting data through document reviews, interviews and observations (number of interviews not reported).
Year of data collection not presented. [LT9] x A case study of the UK National Health Service studying a project to introduce BM in NHS trusts. The case study was one of 12 similar studies by the authors between 1993 and 1996. [LT10] x A document study of BM in 48 e-government projects in Norway. Study commenced in 2005. [LT11] x Nordic survey of IT evaluation methods in use in Sweden (312 responses, 34% response rate), Finland (52 responses, 15% response rate) and Norway (63 responses, 17% response rate). Data collected in 2005. [LT12] x A document analysis of six different BM methods in use in the Norwegian government, using the BM model suggested in [LT46]. [LT13] x This study is based on the same document study reported in [LT12] and analyses differences and similarities across six different BM methods in use in the Norwegian government. [LT14] x A field study to investigate determinants of BM acceptance through a conceptual model and exploratory interviews (11 interviews with practitioners and senior executives). Year of data collection not presented, and limited contextual description of the interview participants (such as industry and geography).
[ LT15] x A survey of Norwegian IT professionals attending a seminar on large-scale agile software development in 2018, focusing on the adoption of benefits management and agile practices, and how perceived client benefits varied with different levels of BM adoption (71 responses, 71% response rate).
[ LT16] x A 2018-survey of Norwegian professionals in public and private organizations, focusing on the distribution of work across IT development and maintenance activities, and the relationships with project benefits management and organizational performance (87 responses, 12.7% response rate). [LT17] x Two surveys: a Survey of Norwegian IT professionals attending a seminar on software project management in 2014 (63 respondents, 79% response rate), focusing on characteristics of projects with success in delivering client benefits; and a follow-up survey of the same participants conducted in 2015 (64 respondents, 85% response rate). [LT18] x Based on a description of a real-life software project, 60 software professionals with cost or BM estimation experience attending a seminar in Norway on BM answered questions related to uncertainty of costs and benefits. The participants were randomly split into two groups: one did traditional (minimum-maximum) uncertainty assessment of costs and benefits and the other did alternative assessment by looking back on previous estimation errors. The seminar had in total around 100 attendees. [LT19] x Two empirical studies of the use of different types of contract affecting the outcome of IT projects. Studies based on analysis of a large data set of 407,815 projects, as well as interviews with 107 persons from 35 public Norwegian agencies and two municipalities. Data collected between 2001 and 2012. [LT20] x Cross-industry survey of 183 medium-to large-sized firms in Germany. Response rate and year of study not reported.
[ LT21] x A case study of a large public organization where the aim was to better understand the use of valuation methods. The case study was conducted in 2009 and data were obtained through document analyses and seven interviews. [LT22] x Australian survey of large organizations investigating investment and BM practices, 179 responses, 19.6% response rate. Survey partly based on the questionnaire used in [LT45]. Data collected in 2005. [LT23] x Semi-structured interviews as part of a case study approach from seven Australian pharmaceutical companies (IT, IT procurement and supply chain managers, and system users).
Year of data collection not available.
[ LT24] x Taiwanese  [LT40] x IT procurement managers from 30 large Australian organizations were interviewed in 2005. The study investigates the evaluation of B2B e-commerce investments, the benefits from the investments and satisfaction with the IT systems. [LT41] x A multiple case study approach at four financial organizations in Brazil investigating how BM is adopted. The researchers used interviews, document analysis and a survey to complement the case study (186 respondents, 14% response rate).
[ LT42] x A study of IT project evaluation practices in Australia, 78 interviews in 36 companies. Year of data collection not available.

APPENDIX E Adoption of BM process elements
Ref. -23

Identifying and structuring benefits
[LT46] Most respondents created business cases (4% did not) (Section 5.2) 31% did benefits delivery planning (Section 5.5) -49% did business benefits evaluation and review (Section 5.6) 32% had process for further benefits, and 16% sought to recover missed benefits (Section 5.6) [LT47] Business case creation ahead of IT system purchases: 96% 'agree' or 'strongly agree'. (p. 630) 60% did benefits reviews post go live ('agree' or 'strongly agree') (p. 632) a 'Effective' was not defined. [LT29,p. 91] asked if the IT investment evaluation methodology was effective in ensuring successful information systems.

APPENDIX F Journals and conferences
We identified six and four papers respectively from the International Journal of Project Management and the European Journal of Information Systems. Two papers were found in Information Technology and People. One paper was found in each of the