- Top of page
In recent years, systems engineering and Project Management Bodies of Knowledge have been rapidly growing. However, despite the vast amount of literature available on systems engineering and project management, about two-thirds of all projects still fail. A review of both project management and systems engineering publications reveals that most of these works focus on processes. We suggest focusing on people—project managers and systems engineers. One of our previous studies dealt with project managers; this article focuses on systems engineers. This article presents findings of a study aimed at exploring the relationship among systems engineers' capacity for engineering systems thinking (CEST), project types, and project success. The instrument used in this study was a self-report questionnaire, composed of three parts. The first part assessed the participants' CEST, the second part assessed several measures of project success, and the third part assessed four dimensions of project type. The simple random sampling method was used, and the sample included 114 senior systems engineers who were randomly selected from the sampling frame. The study findings show that there is a statistically significant correlation between CEST and project success. The extent of the project's novelty, complexity, and technological uncertainty are moderator variables that affect this correlation.
- Top of page
According to the Standish Group report (2009), 68% of all projects failed. Forty-four percent of the projects were late, had an overplanned budget, and/or had fewer than the required features and functions. In addition, 24% were cancelled prior to completion or delivery and never used. Only 32% of all projects succeeded—that is, were delivered on time, remained within the planned budget, and had the required features and functions.
Numerous studies have been conducted in an attempt to answer the following question: What causes projects to fail? Many reasons have been found. We present only three examples. In a survey conducted among 256 UK companies, it was found that 32% of information technology (IT) projects failed due to poor project management, 20% due to a lack of communication, 17% due to the failure to properly define objectives and requirements, 17% due to unfamiliar project scope or complexity, and 14% due to the inability to cope with new technology (KPMG, 2001). According to another survey (ProjTech, 2003), the main reasons for project failure were incorrect requirements, insufficient planning, poor risk mitigation, and use of incorrect technical solutions. In a study conducted by Standing, Guilfoyle, Chad, and Love (2006), it was found that the top five reasons for IT project failure were lack of user support and involvement, lack of properly defined project scope, lack of executive management support and commitment, imprecisely defined objectives, and poor project management and leadership.
On the other hand, many studies try to identify what causes projects to succeed. The search for factors that lead to better project performance and success spans many years of research. A literature review (e.g., Chua, Kog, & Loh, 1999; Dvir, Raz, & Shenhar, 2003; Holland & Light, 1999; Nah & Lau, 2001; Pinto & Slevin, 1988) reveals that the top ten critical success factors to ensuring project success are clearly defined objectives and requirements, top management support and involvement, proper planning, vendor and customer involvement and partnership, appropriate staff selection and training, the existence of the required technology, customer and end-user satisfaction, good control, monitoring and feedback, and high levels of communication and proper risk management. A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (2008) provides general guidelines for better project management.
Systems engineering is an interdisciplinary field of engineering that mainly, but not only, relates to design and management of complex engineering projects. Systems engineering overlaps with both technical and human-centered disciplines, such as electrical engineering, mechanical engineering, computer science, control engineering, industrial engineering, organizational studies, and project management. Many definitions for the term systems engineering can be found in the literature, including several official definitions (e.g., International Council on Systems Engineering, 2004, 2010; National Aeronautics and Space Administration, 2007).
The systems engineering and project management bodies of knowledge have experienced rapid growth in recent years. A huge number of scholarly books and papers have been published, many tools are offered by various vendors, and an impressive number of conferences related to systems engineering are held every year. Yet, despite the vast amount of project management and systems engineering literature available, about two-thirds of all projects still fail (Standish Group, 2009).
Upon reviewing both project management and systems engineering standards, papers, books, conference proceedings, and tool manuals, it appears that most of this material focuses on processes. For example, ISO 15288, IEEE 1220, EIA 632, CMMI, the INCOSE Handbook, and the PMBOK® Guide (Project Management Institute, 2008) are all process-centered. The common systems engineering and project management tools offer better ways to manage processes. We suggest focusing, instead, on people—project managers and systems engineers. In their research, Dvir, Sadeh, and Malach-Pines (2006) deal with project managers; in this article, we focus on systems engineers.
Findings regarding the competencies of successful systems engineers have been reported in the systems engineering literature (i.e., Arnold, 2006; Derro & Williams, 2009; Frank, 2006; International Council of Systems Engineering, United Kingdom Chapter [INCOSE UK], 2010; Jet Propulsion Laboratory, 2006). However, most of these works do not distinguish among different types of projects or attempt to match the competencies of successful systems engineers to specific types of projects and project performance or success. In this article, we present findings of a study aimed at exploring the relationship among systems engineers' capacity for engineering systems thinking (CEST), project types, and project success. Since we deal here with three components—(1) capacity for engineering systems thinking (CEST), (2) project types, and (3) project success—we will begin by explaining these terms.
Capacity for Engineering Systems Thinking
Systems thinking, according to Senge (1994), is a discipline for seeing wholes. Engineering systems thinking is a major high-order thinking skill that enables individuals to successfully perform systems engineering tasks. To successfully perform systems engineering roles, systems engineers need a systems view or a high capacity for engineering systems thinking. It was found that this ability is a consistent personality trait, and that it can be used to distinguish between individual engineers. Systems engineers with high CEST are more capable of (1) analyzing customers' needs and requirements, (2) developing the concept of operation, (3) conceptualizing the solution, (4) generating a logical solution (functional analysis) and a physical solution (architecture synthesis), (5) using simulations and optimization, and (6) implementing systems design considerations and conducting trade studies wherein it is necessary to generate several alternative solutions.
In organizations and projects, there are many different kinds of job positions that may be included in the systems engineering category. Different positions require different competencies, characteristics, abilities, traits, and attributes. Despite this fact, it was found that a set of core characteristics, abilities, traits, and attributes does in fact exist, and is required of all systems engineers, independent of their specific position. Fourteen cognitive characteristics, twelve capabilities, nine behavioral competencies, and three items related to education, background, and knowledge were found (Frank, 2006). A tool for assessing the CEST of systems engineers was built based on these findings (Frank, 2010). This tool was used in the study presented later in this article for assessing the CEST of the study participants.
One of the common misconceptions regarding projects is that all projects are the same, and similar tools can be used for all project activities. In reality, projects differ in many ways. Indeed, several authors have criticized the universal, “one-size-fits-all” idea and recommended a more contingent approach to the study of projects. Because projects can be seen as temporary organizations within organizations, in order to understand project classification, it is advised to first refer to organization theories (Dvir et al., 2006).
Classical contingency theory asserts that different external conditions might require different organizational characteristics, and that the effectiveness of the organization is contingent upon the amount of congruence or goodness of fit between structural and environmental variables (Drazin & Van de Ven, 1985; Lawrence & Lorsch, 1967; Pennings, 1992). While correlations of structural and environmental attributes have been studied extensively when the organization is the unit of analysis, these have been investigated far less within the context of projects. Project management literature has often ignored the importance of project contingencies. Any search for a framework for categorizing projects must address the case of projects that are temporary, part of an organization and its culture, and performing new tasks that have not been done before. In addition, such a framework should be context-free and independent of any particular industry or organization (Dvir et al., 2006).
Shenhar and Dvir (2007) identified four dimensions to distinguish among projects: novelty, technology/uncertainty, complexity, and pace. Together, these four dimensions create the NTCP Model (novelty, technology/uncertainty, complexity, and pace) and form a context-free framework for selecting the proper management style.
Novelty: This dimension refers to how new the product is to its potential users. There are three levels in this dimension: derivative projects produce new products that present only modest improvements relative to older products; platform projects produce a new generation of products; and breakthrough projects demand the highest level of innovation management.
Technology/Uncertainty: Different projects present, at the outset, different levels of uncertainty (mainly technological uncertainty), and project execution can be seen as a process aimed at uncertainty reduction. Uncertainty determines, among other things, the length and timing of front-end activities, how well and how fast one can define and finalize product requirements and design, the degree of detail and extent of planning accuracy, and the level of contingency resources.
Complexity: Project complexity depends on product scope, number and variety of elements, and the interconnection among them. The level of complexity will determine the organization and the process, as well as the formality with which the project will be managed.
Pace: This dimension involves the urgency and criticality of time goals. The same goal with different time constraints may require different project structures and different management attention.
This NTCP model can guide project managers and systems engineers in selecting their project management style during project initiation, recruiting team members, structure, processes, and tools. A profile can be portrayed for each project. The shape produced is like a diamond (in fact, this model is sometimes called the diamond model). For example, the diamond profile of a project for developing a platform system that is high-tech and time-critical is depicted in Figure 1. This model was used in the study presented later in this article for project classification.
Assessment of Project Success
According to Dvir et al. (2006), most projects are conceived with a business perspective in mind, and often with a goal that focuses on better results and organizational performance—more profits, additional growth, and an improved market position. Ironically, however, when project managers, systems engineers, and project teams are engaged in day-to-day project execution, they are typically not focusing on the business aspects. Their attention, rather, is operational, and their mind-set is on “getting the job done.” Most project managers and systems engineers see their job as being successfully completed when they finish the project on time, within budget, and according to specifications. This operational mindset is clearly reflected in the project management literature, which has traditionally used time, budget, and performance as the main indicators of project success. Any of these measures, however, even when taken together, are incomplete and may be misleading. They may count as successful projects that met time and budget constraints but did not meet customer needs and requirements, or projects that experienced great difficulties in the commercialization process of the final product.
Several prior studies have suggested adding new elements to the assessment of project success, such as stakeholders' satisfaction (Baker, Murphy, & Fisher, 1988); efficiency of the implementation process and the perceived value of the project (Pinto & Mantel, 1990); technical performance, efficiency of execution, managerial and organizational implications, personal growth, and business performance (Freeman & Beale, 1992); financial performance, the creation of new opportunities for new products and markets, and market impact (Cooper & Kleinschmidt, 1987); and meeting design goals, benefit to the end-user, benefit to the developing organization, and benefit to the defense and to both the firm and national infrastructure (Sadeh, Dvir, & Shenhar, 2000).
- Top of page
Table 3 presents the correlations between the subjects' capacity for engineering systems thinking (CEST) and the projects' five success criteria (see the Method/The Instrument/Part B section).
Table 3. Correlations between CEST and success measures.
Four correlations were found to be statistically significant. The findings indicate that there is a positive significant correlation between subjects' CEST and project success in four dimensions—overall success, meeting design goals, benefit to the developing organization, and benefit to the national infrastructure. The success indicators of customer and team satisfaction were also positively, but not significantly, correlated with CEST. We will discuss these results in the Discussion section.
An ANOVA test was performed in order to examine whether the project type (according to the NTCP model) is a moderator variable (Baron & Kenny, 1986; Frazier, Tix, & Barron, 2004) that affects the correlation between the subjects' CEST and project success. The idea was to test whether the variable project type dominates the significant correlation between the dependent variable—project success (PR_Succ)—and the independent variable—CEST (strengthens or weakens the correlation). The results are presented in Table 4.
Table 4. Two-way ANOVA: Examination of the effects of CEST and project type (dependent variable-project success).
|Source||Type III Sum of Squares||df||Mean Square||F||Sig.|
|CEST * Project Type||0.204||4||0.051||0.508||0.730|
|Error||10.528||105||0.100|| || |
|Total||1,238.831||114|| || || |
|Corrected Total||11.583||113|| || || |
We can see that there is a statistically significant difference between the three CEST groups. In other words, we see again that there is a positive significant correlation between subjects' CEST and project success. The project type does not significantly affect the correlation between the subjects' CEST and project success (p = 0.739 > 0.05; p = 0.730 > 0.05).
However, in order to test whether there is a specific dimension (novelty, technology, complexity, and pace) that affects the correlation between the subjects' CEST and project success, four additional two-way ANOVA tests were performed—one test for each dimension. Table 5 presents the results for the novelty dimension.
Table 5. Two-way ANOVA: Examination of the effects of CEST and novelty (dependent variable—project success).
|Source||Type III Sum of Squares||df||Mean Square||F||Sig.|
|SE_Grade_Cat * Novelty||1.739||4||0.435||2.579||0.042|
|Error||16.862||100||0.169|| || |
|Total||1,246.606||109|| || || |
|Corrected Total||21.008||108|| || || |
We can see that there is a statistically significant difference (p = 0.047 < 0.05) between the three novelty groups (derivative, platform, breakthrough). The variable novelty does significantly affect the correlation between the subjects' CEST and project success; there is a significant interaction between the two variables (p = 0.042 < 0.05). Post-hoc tests revealed that the more innovative the project, the higher the correlation between the subjects' CEST and project success. In other words, successful systems engineers (systems engineers with high CEST) are needed most in platform projects (projects that produce a new generation of products) and breakthrough projects (radical innovative projects).
Similar results were obtained when an ANOVA test, with complexity as an independent variable, was run. It was found that the more complex the project, the higher the correlation between CEST and project success. This implies that organizations whose core business is complex projects should employ systems engineers with a high level of CEST. In other tests, a significant correlation between the level of technological uncertainty and project success was found. One of the main characteristics of successful systems engineers is “tolerance for ambiguity and uncertainty” (Frank, 2006). Therefore, organizations whose core business is high-tech and super-high-tech projects should nominate systems engineers with a high level of CEST for technical management of these projects.
No significant correlation was found between CEST and the satisfaction level of the projects' teams and between CEST and the satisfaction level of the customer and end-users. Organizations that pursue project-team and customer satisfaction should nominate project managers who are committed to success in these measures.
- Top of page
The current study combined cognitive aspects and measures with research issues from the fields of systems engineering and project management. Based on the findings of previous studies, it was hypothesized that there is a positive correlation between the capacity for engineering systems thinking of systems engineers employed in a given project and the level of success of that project. In order to test this hypothesis, the study examined the relationships among project profiles, project success, and the CEST of 114 systems engineers. The findings of this study clearly show that there is a significant correlation between CEST and project success, and that the extent of the project's novelty (derivative, platform, or breakthrough) is a moderator variable that affects this correlation. The more innovative the project is, the higher the correlation between the subjects' CEST and project success. In other words, successful systems engineers (systems engineers with high CEST) are needed most in platform projects (projects that produce a new generation of products) and breakthrough projects (radical innovative projects).
When interpreting these findings, two major cautionary measures should be considered. First, the findings of the current study show that the coefficient of determination, R2, is relatively low. This means that the prediction of project success can be only minimally based on CEST. In other words, only a low percentage of the variation in project success can be explained by CEST. The remaining percentage should be explained by other variables. Of course, this finding makes sense, as many other variables might explain project success, including the personality of the project manger (Dvir, Sadeh, & Malach-Pines, 2006), the competencies of the project manager (Boston University, 2010; Edum-Fotwe & McCaffer, 2000), other competencies relating to the project's systems engineers (Frank, 2006), and all critical success factors (Chua et al., 1999; Dvir et al., 2003; Holland & Light, 1999; Nah & Lau, 2001; Pinto & Slevin, 1988). Second, we must bear in mind that correlation does not imply causation. Correlation is a necessary but not sufficient condition for causation. The significant correlation between CEST and project success does not automatically imply that CEST causes project success.
In any case, a significant correlation between CEST and project success does exist and, because correlation is necessary for causation, it is clear beyond all doubt that, with regard to systems engineering job positions, organizations should select engineers who possess a capacity for engineering systems thinking and create a supportive environment for enabling systems thinking development in engineers. Before discussing the best ways to create such a supportive environment, however, we must first discuss whether the engineering systems thinking capability is acquired or innate. A direct answer is not possible at this stage, because in order to cope with this question, one should probably run controlled in vitro experiments, and maybe even neuro-physiological brain research studies. However, research shows that CEST is presumably a combination of innate talent and acquired experience. Davidz and Nightingale (2008) refer to a “wide and varied background” in the “individual characteristics that enable systems thinking development.” Frampton, Thom, and Carroll (2006) found that successful IT architects have broad experience in all facets of the software development life cycle. They found that more than ten years of significant experience was usually required to perform the role.
Frank (2006) found that successful systems engineers usually execute a wide range of jobs, which enable them to become acquainted with systems and technologies, learn from others' experiences, be involved in systems-related issues, work with senior systems engineers, and develop a capacity for engineering systems thinking. Engineers who have some innate potential can acquire or improve their CEST through the following processes: accumulating experience and performing varied engineering and systems engineering job positions, working together with successful systems engineers and learning from their experience and self-learning, and participating in systems engineering educational programs. Systems thinking may also be acquired or improved by learning the general systems theory principles and systems thinking principles (Kim, 1995; O'Connor & McDermott, 1997; Richmond, 2000; Senge, 1994; Waring, 1996), analyzing already developed systems (including design considerations), and studying selected topics in various engineering disciplines. In another study, Frank and Elata (2005) found that freshman engineering students may develop approaches and strategies related to systems engineering and systems thinking. These results also imply that systems thinking may be developed through learning and experience. Frank and Kordova (2009) introduced an engineering management course aimed at developing systems thinking capabilities through active learning in a project-based learning environment. The study findings have shown that the final project contributed to the development of CEST among learners. Perhaps this is additional evidence that supports the notion that CEST may be improved and acquired through learning.
To conclude, previous studies show that CEST can be developed through experience and learning. Therefore, organizations should create a supportive environment for enabling systems thinking development in engineers and managers. Engineers and managers with a high CEST may lead to better performance in general, and especially in regard to meeting design goals, overall project success, and making a contribution to the organization and national infrastructure.
- Top of page
The systems engineering and project management literature has been rapidly growing in recent years. A huge number of scholarly books and papers have been published, many tools are offered by various vendors, and an impressive number of conferences related to systems engineering and project management are held every year. Yet, despite the huge bodies of knowledge related to systems engineering and project management, about two-thirds of all projects still fail.
A review of both project management and systems engineering standards, papers, books, conference proceedings, and tool manuals clearly shows that most of these sources focus on processes. We suggest focusing, instead, on people. Dvir et al. (2006) dealt with project managers; in this article, we focused on systems engineers. Findings regarding the competencies of successful systems engineers have been reported in the systems engineering literature. However, most of the works do not distinguish among different types of projects or attempt to match the competencies of successful systems engineers to specific types of projects and project performance or success. This article presents the findings of a study aimed at exploring the relationship among systems engineers' capacity for engineering systems thinking, project types, and project success.
The instrument used in this study was a self-report questionnaire composed of three parts. The first part assessed the capacity for engineering systems thinking of the participants and was based on a reliable and valid tool for assessing CEST and engineers' interest regarding systems engineering positions (Frank, 2010). The second part assessed project success and was based on reliable and valid findings of a previous study that identified five success dimensions: meeting design goals, benefit to the customer and end-user, benefit to the project team, benefit to the developing organization, and benefit to the national infrastructure. The third part assessed project type and was based on the reliable and valid NTCP model (Shenhar & Dvir, 2007). This model enables portraying a project profile in a four-axis diagram format—novelty, technology, complexity, and pace.
The population in this study included all senior systems engineers employed in the high-tech electronics systems industry in Israel. The sampling frame included all senior systems engineers employed in the 16 largest high-tech electronics systems companies in Israel. The simple random sampling method was used, and the sample size was 114 senior systems engineers randomly selected from the sampling frame.
The study findings show that there is a statistically significant correlation between CEST and project success. The extent of the project's novelty, complexity, and technological uncertainty are moderator variables that affect this correlation. The more innovative, complex, or technologically uncertain the project is, the higher the correlation between the subjects' CEST and project success. Therefore, it is recommended that for systems engineering job positions—especially those in more innovative, complex, and technologically uncertain projects—organizations should select engineers that possess a capacity for engineering systems thinking and create a supportive environment to enable and encourage systems thinking development in engineers. It was found that CEST may be improved and acquired through learning and experience; hence, it is assumed that CEST is a combination of innate talent and acquired experience. The correlations obtained in this study are statistically significant on the one hand but relatively low on the other hand. Of course, this finding makes sense, as many other variables might explain project success. However, a series of additional and more extensive tests must be conducted in future studies among other cultures, sectors, and organizations with a larger number of participants. It is suggested to consider the work presented in this article as a trigger for initiating a strand of studies aimed at exploring the relationship between processes and personal competencies; this is vitally important to the field of project management.