SEARCH

SEARCH BY CITATION

Keywords:

  • Software Engineering education;
  • web-based tool;
  • graduate level;
  • software industry collaboration

ABSTRACT

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

In recent years, there has been an on-going demand for better services and functionality in software products; as a consequence many models, techniques, and tools have been developed such as CMMI-DEV v1.2, TSP, or Scrum. However, software products still suffer from excessive costs, delays in delivery, and low quality. Furthermore, there is a lack of educational material providing high levels of interaction between students and the software industry to learn about how enterprises adopt these models, techniques, and tools into their daily work. This article describes a web-based Tool (EduSysProVAL) to support a graduate course in collaboration with the local software industry. The main goal of this research is to demonstrate that a Software Engineering course may use the EduSysProVAL tool to improve students' practical and professional skills, thus increasing their participation and effort in improvement initiatives, in comparison to traditional educational approaches which are only based on theory classes. This research uses a four-category questionnaire and follow-up interviews to evaluate a satisfactory level of tool effectiveness with undergraduate students and summarizes the industry's positive perceptions about its contribution to the course. © 2013 Wiley Periodicals, Inc. Comput Appl Eng Educ 23:117–136, 2015; View this article online at wileyonlinelibrary.com/journal/cae; DOI 10.1002/cae.21584


INTRODUCTION

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

Since the beginning of the eighties, the software industry has tried to increase its quality and productivity by applying new methods and techniques. It has been recognized that unfortunately the fundamental problem for many companies is the incapacity to manage the software process [1]. According to researchers such as Oktaba [2] and Johnson [3], the software development process is far from being a mature process.” Both large enterprises and small enterprises have made an important effort to improve their software process. One indicator of this is the increasing number of international initiatives related to process improvement, such as the Capability Maturity Model Integration for Development v1.3 (CMMI-DEV v1.3) [4], ISO/IEC 15504:2004 [5], SPICE [6], and ISO/IEC 12207:2008 [7]. In addition, many methods for assessing processes in organizations, such as the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) [8] and ISO/IEC 15504:2004 [5] and improvement lifecycle models such as IDEAL [9] and ISO/IEC 15504:2004 [5] have been developed. Thus a process improvement initiative has become one of the main aims of software enterprises (SEs), because of the fact that the quality of the product is closely related to the quality of the process that produces it. In order to get better products, companies need to improve their software development processes to increase the quality of the produced software and to keep budget and time under control. Research by Komi-Sirviö [10] argued that “The purpose of improvement is often to enhance software development in order to raise the quality of software. On the other hand the goal may be to shorten the delivery cycle, to lower the costs and thus improve profitability, or to strengthen the market position. There may also be a need to prove the maturity of development, which may require changes in software development processes.”

This interest in process improvement in SEs is now being extended to universities aiming to incorporate this topic into their graduate courses' curricula and provide students with the necessary knowledge to support them. However, universities have not paid sufficient attention to develop educational material which supports interaction with the software industry. Process improvement is not explored sufficiently by Computer Sciences departments in many universities; hence any development of an improvement initiative becomes completely chaotic as important concepts are unknown or not applied by students. Consequently, understanding how to implement process improvement initiatives successfully is arguably the most challenging issue facing the Software Engineering field today. The process improvement literature contains some case studies of successful companies and descriptions of their improvement programs [11, 12]. However, research efforts to date are limited and inconclusive and without adequate theoretical and pedagogical justification it is impossible to teach students how to conduct a successful improvement effort.

During a process improvement program, the current state of software development processes is being assessed and improvement actions are being determined. However, these improvement actions are based on process models obtained during interviews and document studies, for example, CMMI-DEV v1.2.1 Thus, a great quantity of knowledge is generated that is necessary to manage knowledge assets such as learned lessons, strong practices, weak practices, problems and solutions, recommendations and plans that contribute not only as individual learning but to the same improvement initiative. Moreover, research by Viana et al. [14] states that understanding how to successfully deal with human and technical aspects involved in process improvement programs is a challenging issue. Technical aspects are not enough to guarantee the success of activities; human factors (e.g., experiences, opinions, and perceptions) impact the effectiveness of process improvement initiatives and graduate students have to learn this through real practice. Students have to learn that an improvement initiative is affected by many factors (e.g., financial resources, human resources, business objectives, organizational culture, and vision), and this topic cannot be learned only with lectures. Furthermore, Figure 1 shows that when starting an improvement initiative, it is imperative that students determine the appropriate scope and tasking through a planning activity, taking into account the size of the enterprise. Later they have to evaluate the current software process status, pilot their corrective actions and finally, present solutions to software staff.

image

Figure 1. Context of an improvement cycle.

Download figure to PowerPoint

By teaching process improvement we can prepare students for the future and simultaneously improve our own understanding and teaching of Software Engineering. However, a main issue is the way in which Software Engineering courses are typically taught. According to Gresse von Wangenheim and Hauck [15], expository lessons are still the dominant instructional technique in, basically, all sectors of education and training. While these are adequate in presenting abstract concepts and factual information, they are not the most suitable for higher-cognitive objectives, aiming at the application and transfer of knowledge to real-life situations [16]. Research by Mead [17] argues that professionalism continues to be a hot topic in Software Engineering education, and it is still unpredictable whether licensing or certification will become a major trend in the future. Related to that are discussions about whether litigation over software quality will become more common. In this sense, in order to improve Software Engineering education, a general trend is to emphasize hands-on” experience for the students, related either to industry or a simulated environment. Yet, so far, there have been only a few proposals made for teaching process improvement effectively, in recreating an authentic context in which software processes are engineered in the classroom. In other words, it is necessary to use tools which conduct the improvement activities as a matter of primary educational concern. Thus, the main objective of this article is to design and implement a web-based tool for planning, conducting and piloting process improvement initiatives in software development enterprises. With this computational support, the purpose of the research is to establish an iterative educational approach within graduate programs for continuous improvement through realistic improvement projects in collaboration with the local software industry. The remainder of the article is organized as follows: the second section analyzes some of the related work and the third section provides detailed information about the functionalities and structure of the proposed web-based tool (called EduSysProVAL, EDUcational SYStem for PROcess eValuation and Action pLans). The fourth section summarizes how this tool is integrated into a Software Engineering course, and the fifth section shows some results obtained from the evaluation of our approach in a graduate course. Finally, conclusions are summarized in the sixth section.

PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

According to Dingsøyr et al. [18], a general approach to teaching Software Engineering is to relate the course to either industry or a simulated environment. In the context of process improvement, some approaches have been proposed following this approach. For example, the Georgia Institute of Technology established a course called Real World Lab [19], where undergraduate students were involved with real industrial projects with products and customers. In addition, students took part in performing a Capability Maturity Model (CMM) [20] assessment on local industry through interviews. Other examples were the class designed for students of the University of Texas [21], where CMM and ISO 9000 standard [22] were introduced to teach and apply quality improvement ideas; and the Software Engineering course held at the Ecole Polytechnique de Montreal [23], where students used a simplified version of the Trillium model [24] to assess their projects.

Research by Ref. [18] described the main design choices of a process improvement course. The course was organized around an industrial case study, where students come in contact with actors from local enterprises. The results obtained showed problems related to quality and process issues, and that the insight was too superficial; in addition, problems with student involvement were reported.

Hislop discussed in Ref. [25], the experience at Drexel University, in using the Personal Software Process (PSP) [26] to teach process improvement in a graduate Software Engineering course. The research described the context in which the PSP is used and discussed issues related to students, course structure, PSP features, and instructor load. Hislop's work included some preliminary results that provided an insight into the impact of the course on student attitudes toward Software Engineering.

Jaccheri [27] described a redesigned course in software quality and process improvement. The Jaccheri course encompassed lectures and project work in groups in collaboration with the local software industry. The main contribution of Jaccheri's work is related to the interaction pattern proposed to collaborate with industry. The interaction model was based on research questions, presentations from industry, and subsequent presentations and documents written by students.

Calvo-Manzano et al. [28] present a case study that describes TSPi teaching (Introduction to the Team Software Process) [29] to fourth year students, grouped by teams, at the Computer Science School, Polytechnic University of Madrid. The achievements of the teams after receiving training, and the use of TSPi, were analyzed and discussed.

Another example of research in process improvement education is a graduate Software Engineering course to educate students in the basic concepts of the field, proposed by Hawker [30]. This course design is based on the OMG Software Process Engineering Metamodel [31], and the IEEE Standard for Developing a Software Project Life Cycle Process [32] as ways to model and compare process design alternatives, and to provide mechanisms to assemble reusable process components into enactable processes.

Figueiredo et al. [33] created an undergraduate course in Software Engineering in the University of Brasilia, that focuses on the development and quality of software, which is based on models for process improvement such as, the Capability Maturity Model Integration (CMMI); the Reference Model of the Program for the Improvement of Brazilian Software Process (MR-MPS) [34]; and on the recommendations provided by specialized documents like Software Engineering Education Knowledge (SEEK) [35].

A different approach is proposed by Gresse von Wangenheim and Hauck [15], adopting a constructivist approach for teaching process improvement and assessment in a graduate course in combination with an undergraduate capstone project course. They applied the proposed course design and investigated its impact on learning effects, its adequacy, and its strengths and weaknesses, by administering a pre- and post-test and applying a questionnaire at the end of the course.

Research by Mendoza et al. [36] focuses its efforts on the development of a tool based on e-learning standards: the HEPALE! Project. The e-learning system attempts to solve recurring problems in the application of a new process improvement model related to the lack of knowledge about how it works, its dissemination, and the tools available, which make it difficult to acquire the model and then understand it. In this context, Rodriguez et al. [37] introduces a 3D tool based on the Scrum [38] life cycle to support virtual meetings regardless of physical locations of team members. This tool provides a virtual world consisting of technology to hold meetings in a Scrum context. This research used a questionnaire to evaluate the usability and effectiveness of the tool with undergraduate students.

The last study reviewed was our previous work presented in Ref. [39] as an approach that integrates a student teamwork methodology (TSPi) and Project-based Learning (PBL) to improve software project managerial skills in undergraduate students. As complementary support, a computational tool, the Teamwork Platform for Software Engineering education, was developed to establish an interactive Software Engineering course with local software industry collaboration.

In summary, the studies reviewed here show a clear tendency for improving process improvement teaching based on course design, and the active collaboration of software industry in the training process. However, process improvement content is more typically taught in professional training courses and/or formal preparation courses for professional certifications, that is, Software Engineering Institute's SEPM Certificate Programs or the International Software Process Improvement Certification (ISPIC). Therefore, it becomes important to provide opportunities for students to learn the knowledge and skills required through using software tools as part of their formal education [40]. Thus, the topic of educating software engineers from an industry point of view is addressed in almost all the reviewed literature. However, we believe that the introduction of an educational framework in which students can better manage a software process improvement initiative through more precise orientation when working with a real SE and its goals, will promote the main skills that the software industry requires from newly hired individuals: team work, testing and evaluating capabilities, effective communication skills, quality measurement, and process improvement.

THE EduSysProVAL DESIGN

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

Our research provides a web-based tool, called EduSysProVAL, which establishes a collaborative approach to learning how to control process improvement activities within a Software Engineering course at graduate level. Following this approach to process improvement, the focus of the first step would be that the student understands what exists in the enterprise and determines what causes significant problems. Solutions can then be devised in the action plans and evaluated in pilot studies or even controlled experiments. Students learn that only after a solution is found to be effective and efficient should it be integrated into the SE.

The architecture of EduSysProVAL is based on three layers separating the logic, the assessment and improvement mechanisms, and the information retrieval. This type of architecture enables that through the logic layer, the student takes control over the complete improvement initiative and obtains, at the same time, customized results. The SE processes are evaluated with the assessment and improvement layer, establishing a real mapping of current software processes against the practices of a process model (i.e., CMMI-DEV v1.3). The information retrieval layer manages all the generated information and uses the plan generator to send the complete information to the students. Looking into the design details, the web-based tool consists of three functional modules. The use cases for these three modules are illustrated in Figure 2.

image

Figure 2. Major use cases and relevant stakeholders for the three functional modules.

Download figure to PowerPoint

As the diagram shows, these modules are designed for the following activities:

  1. The first activity is user identification according to three types of different end users: top management (software industry), project managers (students and software industry), and tool administrator (teachers). Our main objective was to teach students how a process improvement initiative works for the two organizational levels: top management (who manages the entire organization) and project managers (who know how the organization works and is probably the role that a student can play in the software industry).
  2. The second activity is related to the logic layer that was developed to set privileges and roles for each type of user. To avoid information loss, information is managed for each session to avoid inserting, removing, and modifying without previous identification.

Regarding the assessment and improvement layer, a top manager is in charge of controlling the project manager's performance in EduSysProVAL. Top management is the only user with authorization for managing the original configuration of the system; adding or removing processes for assessing, selecting project managers for participating in the process improvement initiative, controlling the recommendations, and managing and controlling all information in the organization's repository. Project managers and students can perform the assessment and verify their results. Results and reports can be generated using the methods developed in the third increment of the lifecycle, the information retrieval layer. Thus, functionalities can be grouped into the three layers: (1) the logic layer, related to having access to the tool, (2) the assessment and improvement layer, just the core of the tool, enables students to perform assessments, generate plans, follow tutorials, receive recommendations, and close sessions; it also enables top managers to have access for selecting project managers who will be assessed and which processes will be assessed, and (3) the information retrieval layer that enables top management members to manage recommendations and plans; it also enables students to manage their performance.

With this architecture, EduSysProVAL provides three main functionalities: (1) the top management functionalities, (2) the project manager functionalities, and (3) the EduSysProVAL functionalities. These functionalities are related to the information that each role is authorized to see and manage. For example, top management members and project managers have access to layers 2 and 3, but they access to different information. The following sections show the different functionalities by user type.

The Top Management Functionalities

The purpose of top management functionalities is to support the whole range of process improvement activities in an integral manner in terms of necessary guidance for process improvement activities, communication support and learning support. These functionalities provide access for:

  1. The Process selection browser, which provides top management a checklist with the processes that can be selected for assessing. Each selected process is stored and is associated with a description.
  2. The Role selection browser, which allows top management to select who are the project managers to be assessed. They can also decide whether to delegate the internal role to the EduSysProVAL manager, or address the process improvement initiative with an external student.
  3. The Communication module, which allows top management to send email messages using links and incorporating objects from other modules, such as reports, questionnaires, or new assignments. Through this functionality, top management can be informed about the progress of the process improvement initiative, consulting results, reports, or the progress status.
  4. The Reports generator module, which allows top management to obtain any information about the improvement cycle at any time. They can control the performance of their project managers through the assessment process and obtain final results and graphics derived from the entire process.

The Project Manager Functionalities

The purpose of project manager functionalities is to obtain all information about organizational practices to develop software products, and to allow students to have a good understanding of what is going on in the development process (weaknesses and strengths). It provides access to:

  1. The Tutorial module, which helps project managers and students to obtain all knowledge needed to use the tool, interpret the results, understand each question, and improve their work.
  2. The Visualization module, which provides project managers and students the tools to draw a new process or reuse the existing ones (in accordance with a model reference).
  3. The Current status report module, which presents a categorized level of performance, in accordance with the assessed processes. The project manager can meet his own level of performance. This functionality only provides performance level results for project managers. The entire results of the organization can only be reviewed by top management through the reports generator.

The EduSysProVAL Functionalities

The purpose of the EduSysProVAL functionalities is to allow the top management of a SE and students to explicitly state the goals of the process improvement activities, and to have a good grasp of the current progress of activities in the improvement project as a whole. Moreover, students can directly participate in the process improvement activities through assessments and the Improvement Plans generation. The functionalities provide access to:

  1. The Questionnaires module, which guides students in obtaining all knowledge of the enterprise's daily labor. Previously, top management has selected which processes were to be assessed. This module uses the “levels of performance” from the two-phase mechanism proposed in Ref. [41]. Questionnaires are usually based on the practices of the process reference model. EduSysProVAL uses a weighted average to calculate the level of coverage for each question. So, it moves from a qualitative view (answers to questions—Always, Usually, Sometimes, Rarely if ever, Never) to a quantitative view (coverage level by question). Also, deviation is also taken into account for calculating the coverage level for each process.
  2. The Improvement Plan module, which generates the improvement plans for each improvement project conducted. This module applies management knowledge using databases to manage both information and people.

METHODS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

Participants

Forty-eight students (aged 23–25 years) from a Master's Program course in Applied Computer Technology at the UTM—Universidad Tecnologica de la Mixteca, Mexico, participated in the study during 2010–2013. This advanced course ran for one semester and in the beginning was concentrated on theoretical topics. A classical educational method was adopted using expositive lectures, discussions, and case study reviews. As a result, students gained a surface knowledge of basic topics, but had problems applying them, as well as achieving higher cognitive levels as a basis for the process improvement projects. Following a quasi-experimental design and given the number of students who participated in the EduSysProVAL evaluation, 12 teams were formed with 4 students in each team.2 The student teams worked on the theoretical process improvement ideas to implement various initiatives within the created software industry network. This network is composed of 13 small- and medium-sized SEs (from 11 to 50 employees) and each one is provided two project managers to participate in the improvement initiatives, and two small projects3 to be assessed. The characteristics of these enterprises are:

  • The average of number of staff for enterprises is 20.
  • Organizations are focusing on developing customized software and web-based portals in order to provide software solutions and consultancy.
  • Regarding the previous process improvement experience, enterprises have limited knowledge of the CMMI models and some practices related to planning and control software projects have been followed (not implemented) in its current process. All of these practices only used a project repository with effort data to manage the process.

In fact, the main criteria of selecting the enterprises are related to their interest in collaborating, their size, and their antiquity (at least 5 years of experience); our research is focused on small- and medium-sized enterprises which comprise about 95% of all SEs in Mexico; the big ones have too many resources, skills, and staff to deal with a process improvement initiative. To maintain the interest and commitment of these local industry collaborators, teachers from the network provide enterprises with continuous training and support in the process improvement field. As an added value, SEs obtain the knowledge to perform self-assessments of their processes and, if they wish, they are trained for a formal evaluation or official certification free of charge. Finally, nine teachers, seven with PhD degrees in Software Engineering and two with Master's degrees and between 6 and 8 years of teaching experience, participated in the study.

Instruments

This study incorporated an evaluation mechanism into the tool to assess the software process in small- and medium-sized SEs [41]. Teachers randomly selected a SE for each team and software projects were submitted to analyze objective evidence (project plans, schedule, budget, analysis and design artifact, deliverables, etc.). Monthly tests were conducted by teachers during the school year for a total of 36 tests within the study and reports of work were delivered to determine the students' grade enhancement. We decided to adopt the IMPROVE metacognitive self-questioning method by Ref. [43] that represents the acronym of all classroom teaching steps: Introducing new concepts; Metacognitive questioning; Practicing in small groups; Reviewing; Obtaining mastery; Verification, and Enrichment and remediation. This metacognitive questioning encouraged students to actively engage in self-regulating their learning by using four types of questions: Comprehension (e.g., “What is the problem in the observed simulated environment?”); Connection (e.g., “What is the difference between these two verbal representations?”); Strategy (e.g., “What is the strategy to solve the problem?”) and Reflection (e.g., “Does the solution make sense?;” “Can the solution be presented otherwise?”). Similarly, a questionnaire was distributed to the students to explore attitudes toward using the EduSysProVAL tool in their courses. Four categories were evaluated: guiding teamwork, creating a positive team environment, streamlining industry's involvement, and sustaining the collaborative environment. This assessment instrument was designed for understanding student learning when exposing them to real SE problems to obtain practical experience. The ratings followed a 4-point Likert scale [44] with Strongly disagree,” Disagree,” Agree,” and Strongly agree.” A 4-point scale was intentionally selected to eliminate the indecision factor among students using “socially desirable elements” as a semi-forced measure to allow the student to choose one side (positive or negative) [45].

Research Questions

According to Kitchenham, in order to evaluate, examine and interpret results in Software Engineering experimentations, it is necessary to formulate research questions, topic areas or a phenomenon of interest [46]. We looked at the phases from the IDEAL (Initiating, Diagnosing, Establishing, Acting, and Leveraging) model [9], one of the most popular improvements models used around the world, to formulate the following research questions within an educational context:

  • RQ1: What activities would be performed before a process improvement initiative begins?
  • RQ2: What are the strengths and weaknesses in each enterprise's software process?
  • RQ3: How general is the understanding of an improvement plan in the general context of the enterprise?
  • RQ4: How can enterprises achieve an increased level of performance in their software processes after the process improvement initiative?
  • RQ5: Is there a correlation between the amount of effort required to understand process improvement and the average improvement gained?

Figure 3 gives an overview of how our five research questions together give a comprehensive view of our approach covering four of the five IDEAL phases (or Planning, Evaluating, and Piloting phases from Fig. 1).

image

Figure 3. The relationship between the five research questions.

Download figure to PowerPoint

By analyzing the five research questions, we aim to gain a broad picture of what students' experiments are reporting on using EduSysProVAL. We collected information about what activities the students would perform before a process improvement initiative begins (RQ1); what are the strengths and weaknesses in each enterprise's software process (RQ2); how general is the understanding of an improvement plan (RQ3); how can enterprises achieve an increased level of performance in their software processes after the process improvement initiative (RQ4), and, if there is a correlation between the amount of effort required to understand process improvement and the average improvement gained (RQ5). The following section shows how each of the five research questions were answered through practical exercises.

PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

Students work in collaboration with SEs incorporated in the “software industry network” at the process and quality level. Inside this network, students have to join the role (or at least they have to learn how to perform the process improvement work) of project managers. Through this approach, students and teachers would learn about the software industry. A high level overview of the process carried out by students in response to the research questions, with collaboration of the software industry network through the web-based support during a course, is presented in Figure 4. The implementation begins defining the scope related to what process areas are included in the process improvement initiative (i.e., Project Planning (PP) and Project Monitoring and Control (PMC) areas from CMMI-DEV v1.2). This selection enable students to identify any weaknesses for the selected processes and then generate improvement plans to implement the related improvements. Once the improvement plan is generated, SEs will have received the required guidance to complete the improvements, and the improvement will have needed to be tracked.

image

Figure 4. Process improvement initiative setup in software industry network.

Download figure to PowerPoint

Students and teachers have been the advisory group that provides guidance for the initiative and insight into the SEs. An assessment team (composed of an external core—advisory group—and two project managers for each enterprise) and project managers (in order to assess the selected processes) are chosen. These project managers (whose projects are going to be assessed) are professional members of staff who know the enterprise's culture and the way the development projects are conducted. The project managers receive the CMMI-DEV v1.2 basic training through the EduSysProVAL tool. In a similar manner the project managers answer a structured questionnaire related only to the CMMI-DEV v1.2 Capability Levels 1 and 2.4 The answers are in conformance with the way the enterprise works [8]. A detailed explanation of how the research questions are answered by students is provided as follows.

Question 1: What Activities Would Be Performed Before a Process Improvement Initiative Begins?

The Initiating phase of the improvement initiative is covered and supported by the executives of the enterprises and students' participation. Therefore, it is not necessary to perform any additional internal/external work to get top management support and commitment.

Initially, participants and students are introduced (through a 2-h workshop) to process improvement and how to use the tool. This is where EduSysProVAL enables top management to establish the initial improvement characteristics: selection of the assessment team, the project managers and processes to be assessed, and the selection of the leader (see Fig. 5).

image

Figure 5. Planning the improvement work in software industry network.

Download figure to PowerPoint

The approach of a practitioner as a leader of the process improvement effort has been implemented and is currently providing good results, considering that the practitioner should dedicate less effort to the process improvement due to the automation of the work provided by EduSysProVAL. Because the enterprise is probably just starting to learn about the process improvement and how to go about launching an improvement program, the initial step requires substantial education and training. In this sense, teachers from the UTM provide programmed workshops and expository classes to organizations' top management and project managers before beginning the improvement work. The covered topics are: planning, team development, managing technological change, software process improvement benefits, consulting skills, CMMI-DEV v1.2, and software process improvement processes. Figure 6 shows that the EduSysProVAL tool incorporates a tutor that satisfies the knowledge acquisition process according to the necessities of SEs and students.

image

Figure 6. A screenshot of the EduSysProVAL tutorial module.

Download figure to PowerPoint

This is not a certification course or even a formal course. This tutorial manages a set of lessons/topics and promotes the inclusion of new topics according to the improvement project progress. If the performance of the project manager or student is low during the assessment, the tutorial recommends topics and alerts top management in order to monitor and control this project manager. A continuous monitoring process is performed by teachers in the network to evaluate the initiative progress and students' and project manager's performance, corrective actions are suggested to top management and a PDF file is submitted by teachers to teach how to fix the problem. Thus, EduSysProVAL establishes a configuration module to automate the Planning phase through the following activities:

  • After the initial workshop, the tool administrator begins the improvement initiative adding the top management roles and names. Top management is also provided with an idea of the amount of work, effort, and investment necessary to implement the initiative. E-mail information is necessary to send a commitment request to each member of top management involved in the initiative. This e-mail provides a link to accept or decline the commitment through the Communication module.
  • Once the commitment is accepted, top management members should select an assessment team composed of students and project managers to be assessed using the Role selection browser.
  • The Process selection browser requires that top management establishes the process areas related to the improvement initiative that are going to be assessed.
  • Finally, the assessment team should be trained and topics for formation (training) are included in the Tutorial manager, as well as their individual profiles. EduSysProVAL manages the progress of training and alerts the leader when this has finished.

Question 2: What Are the Strengths and Weaknesses in Each Enterprise's Software Process?

In order to characterize the initial state of the enterprise's software processes, as well as to assess the improvements done through the improvement cycle, an assessment mechanism [41] was implemented inside EduSysProVAL, as a seek step” of process improvement efforts (the Diagnosing phase). Once the seek step” is performed, EduSysProVAL is able to identify weaknesses and strengths with the aim of determining the current situation of SEs and proposing an adequate improvement plan. After this plan, efforts should be concentrated to improve the quality of produced software. Project managers should answer, for example, the PP and PMC questionnaires to obtain the coverage level per area. Consequently, EduSysProVAL analyzed the results of the evaluation giving a specific weight to each type of answer and identified which practices are institutionalized within the whole SE, and which ones were not performed at all (see Table 1).

Table 1. Performance Level Classification in EduSysProVAL
Possible answerPerform levelDescription
Always4The activity is documented and established in the organization. It is always performed, between 76% and 100% of the time, in organization software projects
Usually3The activity is established in the organization but rarely documented. It is usually performed, between 51% and 75% of the time, in organization software projects
Sometimes2The activity is weakly established in the organization. It is performed sometimes, between 26% and 50% of the time, in organization software projects
Rarely if ever1The activity is rarely performed in the organization. It is rarely performed, between 1% and 25% of the time, in organization software projects
Never0The activity is not performed in the organization (0%). No person or group performs the activity in the organization
Don't knowThe person is not sure how to answer the question
Not ApplyThe question is not applicable to the organization

The assessment mechanism used here is part of the Questionnaires module and it has been based on the two types of practices established by the CMMI-DEV v1.2 (see Fig. 7). The first phase is related to specific practices (related to project managers or practitioners) while the second phase is related to generic practices (related to top management). In order to have input from all stakeholders, one developer (or practitioner) and one member of top management are selected to start the improvement cycle.

image

Figure 7. A screenshot of the EduSysProVAL assessment.

Download figure to PowerPoint

As an example, Figure 8 shows the technique used to identify strengths and weaknesses in the enterprise's PP process. In the software process improvement context, a questionnaire-based technique is usually chosen because it provides a quick fix for a research methodology and because the researcher can determine the questions to be asked and the range of answers that can be given. This makes it more precise and easier to analyze from the researcher's point of view [47-49]. EduSysProVAL uses a weighted average to calculate the coverage level for each question (number 1 in Fig. 8). Thus, it moves from a qualitative view (answers to questions—Always (A), Usually (U), Sometimes (S), Rarely if ever (R), and Never (N)) to a quantitative view (coverage level per question) (number 2 in Fig. 8). Also, the deviation is also taken into account by calculating the coverage level for each process (number 3 in Fig. 8).

image

Figure 8. Example of processes performed by students.

Download figure to PowerPoint

The percentage of coverage level for each question (Cq) is calculated taking into account all project managers' responses, as follows:

  • display math(1)

where Cr is the response coverage, i is the number of the question, j is the number of project managers, and numbers represents the percentages defined in Table 1 for each type of answer.

The reason for the weighting is because it is considered that stronger assertions should have more importance than those that are weaker, given that, for example, something that is always done cannot influence in the same way as something that is rarely done. Without it, all the questions would always have 100% coverage, which would not clarify anything.

In addition, for each question, the mathematical mean and standard deviation are calculated as follows:

  • display math(2)
  • display math(3)

where i is the number of the question, pm is the number of project managers that participate in the process assessment and letters with # are the performance levels defined in Table 1. The values 1, 0.75, 0.5, 0.25, and 0 can also be used and the maximum value of the mathematical mean rather than 4 would then be 1. However, it is recommendable to use higher weighting values (4, 3, 2, 1, and 0) by applying a factor, thus the existing differences are more visible. It is important to note that there is a biunivocal correspondence or the same meaning between the percentage of coverage of the question and its mathematical mean (the only difference is that the scaling factors are different), such that a high percentage of coverage will also reflect a high mathematical mean and in the same proportion as that percentage. Finally, the total coverage level (CL) of the process is calculated as follows:

  • display math(4)

where p is the assessed process, i is the number of the question, and n is the total of questions.

In this way, students determine the coverage level in each enterprise and the coverage average by process to identify strengths and weaknesses in the selected processes. In this sense, to analyze strengths and weaknesses (or problems) students have to use the project managers' responses obtained in the assessments. Those questions with coverage less than 75% (number 4 in Fig. 8) imply that the related reference model's activity (number 5 in Fig. 8) is not well implemented in the SE and has to be considered as an issue to improve (a weakness) in the process. On the other hand, those questions with coverage equal or more than 75% would be analyzed by students through the standard deviation as follows:

  • If the standard deviation is less than 0.8, the reference model's activity related to that question represents a strength of the assessed process (number 6 in Fig. 8).
  • If the standard deviation is equal or more than 0.8 (number 7 in Fig. 8), the reference model's activity related to that question would be deeply explored by students through interviews to clarify if it is a strength or an issue to improve. This case means that there have been strong disagreements in the answers given by the project managers in completed questionnaires (project managers' responses to that question were very divergent and, therefore, it should explored in greater depth).

Thus, Figure 8 summarizes the process that students have to perform to make a decision and formulate improvement recommendations taking into account all the detected weaknesses. At the end, the obtained information would be checked with the documentation and through mapping with student interviews as part of the objective evidence.

Figure 9 shows the obtained results of the initial EduSysProVAL assessment for one enterprise through the Current status report module. The shown charts summarize the percentage of effective practices addressed in the existing processes of the SE, and the results are automatically grouped by EduSysProVAL according to CMMI-DEV v1.2. This information was sent to the top management by the Reports generator module as an “improvement step.” Top management analyzes all the recommended improvements and prioritizes them according to their business objectives.

image

Figure 9. Results per project manager in EduSysProVAL.

Download figure to PowerPoint

Question 3: How General Is the Understanding of an Improvement Plan in the General Context of the Enterprise?

The Establishing phase begins with the improvement step.” EduSysProVAL sets the priorities, then develops the approach and finally, generates the action plans to improve the current process with the obtained data from the previous phase. The outcome of the improvement step” is a detailed implementation plan that is composed of eleven domains: specific actions, schedules, milestones, deliverables, decision points, resources, responsibilities, measurements, tracking mechanisms, and risk management and mitigation strategies.

Using the results obtained in the Evaluating phase, the improvement plan recommends the process redefinition as shown in Figure 10.

image

Figure 10. A screenshot of the EduSysProVAL improvement plan module.

Download figure to PowerPoint

This action plan is similar to a checklist which provides a verification tool that is checked when an improvement activity has been done. The Improvement plan module of the EduSysProVAL tool, when applied to software projects in SEs, helps students to produce a detailed plan. Nevertheless, given the organizational experience of students using reference models, it is natural to consider teacher knowledge and project manager experience as the supporting process for planning, tracking, and controlling the implementation activities for the improvement team. The Improvement Plans generator organizes the obtained information from seek step” using databases. The most important preliminary requirement for knowledge management in EduSysProVAL is to manage the obtained data from the Evaluating phase. The contents are obtained through the recording of events and experiences in a specified format and the capturing of good and bad practices from the assessment. Conversion of performance levels in the form of information and storing them in the database is achieved through the engineering technique of data-mining. The transformation identifies important aspects of the knowledge domain, for example, products, attributes, activities, and associations with other process areas.

EduSysProVAL identifies similar notions and these are cluster-based on their attributes. Giving this information to EduSysProVAL, the captured activities are represented in the form of eleven knowledge domains and are stored in an object-relational data model [50] of the SEs processes.

Question 4: How Can Enterprises Achieve an Increased Level of Performance in Their Software Processes After the Process Improvement Initiative?

An initial assessment (before implementing the action plans) with EduSysProVAL is conducted by students to establish the baseline capabilities of the enterprise's processes according to CMMI-DEV v1.2. Finally, a second assessment (after implementing the action plans or Acting phase) using EduSysProVAL is applied to each enterprise. Enterprises have to achieve an average increase in their processes' capability levels. For example, an enterprise can have an initial coverage of 54% in PP process when the first assessment is conducted. A re-assessment after using EduSysProVAL has to show an increase of k in the PP process (in first iteration). Therefore the PP process will have an improved level of 54% + (54k)%. Personal bias is controlled using verifications through documentation and interviews with top management and groups of people as in appraisals.

Question 5: Is There a Correlation Between the Amount of Effort Required to Understand Process Improvement and the Average Improvement Gained?

Using the number of employees, the total improvement effort in hours and the effort per person for each enterprise, students can respond to this question. In fact, it is interesting that students note the relationship between the effort per person and the average process improvement using EduSysProVAL. Normally, enterprises which invest the largest number of hours per person will achieve the greatest increment in its process improvement.

Commitment, initiative, and responsibility are factors that strongly contributed to the data analyzed with EduSysProVAL. This web-based tool is greatly concerned about communicating the importance of the improvement initiative, not only to the leader, but to the entire team involved with the improvement program. The level of commitment and rationale needed to achieve a better education are crucial to report successful experiences. This empirical implementation of improvement initiatives also enabled us to obtain a combination of valuable experiences, “more knowledge on how to do it” and observation of behavior during the improvement process within student groups.

RESULTS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

Understanding how to successfully implement improvement initiatives is arguably one of the most challenging issues facing the Software Engineering educational field in universities today. Software process improvement literature contains many case studies of successful companies and descriptions of their programs. However, there has been no systematic attempt to synthesize and organize a practical software tool to teach the “how to” lessons through realistic experiments. We conducted an evaluation of EduSysProVAL to validate the assistance in process improvement education with local industry collaboration. The evaluation included a survey and follow-up interviews. This survey consisted of fourteen questions, along with instructions to the students on how to understand and answer the questions. The design of the survey questions was based on the research questions raised in this article and considered for the course development, which focus on how to encourage students and sustain collaboration with local industry throughout process improvement initiatives. Accordingly, four question categories specific to characteristics of process improvement initiatives and relevant teamwork settings were further developed. The four categories were the following:

  1. Guiding teamwork.
  2. Creating a positive team environment.
  3. Streamlining industry's involvement.
  4. Sustaining the collaborative environment.

Student teams were asked to respond to the survey questions on a Likert scale four-point questionnaire ranging from “Strongly agree” to “Agree,” “Disagree,” or “Strongly disagree.” After the survey, follow-up interviews were conducted to review and to have an in depth discussion of the obtained results. This was considered necessary because the survey was designed to show only the degrees of agreement regarding the questions; thus interviews were conducted to further explore how EduSysProVAL helped to address the research questions. The procedure conducted to analyze the data obtained from the survey questions depicts the range of responses in terms of percentage of respondents who expressed agreement, disagreement, etc., in each category covered in the survey. It is important to mention that, until now, no practical experiment with EduSysProVAL has been repeated in two consecutive scholar years. In this sense, CMMI-DEV v1.2 Levels 1 and 2 are composed of different process areas to evaluate in a SE and, as every enterprise is different, all the improvement initiatives (and, as a consequence, all the experiments) are different. The established research questions enable students to obtain specific information about the assigned SE and the current situation of its processes, and provided recommendations are customized according this. We are introducing a different reference model to increase the number of SEs in the network. The following tables summarize the questions and survey results. Findings and lessons learned from the interviews are described in the following sections.

Guiding Teamwork

As Table 2 indicates, most of the students (about 90%) indicated that they agree” or strongly agree” that EduSysProVAL was beneficial in guiding their teamwork participation in the process improvement projects. In the discussion, questions 1, 3, and 4 gained a positive response. These questions are related to students' learning of how to design and carry out process improvement initiatives as a team. As the students replied, the constant visualization of the improvement initiative helped them to better understand the current status of the enterprises' software process (question 2), which in fact have fortified more ad hoc and informal teamwork activities than commonly are conducted in practice by formal meetings [51]. Finally, tutoring sessions in EduSysProVAL ensured that the collaborations were clearly supported inside a process improvement project. This support is established since the start of the Planning phase, where the commitment status of individuals is better visualized, thus helping to explore and reduce the cultural rejection that enterprises suffer when evaluated. This is particularly confirmed by answers to questions 2 and 5, which were designed to determine whether EduSysProVAL helped to solve the issues related to misunderstanding between students and industry in the software process.

Table 2. Results From Students' Attitudes Toward Guiding Teamwork
Evaluation itemsStrongly agreeAgreeDisagreeStrongly disagree
#%#%#%#%
Questions for guiding teamwork category
1. EduSysProVAL helps me learn how to design and implement a software process improvement initiative3573.0817.024.036.0
2. EduSysProVAL helps visualize the process improvement progress3879.048.0511.012.0
3. EduSysProVAL establishes an interactive environment for collaborative work with the software industry network3063.01225.036.036.0
4. EduSysProVAL helps me know better how and what professional teams members are doing during a real process improvement initiative2858.01532.036.024.0
5. EduSysProVAL helps direct the group work step by step by utilizing the tutorial sessions3675.01021.024.000

Creating a Positive Team Environment

Regarding question 1 of Table 3 as to whether the students are enthusiastic to work as a team again, we further discuss how EduSysProVAL helped to encourage collaboration and create a positive team environment. We also used question 2 to cross-verify the confounding factor that good friends on a team may bias the evaluation [52] which is intended to determine how EduSysProVAL helps to create a teamwork environment. Nowadays, in Software Engineering courses it is more common that students solve problems communicating and working with others [53, 54]. To encourage greater participation and hands-on enterprise quality and process issues, students played process and quality roles such as team leader, process improvement leader, and evaluation team member. The objective of such encouragement was to help students gain knowledge and experience from their teammates [55]. Question 3, was used to investigate how student individualism and team development, along with process and contextual variables, were related to the effectiveness of teams. In particular, the positive relationship between students and team job motivation was increased as teams worked under more realistic conditions.

Table 3. Results From Students' Attitudes Toward Creating a Positive Team Environment
Evaluation itemsStrongly agreeAgreeDisagreeStrongly disagree
#%#%#%#%
Questions for creating a positive team environment category
1. I would like to work with my teammates again2655.01837.048.000
2. I know my teammates well since the process improvement project began1225.02246.01021.048.0
3. I feel confident to contribute my ideas to the team3675.01123.012.000

Results demonstrated that the relationship between job motivation and team process behaviors (helping, sharing, contributing, and innovating) was more positive in teams who were developmentally mature. These results add to prior research [56] corroborating that team members' behaviors were positively related to effectiveness, but those relationships became more positive in the presence of certain contextual factors (i.e., realistic goals from the software industry network and efficient communication), and less positive in the absence of collaboration.

Streamlining Industry's Involvement

Table 4 shows that students also felt that EduSysProVAL helped strengthen industry involvement and helped organize team discussions (questions 1, 2, and 3). In their replies to questions, two responses emerged. The first response was the following: “In the Piloting phase, the teacher directed our ideas on-line to understand the identified weaknesses in software process before we presented them to enterprise project managers. We were then supposed to obtain the feedback on-line from the discussion with these project managers and discuss it in the next class.” As this statement indicates, EduSysProVAL enabled the teacher to mediate in the collaboration and watch how students work with the enterprise, which helped to keep activities on track. The second response that emerged from the discussion shed light on the reason for this success: “The Evaluating sessions required us to maintain the results of questionnaires; this data and documentation, once collected, presented the evolution of the enterprises' software process agreed on by relevant project managers.” It is evident that through these sessions, students learned to build objective evidence from enterprises in an organized way, and then consistently communicate with the project managers by having proof and evidence.

Table 4. Results From Students' Attitudes Toward Streamlining Industry's Involvement
Evaluation itemsStrongly agreeAgreeDisagreeStrongly disagree
#%#%#%#%
Questions for streamlining industry's involvement category
1. EduSysProVAL helps take advantage of local industry involvement3369.01531.00000
2. EduSysProVAL helps streamline the collaborations and discussions with project managers from software industry network2552.01021.0715.0612.0
3. EduSysProVAL supports how to” lessons through realistic experiments in the software industry network3063.01837.00000

Sustaining the Collaborative Environment

According to Iversen et al. [57], software process improvement is highly demanding and dynamic. During its development, a variety of issues makes it difficult to develop and implement new processes. For example, the project manager may request specific knowledge for the reassignment of people possessing the dynamic knowledge about how to apply mature processes and improvement methods. EduSysProVAL provides an educational workspace for students to handle such dynamic activities. In the discussion of sustaining collaboration, which corresponds to questions 1, 2, and 3 from Table 5, students replied that they utilized the modeling module of EduSysProVAL to control changes in enterprises' software processes which enabled reaching a better understanding of industry. Students liked the tool because it can automatically notify the students and project managers (according to the teacher's assignment) and can also request that they check whether a previous process or a modification on it needs to be revised (a minor update) or reworked. We found this information useful in helping students to sustain collaboration. EduSysProVAL is used not only for performing collaborative modeling tasks, but also for helping to sustain the required group effort in the dynamic environment of the process improvements.

Table 5. Results From Students' Attitudes Toward Sustaining the Collaborative Environment
Evaluation itemsStrongly agreeAgreeDisagreeStrongly disagree
#%#%#%#%
Questions for sustaining the collaborative environment category
1. I feel that the collaborative network via EduSysProVAL is effective and continuous throughout the process improvement initiative3165.01225.024.036.0
2. I feel that the modeling module, particularly the collaborative layout, makes process understanding more effective3879.0919.012.000
3. I would like to use EduSysProVAL in my next group project related to process improvement in software enterprises2960.01225.0715.00 

As for the survey result of question 3, 85% of the survey participants felt like trying the tool again. In further discussion of how EduSysProVAL had assisted in sustaining collaboration, we found that it was mostly recognized as helping establish and conduct process improvement initiatives with industry collaboration. For example, most students (90%) liked the collaborative network functions such as sending notifications, creating models, providing the status of enterprises' software process, etc. Moreover, they recognized the fact that EduSysProVAL provided more guidance on how, not just what, to improve their software skills.

In addition to these results, the evaluation on students' grades confirms the improvement in students' performance since EduSysProVAL was incorporated into the course. Figure 11 shows that the mean score for the experimental group in the 2008/2009 pre-course was 7.5, and the scores for the following years were 8.7, 9.1, 9.0, and 9.4, respectively (obtaining an increase of 25.4% in the final assessment). These results satisfied one of the main objective of our research related to improving the quality of classes in a manner which does not affect (negatively) the percentage of passing students.

image

Figure 11. Students' grades for pre-course and posterior years.

Download figure to PowerPoint

Industry Collaborators' Perceptions

As additional results, we collected information to examine the sustainability of this collaborative experience by focusing on the “give-and-take” outcomes between UTM and local SEs in the network. The approach taken is to focus on “behavioral outcomes” (e.g., degree of satisfaction) [58]—that is, the perceived benefits. In this sense, the most significant benefit achieved by participating SEs is an increased access to new UTM research and discoveries in the field of process improvement. Furthermore, all the participating SEs state that through this collaboration, they have gained insights from the practical application of UTM's own research and theory. In relation to the use of EduSysProVAL and its incorporation in the Software Engineering course, SEs believe that the use of the tool will help them recruit more graduates skilled in the software process improvement field. Local industry agrees that the approach of our course enables students to gain knowledge about practical problems and it is useful for teaching. In a similar way, the perceptions of the SEs after three years of collaboration relate the use of EduSysProVAL with the available support to help them to learn how to solve specific organizational and technical problems related to process improvement initiatives. Finally, the enterprises maintain their commitment with the collaborative network because they consider that the provided training (seminar and workshops) has motivated them to improve the quality of their products.

Comparison of Related Works

As we said before, the process improvement field is poorly explored within an educational context; almost all research is related to undergraduate and graduate courses defined under the topic of software quality.” In spite of this, there are a few tools that implement software process assessments and apply statistical analysis to the collected data. We have only focused on process improvement educational tools (all-in-one” tools), leaving out of the study generic tools (e.g., Microsoft Project®) for planning or managing any kind of projects because they do not have an educational approach nor are focused on controlling process improvement initiatives. Thus, research efforts to date are limited and inconclusive and without adequate support for learning to conduct a whole process improvement cycle. Finally, our results included a comparison that summarized the existing related tools published in Computer Applications in Engineering Education literature. Two classes are revisited due to their relevance to process improvement education: theoretical models for education and computational tools to support education. Existing theoretical models concern students' work in collaboration with organizations at the process and quality level. In these models, the students have to play the role or at least they have to observe the work of quality and process managers. By contrast, existing tools for software quality education facilitate a comprehensive learning environment for software development that takes the form of projects. Although these approaches assume different perspectives, they are both beneficial in Software Engineering education at undergraduate and graduate levels. Hence, we compare them with EduSysProVAL in terms of their values and the roles they play in process improvement education. These comparisons are presented in Table 6 and described as follows.

Table 6. A Comparison of Existing Theoretical Models and Tools to Support the Process Improvement Education
WorksIntended operating environmentRole in collaborative workProviding tools for process improvement activitiesShowing contexts among students and industryEngaging industry involvement
Software Process Improvement Teaching through a Case Study [18]ClassroomA group work environment for executing project activitiesN/AStudents directly interacted with actors from the IT industry by interviewing them by means of a set of research questionsShowing direct contact between students and project managers when participating in a task
Model for a Software Process Improvement Course [27]ClassroomSame as aboveN/AStudents directly interacted with local industry through research questions, presentations from industry and subsequent presentations and document writing by studentsStudents have to play the role or at least they have to observe the work of quality and project managers
Using PSP an XP [60]ClassroomN/AN/AN/AN/A
Teamwork Platform for Software Engineering education [39]Virtual classroomEducation based on projects from an established software industry networkThe design and execution of collaborative software development activities in teamsEnterprises collaborate with one project manager to lead the projects and provide feedback and coaching to studentsDesigning and developing collaborative software projects based on teamwork and local industry involvement
EduSysProVAL (this implementation)Virtual classroom. Project environmentsSame as aboveThe design and execution of collaborative process improvement activities in teamsEducation in establishing improvement lifecycle activities of a collaborative real projectEducation promoting and sustaining various enterprises' involvement in improvement projects

From the perspective of collaborative project education, most of the existing models and tools are focused on real SEs interaction within a collaborative network. In a student team project environment where various collaborations make up the elaboration of the project, there is a need for training students to be able to design new software process and go through various collaborative and participative functions of the improvement initiatives. EduSysProVAL is designed to serve this need by providing training based on on-line tutorials, teamwork and a software industry network. Regarding the role of process improvement tools, although the existing tools also promote collaboration, their contents refer mostly to improving individual skills related to software development and software quality inside real projects. Thus, the contents exist in the system without contextual linkages regarding the current development of a team in a specific project. EduSysProVAL is designed to monitor the interactive process inside a team and between students and project managers as well as recommend customized topics.

Such a function refers to the assimilation of the improvement lifecycle in real projects, which promotes the evolving nature of the teams through the projects. As for the engagement of local industry involvement, the tool in the table (the Teamwork Platform for Software Engineering education) allows for the students' appearance and identity (sometimes using a username instead of a real name) in the computerized workspace. According to the social presence theory [59], the presence of the collaborators helps students in communicating and performing a team task effectively; thus, many collaborative tools require the presence of the participants' identities. In a process improvement environment that contains many interrelated and continuous collaborative tasks, human involvement should be further institutionalized to ensure the consistency of the initiative and the level of commitment. In this regard, EduSysProVAL engages students and projects managers who are supposed to be involved in a process improvement project.

Limitations

Early experiences in using a web-based tool as a foundation for teaching the main activities of a process improvement lifecycle in a graduate course have been reported. Our research has shown that EduSysProVAL supports the traditional educational approach used in Software Engineering courses that include topics related to software quality. These aspects have been analyzed based on subjective data collected from the students' teams at the end of the course. EduSysProVAL has good coverage5 in improvement lifecycle phases at the course level. However, it is important that teacher and students ensure that the action plans are followed faithfully by all the teams and project managers within the software industry network.

As mentioned above, EduSysProVAL has good coverage of the Evaluating and Piloting phases at the team level. However, most of the reference models (i.e., CMMI-DEV v1.2) implementations have to be at the organizational level. Even if all the teams are following them, there are still many organizational aspects that need to be covered. Thus, we believe that these issues should be reinforced by a theoretical part of the course and supported by industry coaching. Teaching how to conduct a process improvement initiative by means of a computerized tool, oriented to establish collaboration with local software industry and focusing on CMMI-DEV v1.2 in a graduate course was a good decision. It means students can learn how a widely known reference model works from an organizational context, and learn how the cost for an improvement initiative can be reduced leading it in a systematic way. Students can learn how the productivity can be improved within SE when they are exposed to realistic improvements that are proven and monitored in pilot projects. Nowadays, we are introducing the Mexican MoProSoft model to EduSysProVAL to include small- and medium-sized SEs into the local network and increase the opportunities to teach graduate students different reference models.

Thus, this implementation of EduSysProVAL defines an improvement lifecycle, and an appraisal method. These products provide a consistent and complementary set of tools that facilitate learning the implementation of a process improvement initiative. Although EduSysProVAL can be a huge shortcut to learning this, it is still too difficult to learn it without a specialized guide.

CONCLUSIONS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

Though process improvement efforts have exploded onto the market as a strategy to follow when software organizations want to improve the quality of the produced software, there are many universities that are still not teaching how to do this work. The CMMI is considered to be one of the best-known models that focus on software process improvement for achieving software quality in enterprises. However, the CMMI-DEV models are relatively new, so there has been little research written on which data collection and action plan generation tools can be employed when using this approach in education.

In the Master's Program in Applied Computer Technology at the UTM a Software Engineering course has been held for Master's students since 2008. Since the middle of 2009, the course has been supported by the EduSysProVAL tool inside a collaborative network of local SEs, with the objective of increasing the learning effect, specifically, at the application level, and reinforcing the understanding of theoretical concepts. However, teaching the real use of process improvement concepts is challenging. Therefore, this research has developed an instrument to teach and implement improvement activities to improve the current status of Software Engineering education. Its purpose was to investigate its feasibility inside a graduate course and to influence the direction of future research. As a direct result of this research, information regarding the educational community and the issues associated with software development were generated through the conformation of a software industry network.” The concept of university–industry collaboration is an important social experiment in countries' innovation systems. We have observed that while collaborating with the SEs on an improvement initiative, UTM teachers and students may serendipitously gain valuable insight into a personal research area.

Thus, experiences in incorporating enterprises into the practical exercises provide a preliminary indication for a positive impact on learning. However, this information needs to be extracted, organized, and analyzed in order to be useful in addressing the objective indicated at the beginning of this article. We intend to repeat the application of the EduSysProVAL in more graduate courses, as well to collect feedback on a larger scale.

ACKNOWLEDGMENTS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies

In order to close our study, we wish to express our gratitude to all 13 enterprises which actively collaborate with our program. Particularly, those project managers assigned by enterprises to participate in the software process improvement initiatives, to help and guide teams, and to evaluate and offer advice to all students. As we said in previous work, this kind of educational approach is strongly based on the commitment of all involved parts (students, teachers, and the software industry). Even more important is the commitment to students to change the way of education in Software Engineering areas and to provide them with more specific and useful skills.

REFERENCES

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies
  • 1
    H. Oktaba and M. Piattini, Software process improvement for small and medium enterprises: Techniques and case studies, Information Science Reference Publisher, Hershey, NY, 2008.
  • 2
    H. Oktaba, Moprosoft: A software process model for small enterprises, First International Research Workshop for Process Improvement in Small Settings, Software Engineering Institute, 2006, pp. 93–100.
  • 3
    J. Johnson, My life is failure: 100 things you should know to be a better project leader, Standish Group International Publisher, Boston, MA, 2006.
  • 4
    CMMI Product Team, CMMI for Development (CMMI-DEV, v1.3), CMU/SEI-2010-TR-033, Software Engineering Institute, Carnegie Mellon University, 2010.
  • 5
    ISO. ISO/IEC 15504:2003/cor.1:2004(e): Information technology—Process assessment. Parts 1–5, International Organization for Standardization, Geneva, 2004.
  • 6
    ISO/IEC. ISO/IEC TR 15504:1998(e): Information technology—Software process assessments. Parts 1–9, International Organization for Standardization, Geneva, 1998.
  • 7
    ISO/IEC. ISO/IEC 12207: 2008, systems and software engineering—Software life cycle processes, International Organization for Standardization/International Electrotechnical Commission, Geneva, 2008.
  • 8
    Members of the Assessment Method Integrated Team, Standard CMMI® Appraisal Method for Process Improvement (SCAMPI), version 1.2, CMU/SEI-2006-HB-002, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, 2006.
  • 9
    B. McFeeley, IDEAL: A user's guide for software process improvement, CMU/SEI-96-HB-001, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, 1996.
  • 10
    S. Komi-Sirviö, Development and evaluation of software process improvement methods, Espoo 2004, VTT Publications 535, 2004, p 175.
  • 11
    T. Dyba, An empirical investigation of the key factors for success in software process improvement, IEEE Trans Softw Eng 31 (2005), 410424.
  • 12
    M. Unterkalmsteiner, T. Gorschek, A. K. M. M. Islam, C. K. Cheng, R. B. Permadi, and R. Feldt, Evaluation and measurement of software process improvement—A systematic literature review, IEEE Trans Softw Eng 38 (2012), 398424.
  • 13
    CMMI Product Team, CMMI for Development (CMMI-DEV, v1.2), CMU/SEI-2006-TR-008, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 2006.
  • 14
    D. Viana, T. Conte, D. Vilela, C. R. B. de Souza, G. Santos, and R. Prikladnicki, The influence of human aspects on software process improvement: Qualitative research findings and comparison to previous studies, 16th International Conference on Evaluation & Assessment in Software Engineering, IEEE Computer Society, 2012, pp. 121–125.
  • 15
    C. Gresse von Wangenheim and J. C. R. Hauck, Teaching software process improvement and assessment, 17th European Systems and Software Process Improvement and Innovation Conference, Grenoble Institute of Technology, 2010, pp. 25–34.
  • 16
    A. Cater-Steel, M. Toleman, and T. Rout, Process improvement for small firms: An evaluation of the rapid assessment-based method, Inf Softw Technol 48 (2006), 323334.
  • 17
    N. R. Mead, Software engineering education: How far we've come and how far we have to go, J Syst Softw 82 (2009), 571575.
  • 18
    T. Dingsøyr, M. L. Jaccheri, and A. I. Wang, Teaching software process improvement through a case study, Comput Appl Eng Educ 8 (2000), 229234.
  • 19
    M. M. Moore and T. Brennan, Process improvement in the classroom, SEI Conference on Software Engineering Education, Springer Verlag, 1995, pp. 123–130.
  • 20
    M. Paulk, V. Weber, S. Garcia, M. Chrissis, and M. Bush, Key practices of the capability maturity model (Version 1.1), Technical Report CMU/SEI-93-TR-25, Software Engineering Institute, Pittsburgh, PA, 1993.
  • 21
    L. Werth, An adventure in software process improvement. In: Software Engineering Education, Vol. 750, J. L. Diaz-Herrera (Ed.), Springer–Verlag, Berlin Heidelberg, Germany, 1994, pp. 191210.
  • 22
    D. Marquardt, ISO 9000: A universal standard of quality, Manag Rev 81 (1992), 5052.
  • 23
    P. N. Robillard, J. Mayrand, and J. N. Drouin, Process self-assessment in an educational context, In: Software Engineering Education, Vol. 750, J. L. Diaz-Herrera (Ed.), Springer–Verlag, Berlin Heidelberg, Germany, 1994, pp. 211225.
  • 24
    F. Coallier, TRILLIUM: A model for the assessment of Telecom product development & support capability, Softw Process Newsl 2 (1995), 38.
  • 25
    G. W. Hislop, Teaching process improvement in a graduate software engineering course, 29th Annual Frontiers in Education Conference, IEEE Computer Society, 1999, pp. 19–21.
  • 26
    W. Humphrey, Introduction to the personal software process, 1st ed., Addison-Wesley Professional, Reading MA, 1996.
  • 27
    M. L. Jaccheri, Software quality and software process improvement course based on interaction with the local software industry, Comput Appl Eng Educ 9 (2002), 265272.
  • 28
    J. A. Calvo-Manzano, S. Bayona, G. Cuevas, and T. S. Feliu, Teaching team software process in graduate courses to increase productivity and improve software quality, 32nd Annual IEEE International Computer Software and Applications Conference, IEEE Computer Society, 2008, pp. 440–446.
  • 29
    W. Humphrey, Introduction to the team software process, 1st ed., Reading, MA, Addison-Wesley Professional, 2000.
  • 30
    J. S. Hawker, A software process engineering course, 2009 American Society for Engineering Education Annual Conference, American Society for Engineering Education, 2009, pp. 25–31.
  • 31
    J. Bézivin and O. Gerbé, Towards a precise definition of the OMG/MDA framework, 16th International Conference on Automated Software Engineering, IEEE Computer Society, 2001, pp. 273–282.
  • 32
    Institute of Electrical and Electronics Engineers, IEEE Standard for Developing a Software Project Life Cycle Process, IEEE, New York, 2006.
  • 33
    R. M. Figueiredo, A. B. d, L. R. Sales, Jr., L. Laranjeira, and A. Rocha, Teaching software quality in an interdisciplinary course of engineering, 7th International Conference on the Quality of Information and Communications Technology, IEEE Computer Society, 2010, pp. 144–149.
  • 34
    A. Rocha, M. Montoni, G. Santos, S. Mafra, S. Figueiredo, A. Albuquerque, and P. Mian, Reference model for software process improvement: A Brazilian experience, I. Richardson, P. Abrahamsson, and R Messnarz, (Eds.), Software process improvement, Vol. 3792, Springer–Verlag, Berlin Heidelberg, Germany, 2005, pp. 130141.
  • 35
    A. E. K. Sobel, Computing curricula—Software engineering volume. Final draft of the software engineering education knowledge (SEEK), IEEE with ACM Press, Miami, FL 2003.
  • 36
    R. Mendoza, M. Morales, M. Morgado, H. Oktaba, G. Ibargüengoitia, F. Pino, and M. Piattini, Supporting the software process improvement in very small entities through e-learning: The HEPALE! project, 2009 Mexican International Conference on Computer Science, IEEE Computer Society, 2009, pp. 221–231.
  • 37
    G. Rodriguez, A. Soria, and M. Campo, Supporting virtual meetings in distributed scrum teams, IEEE Lat Am Trans 10 (2012), 23162323.
  • 38
    K. Schwaber, The enterprise and SCRUM, Microsoft Press, Redmond, WA, 2007.
  • 39
    I. Garcia and C. Pacheco, Using TSPi and PBL to support software engineering education in an upper-level undergraduate course, Comput Appl Eng Educ, Published online in Wiley Online Library, DOI: 10.1002/cae.21566
  • 40
    E. Biberoglu and H. Haddad, Survey of industrial experiences with CMM and the teaching of CMM practices, J Comp Sci Coll 18 (2002), 143152.
  • 41
    I. Garcia, J. A. Calvo-Manzano, G. Cuevas, and T. San Feliu, Determining practice achievement in project management using a two-phase questionnaire on small and medium enterprises. In: Software process improvement, Vol. 4764, P. Abrahamsson, N. Baddoo, T. Margaria, and R. Messnarz (Eds.), Springer-Verlag, Berlin Heidelberg, Germany, 2007, pp. 4658.
  • 42
    Ö. Göl and A. Nafalski, Collaborative learning in engineering education, Global J Eng Educ 11 (2007), 173180.
  • 43
    B. Kramarski and S. Mizrachi, Online discussion and self-regulated learning: Effects of instructional methods on mathematical literacy, J Educ Res 99 (2006), 218230.
  • 44
    R. Likert, A technique for the measurement of attitudes, Arch Psychol 22 (1932), 155.
  • 45
    H. F. Weisberg, The total survey error approach, Chicago, University of Chicago Press, 2009.
  • 46
    B. A. Kitchenham, Procedures for Performing Systematic Reviews. Joint Technical Report Software Engineering Group. Department of Computer Science Keele University (UK) and Empirical Software Engineering, National ICT Australia, 2004.
  • 47
    H. Young, T. Fang, and C. Hu, A successful practice of applying software tools to CMMI process improvement, J Softw Eng Stud 1 (2006), 7895.
  • 48
    N. Habra, S. Alexandre, J.-M. Desharnais, C. Y. Laporte, and A. Renault, Initiating software process improvement in very small enterprises: Experience with a light assessment tool, Inf Softw Technol 50 (2008), 763771.
  • 49
    F. J. Pino, C. Pardo, F. García, and M. Piattini, Assessment methodology for software process improvement in small organizations, Inf Softw Technol 52 (2010), 10441061.
  • 50
    M. Syazwan, I. Benest, A. Evans, and C. Kimble, Knowledge modeling for developing knowledge management systems, 3rd European Conference on Knowledge Management, 2002, pp. 15–25.
  • 51
    D. Hamann, Towards an integrated approach for software process improvement: Combining software process assessment and software process modeling, PhD Thesis in Experimental Software Engineering, D. Rombach, F. Bomarius, and P. Liggesmeyer (Eds.), Vol. 19, Fraunhofer IESE, 2006.
  • 52
    L. Wichstraum, Harter's self-perception profile for adolescents: Reliability, validity, and evaluation of the question format, J Pers Assess 65 (1995), 100116.
  • 53
    Z. Budimac, Z. Putnik, M. Ivanović, K. Bothe, and K. Schuetzler, On the assessment and self-assessment in a students teamwork based course on software engineering, Comput Appl Eng Educ 19 (2011), 19.
  • 54
    C. Y. Chen and P. P. Chong, Software engineering education: A study on conducting collaborative senior project development, J Syst Softw 84 (2011), 479491.
  • 55
    D. A. Fuller and A. F. Moreno, Experimenting with a computer-mediated collaborative interaction model to support engineering courses, Comput Appl Eng Educ 12 (2004), 175188.
  • 56
    B. D. Janz, J. A. Colquitt, and R. A. Noe, Knowledge worker team effectiveness: The role of autonomy, interdependence, team development, and contextual support variables, Pers Psychol 50 (1997), 877904.
  • 57
    J. H. Iversen, L. Mathiassen, and P. A. Nielsen, Managing risk in software process improvement: An action research approach, J MIS Q 28 (2004), 395433.
  • 58
    Y. S. Lee, The sustainability of university-industry research collaboration: An empirical assessment, J Technol Transf 25 (2000), 111133.
  • 59
    K. Kreijns, P. A. Kirschner, and W. Jochems, Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research, Comput Hum Behav 19 (2003), 335353.
  • 60
    L. G. Martínez, G. Licea, J. Reyes, and L. Aguilar, Experiences using PSP and XP to support teaching in undergraduate programming courses, Comput Appl Eng Educ, Published online in Wiley Online Library, DOI: 10.1002/cae.20581
Notes
  1. 1

    According to the Software Engineering Institute (SEI), “CMMI-DEV v1.2 is a process improvement maturity model for the development of products and services. It consists of the best practices that address development and maintenance activities that cover the product lifecycle from conception through delivery and maintenance” [13].

  2. 2

    According to Göl and Nafalski [42], collaborative groups are small, normally not exceeding five members, and are intended to be conducive to achieving a successful learning outcome that may be demonstrated by the acquisition of knowledge and skills or the completion of a set task.

  3. 3

    In the context of small- and medium-sized software enterprises, a small project tends to perform better in its capacity to achieve the schedule, budget, and productivity. However, there is no evidence of its quality. According to the Software Engineering Institute “small projects are identified as projects that have 50,000–100,000 LOC, 6–12 months, and 10 or fewer programmers.”

  4. 4

    Our study focused on CMMI-DEV v1.2 Levels 1 and 2 for two reasons: (1) all the enterprises in the local network are small-sized enterprises which have no experience in software process improvement initiatives and are trying to learn how to prepare them for official assessments, and (2) graduate students want to learn how a reference model works, and the elemental basis for this, and any process improvement effort is found in early levels of these models. Levels 1 and 2 ensure that software organizations define and manage their software process. Higher levels are recommended when organizational maturity levels have been achieved through experience.

  5. 5

    Taking as reference the IDEAL model [9], EduSysProVAL provides an educational context for four of the model's five phases: Initiating, Diagnosing, Establishing, and Acting. We are currently working on the incorporation of the last phase (Leveraging).

Biographies

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. PROCESS IMPROVEMENT EDUCATION IN SOFTWARE ENGINEERING COURSES
  5. THE EduSysProVAL DESIGN
  6. METHODS
  7. PROCESS TO ANSWER THE ESTABLISHED RESEARCH QUESTIONS
  8. RESULTS
  9. CONCLUSIONS
  10. ACKNOWLEDGMENTS
  11. REFERENCES
  12. Biographies
  • Image of creator

    Ivan Garcia holds a PhD in Software and Systems at the Universidad Politécnica de Madrid and is a full-time professor at the División de Estudios de Posgrado at the Universidad Tecnologica de la Mixteca, Mexico. He is the author of international papers on Software Engineering and, more specifically, Software Process Improvement. Garcia is a member of the Technical Committee of the International Association of Science and Technology for Development (IASTED) and collaborator of the “Research Chair in Software Process Improvement for Spain and Latin American Region”, and he is also a reviewer of international journals and conferences in Mexico. He has participated in more than 15 software project management and software process improvement projects in the Mexican software industry. In Latin America, Garcia has participated in several institutional research projects in industry (IBM Global Services) and in academia. Finally, he was a member of the team that translated CMMI-DEV v1.2 and CMMI-DEV v1.3 to Spanish.

  • Image of creator

    Jose A. Calvo-Manzano holds a PhD in Computer Science. He is an assistant professor in the Computer Science School at the Universidad Politécnica de Madrid. He is teaching in the area of Software Engineering, specifically in the domain of software process management and improvement. He has participated in more than 20 research projects (European and Spanish Public Administration). He is an author of more than 50 international papers. He is an author of books related to software process improvement and software engineering topics also. He is the Director of the “Research Chair in Software Process Improvement for Spain and Latin American Region”, where he has been a member of the team that has translated CMMI-DEV v1.2 and CMMI-DEV v1.3 to Spanish. He also holds the ITIL® v2 and v3 Foundation, and CMDB Certification.

  • Image of creator

    Carla Pacheco has a PhD in Computational Languages and Software Engineer and is full-time professor at the Postgraduate Division of the Universidad Tecnologica de la Mixteca, Mexico. She is author of international papers related to Software Engineering and, more specifically, Requirements Engineering. She is a reviewer of the International Journal of National Polytechnic Institute of Mexico. Pacheco has participated in several Mexican projects related to software requirements as a part of process to improve the Mexican software industry.

  • Image of creator

    Carlos Perez is currently a M.S. student in Computer Science at the Universidad Tecnológica de la Mixteca, in México. His research interest is software process improvement in small companies.