MBSE adoption experiences in organizations: Lessons learned

Lessons learned through MBSE adoption efforts is one of the key ways of communicating best practices and recommendations for MBSE. This study compiles lessons learned from published case studies and practitioner interviews. Lessons are summarized into categories such as adoption strategy, modeling practices, and communication. This paper provides a source for future adopters of MBSE to review best practices and recommendations from a multitude of different experiences. This should improve the adoption and implementation of MBSE.


BACKGROUND
This study builds on a line of research that has been conducted related to MBSE/DE adoption and value.The first relevant study is the MBSE Maturity Benchmark Survey. 9It was a joint effort between the International Council on Systems Engineering (INCOSE), the National Defense Industrial Association (NDIA), and the Systems Engineering Research Center (SERC).The purpose driving this study was to better understand MBSE use and expectations in industry.As such, this survey collected responses related to obstacles and enablers of MBSE adoption.
Some of the top responses in this category are shown in Table 1.
Another related study conducted a literature review to determine how the value of MBSE was being presented in the existing body of literature at the time. 8The study concluded that the majority of benefits claimed of MBSE in 360 literature sources were not supported by empirical evidence.A majority of the benefits were claimed based on the authors' expectations of what the benefit should be.
This literature review specifically targeted "lessons learned" or "best practices"* from adopting MBSE.Wiley Online Library was used as the main source for the search.The search terms that were used were [MBSE OR "digital engineering"] AND ["lessons learned" OR "best practices"].This resulted in 230 results from journals, books, and conference proceedings.
The 230 search results were read to determine if actual lessons learned were presented.† A total of 46 papers were identified that discussed MBSE adoption lessons learned.The different classifications of those papers are shown in Table 2.The majority of the lessons learned come from single case study experiences, so the synthesis of those experiences should provide a more reliable narrative on MBSE adoption.
Each paper was evaluated for lessons learned, which were recorded in an Excel spreadsheet.Over 600 lessons learned were extracted from the literature sources.Once all of the papers had been evaluated, an inductive coding process revealed the various categories seen in Table 3. Table A1 in Appendix A includes examples of the literature lessons learned for each of the categories.
There are several limitations of relying on lessons learned for prescriptive recommendations for MBSE adoption.Some of the lessons are provided as prescriptive (cannot) instead of recommendation (should not), even though there may not be evidence provided for why it cannot be done.All lessons learned are presented in this paper as suggestive (e.g., should), although many originate as prescriptive statements (e.g., must) from the authors.This change has been implemented to take into account that many sources may not have adequate evidence to support definitive statements.Furthermore, many of the literature sources were categorized as experience from a case study, which are often self-reported.In these cases, especially since there is a lack of metrics that can quantify what is a "good" or "bad" implementation of MBSE, it is possible that what is overall a poor implementation of MBSE could be documented as a "best practice."As the popular saying goes, correlation does not equal causation.These various factors are some of the reasons why we believe it is critical to establish metrics to evaluate MBSE. 63 should be noted that many of the resulting "lessons learned" from this literature review are in line with existing knowledge found in change management literature.Factors such as organizational culture, organizational readiness for change, leadership support, organizational structure, and technical expertise have all been shown [68][69][70][71][72][73][74][75][76][77]

Method
Interviews were conducted by the first author as part of her dissertation work.The purpose behind the interviews was to capture a detailed look at the MBSE adoption process.Participants were recruited through a survey that was distributed over email and social media to different SE organizations/groups. Anyone who had been part of an organization that attempted to adopt MBSE was eligible to be included as a participant.The interviews were conducted and recorded over Zoom.Permission to record the interview was obtained verbally before the interviews began.Participants had the option to turn off their video if they did not want their face recorded.These recordings were then transcribed after the interviews are completed.
Participants were asked to generally recount their adoption experience and also asked to explain how their organization and project are structured and how MBSE performed in those settings.The semistructured interview format provided the opportunity for the interviewer to ask follow-up or clarifying questions based on the responses given.A pilot interview was conducted before participant interviews in order to refine the protocol.Table 4 shows the interview prompt questions that were used to guide the interview when needed.The first and last questions listed in the table were posed to every subject.
Since the only qualification for an interviewee to be included was experience trying to adopt MBSE, there is a similar limitation as to the self-reported case study examples discussed in Section 3.These interviews are also self-reported, so there was no "quality metric" for how well or poorly the organizational unit that the interviewee

How is MBSE defined or understood in your organization?
What Systems Engineering tasks do you use MBSE for?
What does your day-to-day use of MBSE look like?
Is MBSE used on all projects?Does everyone in an MBSE project use MBSE?
Has there been any effort to standardize the use of MBSE across the organization?
How was the process of first adopting/learning MBSE?
Was there a defined plan/process to adopt/learn/teach MBSE?
How often does your organization adopt new technologies/methods? How does your organization determine the success of an adoption effort?
Is there a future plan on how to adapt to new/improved MBSE tools/capabilities?**If not adopted** At what point did you stop using MBSE?**If not adopted** Is there a plan to implement or adopt anything in place of MBSE?
Is there anything else you want to tell me about your experience with adopting and using MBSE?represents adopted MBSE.Once again, these are limitations we have to accept until more mature metrics are developed and employed in organizations.
Interviews were conducted April 2022 to June 2022 utilizing the video conferencing platform Zoom.The target length for each interview was 30 min, but several interviews were shorter or longer depending on the participant's availability.A total of 18 interviews were conducted, representing 18 different organizational units.The term "organizational unit" refers to the participant's department, division, or other suborganization they work most immediately with.
Interviews were recorded through the Zoom platform and were transcribed after they were completed.This resulted in a transcript of the conversation that took place over the interview.Responses were transferred to Microsoft Excel for processing.First, responses were separated into separate rows by each question.Then, within the response to a single question, the data were further separated so each row contained a unique idea/topic, as viewed by the researchers. of the response.The three possibilities for this category were: problems/challenges, reports of experience, and recommendations.So, if the response was discussing a challenge with adoption, that row would be coded as "Problems/challenges." The options for organizational structure variables were seven identified characteristics of organizational structure discussed in Section 3, or "None."The options for the adoption components category included: training/resources, reception, process changes, maturity of MBSE, use of MBSE, and influence on organizational outcomes.The final category was the key theme of the quote.This was coded using two schemas.One was created inductively as responses were being processed.The codes that emerged from this method are shown in Table 5.

Results
Certain demographical information was extracted from the interview transcripts.Figure 1 shows the different industries that the organizations represented by the interviews are a part of.Nearly half of the organizational units were in the Aerospace/Defense industry, which is a typical distribution in SE practice.The rest of the organizations were in various stages of the adoption process.
Every interview began with the same question: "How is MBSE defined or understood in your organization?" Figure 3 shows the dif- One of the defining characteristics of an MBSE adoption effort is where the effort originated from.Figure 4 shows the distribution Up" refers to an adoption effort that originated from the engineers and other technical personnel.In these cases, people often started using MBSE because they wanted to or saw the potential value, and then had to push the effort up the hierarchy.This typically involved having to convince management of the value of MBSE, which is a difficult task.In some organizations, the adoption effort came from "Both Directions." In these organizations, not only were engineers and others organically using MBSE because they wanted to, but there was some direction from management also pushing for adoption.
Figure 5 shows a summary of key factors that were catalysts behind the decision to adopt MBSE.There are some similarities between the "Adoption Direction" categories from Figure 4

Workforce knowledge/skills
42][43]58 Stakeholders also need to have a level of familiarity with the tools. 36,40,41As one participant noted: "if they aren't familiar with MBSE, they're not going to use it". 29However, there is still an associated learning curve/ difficulty in learning MBSE. 9,37,41,43,58Stakeholders need training at different levels based on their role. 3,5,37,43

Integration
One of the critical MBSE adoption issues revealed in the interviews was the concept of integration.Integration ideally occurs at both the organizational level and the technical level.The technical level refers to the fact that tools, models, and/or data repositories need to be linked together in some way.This is how an authoritative source of truth is established.Some tools have to integrate with others by virtue of their functionality.One participant discussed an architecture tool which "has to be able to integrate with other tools" 55 because it cannot do everything, such as requirements management.Some organizations also want to accommodate engineers from "classically trained disciplines" 40 that have their own tools they prefer to use, so there needs to be a way to transfer the data between those tools.Another occasion for integration is if many systems are spread out over different companies/locations.In this case, the different companies/locations each create a part of a system that has to integrate together for the final product.One participant described the different scenarios where this type of integration can currently be achieved:  3,10,16,30,31,37,43 As with the literature examples, 26 the "crispness and precision" of MBSE in uncovering errors makes it difficult to prove ROI. 14 Metrics is an area that many are trying to address to try and empirically show value, but is difficult in practice. 3,9,36,37,40,55e impact on decision-making is a key way to show the value of MBSE. 16,40As one participant reported: "The only way to really evaluate and measure effectiveness is how well the program succeeded, how much better can you make decisions, and are those decisions supportable and justifiable". 91.4Organizational structure SE as a discipline is involved throughout the lifecycle of a project and thus should involve communicating with multiple subunits.A high level of communication is necessary within the SE group, and between related functions for a given project.21,22 So in order to gain all the advantages MBSE has to offer, the organizational form has to be adjusted.23 According to Kellner et al., 23 "walls and borders" between different subunits need to be broken up to create a cohesive team where all disciplines are involved from the start of a project.This reduces misunderstandings and the rework that comes from those mistakes.
A case study discussing the implementation of MBSE in an organization discussed the creation of a liaison role as a coordinating mechanism.This position, referred to as business system manager, formed the bridge between the engineering division and IT division. 24is new role assisted in the successful adoption and implementation of MBSE through addressing the lacking communication structures between subunits on the project.One of the interview participants discussed the difficulty coordinating between the systems engineers and the IT department.The lack of coordination with IT was one of the contributing factors to the adoption effort failing.
A frequent recommendation in the literature was to establish a Core MBSE team, 4,23 dedicated modeling teams, 18,25 communities of practice, 4 or more formal teams that provide guidance, training, and technical support. 1 A common adoption activity amongst the interview participants was establishing a Core MBSE team.These Core MBSE teams took the form of all of the examples from literature listed above.However, it is still important to make sure the modeling team is well-integrated, and knowledgeable of the system they are modeling.Additionally, a modeling team would need to be communicating insights gained during modeling activities, which can provide significant value in the system development.

Leadership/management support/commitment
,58 For both organizations that were not able to successfully adopt MBSE, the driving barrier was a lack of management support.But when there is management support, the adoption effort is vastly improved.One participant discussed the efforts their organization's corporate management took to support MBSE adoption.Because of the contributions of corporate management, "there is a complete system put together on how to execute the [adoption] strategy at a high level, [and] there have been a lot of activities going on." 37Oftentimes, the positive accounts of leadership support from the interviews involved leadership that had an understanding of what MBSE/DE is and the value it can provide. 6,10,37,40,41,43,58veral participants noted specifically having problems with middle management.One participant recounted their organization's experience: "we're trying to convince our managers and follow through on it, since the executives have said they want it.But middle managers can ignore that because it doesn't impact their day-to-day." 41Efforts to convince this level of middle management were often unsuccessful.It was "through the natural progression of people retiring, moving on, or moving sideways" 41 that the void of middle management began to disappear.

Training
Regardless of the adoption strategy, the participants agree that training is a crucial part of the adoption process.9][20] Over half of the interview participants specifically mentioned the importance of training.According to the participants, there are four categories of people who need to be trained to various levels (see Table 7).The first group of people are the Model Reviewers.
These are leaders, stakeholders, or customers who need to know how to use the models to make decisions.The second group is the Developers, or the Modelers.These are the people who are building and maintaining the models in the tools, so they will likely need detailed knowledge of how the tool works.The third group is the Architects.
These are the people who will be working on the model to some extent.Several participants point out that training that is too generic may not be actually useful.As one participant noted, a basic training class that says "let's build a flashlight" will not be helpful when the product is as complicated as a helicopter or a missile. 5

Resources for implementation
One fact commonly noted by the interview participants 3,9,14,16,30,36,37,40,43 as well as the literature sources 4,6,17,29,40 is the large cost that comes with adopting MBSE, especially with respect to the tools.Not only are the tools expensive, but they require time to get approval, install, establish infrastructure, and ensure that the tools are secure. 3,10,31One interview participant noted that their organization's machines do not have the capability to support using an MBSE tool, 5 which is an additional resource consideration.Another interview participant remarked that these software administrative "background" costs are often not budgeted. 9e further resource that needs to be implemented is training, both in terms of time users need to spend taking the training as well as the cost of the training itself. 37It also requires a substantial effort upfront to set up a model-based environment. 36,43If there is not sufficient cost/resources allocated to allow the necessary MBSE efforts, many people will prioritize their "actual" project work, neglecting modeling and other MBSE activities. 37,43As one participant put it, "when you get the project rush the model is not really the priority, so the model ends up being not useful because it is not really reflecting the system". 55

Tool infrastructure
As one interview participant noted, "tools and methodologies are always changing". 41Often multiple languages and/or multiple tools are used in an organization, 3 because none of the tools "do it all". 10e interview participants report various capabilities that the tools do not provide or are not mature. 10,14,29,37,55SysML itself is also hard to understand and cumbersome to use. 6,40,41The right equipment is necessary to be able to open/run a model. 5,31As in the literature examples, 27,28 open standards are reported to be important for integration purposes. 10,40,55Tools are often the sole focus of an MBSE adoption effort, 9,31,42,58 which speaks to the lack of understanding of MBSE conceptually.

Application of MBSE methods/processes and modeling practices
Many literature sources discussed the importance of having an established, and documented methodology for MBSE. 4,17,27,36,51,52,57,61This is a fact that interview participants agreed with. 10,14,16,55One participant particularly noted that "the lack of off the shelf methodologies" is a substantial barrier to MBSE adoption. 14A benefit of having a method/process that is followed is it can make integration between different modelers, organizations, and/or tools easier. 5,10ecific modeling recommendations were frequently mentioned as lessons learned in the literature sources (see Appendix B).

Adoption/implementation strategy and design
There were differing opinions among the interview participants on the adoption strategy that should be taken for MBSE.Some participants believed that "MBSE only works if you go all or nothing." 43According to one participant: "You have to make a clean cut for MBSE.You have to build it up, consolidate the data, ensure that the models are built right with the right information.We cannot just dump everything in there and recreate a mess." 36On the other hand, some advised starting small with MBSE adoption.These participants suggested to "find out where that organization's needs are and tailor your MBSE delivery to that particular aspect." 31Certain programs should be selected to adopt MBSE "based upon the monetary value, the probability of success, and the mustwin aspect. 37" As one participant recommended, "Start with one project, identify what SE concerns you want to improve with MBSE, [and] focus modeling activities on only this concern.Then you learn and grow from that." 55 This division is echoed in the literature, with various authors arguing for an either incremental adoption approach or a wholesale adoption approach. 4,11,12,14fferent perspectives on adoption strategy were also reflected in the literature.For the adoption of SE, Bretz et al. 13 stated that SE should be introduced simultaneously across an organization.If SE is introduced from one department, there is a risk of not considering all stakeholder's needs and missing support from other departments. 13SE adoption literature also considers different approaches: offcycle or on-cycle. 4The off-cycle approach occurs "in a sandbox environment," and is considered ideal but expensive.The on-cycle approach is when MBSE is implemented on productive projects and is much more challenging, since the adoption needs to occur while normal project activities are also occurring.
Another common adoption activity amongst the participants was establishing a Core MBSE team.Some of these Core MBSE teams were created to embed a common way of working across the organization.
One participant described their MBSE team that has personnel based out of the different sites in the organization.According to them, "it is kind of putting the tendrils out in those different areas and [promote] a common way of working instead of an ad hoc type of modeling approach." 14her organizations established Core MBSE teams as a place to go for help and to share ideas.These teams were often categorized as communities of practice or working groups.According to one participant, the purpose of their organization's MBSE team is "to form a center of excellence where you can help people in need and coordinate ideas." 16other participant described the informal network across the organization that was established through a core MBSE team.According to that participant, "if I have a question or am struggling, I tend to be able to know who has done this in the past with the help of the communities, various forums, [and] meetings." 4142][43]58 However, standardization can be slow/difficult. 9,14,16,29Consistency/standards are necessary for organizations that are spread out into different units, 9,10,14,29,36,41 which is especially relevant for interfaces where integration is required. 40An organization where subunits all "d[o] things their own way" can result in missed opportunities/capabilities. 9

Culture change management
Cultural and organizational challenges were frequently cited by the participants.6][17] Several participants found a cultural resistance to adopting MBSE. 5,9,14,16,29,40,42,58As one participant noted, "MBSE requires a mindset change," 5 and that was often difficult to achieve.In one organization, leadership was particularly concerned about the cultural changes necessary.According to the participant, "new people [often] expect everything to already be in a digital realm, and the generation that is phasing out" typically believes they can still use slide rules to do their engineering. 16One participant disclosed that "there's very little resistance from our direct customers, now they have resistance from their customers sometimes that we have to deal with." 6However, there were also reports of people being "onboard" for MBSE, 6,43 particularly the "new generation" of engineers. 14,16ganizational factors were another common issue for adoption.
According to one participant, "it's not about the technical problem of introducing MBSE, it's about company change." 36"Organizational change management" 40 is a critical component of the adoption process.Several participants noted that it was difficult to get something standardized or consistently applied across their organizations due to their fragmented structures.Without consistent oversight, learning MBSE was not always prioritized in organizations by teams who wanted to focus on their typical work.One participant noted that people create issues "due to a lack of holistic systems thinking." 37Teams prioritize their other tasks over learning MBSE, and especially when project costs are considered, "the model building work is going to be the first thing that is going to go." 43

Willingness to use tools
Several of the previously discussed factors contribute to a person's willingness to use MBSE and its tools.Resistance to change (cultural change management) has to be overcome to start.A common complaint that is reported comes from people who do not want to learn a new way of doing something because it is not the way they have always operated. 5,9,16,40,58Another factor is having adequate training to promote the necessary knowledge/skills needed to actually use the tools. 5,9,29,37,42,58other perspective of MBSE users involves stakeholders aside from those doing the actual modeling.For a model to be truly representative of a system, all relevant stakeholders need to be involved in the model creation process.Several literature sources found that the more that stakeholders were exposed to the model, the more benefits they saw, leading to them buying-in to MBSE as a concept. 6,32,34

CONCLUSIONS
The observations from literature and the interview experiences can be summarized into four classifications: organizational design, organization enablers/barriers, organizational change management, and organizational experience.

DATA AVAILABILITY STATEMENT
Data are available on request due to privacy/ethical restrictions.

ENDNOTES
* For the rest of this paper, "lessons learned" will refer to both lessons learned and best practices.† Even though all of the results had the term "lessons learned" or "best practices," they did not all convey actual lessons learned.For example, often authors referenced the fact that "MBSE best practices" were used without stating what those practices were.‡ Direct quotations from interviews are italicized with the ID number related to the participant in brackets [example] at the end of the quote.§ Resources for MBSE implementation.

Lessons learned categories Literature examples
Adoption strategy When adopting MBSE, a "clear purpose and scope" should be defined from the start. 4,29The transition should not be conducted in "an ad hoc manner," 15 there should be a plan on how the transition will occur and how change management will be employed at all organizational levels.The projects where MBSE is first adopted should be intentionally selected to have the best chance of success and impact. 26,27,30Many reports recommend to "start small" with the MBSE implementation, 27,30,31 and build up capabilities and employee skills overtime.Pilot projects are recommended in order to "validate the MBSE approach," 15,32 "acquire some experience," 27 and "demonstrate the value of system modeling." 26Others argue that "the best way to figure out how to apply MBSE is to do it for real" 18 ; modeling skills develop as a result of "practical work". 33thoritative data and models, Authoritative Sources of Truth (ASoT) In order to establish the model as an authoritative source of truth, all stakeholders should be involved with contributing to models early on. 32,34"The model [should not] be considered as the reference if all stakeholders are not involved." 34Multiple copies of data should not be propagated, model instances should refer to a centralized source of data. 4,31The digital artifacts within the ASoT should be standardized as much as possible to facilitate interoperability, 28,35 and the standards/policies/guidelines governing the ASoT should be communicated to all stakeholders.

Automation
Artifacts should be generated automatically from the data/models, 31 diagrams should not be manually maintained. 26,34Automation that is made possible using MBSE may be used to convince stakeholders of the direct benefits of MBSE. 64][45] "Keep communications open and regular," 31,44 information should be accessible to everyone. 18,31,46mplexity The complexity of MBSE tools and methodologies make adopting MBSE difficult and time-consuming. 4,17,33,43Despite this, MBSE aims to assist in managing complexity in systems. 47nfiguration management Configuration management is different for descriptive models, which "share some features with analytical models (software) and some features with program documentation (static documents)."16,26,33 Consistency of MBSE practices Consistency/standardization is necessary to assist with communication between units and integration of models/artifacts.1,18,33,40,41,46 However, some find that there is a "resistance to enforced consistency."43,48 Different organizational units choosing their own methods, tools, languages, and so forth can lead to "integration and model interchange issues" down the road.29 A formalized framework or selection of practices is necessary to avoid these issues.17] It is important to involve everyone in the adoption process, 27 since it "affects the entire set of engineering disciplines and overall business enterprise." 1 Resistance to change is a common problem in the innovation process, 4,29,48,52 stakeholders and project members often need to be convinced that using MBSE benefits them in some way.6,17 Data It is important to focus on the data, not just diagrams/models.26 Models should be built to support the input and exchange of data from different sources.6,36,41,53 Data should be designed and managed 1,31 ; "data does no good if you cannot find it or do not have the rights to use it." 28 "The more a mdel is used, the more data it contains and integrates, the more valuable it is."26 Dedicated modelers "The time to train a MBSE modeler and to become proficient at the tool is beyond the ability of any single project."6 Many recommend having a core modeling team of dedicated modelers that can be matrixed into different projects.6,18 Technically capable people with methodological knowledge of MBSE should be provided to support the engineers.27,39 (Continues)

Lessons learned categories Literature examples
Deliverables "Keeping focused on real engineering deliverables is important to avoid the pitfall of delivering a modeling solution everyone thinks is finished but which does not provide the required engineering answers." 18,31agram layouts A consistent set of approved representations for diagrams should be adopted and adhered to Smith et al. 28 Having a consistent layout across diagrams should assist in their readability, 34 especially since stakeholders often have a hard time understanding diagrams. 45,54main/tool knowledge Lack of knowledge related to MBSE is a cited inhibiting factor to MBSE adoption and use. 4,17,38,54,55There is a significant learning curve for MBSE tools, languages, diagrams. 43,50,56All stakeholders may need training at different levels based on how involved they will be with the modeling. 18,33portance of mentorship Mentor support is advocated due to the complexity and steep learning curve of MBSE. 6,15,27,33,34,56Specifically, one experienced mentor is recommended for every 5-7 developers. 33portance of tool support MBSE tools are often complicated and time-consuming to figure out, so having dedicated tool support that can assist engineers and other operational users is critical. 6,27,34,57,58portance of training Not all stakeholders need to be proficient in MBSE, but they likely have to have enough of an understanding to be able to interact with the models in whatever role they have. 1,4,15,18,27,33,59There could be an issue due to the relatively simple examples used in training compared to the complexities of an actual project. 33This may be offset through continued mentorship. 15,33frastructure Infrastructures for tools and the ASoT should be planned and established early. 1,32How data will be stored, accessed, and exchanged should be thought out as well. 28,53eadership/management support "creates a strong MBSE infrastructure." 6,5742]47 The interfaces where integration needs to occur should be well-defined. 28,33,41Consistent modeling standards/methodologies across teams/groups are reported to assist with integration. 36,46owledge capture/representation Knowledge capture in models in an area that needs further development.16,50,60 Leadership Commitment/support from leadership/management is frequently cited as a key enabling factor (or inhibiting factor) for the successful adoption and use of MBSE.4,15,17,18,27,34,57,58 Leadership/management should also have an appropriate level of understanding of MBSE.17,59 Legacy systems The issue of how to deal with legacy systems is difficult. There ist a simple process to translate old artifacts into a model-based environment.17,27,48 Additionally, the document-based way of working often still needs to be maintained, 26,40,52 and therefore models need to be able to produce the expected document-based artifacts.26,40 Maturity of MBSE MBSE tools, methods, terminology, and standards are often seen as immature.17,52 MBSE over the whole life cycle MBSE should be implemented across the full system lifecycle. 1,45,55 Measurement/vae Sharing lessons learned/experiences with MBSE are an effective way to build support for MBSE.6,27,56 The improved communication and understanding that comes from MBSE are extremely beneficial.6,18,38 "Measure success in the problem space, not the solution space. Measuent should focus on the quality of the questions and decisions that can be supported by the models."26 Model analysis Change impact analyses are assisted as a result of MBSE and the traceability it enables.31,38 Model definition/guidelines It is important to "set clear modeling objectives" right from the beginning of a project clearly defining the purpose behind the model.15,26,34,36,41,50,53 Stopping criteria should be defined so the model is not over-detailed.4,18,33,45,53 Modeling guidelines should be formalized to ensure consistent practices.4,18,33,34,50,61 Model reviews Regular model reviews should involve model contributors and domain experts.34 Model reviews should be conducted by going through model files instead of static images. 26 Howevr, there is still many questions on how model reviews should be conducted.16 (Continues)

Lessons learned categories Literature examples
Modeling practices Pitfalls: -not separating need and solution modeling 34,39 -not associating model elements with proper descriptions 34 -do not keep several engineering levels in the same model (systems and subsystems have different lifecycles, contributors, design drivers, etc.) 26,34,47 Best practices: -provide a guide for others to be able to navigate the models 34 -build in flexibility to adapt to future needs 18,26 -generate artifacts directly from the model 38,61 -use dynamic references when possible 31

Necessity of MBSE method
An MBSE method should be defined and used as widely and consistently as possible. 4,6,15,17,28,29,32,34,43,50,51,57ganizational design It is important to consider all domains/departments when adopting MBSE. 1,27Interfaces between organizational subunits should be identified and standards applied across those subunits as much as possible. 41,46Additionally, it may be difficult to bring staff up to the skill level necessary for MBSE immediately. 17,18,46Some organizations approach this issue by hiring as many outside skilled MBSE practitioners as possible. 18Another strategy is to focus staff development efforts on a select group that will become "expert modelers that can be provided by the organization to any project." 6,18ocess changes Business processes should be re-evaluated and potentially changed to best integrate MBSE. 1,42,48,57,59New processes should be well-documented and visible/accessible to everyone. 27,31,41ovide examples One recommendation to assist the learning curve of MBSE is to provide "real project examples" (e.g., model an existing artifact). 18,27,33,39,54,61Showing examples of how MBSE has created value is also an effective way to convince others to engage with MBSE. 6,18,38,41,56usability If data is "copied and pasted" for reuse purposes, if that data is then altered the model's status as a "source of truth" is lost. 4Establishing a library of reusable model elements helps get projects initially started and help with modelers in adopting MBSE. 6The more the analysis is separated from the model, the more reusable it may be. 18curity Security policies need to be created to enforce authorized access to certain classified portions of the model. 1,26,28ills/abilities There is a reported significant learning curve for MBSE tools, languages, and diagrams. 43,50,56ll stakeholders may need training at different levels based on how involved they will be with the modeling. 18,33akeholder involvement All stakeholders should be involved throughout the duration of the project, 15,28,31,34,39 but do not need to be exposed to the model in greater detail than necessary. 39,53It can also be a challenge to "align all stakeholders with the selected modeling objectives and methodology." 43ols MBSE tools are reported to have a significant learning curve. 52,56SysML is complicated and requires a large number of elements to be created in order to complete a model, 17,33,50 which is said to be "not very resource-effective" by modeling teams. 43Tools for requirements management may allow for more data consistency and traceability. 47Open standards are important for integration when multiple tools are used in a project. 27,282]47 The use of a table, matrix, or other representations of requirement traceability assists in managing requirements and system changes, 53,62 MBSE tools may provide this capability. 31,38,47aditional SE artifacts There needs to be a way to extract "traditional SE artifacts" from models because not all stakeholders will be able to/want to engage with the model as a decision-making tool. 6,16,26,33,40front investment There is a significant upfront investment of not only cost to implement, 4,17,18,27,29,40 but time/effort to learn MBSE. 6,17,18,33,40There is also a substantial initial effort that should be put into creating and populating the model, 40,61 and building up an "MBSE-friendly" infrastructure.which we analyze for technical resources, performance, and cost.The models that we build to address these two disparate viewpoints must of necessity be partly conceptual and partly realizational.This should not mislead one into thinking that the space between them must be entirely filled in: it has proven very workable to have some parts of our model be treated as strictly conceptual, and other parts be treated as realizational (e.g., for the mass margin analysis)." 18odels Evolve.The model needed in concept formulation is very different than the model needed in detailed design, or in operations.
Models need to evolve and grow, and sometimes shrink.This should be the focus of model reuse along the project lifecycle.It also helps to answer the people who will suggest that building a detailed model of the last flown mission will help you formulate the next.It all goes back the principle of modeling for a purpose, and not more.While the models may change, these changes can be evolutionary and cumulative as long as they are connected by a common set of ontologies and methodologies." 18odels need to be architected and peer-reviewed.The conceptual model and the organization of models into modules can significantly impact their usability, consistency, and maintainability.It is important to have experienced architects familiar with both the problem space and the capabilities of the tools to lead that effort." 26he first infusions will not have the benefit of an engineering pool with ubiquitous modeling skills. . .we found that the best way to get started on the right path was simply to hire as many of the existing cadre of skilled MBSE practitioners as we could afford." 18et Outside Expertise.From the very beginning, visionary managers . . .brought in world-class outside expertise to teach, advise, and guide our adoption of MBSE.These experts imbued our efforts with a maturity and credibility which would otherwise have been achieved through expensive trial and error.In addition, because some of these experts have also been in positions of executive leadership, they were also extremely helpful in helping executive leadership . . .understand the value proposition for MBSE.So outside expertise has proven invaluable both for the quality of the infusion itself, as well as the institutional support for the infusion." 56refer methodological knowledge over domain knowledge -While it would be optimal to have practitioners with a sound understanding of both MBSE methodology and the domain being analysed, this will not be achievable in all instances.If this is the case, it is recommended that an experienced MBSE practitioner be preferred to a domain expert.To extend Logan's (2011) observation on MBSE enforcing good SE: expert application of MBSE on a group of domain experts will elicit a better product than a less than expert application of Systems Engineering on the same domain savvy personnel." 39kill in modeling comes mainly from practical work and the learning curve to become highly productive seems to be 3-6 months, depending on modeling focus, engineering back-ground, and ambition.A wide range of learning curves was observed; one engineer with a good background in object-oriented methods became highly productive in 1 month.Hence, pilot projects for MBSE introduction should be allowed to have longer calendar time allotted, until the first increment of system analysis and design has been completed.For following increments the calendar time can be "paid back" through increased efficiency gained by the MBSE approach.With this initial investment in the "core development" phase, . . .we believe that there is a great potential for cost and time savings in the following phases." 33ools and modeling languages evolve very quickly.These modeling tools and modeling languages are evolving quickly, so any tool assessment, no matter how disciplined the decision process, will also "There should be methods to capture, store, access, and share both artifacts and the central model." 57ool selection should be driven by needs.Stakeholder concerns and needs for the models should be solicited first, then the desired outcomes for the project, as these should be the ultimate drivers for the project.The questions that the models are intended to answer to achieve these outcomes should be identified, followed by the products that the model set needs to contain or produce to answer these questions.Then, the methodology to be followed to develop those products through the sequential application of system modeling concepts should be fleshed out to the point where the needs of the modeling language and the tools are made explicit.This process provides a traceable framework for identifying tool needs based on the stakeholders' needs for the integrated set of models." 32ommunicate the interfaces approved for access to other stakeholder ASoTs, such as OSLC.Our tool survey revealed that of the two common approaches for tool integration (either build a custom interface or build to a common standard), building to a common standard is more scalable and better supports future capabilities." 28ools that support standardized, interoperable data representations and interfaces provide flexibility and enable the credible threat of recompete."28 "Challenge: Tool Dependency and Integration.Companies need to pick a set of tools and train employees accordingly.Such a decision is not an easy task and there is no tool that satisfies all needs.Moreover, integration between systems modeling tools and others, such as simulation or requirements, is still solved with specific solutions." 4 third lesson learned is the division of modeling between two geographical diverse teams. . . .Each team has a lead modeler which has some background in MBSE, and thus, their own ideas of how models should be constructed and how information should be represented.This is a problem since one of the main goals of this modeling work is to link information from the software diagrams to the operational process flows, and vice versa.If two teams are not on the same page, as far as modeling methodology, this could present a problem when it comes time for integration/linking. . . the modeling team has gone through several iterations of methodology discussion before finally settling into a final methodology.Even with everyone seemingly on the same page, the team still finds it important to meet on a regular basis to reevaluate the modeling standards for the trade study and discuss any issues occurring or that we might see on the horizon."46 "Invest in open standards.

Demonstrated benefits/results
"In contrast, the discipline specific Subject Matter Experts (SME) generally are not very interested in the MBSE approach since system integration is not their primary responsibility.The benefits to the individual discipline are often overshadowed by their direct responsibility.
To win over the SMEs, there needs to be some concrete added value provided to them.By demonstrating some of the tools described below, the SMEs can be convinced to give MBSE a try.For instance, producing documents, and requirements compliance matrices from the information captured in the models, and showing the ability to produce the reports that project management and design engineers utilize during the design process can demonstrate how MBSE based approach can assist the SMEs in their daily activities.Demonstrating the capability to support the communication between all the project stakeholders is also important to obtain acceptance of the MBSE approach.Showing evidence of successful projects that have benefited over time can help the project justify the additional costs." 6he project team members need to be convinced that the extra effort required to develop the models provides some direct benefits.Generating products such as a parts list, connectivity information, telemetry and command data, requirements and traceability from the model is a considerable help for the project team.SMEs and other stakeholders can also benefit from these products." 6Resolving disconnects is a key benefit of MBSE.These disconnects linger unnoticed because the document-centric approach for knowledge management results in many information stovepipes that may only intersect due to serendipitous circumstances.However, building models based on the available data sources was found to be very useful in exposing the disconnects, driving stakeholders to recognize the disconnects and beginning the process of resolving them.The state of consistency of the program's technical baseline will determine how much time is spent resolving these issues.If the MBSE effort is being tracked against schedule milestones, it is important to realize that this time spent is more a benefit of MBSE than a cost of MBSE.MBSE found the latent problems--it did not create them." 26irst Description, Then Analysis.Another common misconception is that models are not really useful until they can be subjected to quantitative analysis.This is simply not the case.Capture and description are powerful and far-reaching first steps.Just describing something in a formal modeling language like SysML immediately improves communications and understanding.The benefits of this would be difficult to overstate." 18hange impact assessments.Changes to the conceptual design continually occurred during early supplier engagements.The Operations Concept Definition (OCD) and Maintenance Concept Definition (MCD), System, Subsystem requirements and interface requirements were analysed by various SMEs and refined.The inherent traceability of the MBSE approach significantly assisted these impact assessments with a near end-to-end visibility from project business requirements to functions and interfaces to system and subsystem requirements.
This has enabled trade-offs to be made against the user's operational requirements and commercial off-the-shelf (COTS) products offered by suppliers, with the intent to minimise customisation in products." 38sing Enhanced Functional Flow Block Diagrams (EFFBD) for scenario modelling were found to be novel to the . . .system acquirer, supplier and user, and considerable time was spent on advocating the benefits of using this method.The best EFFBD scenario review results came from preparing the reviewers with a simple flow block diagram example, explaining the purpose of the information captured in them, and keeping the review session numbers low (1 -3 people).Following a few sessions, majority of the reviewers were able to utilise the diagrams to create a shared understanding of sociotechnical interactions between multiple operational and technical stakeholders." 38monstrated "ability to link system elements and components to the requirements and provide end-to-end traceability from the final system architecture, back to the original customer goals." 49 model-based architecture [is a] valuable communication device, especially for stakeholders who desire system information to be conveyed quickly and efficiently."49 ". . .some short time benefits can be easily obtained. Fonstance, communication between cross-functional teams, single source of information, traceability between requirements and system artifacts etc.
Project planning need take this into account and commit sufficient resources in advance avoiding budget over-runs later.

" 40
Organizational structure "There is a need for a team of expert modelers that can be provided by the organization to any project.In the past it was identified as a critical core function to develop and maintain a cadre of users for applications proficiency.With less mentor support, the risk increases for a diverging model and/or insufficient stringency or quality in the model/system." 33eam Organization Matters. . .The pattern we have found that works well is a three-tiered one involving a small set of core modelers within a larger set of modeling-savvy systems engineers, within a larger set of all project personnel.While we have found that descriptive modeling can be done by almost anyone with the basic training, the additional rigor and consistency needed for quantitative analysis requires us to designate a smaller team of people who are modeling experts and who can apply best practice to the official configuration managed project models.Presently we have a core modeling team of a half dozen or so, within a larger team of 20 or so engineers.The experienced systems engineers provided guidance to keep the modeling focused on providing useful information, as well as mentoring of the core modelers who tended to be more junior.Frequent (daily) interactions were crucial to getting useful products: we were pathfinding so we had to stay very closely in touch.As important as it is to have a core modeling team, it is just as important to avoid fencing them off from the rest of the project.If the models are to be useful to the project, the project must understand and interact with the modeling team regularly, and likewise the modelers must be engaged in the larger engineering effort.Modelers who are also capable systems engineers will naturally employ their modeling skills to deliver engineering products in a model-based way." 18 "To counter the pitfall of capturing several engineering levels in the same model, use models as a means to perform co-engineering.In the case of a transition from a system to a subsystem for example, subsystem stakeholders have to be involved in any decision related to their subsystem and must validate that the high-level view the system stakeholders have is accurate, relevant and feasible.

Resources for implementation
"MBSE does imply heavy investment as well as a steep learning curve, especially in the initial phases, and is beyond the scope of a single project." 40BSE adoption requires a substantial upfront investment, especially if it has not been considered before.This also includes determination of an effective investment strategy, accurate cost estimation and quantifying its return on investment." 4 ". . . it is clear that the resource needed to align large teams to a coherent modelling methodology has been underestimated.Modelling opens up many new possibilities in terms of capturing design information and some stakeholders see the opportunity to revolutionise their engineering process.Here, the challenge is to communicate the cost associated with meeting ambitious objectives and the risk imposed by setting ambitious objectives that cannot be readily met by the current generation of SysML tools." 43For instance, companies with high available upfront investment might suffer from having the freedom that each department starts to define its own MBSE adoption solution.This brings later integration and model interchange issues." 29ood tools are costly; time to implement MBSE before starting the project is costly; until it becomes mainstream or someone demonstrates value at no cost to them, many will resist."17 "The second challenge mentioned, the additional costs and efforts, requires an organizational proactive approach to put in place an overall infrastructure including training, toolset and MBSE support team.
The costs mentioned in the challenge section have to be fully or partly supported by the overall organization rather than by the project itself.Training is already funded by the JSC Human Resource training department.Similarly, tool costs should be subsidized by the overall engineering organization in order to facilitate the insertion into a project and ensure standardization across projects." 6aving enough skilled resources to support the MBSE efforts is critical to its successful adoption.Mentors should be made available to partner with the Systems Engineers in order to support the adoption of modeling practices and tools.Effort and budget is required to provide project mentors who can participate as part of the project team and be involved in the system design and integration." 6

Tool infrastructure
"Security classification issues complicate model management.While one of the strengths of the MBSE approach is that the model can be queried and transformed to construct a near-infinite variety of customized views, some of these views may result in classified associations of information.This may pose a significant challenge for security review of models." 32o build these MBSE tool suites in house, it was necessary that [we] develop a set of processes to guide the specification, development and deployment of the tool suites, and a unique set of SE skills within the cadre of engineers who execute these processes."24 "The project team recommends creating an appropriate structure and defining where each type of textual component should be stored, since there are various locations available."Acquire required metadata for each digital artifact needed in order to support access control, search, approval, and recompete.Develop and adhere to a standard for the metadata collected"."The ASoT requirements call out collecting artifact expiration dates, countryof-origin, country-of-delivery, information criticality, non-functional requirements such as manufacturing constraints, and cost and scheduling metrics.The ASoT requirements also call out evidence to demonstrate the provenance of digital artifacts, such as the tools used to build or generate the artifact, the contract guidance used to produce the artifact, marking and licensing information (even from previous contracts), template models used to produce the artifact, and analysis results and certification results associated with specific versions of the artifact."28 "Models Evolve.The model needed in concept formulation is very different than the model needed in detailed design, or in operations.
Models need to evolve and grow, and sometimes shrink.This should be the focus of model reuse along the project lifecycle.It also helps to answer the people who will suggest that building a detailed model of the last flown mission will help you formulate the next.It all goes back the principle of modeling for a purpose, and not more.While the models may change, these changes can be evolutionary and cumulative as long as they are connected by a common set of ontologies and methodologies." 18hat constitutes a complete set of models is a fundamental gap that is beyond the purview of MBSE.Also, the specification of model uses and how to use models is an overarching concern for model-based approaches." 16ocus on the underlying data, not just diagrams.While diagrams are the most visible products of a model, a well-architected model can provide many more insights through queries and automated report generation than the hand-assembled diagrams can by themselves, and additional views can be generated from that model through automated model transformations." 26t the commencement of every project, it is essential to establish a package structure to enable model elements storage, data accessibility and control, model management, and data exchange."53 "But if the first mission element took longer than expected to analyze . . . the second and third ones showed the power of developing reusable methods: they each took a fraction of the time of the first.For the first mission element. . .both model capture and analysis were performed in the SysML model and the mass report took approximately two work-months to complete.The subsequent two concepts we modeled . . .each took about one half work-month each.And, a subsequent change of . . . den was accomplished in a fraction of that time.So, our advice is to first focus on description, and then implement analyses.A large part of the benefit accrues as soon as people start using the descriptive models, and this gives time and support to allow the more difficult work of analysis to be done."18 "Separate the Model from the Analysis. . . .Two troublesome characteristics worth mentioning in this context are: as it is commonly used, the model and the analysis are inextricably intertwined; and by the nature of the tool, the model is forced into a form which facilitates the analysis.It is clear that the more a model can be a self-contained, internally self-consistent, and an intuitive description of the concept, the more informative it will be.Moreover, the more the analysis can be separated from the model, the more reusable it will be. . .The corollary to this is "keep the model aligned with the concept rather than with the analysis".We initially found ourselves adopting modeling patterns which made the analysis scripts easier (drifting back into the Excel mindset).But we soon discovered that in order to further expand and refine these analyses, we would be forced to model in more and more non-intuitive ways.Therefore, we discovered, and adopted, the principle that the model should be kept intuitive and aligned with the concept.We are convinced that the extra work required to make the analysis tools work is well worth it in the long run."18 "Challenge: Modularity and Reusability.Many organizations still follow an opportunistic and isolated reuse approach, where a set of data is copied and pasted from one context to another.Unfortunately, this still happens even with system models and results in losing the "source of truth" as soon as the copied source or pasted target is changed." 4 "Provide Versioning, Variants, and Backups.If possible, this should be done on a whole model basis.Again, this ensures that impact and traceability can be assessed against the whole model.If done on a section-by-section basis, it becomes more likely that version skew will take place."31 "Model configuration management is different.Configuration management of these descriptive models is a little different than for typical analytical models.The reason for this is that these descriptive models share some features with analytical models (software) and some features with program documentation (static documents).Like software, these models are contained in electronic files.Like program documentation, these models are intended to describe the system primarily for human understanding across multiple perspectives, and therefore need to reflect the system more precisely than typical analytical models, which only need to capture the essential abstracted characteristics of the system as it pertains to that analysis perspective.However, some of the unique properties of these models may add complexity to the configuration management process.For example, software configuration management typically relies on textual comparisons of software source code and checksum calculations performed on binary files.In contrast, system models are neither textual nor linear in nature, so model comparison will be much more complex.For example, moving a graphical model element within a diagram constitutes a change to the model file, but may or may not represent either a syntactic or semantic change.Because of these unique characteristics, configuration management processes for models will need to incorporate and improve upon the appropriate features from both software configuration management and document configuration management practices."32 "Another part of the MBSE infusion approach that has been successfully implemented . . . is the creation of a library of reusable models.ings from within the model, while maintaining consistency of the model with these drawings inevitably leads to parallel systems that need to be individually maintained while remaining mutually consistent."16 "Tools are perceived as not really mature at least for SysML and language is found too complex so people prefer using MS Visio." 17

Application of MBSE methods/processes and modeling practices
"More time should be spent on properly defining how to best use existing methodologies, which therefore appear to cost much more time and resources than necessary.Time must be set aside to build up an MBSE-friendly infrastructure; otherwise MBSE practices will also be considered a burden, because they are not well defined." 61nsure that the model is centralized and distributed.This does sound like a contradiction, but it is possible.Mostly it involves making MBSE a reality.A centralized repository for the information as opposed to a file-based system ensures that people are able to access the model as a whole rather than snippets.In addition, because of the pervasive nature of the model elements and the need for cross-references, much of the model is required for most operations.Keeping as much information in the model simplifies traceability considerably and helps ensure completeness, correctness, and consistency.It also provides a means for impact analysis." 31n some cases, the technical debt that has accumulated in the models makes reuse of the model untenable, and significant refactoring of the models-if not complete reconstruction from first principles-may be the most effective approach to take in the long run.""Do not use SysML for requirements handling only!Since it was not meant to be used as such, it becomes a bit awkward and unhandy as compared to other tools, which are better suited to this kind of use." 45 "Keep the Focus on Engineering Products.Keeping focused on real engineering deliverables is important to avoid the pitfall of delivering a modeling solution everyone thinks is finished but which doesn't provide the required engineering answers.After all, the engineering deliverables are the whole point of the exercise.Our early attempts at "rolling up the mass" for the mass margin report showed this in stark relief: getting the numbers to add up, make sense, and be reliable turned out to require significantly more modeling and scripting than expected."18 "A well-defined MBSE method is essential.An MBSE method must be clearly defined to support the model development.The method should also provide guidance on how to organize the system model to ensure it can be navigated, managed, and controlled."57 "By having the project team clearly identify their goals in adopting MBSE, one can better advise them on the method to follow.The method defines the concepts and rules to model the system.Using the method, the engineers can ensure that the models used to describe the structural and behavioral aspects of their systems are created accurately.The JSMT developed a meta-model, as a foundation to the modeling method, to capture the system architecture, hardware interfaces, and command and telemetry interfaces.The tools that extract data from the system models are used to generate multiple target products.This is aimed at providing added value to the project team."6 "Have the new MBSE processes well documented so you better understand what tool you will need." 27he project team found that SysML-based MBSE practice provides a great level of flexibility, providing systems engineers with numerous options for facilitating and capturing systems engineering knowledge throughout the system lifecycle.However, this high-level of flexibility imposes the need to design the model structure carefully and select the most appropriate metamodel, SysML diagrams, and model elements to suit the project.This selection needs to be formalised and supported with model design rationale, in order to ensure consistent practices between team members, and across projects internal or external to an organization."50 "Models are Meant to be Abstractions.A common misconception of MBSE is that in order for the model to be useful it must describe everything and describe it to a fine level of detail.This misconception needs to be corrected for an infusion to succeed, because otherwise resources will run out long before the job is complete.A key principle we have followed is to model only as far as we need to answer the question at hand.Assuming this is done on an infrastructure of common languages and tools, then the model can grow over time, as necessary, and each new model element will add synergistically to the body of work."18 "It is important to realize the necessity of different levels of abstraction, their relation and how to effectively use them."61 "One must be mindful when conducting the modelling task of the level of model fidelity needed, the purpose of the model, the questions that need to be answered, and not to model for the sake of modeling.Thus, the purposes of creating the model must be clearly defined upfront."53 "Establish a consistent approach towards the definition of equivalency relationships within the Model Based Engineering Environment.
Specifically, a rigorous process must be in place to establish equivalency relationships, and to modify or remove equivalency relationships when associated artifacts change, undergo versioning, or are removed.
Without a consistent process, equivalency relationships can become confused, corrupted, or lost, leading to unreliable traceability throughout the ASoT implementation."won't be used). . .(virtuous cycle, in which the models are so highly valued and frequently used that the enterprise is compelled to commit the resources to maintain them, which allows them to retain and expand upon that value, and continue to increase in value)." 26ust Do It.We've found that the best way to figure out how to apply MBSE is to do it for real: make the commitment to adopt MBSE as the way to produce (at least some subset of) the project products, and then figure out how to accomplish this.This is in contrast to the suggestion sometimes made by skeptics, that a "safer" or "more gradual" approach would be to conduct a "shadow" or "parallel" pilot that allows side-byside comparison of benefits and drawbacks, including cost."  an ASoT that is under the DoD's control.Mark the digital artifacts approved for integration, and associate with each digital artifact the evidence that justifies that approval.Track the system throughout its lifecycle to identify the as-approved, as-built, as-maintained, and asdestroyed versions of the system.Acquire models to represent legacy components." 28M Can Start Modestly.In thinking about the needs of a Configuration Management (CM) system for our models, we found that the "We let the discovery of the need drive the solution.There was 'top down' innovation but not in the traditional sense of pre-ordained specifications: it consisted mainly of constant guidance during the modeling process to keep the effort focused on satisfying the end objectives." 18eal Examples are Powerful.Trying to describe to stakeholders and potential collaborators what MBSE looks and feels like has proven to be rather difficult and not very effective.We have found that many people 'get it' for the first time only when they see an actual example." 18eer Pressure Pays.The outside experts mentioned above could convey to our executive leadership their informed assessment of the state of the industry in a way that we practitioners could not.Their assessments are far more authoritative than ours could be, so that when they warn of JPL falling behind and becoming less competitive if it does not proactively engage with MBSE, the message is compelling and believable.In this way we have found that peer pressure pays in terms of building institutional support for MBSE.Likewise, we have found

Willingness to use tools (employee and stakeholder buy-in)
"Seek the "killer apps" (specifically identified high impact applications of the methodology that can motivate each stakeholder to make the leap from skeptic to advocate)." 32haring the model with all stakeholders and making it the reference.Once a model is recognized as the reference, it is used as a source for other engineering activities and its existence becomes therefore less likely to be challenged.Evangelization, coaching and MBSE commitment from the management are necessary to reach this goal." 34s more stakeholders were exposed to the modeling method and tools, they requested additional capabilities to extract an increasing number of system design artifacts.Expanding the tool suite and generated products makes MBSE useful to a broader audience and increases the stakeholder involvement." 6rovide expected SE products to accelerate adoption.Once the MBSE effort has proven that it can support existing processes with familiar products, these stakeholders may become more amenable to exploring more transformational approaches to improving processes and products, enabled by MBSE capabilities." 32ngage early and often -If. . .MBSE has resulted in the discovery of non-obvious issues and problems, the earlier empowered stakeholders are engaged, the earlier these issues can be addressed.Through the use of the model, the implications of a given decision or option can be quickly (relative to not using a model) analysed and then returned to the decision makers for resolution and further guidance if required.
The early and regular engagement also militates against producing a perfect solution for the wrong question." 28he models afforded by MBSE, specifically SysML, were shown to bridge the gap in communication between engineers and medical professionals.This suggests that the same benefit may be seen when working with professionals in other non-engineering domains.The ability to formally model aspects of the system and display them simply enough for all stakeholders to understand, proved invaluable." 37xpose the stakeholders to the model, but not in more detail than absolutely necessary -The model is a tool to enable design decisions to be made not an end in itself.Depending on the stakeholder group and the level of experience of the MBSE practitioners, the early and frequent engagement referred to above, may devolve into a discussion on the minutia of the model or the modelling process rather than being used as a forum to make decisions or discuss options and trades.Some stakeholder groups may, if allowed, become enamored of the model or the modelling process rather than focused on making the decisions the model was created for." 50he process itself needs to be visible, with a clearly defined purpose and observable benefits of using supporting models -using models or diagrams to communicate complex use cases and system functionality is likely to be beneficial and improve communication of design information, but the process by which this is done must also be clear."41 "The tool must be used as a communication aid, as well as for storing and executing design information.It would require the development of a system model with the appropriate structure to communicate data between multiple domain-specific tools in order to execute and analyse the validity of the proposed system."41 "Benefit for project: The ability to embed recognizable graphical icons within the models to represent system elements allowed for the models to be intuitively understood without a great deal of additional explanation.This enhanced communication was perhaps the most outstanding benefit of employing a model-based approach to engineering a system."49 "Problems arose to a higher extent between system engineering and other specialty engineering disciplines.In spite of special training provided for, e.g., safety engineers, the feedback was that the models were very difficult to interpret.A conclusion is that model based projects should be prepared to provide 'traditional style' information in the form of documents in order to, at some level, serve specific engineering disciplines when requested."33 "It is not possible to make all stakeholders (of the model content) proficient in interpreting and navigating the model.Consequently, a way of extracting information from the model to traditional documents must be established so that stakeholders/colleagues can view and review model content as effective as with a traditional document approach, without extensive mentor support." 33

1
Separated responses were coded in four categories: content orientation, organizational structure variables, adoption components, and key themes.If one row had multiple codes it could represent for a single category, that response was divided so that each row was represented by one code per category.Content orientation refers to the nature TA B L E 5 Key themes inductive codes.Industries of organizations from interviews.

F I G U R E 2 F I G U R E 3
Adoption status of organizations from interviews.Interview organizations understanding of MBSE.MBSE, model-based Systems Engineering.adopted and are using MBSE.In Figure2, which shows the adoption statuses of the different organizations, these SE organizations are represented by the "N/A" category.Two of the organizations had attempted to adopt MBSE and were unsuccessful.Another organization had started the adoption process but is currently at a stand-still.

F I G U R E 4 F I G U R E 5
Direction of MBSE adoption from interviews.MBSE, model-based Systems Engineering.Key reasons behind the initiative to adopt MBSE from interviews.MBSE, model-based Systems Engineering. of adoption directions in the organizations."Top-Down" refers to an adoption effort that originated from upper levels of leadership or management.These efforts were often due to someone at the executive level supporting the adoption of MBSE.Another source of the Top-Down effort was some type of mandate or policy at a high level, especially for organizations who work with the government."Bottom- and several influencing factors."Personal preference of personnel" is in line with the Bottom-Up organizations, while Mandate/Customer requirements overlaps with the Top-Down direction.The other two key factors are related to the potential capabilities MBSE could provide.Project or system complexity was a frequently cited reason for looking to MBSE.This differs from the final category "desire to improve" in the mindset of the organizations.Having a desire to improve the organization through MBSE has a more positive point of view.Whereas system complexity is a problem, and the organization is hoping that MBSE can be the solution.In other words, one is motivated by the presence of negative factors, and the other is motivated by the potential of positive factors.
However, future work should explore more organizations that are currently employing MBSE and DE in an exemplary way in order to support the generality of the findings in this paper.Many of the factors found from this research are also consistent with findings well established from change management.For example, leadership commitment to change, communicating the value of the change, and dedicating resources to support the change effort are all factors that have been shown to assist organizational change management, 64 all of which were also "lessons learned" for MBSE adoption from the sources discussed in this paper.This further illustrates the importance of gathering empirical evidence regarding MBSE adoption.Through the development of metrics specifically targeted at MBSE, we can further isolate aspects of MBSE adoption that are worth cementing as best practices.

" 46 "
The final lesson learned is the addition of new team members to the trade study.As the trade study matures and goes through cycles, team members come and go.One of the biggest challenges for the team is bringing new team members up to speed on the Trade Study itself.Background knowledge . . . is critical.Since the Trade Study continues moving forward, bringing any new person into the team would require them to come up to speed on all existing [knowledge], the trade study work done so far, as well as keeping up with current trade study work being done. . .The Trade Study team has found an incredible value of MBSE and this modeling methodology when bringing new team members up to speed rapidly.Tradition SE practices would have a new team member reading thousands of pages of documentation on each of the network processes and systems.Using MBSE and our specific model, new members are able to go to one source for all information they would need to come up to speed on the study as the existing network diagrams have all been created already.At the same time as the new team member is parsing through previous cycle diagrams to learn the working of each network and what the integrated solution might be,the team member can also browse models which are currently being created so the transition from keeping up to speed on the study, to producing for the study, can be as smooth and painless as possible.This has enabled our team to bring in new members and quickly get them up to production capability for the team, which in turn allows the team to shift off veteran members to further work in any one of the specific cycle programs."In the early formulation phase in which we find ourselves there is a curious duality.On the one hand, the key work in early formulation centers around conceptual thinking.The spacecraft we propose are mere sketches, and a critical function of models is to describe the design space generously, in which the concept can evolve and take shape.On the other hand, we must always show that our concepts are feasible, and one of the ways we do this is build and analyze a 'point design'

6 "
such as CAD and Finite Element Analysis since an individual project could not afford the time to train project members on the proper use of these detailed modeling tools.MBSE should be treated in the same manner.The time to train a MBSE modeler and to become proficient at the tool is beyond the ability of any single project.Another problem is that the MBSE modeler on one project may not continue with MBSE after that particular project.The time invested to train that modeler is lost when the project is completed.By having a team of MBSE modelers matrixed into all projects, the skills can continue to improve on each new project."No matter how much UML/SysML and tool training is performed, there is a critical period in any project where engineers are getting frustrated in their attempts to apply the MBSE techniques.This is partly due to the gap in complexity between the comparably simple examples used in training and the size and complexity of the product under development. . .experienced mentors were engaged to assist developers to overcome such frustration and to ensure that modeling guidelines were applied by the individual developers.The mentors also had the responsibility to verify and approve implementation of tool/method "add-ons," such as report generator implementation, document templates, and modeling guidelines.By experience, at least two members in the project team need to have experience in development with the use of object-oriented methods, the modeling tool, and large-scale models.Having two experienced persons reasoning about how to plan the modeling approach and how to partition the model, brings stability and safety to the proposed and adapted method.Initially, in a project, it seems desirable to have one experienced mentor for every 5-7 developers. . .Naturally, mentor support can be decreased as developers gain

" 34 " 57 " 18 " 4 " 59 " 58 Training" 27 " 15 " 18 "
Always take extreme care when showing diagrams to persons not introduced to SysML! Systems Engineers are nowadays used to MS Visio diagrams in which an exact and precise meaning is not associated with each type of arrows and blocks, and in which the purpose of the diagram is supporting a paragraph, or a document.This leads to misunderstandings, confusion and misinterpretations of the SysML diagrams.When a modeler presents a diagram to other engineers, he or she needs to make sure, before even explaining the diagram itself, that the others understood: the purpose of the diagram (e.g., showing architecture and not functions), what each type of element represents (e.g., physical block), and the meaning of each link (e.g., composition)." 45ORGANIZATIONAL ENABLERS/BARRIERS Leadership/management support/commitment "Leadership sets direction, supports staff development, organizes for project and infrastructure development, support and sustainment (ask for artifacts, implement MBSE-based reviews, establish and reward milestones, measure progress)."Unity of Leadership is Essential.In the first infusions, management support for the effort [should] be clear and consistent.Management must be willing to pay the startup costs and to give time for the effort to pay dividends.In addition, the engineering leadership must be reasonably unified in their willingness to work together to figure out how to do this."Executive Level Sponsorship.Although increased MBSE popularity has strengthened executive support, there are still conflicting MBSE adoption goals between short-term driven employees who care about low adoption cost, with others aiming at more adoption quality and long-term solutions."As a consequence of not adapting how projects were planned and executed, the MBSE initiative often became an isolated effort, not part of the generic planning process, leading to unclear goals and objectives.Successful implementation is a management issue, and requires managers have the relevant understanding/competence.However, this necessary education/training is often not taking place on all levels."Failure to consider the broader engineering organizational needs, as well as those of the enterprise itself, results in lack of enterprise leadership support, including MBSE adoption decisions and choices that fail to operate within the overall engineering environment and fail to achieve their goals even within the systems engineering organization." 1 Organizations should be "aware of potential target programs and [have] just-in-time availability of mature tools, training, support, expertise, tailoring approaches, and troubleshooting prior to actually beginning the engineering.It also requires the encouragement of program leadership to recognize that the competitive environment likely requires changes to the development process for the sake of improved cost, schedule, and capability."All engineers should get, at least, basic training in MBSE."New practitioners need training in the language methods and tools of MBSE.The training should also be adapted to different members of the project team.In particular, a small core modeling team may require more significant MBSE training, while the larger project team may only require sufficient training to understand the modeling artifacts.After the initial training, ongoing mentorship is essential to provide the support needed to help the team climb the learning curve."Everyone Needs Training, but to Different Levels. . .three groups need to receive training commensurate with their level of interaction with the models.Different levels of modeling familiarity are required, thus resulting in different levels of training. . .we have constructed a set of classes that addresses all three user-type groups.The classes are sequential, each one building on the one before it.We start with the basics: Architecting, AFT, MBSE and SysML familiarization for everyone.The second class is the advanced SysML and Tool-Specific training,for the engineers who will be working with the models (MagicDraw and AFT).Then finally the third class, really a continuing series of special topic sessions, is for those engineers who actually construct, maintain, and analyze the official project models."To master a modeling tool for UML and/or SysML, training and practice must provide some understanding of the definition of these model elements, how they relate, how to apply them efficiently in modeling tasks, but also what not to use of all available features.Even if only a subset of UML/SysML is used in the project, the users-and in particular the mentors-need to have understanding of the underlying meta-model and its implementation in the tool.The fact that SysML is based on UML, and that UML is in turn based on a "merge" of several other modeling notations/standards, makes the meta-model cumbersome and non-transparent."33

" 50 "
Challenge: Large Models Visualization.Different team members are involved in querying the model contents.Unfortunately, existing tools require additional training effort, and customizing the layout of model elements and diagrams is time consuming.Additional challenges appear in large models, where model navigation and understanding become highly complicated." 4

26 " 26 " 59 " 29 " 4 " 59 " 52 "
Initial exploration in the IMCE Concept of Operations was helpful.Initially setting up the model to support collaboration, we focused on: structuring modules and packages with collaboration in mind; and we emphasized single owner packages in topically defined modules.Model access permissions were set loosely for the time being.Lightweight versioning was found to be sufficient: Teamwork was used to track changes to model elements; DocWeb reports captured snapshot of full model and resource reports; reviewed and baselined versions are tagged as such in DocWeb.Quality Control is developing as needed:scripts are now doing some rudimentary model validation; a hand calculation is used before report release as final correctness check."18"Exploit "network effects" to accelerate adoption (the more a model is used, the more data it contains and integrates, the more valuable it is)."Model the 'T' to explore both breadth and depth.Modeling needs and issues driven by the breadth of model scope were found to differ significantly from those driven by depth of modeling detail.Exploring both of these dimensions of modeling early in the model life cycle was found to mitigate the risk that poor decisions made early in the model's life are allowed to propagate far and wide as the model grows."Alongside with model progress monitoring, it is crucial to organize regular model reviews involving not only model contributors but also domain experts.The model cannot be considered as the reference if all stakeholders are not involved."34"Iterative Design Approach -Through regular stakeholder meetings to confirm the results of the operational and functional analysis, divergent (and at times conflicting) stakeholder expectations were able to be managed.Presenting previous decisions and the resultant analytical consequences enabled convergence to be achieved when describing the physical domain.A strength of using a model based, vice a paper based, approach enabled a relatively fast turnaround on the effect of decisions made by the key stakeholder group." 39Cultural change management "There is a learning curve associated with reprogramming engineers.It can be more difficult to learn a new way of doing a familiar task than to learn it fresh."The most challenges related to MBSE adoption are noticed to be based on the human and technological factors.It starts with the awareness and change resistance on both executive and engineering levels within an organization.It goes over having the right MBSE resources to define the purpose, scope and method.Additionally, these challenges need to be addressed from the early phases and directly depends on the executive sponsorship and available upfront investment."The human factor plays a central role, particularly if key players have different levels of MBSE knowledge and adequate time for training is not granted.Consequently, change is not always accepted, compared to existing approaches, it creates strong resistance due to the lack of expertise to deliver the required artifacts."There is resistance to enforced consistency.Smart people like to do things their own way, which makes collaboration and knowledge transfer more difficult."Tooling inertia describes phenomena of the current in-house tooling environment that made our participants refrain from adopting MBSE.Tooling inertia includes resistance against learning new tools as well as potential incompatibilities of MBSE tools with current tools, and resistance to integrating new tools."Inertia cannot be easily offset without a change agent or a technology champion.Understanding where to champion the efforts and at what level of the project, is important to successful adoption of MBSE." 6 peer pressure within JPL to be an effective driver of infusion.IMCE has organized several lab-wide opportunities for emerging MBSE-based efforts to showcase their work and share lessons learned.The obvious benefit of course has been to cross-fertilize and share learning across many efforts.The additional benefit is the spirit of healthy competition which has been fostered."56 Literature lessons learned categories.
TA B L E 3 TA B L E 4 Interview prompt questions.

Table 6
shows the different categorizations used throughout this study (from Tables 1, 3, and 5) synthesized to factors that fall into three different classifications: organizational design, organizational enablers/barriers, and organizational change management.These classifications/factors will structure the discussion of lessons learned gleaned from this study.TA B L E 6 MBSE adoption "lessons learned" categories.

Table 8
Categories of lessons learned identified in this research.
shows some of the most prominent factors compiled from this research organized into those three categories.Appendix B shows specific lessons learned from literature applied to each of these categories.Based on reports from the literature and the practitioner interviews given in this study, there is likely still a lack of understanding or confusion about what MBSE is.This is reflected in the difficulties reported by those that have tried to adopt MBSE.The lessons learned presented in this study are a consolidation of many different organizations' experiences with MBSE.Since the measurement of empirical metrics is still an area that needs development, the information gleaned from these experiences is the main source of data we have about MBSE adop-TA B L E 8 tion and implementation.Future work needs to continue to develop measurement frameworks and empirical metrics in order to provide more substantiated evidence to convince those in industry to adopt MBSE/DE.The data obtained through these MBSE adoption interviews are consistent with not only the literature lessons, but other previous studies such as the MBSE maturity survey as well.The consistency of these important adoption categories across different studies and samples provides validation that these topics are important to MBSE adoption.
17other major inhibiting factor is the availability of skilled practitioners; not just to actually execute MBSE-based projects, but the advocacy and energy of our most talented modelers is now spent in projects (a good thing) without much time left over to educate management or practitioners in other parts of the adoption curve than early adopters (not such a good thing)."17 Statement of Work and CDRL should identify negotiated data rights for each digital artifact to be delivered. . .Communicate data rights and distribution marking policy for all types of digital artifacts that the ASoT will manage.Communicate the granularity with which markings are to be applied within diverse types of artifacts.For example, policy might call for data rights markings applied at the level of blocks in a SysML "Acquire the data rights for each digital artifact . . .Consider technical data, computer software, and computer software documentation data rights and communicate the DoD's desired rights in the solicitation for each procurement based on the TD and CS strategy according to Defense Federal Acquisition Regulations Supplement (DFARS) 207.106 in the Acquisition Planning Phase of the procurement.The 26

"
It is easy to fall into the trap of putting semantics into diagrams: the semantics must be instead fully defined by the model behind in order to create more artifacts from the same source in an automated way: this requires an expensive initial effort which pays back many more times 28 16ditional purposes can be served with additional diagrams or reports.Most of these large diagrams can-and should-be built hierarchically, in many smaller diagrams that each attempt to capture a small number of key concepts.Capturing too many layers of hierarchy in a single diagram makes it difficult to read and understand.A good rule of thumb was found to be to limit the scope of a diagram to what can be viewed on the computer screen without scrolling."26Ifnotapplied to the definition of requirements, development of the CONOPS, analysis of organizational impacts, and security concerns and the planning of development and integration, the resulting implementation of tools and the information technology infrastructure will likely fall well short of the strategic needs of the business and the functional and performance requirements."1 44nother major pitfall is keeping several engineering levels in the same model 'for the sake of simplicity' .The architectural design of a system is not the same thing as the architectural design of each of its subsystems: lifecycle are different, contributors are different, design drivers might be different, etc. Performing the design of a system and a subsystem in the same model is a major error that has led to significant refactoring work 2 or 3 times on ...projects these last years."34"Buildinflexibilitytoadapttofutureneeds.Architects of the integrated set of system models should provide some flexibility in the "Build models hierarchically and eschew 'eye chart' diagrams.Each diagram should be built for a specific purpose, and only the information needed to communicate that purpose should be included in the dia-gram."Bringeveryonetoadoption(i.e., avoid creating castes)."27"Operationalusersgetlost in their objectives, face difficulties with the tooling.Damages are there already when help is sought, typically leading to blaming the method and workbench."34"Theadoption of MBSE and the development and implementation of the digital engineering environment requires systems engineering methods and rigor."Acrucialbasis for MBSE adoption is to define a clear purpose and scope (the why and what).Ideally, it must be precisely described before beginning the deployment.However, this is a challenge in real world applications, where modeling can be used in so many ways."4"Fornew programs the challenge is to gain adoption early during the program definition phase.Short time-to-market leaves little time for learning MBSE once the program begins, yet there may be little demand to learn an MBSE approach prior to having a targeted program 18