Model-Based Systems Engineering – Adoption Challenges (Survey Results)

Model-Based Systems Engineering (MBSE) has been challenged concerning its successful adoption in real-world applications. Although MBSE remains to be the focal point of any systems engineering activities, its adoption still faces significant hurdles to demonstrate its return on investment.

Based on our MBSE research and industrial work, we explored the topic of MBSE adoption challenges and found out the importance of alignment among MBSE community members for effective understanding of the actual status of MBSE adoption challenges. Therefore, in early 2017, we decided to identify a set of MBSE adoption challenges and then ask, through an online survey, participants to collect the opinion concerning these challenges, what phase of MBSE adoption they occur in, and the dependencies between them. We presented the publication described in this post at the EMEA Sector Systems Engineering Conference 2018.

With this contribution, we aim to trigger the MBSE and INCOSE community for further discussions and industrial feedback to help in understanding and measuring the challenge space of MBSE adoption. In this way, appropriate solutions could be sought to overcome the existing challenges.

MBSE Survey Objectives and Design

The MBSE adoption challenges survey was conducted using an online questionnaire. We received 42 complete and faultless responses in total. The survey content, especially the challenges list was the result of several interviews with MBSE experts. The aim of the survey was to find out the current state of the MBSE adoption challenges, particularly, to target the following three objectives:

• Objective 1: Review the collected challenges list and collect new ones
• Objective 2: Determine in which phases the challenges are faced
• Objective 3: Determine the dependencies between the challenges

The survey was designed as a one-page survey. It contained its purpose description, a representative example of the expected results, a background information section and the questions section. The questions section was composed of 6 required questions grouped into two parts. Part 1 presented 3 fundamental questions to collect the opinion concerning: (1) MBSE adoption challenges, (2) what phase of MBSE adoption they occur in, and (3) dependencies between them. Part 2 presented 3 questions to reveal the demographics and background of the survey participants.

The survey respondents had the opportunity to provide their contact information to take part with the survey prize draw. To ensure a greater chance that more respondents complete the survey, the time needed to complete the whole survey was estimated to be less than 15 min. Final results showed that the typical time spent was 12 min.

Survey results

Question 1: MBSE Adoption Challenges List

The aim of Question 1 was to target the first objective of the survey by reviewing the set of MBSE adoption challenges and collecting new ones. Respondents were asked to indicate their level of agreement for each challenge whether it is related to MBSE adoption. An “other” answer option, as text box, was enabled to this question to allow participants to specify other challenges which are not listed in the question with their level of agreement.

According to the summary of responses, between 65% and 85% agree and strongly agree that all survey challenges are directly related to MBSE adoption. Whereas 7% and 24% decided to neither agree nor disagree and between 2% and 12% disagree and strongly disagree that the challenges are not related to MBSE adoption.

On the one hand, the majority of individual responses (57%) strongly agree that awareness and change resistance is a principal challenge. Together with those who agree (31%) on that, it forms 88% of the total participant’s responses. This confirms that the human factor (or personnel) and the challenges it brings with respect to change resistance is the one that organizations should be solving first. A similar observation has also been reported for the purpose and scope definition challenge (88% for strongly agree and agree). This further asserts the difficulties of defining the goals and boundaries of MBSE adoption.

The human factor is the first challenge to tackle when introducing model-based systems engineering in an organization Click To Tweet

On the other hand, the technical factors conquered the financial ones. Upfront investment seams not be a major challenge as 17% decided to have a neutral opinion and 4% split among disagree and strongly disagree. Whereas the adoption technical factors (tool dependency and integration, method definition and extension and adoption strategy) got more strongly agree and agree percentage.

Question 2: In what Phase(s) of MBSE Adoption the Challenges Occur

The aim of Question 2 was to target the second objective of the survey by determining in which phases the MBSE challenges are faced. Respondents were asked to indicate their opinion about the phase(s) where the MBSE challenges occur. A rating scale matrix question was used where rows represented the list of challenges and columns the phases (early, middle and late phase). Multiple responses per row were allowed to enable the section that a challenge occurs in more than one phase. Like Question 1, an “other” answer option was enabled to allow participants to add new challenges with the phases they occur.

According to the clear majority of responses received (above 90%) consider that the upfront investment and purpose and scope definition as early phase challenges, 20 and 24 responses respectively as middle phase challenges and 4 and 5 responses respectively as late phase challenges.

An overwhelming majority of responses (above 70%) recognize that adoption strategy (35), executive level sponsorship (33), awareness and change resistance (31) as early phase challenges. Whereas, these challenges continue to play role as middle phase challenges with 20, 27 and 26 responses respectively and with respect to late phase challenges they got lower responses (6, 16 and 18 respectively).  Responses with respect to the method definition and extension challenge tend to spread among the early, middle and late phases. However, all other challenges, i.e., modularity and reusability, complexity management, tool dependency and integration, and large models visualization lean more towards middle and late challenges.

Question 3: MBSE Adoption Challenges Dependencies

The aim of Question 3 was to target the third objective of the survey by determining the dependencies between the MBSE adoption challenges. Respondents were asked to select for each challenge the other challenges it depends on. A rating scale matrix question was used where rows and columns represented the list of the challenges. It was assumed that a row challenge could depends on one or more column challenge(s), so multiple responses per row were allowed. Checked boxes where row and column are same were ignored (no reflexivity).

The percentages shown represent the respondents’ results of dependencies between challenges. The total dependencies per row challenge are also shown. Dependencies are from a row challenge to a column one. For instance, with respect to the upfront investment, the majority agrees that it depends on the purpose and scope definition, executive level sponsorship and adoption strategy. The method definition and extension depends on average on all other challenges however mainly on the purpose and scope definition and modularity and reusability. Whereas the executive level sponsorship depends on the upfront investment, purpose and scope definition and awareness and change resistance.

Interestingly the method definition and extension gathered the highest dependencies’ total percentage value (407%). It was followed by the upfront investment (383%) and adoption strategy (381%). On the other side, the large models visualization gathered the lowest total percentage value (257%). In summary, Figure 3 demonstrates how it supports in tracing the path to solve a particular challenge based on the dependencies. Additionally, having several time-based versions of it would measure the progress of solving, or not solving, a challenge and the proof behind the solution.

Question 4 to 6: Participants Demographics

Survey Findings

The work presented here is a first step in the direction of defining, understanding and measuring the challenges associated with the adoption of MBSE. Most responses came from practitioners in consulting, where personnel are involved in different MBSE adoption projects. The goal is not to get the perfect answers, but rather understand the actual status of the adoption challenges among the received responses.

Finding 1: The most challenges related to MBSE adoption are noticed to be based on the human and technological factors. It starts with the awareness and change resistance on both executive and engineering levels within an organization. It goes over having the right MBSE resources to define the purpose, scope and method. Additionally, these challenges need to be addressed from the early phases and directly depends on the executive sponsorship and available upfront investment.

Finding 2: Although other challenges were less related to MBSE adoption, results show that respondents know that it will come in later adoption phases. This knowledge is very crucial while looking at the dependencies for overcoming the challenges. For instance, to see how a challenge as complexity management is well known from the beginning as a late phase challenge, the question mainly lies if the adoption strategy, methods definition and modeling tools consider solving it from the beginning.  Unfortunately, it not the case yet in practice, as far as we can testify.

Finding 3: After correlating all the data collected together, an important contribution related to setting the challenges’ priorities can be observed. Challenges can be prioritized based on three factors: importance, time and dependency. However, these factors are not unique across organizations. For instance, companies with high available upfront investment might suffer from having the freedom that each department starts to define its own MBSE adoption solution. This brings later integration and model interchange issues. Like time, organizations with different design and development timeline could face same challenges but in different phases. Moreover, delivering specific customer products (e.g., rolling stock) differ from other delivering generic and more configurable products (e.g., automotive). Therefore, the first step to organize these three prioritization factors based on organizations domains and then look for common aspects if they exist.

The results of this survey work demonstrate how the survey objectives have been accomplished, they open the opportunity to overcome MBSE adoption challenges and thus to achieve effective MBSE implementation. We believe that the challenges faced are not only due to MBSE but the way it is adopted. A long-term vision should not only consider the tools aspect but also target and solve methodological, technological and educational perspectives. This is part of our ongoing work.

You can also check a summary of this model-based systems engineering adoption work in the presentation below