After reading the paper “Transition to Model-Driven Engineering – What Is Revolutionary, What Remains the Same?” by Jorge Aranda, Daniela Damian and Arber Borici I immediately contacted the authors and asked them to summarize the main lessons learnt from their work in a guest post. I have no doubt you’ll for sure enjoy it. Enter Jorge.
When it comes to MDE, there has been far more ink (or pixels) dedicated to opinions on its purported advantages or disadvantages than to reports of practical experiences of real-life organizations that have started to use it.
A few months ago, my colleagues (Arber Borici and Daniela Damian) and I presented a paper to try to buck that trend. In the paper, we reported on a case study about the attempts of a large organization (General Motors) to transition to model-driven engineering for its software development efforts. The case study is based on a visit to General Motors, where I interviewed people in several different roles involved with software development in one way or another, and on observations of their software development process. (Details on the study’s methodology can be found on the paper itself.)
Four findings in particular stood out. The first has to do with the question of whether MDE actually helps bring software development closer to domain experts—after all, that is one of the advantages it is supposed to have, the ability to shed the complexity of software development so that domain experts can “code” their models themselves. We found that MDE indeed brings software development closer to domain experts. For example, some control engineers were able to delve straight into their models to specify the behaviour they intended their software to have. However, software engineers were still necessary to perform a “middle-men” role for many tasks, mostly having to do with organizational and process-based tasks needed to move along their projects, and with more arcane issues in software development that modelling tools were not equipped to resolve.
The second finding may perhaps be a letdown for MDE enthusiasts. We found that MDE in practice leaves the essential challenges of software development unchanged. It may bring some incremental improvements in productivity, but nothing more. Software development is difficult because it is hard to resolve stakeholder conflicts, to communicate effectively, and to transform a real-world problem into a computer-based representation, whatever the representation’s language may be. MDE does not change these essential difficulties—in fact the software development process at GM remained largely indistinguishable from the one it had before the transition.
The third finding is somewhat of a cautionary tale. While MDE does nothing to address the essential problems of software development, in some cases it may even bring about additional problems that weren’t there before. For some groups within GM, switching to MDE disrupted the organizational structure and altered its balance, creating morale and power problems. Small teams that were previously working effectively saw their dynamics upended: where before MDE one engineer (the control engineer) worked on the “physics” of a problem and another engineer (the software engineer) worked on its implementation, after MDE the control engineer did most of the technical work and the software engineer was stuck with testing, dealing with change review boards, herding work items, and ensuring that the resulting models adhered to the standards of the organization. The first engineer did the cool stuff, the second saw her job transformed into a clerical position. The resentment and morale problems were still a significant issue at the time of my visit, months after the switch to MDE took place.
Finally, we also found that the pioneering state of many MDE efforts mean that the cultural and institutional infrastructure to support MDE is not fully developed yet. Students do not come out of university knowing MDE, tool support still lags significantly in comparison to the capabilities of traditional software development tools, basic problems are still being uncovered and addressed, and so on. Transitioning organizations may find that the advantages of MDE outweigh these infrastructural frictions, but they should expect those frictions to be there when they make the switch anyway.
Of course, readers should consider that this case study arose from limited observations in a single organization—your mileage may vary, and I’d be interested in hearing your experiences. For those considering a transition to MDE, though, I would say that (a) it is a viable proposition, at least in some domains, but (b) it is still a proposition that is organizationally and technically painful.
FNR Pearl Chair. Head of the Software Engineering RDI Unit at LIST. Affiliate Professor at University of Luxembourg. Â More about me.
Jordi, Jorge,
thanks for this post, I think you are raising a really crucial issue here, which is related to the general problem of mainstream adoption of MDE practices.
I would like to add just another bit to your analysis: I think that there is one aspect that is definitely disruptive when adopting MDE practices: the revolution that it implies at a cultural level.
This goes beyond the organizational issues you are mentioning: adopting MDE implies a disruption on the way of thinking of developers, which is often hard to embrace and that sometimes constitutes a reason for resistance, especially from software developers (while it’s less critical for domain experts, managers and business people).
One point I tend to disagree with in your analysis instead is that “MDE in practice leaves the essential challenges of software development unchanged.”
Honestly, I don’t think this is the general case: if one adopts the proper tools and methodologies, I think that the way several challenges are addressed can be dramatically different. For instance, MDE tools that provide quick prototyping are extremely useful for reducing the conflict and the cost of interaction with stakeholders. Tools that let move from prototypes to final products without loosing the intermediate artifacts can save time and money, as well as acquired knowledge. Finally, MDE brings a huge advantage in maintenance phases.
All these statements come from our extensive experience with customers (also reported in quantitative studies we are going to publish soon) of the WebRatio tool suite, adopted in a diverse set of scenarios, spanning from very small customers and projects, to huge organizations and applications, and from Web-specific applications to BPM applications.
Marco, thanks for your comments, and I look forward to reading your studies!
There are so many ways to adopt an MDE approach that it is not possible to establish general conclusions about MDE itself. I would however agree with that one which is not specific to MDE: “Software development is difficult because it is hard to resolve stakeholder conflicts, to communicate effectively, and to transform a real-world problem into a computer-based representation, whatever the representation’s language may be. ”
This is the very core of the problem, especially the first two, and it is not just conflict, it is also insight. Conflict is secondary, it is very visible, but lack of insight, and difficulty to communicate whatever insight you have is a far greater problem.
And apologies for the shameless plug, but this precisely why I wrote a book about Communication and Insight in relation to the Software Development Process. Because it is indeed possible to communicate in a far better way and it is indeed possible to elaborate insight.
I think it would be important to point to Jorge’s Ph.D. Thesis as well: http://catenary.files.wordpress.com/2010/09/arandagarcia_jorge_201011_phd_thesis.pdf
fascinating work for anyone looking at improving software development processes.
JJ-
Good article!
As Marco says: it’s as much a cultural (/educational) thing as a technical/knowledge thing – maybe the balance is even 80%/20%.
One thing that I see is that there is a high coincidence among practitioners of MDE and tool smiths: we make our own tools (to a large degree anyway, using frameworks or workbenches) because we are able to and like to do so, but that’s quite untypical for stakeholders in general. This leaves a yawning gap with the domain experts whom we so desperately vow to empower, and leaves us to struggle for recognition and adoption. Often enough, MDE is abandoned later in a project because the original “smart guy (m/f)” couldn’t be bothered to remain around anymore and the MDE tooling is abandoned.
Good work, Jorge et al.!
As Jean-Jacques points out, “MDE” covers so much ground that we need to look more specifically at the languages and tools used. In this case it’s Simulink and Rhapsody. These were released in 1992 and 1996 respectively, and have long been widely-used de facto standard tools for embedded systems. So not really a cutting edge case, but still very valuable empirical research.
They’re both tools where the vendor provides the language and generators, so this isn’t comparable to something like Domain-Specific Modeling. Unsurprisingly, the results don’t compare well either: “incremental” productivity improvements rather than the 5-10x common with DSM. Basically Simulink and Rhapsody are similar to the CASE tools of the late 80s / early 90s: research showing that these ‘fixed’ tools didn’t give the desired results was what prompted the MetaPHOR project which led to MetaEdit+.
Meinte’s point about in-house (or consultant) tool smiths is a good one (albeit not related to the GM case). If one smart guy builds the tool, or even builds a language with a tool that requires low-level or complex work, the language and tool use will most likely stagnate and die when that tool smith leaves the project. And most tool smiths love building but hate feedback from “dumb” users, so they don’t tend to hang around once a language starts being used in earnest. This was one of the reasons we decided to make building languages with MetaEdit+ as simple as possible – the other being that then domain experts can define the language.
There are many ways to get a tool and a language, from buying both to building both. I believe the evidence is pretty conclusive that the best way for most cases is a commercially-maintained tool with a language built in-house.
Steven,
Thanks. You’re right that Simulink and Rhapsody are far from cutting edge, from the point of view of MDE as a field, but despite their age they are still quite experimental from the point of view of mainstream software development.
Could you please point me to your sources regarding common productivity improvements of 5-10x with DSMs?
Jorge,
Of course you’re right that Simulink isn’t used for “mainstream software” – nobody would build database apps with it. But in embedded software interfacing with the physical world, my impression is that it is a de facto standard. Finding sales figures for Simulink isn’t easy, but back in 2004 they claimed tens of thousands, with hundreds of thousands using MATLAB for “DSP algorithms or control systems for embedded software” http://eetimes.com/electronics-news/4050334/Matlab-edges-closer-to-electronic-design-automation-world
I’d be surprised if you can find a company making automotive software that doesn’t use Simulink.
I’ve collected references to empirical studies of real-world DSM projects here:
http://www.metacase.com/blogs/stevek/blogView?showComments=true&entry=3446309083
As I said, I believe the key difference between DSM results (5-10x productivity) versus other MDD approaches (0.85-1.4x productivity) is that in DSM, the language is made to fit the problem, whereas in other MDD approaches the problem is made to fit the language.
For a long time I wondered why we weren’t seeing similar results published of projects using other language workbenches. While I thought MetaEdit+ was the best, surely the others could achieve results that were at least close? Last year a group of Eclipse committers on the Papyrus tool did an experiment, implementing the same language with various language workbenches. They found that with MetaEdit+ they could build the language 10-50x faster than with the other tools (Rational Software Architect, GME, MetaEdit+, Obeo Designer, Eclipse GMF)http://tinyurl.com/gerard12 That kind of difference could well explain why fewer projects with those tools make it through to successful adoption. For those that do, the language builder has had to concentrate more on lower-level details of implementing the language in the tool than on designing a good language for the domain.
Another possible factor is what languages people choose to build: tools can affect that choice explicitly or by making some things easy and others hard. We examined 76 DSM projects and found that successful ones tended to avoid certain things in their language and process. The results are published in IEEE Software:
http://www.metacase.com/papers/worstpracticesfordomain-specificmodeling.html
Thanks for the detailed response, Steven.
With regards to the prevalence of Simulink in the automotive industry, I believe you would have been indeed surprised if you visited GM about three years ago. It’s possible that the organization had *bought* licenses to use it; it’s also possible that it was used in small groups within the company, experimentally or for niche demands. But getting to the point where software development as a whole happens mainly (almost exclusively) through Simulink and Rhapsody was no small feat for GM, and in fact the transition was not finished everywhere when we visited, more than a year ago.
“The second finding may perhaps be a letdown for MDE enthusiasts. We found that MDE in practice leaves the essential challenges of software development unchanged. It may bring some incremental improvements in productivity, but nothing more. Software development is difficult because it is hard to resolve stakeholder conflicts, to communicate effectively, and to transform a real-world problem into a computer-based representation, whatever the representation’s language may be.”
This is because MDE is a means to an end, and that end has been been successively deprecated to the point that it has become invisible.
Thirty years ago, there were thirty methods and thirty notations. Today, we have one notation and zero methods. (Most of ‘agile’ doesn’t count, because 85% of it comprises social or management techniques, and the remaining 15% boils down to TDD and refactoring–coding techniques.) A method is all about how we _think about problems_, together with a set of techniques and processes for how we go about applying, verifying and valuating that thinking. Where do we find that kind of information in the context of MDE in 2013? Nowhere.
Moreover, we have confused ‘abstraction’ in the problem domain with ‘abstraction’ away from detail and implementation. “What are the correct abstractions for this domain?” is an entirely different question from “How can I represent the code in this system without getting bogged down in detail?” No one talks about abstraction in that first sense.
The claim, then, that MDE “leaves the essential challenges of software development unchanged” applies because we have focussed MDE on tools, metamodeling and sketching out _solutions_, rather than viewing a model as a vehicle for capturing the results of resolving conflicting stakeholder requirements, _using_ the model to communicate, and representing a _problem_ in an unambiguous manner that may be transformed into a computer system.
In short, we have promised that MDE will address software development challenges at the same time that we have emasculated MDE past impotence. Other than that, I don’t feel let down at all.
That is refreshing to read indeed.
To be fair, one should add that it hasn’t been an easy task in the last two decades to practise, apply and advance MDx technology, and if that study is about the past, then it may be so.
Nevertheless, if the paragraph about “…essential challenges of software development…” really holds until today, honestly, I worry about GE.
Already around 2000, I had the pleasure to participate in an innovative project where a fully functioning insurance broker portal was created automatically from models, with quite the opposite of “incremental improvements”, compared to development around that time.
If you ever have seen the beauty and impact of such fully model driven solutions, which have matured from experiments into products since then, you must wonder.
Maybe one sould not look into the hugest enterprises, but smaller companies for MDx success stories?