Today we present the last version of Collaboro, our approach for the collaborative development of Domain-Specific Languages. Previously, we have published the results of this work in these papers:
- Enabling the Collaborative Definition of DSMLs. CAiSE 2013: 272-287. Slides
- Community-driven language development. MiSE 2012: 29-35. Slides
In this blog post, we present a recap of the features and goals and include a summary coming from our latest publication on this topic, titled Collaboro: a collaborative (meta) modeling tool published in the PeerJ journal.
Collaboration between developers and end-users seems particularly fitting when developing Domain Specific Languages (DSLs) given the fact that DSLs target the experts of a domain. However, current DSL development processes are completely centered on the developers who still lead all phases of the DSL development, while end-users only participate in the very beginning, when the domain is analyzed, and they cannot see how their input was interpreted until the modeling environment for the new language is deployed. The validation is therefore performed at the end, which implies rebuilding the environment (a very costly process, even with current language workbenches) every time the end-user wants to change the language definition (e.g., to add new properties to a metaclass, change their type,…).
To improve this situation, we propose to make the development process for DSLs community-aware, meaning that all stakeholders (i.e., technical and end-users) have the chance to participate, discuss and comment during the DSL definition phase (at the moment, this process only covers the abstract syntax, i.e., the metamodel, part of the DSL, we plan to extend it to cover also the collaboration process for the concrete syntax). Moreover, we believe it’s important to track this discussion so that in the future we can justify the rationale behind each language design decision.
To make this collaborative development process feasible, it is important to facilitate as much as possible the collaboration between all people involved. Our solution is based on a new DSL, called Collaboro, to represent the collaborations which arise among the members of a language community. Collaboro has been implemented as an Eclipse-based tool and as a web-based solution that can be used by all kinds of users willing to have an active role in the evolution of their DSL. You can have a look at the project in this webpage.
The core of the abstract syntax of the language is shown in the following picture. As you can see, with Collaboro you can track all the change proposals, solutions (plus the set of changes to apply on the DSL to adopt the solution), comments or votes. The DSL element can represent both abstract (e.g., concepts and attributes) and concrete (e.g., notation symbol or keywords) elements of the DSL under development so that collaboration can arise on any element of the language.
To show how Collaboro works in practice, imagine a Transport DSL used by a transport company to represent its fleet and calculate the most efficient path between several points. The DSL therefore includes concepts such as Truck and Route to model the distribution maps (top part of the next Figure). Thanks to the fact that we are now following a collaborative development process, by looking at this initial version of the Transport metamodel, End-user 1 detects that Transport is missing the possibility of stating the traffic density of each Route concept and claims this is an important concept for the language. Since the rest agree, an initial solution is proposed and improved (after a comment suggesting a slightly different strategy which is voted and selected by the community).
As can be seen, both the collaboration and the decision processes are modelled. In this case we have applied a full agreement policy so the change proposal and the corresponding solution are marked as accepted only if every community member agrees.
While our first versions of Collaboro supported the collaborative development of textual DSMLs in an Eclipse-based environment, the last version offers:
- Better support for the collaborative development of graphical DSMLs;
- A new architecture which includes a web-based front-end, thus promoting usability and participation for end-users;
- A metric-based recommender system, which checks the DSMLs under development to spot possible issues according to quality metrics for both the domain and the notation (relying on Moody’s cognitive framework (Moody, 2009))
- The development of several cases studies, which have allowed us to improve the general expressiveness and usability of our approach
Additionally, we have also explored and described how our tool could be easily adapted to support collaborative modeling. Check the last paper for further details.
The following slides (from our CAiSE paper, not the journal summary one) may be also useful to get a quick view of the approach:
I’m an Associate Professor at UOC and researcher at SOM Research Team, in Barcelona, Spain. My research interests are mainly focused on Model-Driven Engineering (MDE), Domain-Specific Languages and Collaborative Development.
very interesting. what does “current DSL development processes” refer to?
You’re right, the term “current DSL development processes” may be a bit confussing. With this term we wanted to refer to the existing approaches and recommendations to develop DSLs (e.g,. “When and how to develop domain-specific languages” from M. Mernik, J. Heering, and A. M. Sloane).
This comment make me think that perhaps, in MDE, we cannot still speak about a well-known DSL development process (any comment or suggestion is welcomed).
Interesting.
I see in the use case, it is an Eclipse plugin. Is it possible to use it online in a multiuser enviroment (let’s say, an actual company)? Or is it a proof of concept?
Thanks.
Hi Ricardo,
At the moment it is a proof of concept (as it usually happens with all research prototypes 🙂 ) so synchronous editing of the Collaboro model is not supported yet. We hope to find a company interested in industrializing it (like here http://modeling-languages.com/industrialization-research-tools-atl-case/ )
It sounds like you need to widen your idea of “current DSL development processes” and “current language workbenches”. Is it really the case that in the Eclipse / academic world, language development processes and tools are as bad as you suggest? Or do you just need to portray them in a negative light, to set the stage for the improvements in your paper?
Developing the language in isolation from its users is clearly a bad idea – can you find anyone who suggests that? For a more sensible process, and one proven in practice over the last 15 years, take a look at Domain-Specific Modeling by Kelly & Tolvanen (www.dsmbook.com), or Laurent Safa’s article from the OOPSLA DSM Workshop in 2006, http://www.dsmforum.org/events/dsm06/Papers/19-safa.pdf, where he lists as the first two commandments:
– Get embedded into the practitioner team
– Observe the way people work to understand their context
In which language workbenches is “adding a property to a type” a “very costly process”? In MetaEdit+ it takes less than 10 seconds to add a property and deploy the updated metamodel and tool support to all the modelers.
Recording design rationale is something we experimented with in MetaEdit+ in 1992-1998. Look up research by Harri Oinas-Kukkonen and Janne Kaipala. In the end we didn’t find as much practical benefit from it as we’d hoped – a heavyweight change process didn’t fit in well with an agile approach to changing the language (or models). Rather than one fixed process and set of concepts for design rationale, we decided to let customers make their own, to the extent they felt it useful. That seems to have worked better, and of course it’s more domain-specific, so fits well with our philosophy.
Hi Steven,
Some comments:
> Or do you just need to portray them in a negative light, to set the stage for the improvements in your paper?
Not at all, this is more the task of tool vendors 🙂
> In which language workbenches is “adding a property to a type” a “very costly process”?
As I’m sure you understand, the post uses a simple example to keep it short but, if the value in the new property affects the way the object is displayed we do need to perform some non-trivial changes.
> Developing the language in isolation from its users is clearly a bad idea – can you find anyone who suggests that?
I’m sure nobody recommends this. It’s more that the lack of tool support forces people to work this way (i.e. without “persisting” the discussion and/or letting the user to be a more active, i.e. hands-on, participant). We believe observing is good but not yet enough. Sure, we have in the agenda the need to conduct some experiments to validate this assumption.
Hi Jordi,
Thanks for the clarifications. What is the language workbench, and how long does it take to add a Traffic Type enumeration property to Route, and have the concrete syntax display its value e.g. in a small font at the bottom centre of the symbol? Just do it and time yourself for the first time, not for how quickly you can type once you know what the code should be 🙂
What are the benefits that you see which offset the “very costly process” and “lack of tool support” in that current language workbench, rather than some other language workbench which would make changes on the fly easier?
Do you think Safa’s Joker concept or Oinas-Kukkonen’s Debate Browser would be a useful approach for this language workbench? Having the user’s comment in the context of an instance model, rather than the metamodel, may make it easier for the user to express himself, and easier for the metamodeler to understand the comment.
The concrete syntax for the decision model above is much nicer than the black and white boxes in the paper – is the syntax above really in the tool, or is that just a picture?
Steven,
Thanks for the pointers to these works. I don´t think we were aware of them.
One thing that for sure would be nice is that users can ask for changes “by example”. It would be great if the tool allows them to define the model they would like to be able to define (even if this model is not 100% conforming to the metamodel) and then ask the designers to change the metamodel in a way that the model becomes possible.
Regarding the concrete syntax, the one in the post is not implemented. In fact, as we say in the post, Collaboro at the moment has no concrete syntax yet, it has to be used from the tree view of the (meta)model
In this context, I would kindly refer to my thesis in the area of model versioning. In this thesis, a conflict-tolerant merge approach incorporates all changes of all developers and highlight the occurred conflicts. These conflicts can be resolved following a collaborative approach by prioritizing the individual changes.
http://www.ub.tuwien.ac.at/diss/AC07812234.pdf (Chapter 5, p.67ff)
Cheers,
Konrad
Hi Konrad,
Thanks for posting your thesis, it’s good to see AMOR research here! There are a couple of methodological points about the experiment in Chapter 7 that bear mentioning. Before going through them let me say that I thoroughly believe the overall thrust of the results you obtained; I just want to show how you could have improved the validity.
First, if you are comparing two different tools for doing the same task, you should randomize the order of the tools for the subjects, otherwise the 2nd tool gets a significant benefit. You’ve done well to use different task data for the different tools, but changing the order too would improve the validity.
Second, in your questionnaire all of the questions are effectively “our CT Merge is better than EMF Compare”. Asking the question in such a leading way puts serious doubts on the validity of the results. Better would have been to vary the order of the comparison; best perhaps to simply have the subject score each tool separately on a 5-point Likert scale for how well it carried out the task of that question. By just asking “is A better than B?” you lose the ability to see how big the difference is between them.
Finally, even if you’re not going to claim statistical significance, you should certainly try and stick to some basic statistical and mathematical principles. Stating percentages to three significant figures, on a population of just 18, is kind of overdoing it :).
My biggest qualm however is not academic, but more from the real world. Your experimental setup assumes three people working on the same small model, and you force them to make conflicting changes. That goes against your survey finding, that work is divided up to avoid conflict. It also goes against research on modeling tools back in the 90s, that showed that only 3 out of 1000 operations are conflicts. Your experiment is still worthwhile for investigating ways of dealing with conflicts, but you haven’t really studied whether a better approach, avoiding conflicts in the first place, would be more effective overall. Having talked to rather a lot of people working on large modeling projects, some using a merge-based approach and others using fine granularity locking, I can say with some certainty that locking works far better in practice.
Our research on this topic was back in the 90s, so it’s a shame that your comparison of tools and approaches doesn’t extend that far. If you’re interested you can look at Chapter 5 in my thesis (http://www.metacase.com/stevek.html), or at a recent article from OBJEKTspektrum on Mature Model Management(http://www.metacase.com/papers/Mature_Model_Management.html). The article basically lays out how we’ve made a number of very different decisions from the EMF+UML+XMI&VCS+merge approach, replacing it with a GOPPRR+DSL+repository+locking approach, and what benefits have been obtained from this. It would be interesting to hear your feedback, and whether you’re able to get past the fact that we basically throw out everything that today’s academic modeling research takes for granted!
All the best,
Steve
Hi Konrad,
We are well aware of your work and I do believe it´s complementary in many aspects (in fact, we had a chat with Manuel at the beginning of our work on Collaboro to start exploring this complementary but we decided to wait until our idea of Collaboro was more developed, I guess it’s time to meet again!)
Hello,
Another interesting approach to collaborative modeling is the ComMod (Companion Modeling). Wich focus on iterative modeling to design complex systems. And also aims to collective decision making for collective agreement.
To test the model/metamodel/DSL, you can put in place some role playing games and figure out how the DSL would be used by end users.
It is mostly used in sustainable development and to validate multi agent systems for simulation.
You can find the charter of the ComMod there : http://cormas.cirad.fr/ComMod/en/charter/
Regards,
Xavier
This is obvioulsy interesting and needed in practice, but collaborative language development tools already exist: See 10min video at: http://www.metacase.com/webcasts/multi-user.html. Not only language users can join the DSL definition, but note that also models update immediately based on the changed made. This makes DSL development agile and emphasize participation leading to better DSLs.
Since all are customizable on the langauge side, you could also add here the ’Collaboro’ concepts too. The practice we often see is that language users can apply joker, comment etc concept to give their feedback. Depending on the user rights – as shown in the video – users may also be allowed to do the changes by themselves.
Yours,
Juha-Pekka