Mathieu Acher ( @acherm), Benoit Combemale ( @bcombemale), Philippe Collet (@colletp) present their idea of metamorphic DSLs as the solution to the internal vs external DSLs dilemma.
Domain Specific Languages (DSLs) are little languages with specific notations and constructs for resolving problems in a particular application domain. By trading generality for a focused support, DSLs promise substantial gains in productivity and ease of use in a limited domain. Developers, scientists, end-users: we are all using DSLs. Outstanding examples of DSLs are plentiful: Makefiles for building software, Matlab for numeric computations, SQL (Structured Query Language) for databases, Bibtex for managing references, Graphviz for drawing graphs, etc.
DSLs are now spreading all over the place, in numerous domains, and through different shapes:
- Plain-old to more fluent APIs;
- Internal or embedded DSLs written inside an existing host language;
- External DSLs with their own syntax and domain-specific tooling
Let’s see an example of each.
The basic tradeoffs between internal and external DSLs have been already identified and are subject to extensive discussions and research for several years. If we take the example of SQL, users can rely on the plain-old notation. Some Java API like JDBC or on some fluent API like JOOQ are also available and widely used. LINQ is yet another alternative, built on top of C# . In fact, all shapes of DSLs have strengths and weaknesses – whether you are a developer or a user of a DSL.
Who on earth would like to start learning SQL with a Java-like syntax and environment? The external notation is rather used in an introduction to databases. On the other hand SQL has somehow to be integrated with other general-purpose languages in which programs (e.g., Web applications) are written. It does explain the emergence of fluent APIs for SQL like JOOQ.
A new trend is observed. DSLs are now so widespread that very different users with separate roles and varied objectives use them. Depending on the kinds of users, roles or objectives, an external shape or an internal shape of a language might be a better solution. The case of SQL is quite representative of this trend. Our 4-years journey of providing the “right” support (in the domain of feature modeling: http://familiar-project.github.com) led us to develop an external DSL, different shapes of an internal API, and maintain all these languages. A key insight is that there is no one-size- fits-all solution or no clear superiority of a solution compared to another (https://hal.inria.fr/hal-01061576). We found that it does make sense to continue the maintenance of an external and internal DSL.
To the question:  Which side are you on? Internal DSLs, External DSLs or both?  We thus answer “both”!
What about metamorphic DSLs that change their shape according to your needs? Based on our experience and on an analysis of the DSL engineering field, the vision that we foresee for the future of software languages is their ability to be self-adaptable to the most appropriate shape (including the corresponding integrated development environment) according to a particular usage or task. We call metamorphic DSL such a language, able to change from one shape to another shape.
Being that said, the “metamorphic DSL” vision still needs to be technically realized and scientifically assessed, hopefully before an actual adoption by developers. At the realization level, many approaches are worth considering: projectional technologies (e.g., MPS: jetbrains.com/mps), meta-programming or parsing techniques (e.g., SugarJ: sugarj.org, Conerge: http://convergepl.org/) compilation strategies (e.g., LMS: http://julien.richard-foy.fr/blog/2015/01/06/lightweight-modular-staging-in-5-minutes/), model transformations, etc.  For instance, following a model-based approach, we can consider that there exists an abstract syntax common to all DSL shapes; the challenge is then to address the syntactic and semantic gaps that may exist.
As part of our research, we plan to perform usability studies to deeply understand the importance of syntax and tooling support for the different shapes of DSLs. We also want to explore concrete engineering scenarios in which different shapes of a DSL (being internals or externals) are used for developing an application. We hope we and others will contribute around the metamorphic vision of DSLs!
FNR Pearl Chair. Head of the Software Engineering RDI Unit at LIST. Affiliate Professor at University of Luxembourg. Â More about me.
The programming language Avail has first-class DSL support. I’ve recently ‘abandoned’ Scala as my DSL language of choice, in favour of Avail.
That said, Avail is still lacking development tools (syntax highlighting, IDE’s, etc). It will be a challenge to provide syntax highlighting for Avail (because of its completely free syntax), but surely it can be done.
And a syntax highlighting is just a tip of an iceberg.
There is a close relationship between a language and tools facilitating its use. The accessibility and quality of a language workbench often influence the choice of a language.
Scala is pragmatic and does well in many of the practical needs for a DSLs. One of the idea behind its scalability is in being able to create these dedicated abstractions and compose them together. This is by far not limited just for shallow embedding—i.e. having fragments of code with domain-specific syntax woven within Scala so that it appears different. While this is important, the power comes with deep DSL embedding using macros or lightweight modular staging.
On the other hand Avail looks promising and it might bring some very interesting concepts.
After about twenty years in the making, I’m days from releasing the macro subsystem for Avail. As you can imagine, I’m very excited about this. The macro facility has been present but latent for a couple of years, but what I’m almost done writing is the ability to replace the current compiler that knows the hard-coded parts of Avail’s syntax with one that… doesn’t. Instead, the built-in syntax for variable/constant declarations, assignments, variable uses, statements, functions, etc. will be accomplished through less than a dozen macros. At that point, a module you write in a DSL can choose not to import Avail’s core syntax, allowing Avail to “step out of the way” of your DSL. This new approach exposes the false dichotomy of this excellent article. It will revolutionize the construction of DSLs, since they’re just ordinary Avail programs that don’t happen to import the same library of operations and macros that a more “mainstream” Avail module might use.
We’ll be making a big announcement after everything is working, tidy, committed, and written up. You may want to sign up to our mailing list and/or RSS feed.
I am not sure what a metamorphic DSL might look like but I’ve always wanted a general-purpose programming language that facilitates the creation of DSLs. In other words, a language in which one can create internal DSLs that are every bit as powerful as an external DSL but a lot easier to create and less likely to become a monster.
So what would such a language system look like? It would have a powerful underlying type system and a complete implementation of all the basic data types. However, the most important feature for DSL creation would be the ability to define new syntactical forms and their translation to the base language or some previously defined DSL.
Such a scheme would probably not allow one to create truly oddball DSLs but most people don’t want to do that anyway. A DSL is still going to be used by programmers so it can’t be too alien.
Anyway, that’s what I want. 😉
IMO: You just described Avail!
– powerful underlying type system – check!
– complete implementation of the basic datatypes – check!
– oddball DSLs – hmmm 🙂
Some general-purpose languages like Scala or Avail provide mechanisms / facilities to define new syntactical forms (and therefore to implement some shapes of internal DSLs).
The effort might be more or less important for a developer to have a syntax close to the one encountered in the external shape of a DSL.
In any case syntax is not the only aspect of a DSL (being internal or external), and services should come with; Robbert mention syntax highlighting for instance, but overall many specialized services have to be developed. Again it is not an easy task and the services associated to a shape of a DSL may not be comprehensive or suboptimal at the end.
I have voluntary taken the point of view of a developer of a DSL until now. Tradeoffs also apply when considering the point of a view of a user of a DSL.
With metamorphic DSLs, we would like to transition from one shape to another.
Thanks, Robbert. I will definitely look into Avail.
Indeed, syntax highlighting is just a tip of the iceberg.
I believe that a sufficiently advanced DSL tool hinges on what can be discovered/proved at compile time.
One way of dealing with that complexity is to abstract over all compilers (via a DSL?).
Now here is the thing: Avail lets you strenghten types via semantic restrictions at compile time.
Avail enables the programmer to program ‘types’ (where appropriate) in a very disciplined way.
I haven’t figured out how tool support for Avail should look like because of this.
But it is an interesting challenge nonetheless!
You should give Neverlang a try. It enables changes to language definition (both syntax and semantics) without stopping a possible running application (of course only for semantic changes) .
Lately it also generates IDE support (syntax highlight, completion and debugging) and it is language product line compliant.
Hi Walter,
Thanks for your feedback. In the case of Metamorphic DSL, it’s rather a matter of changing the tool support (based on a given context to be defined) than the syntax and semantics. See for instance the example given in the paper where the same syntax and semantics are used, but with different tool support according to the stakeholder.
Hi Benoît,
I did notice now that this is a quite old blog post (2015) and I didn’t think you already wrote a paper about it. You should link it.
From the blog post (the only thing I read about it) it seems that a syntax change occurs, at least in the three versions of the SQL based example. Also the term “metamorphic DSL” is misleading from your replay should be the DSL IDE support to be misleading. My fault I didn’t dig much about that.
Anyway it still seems something you can solve with language product lines where the possible representations (tool support if you prefer) are all represented in the FM along side the language features. In the upcoming SPLC conference there are a couple of papers on this.