The productivity of software development has improved each time the level of abstraction has been raised. Recently, significant advances have occurred through no-code and low-code development, where developers build applications with specialized languages rather than traditional programming languages. These domain-specific modeling languages are either provided by tool vendors or created in-house.
Together with Juha-Pekka Tolvanen, we report a study analyzing the productivity impact of a domain-specific modeling solution, created internally by a company, for developing web-based enterprise applications. The comparison with traditional programming shows that domain-specific modeling was over 500% as productive. It also changed the way how development was carried out by integrating the development steps.
Raising the level of abstraction through the languages used by developers has improved the productivity and quality of the development work. Today this trend is perhaps most readily seen in interest in low-code, no-code and citizen developer environments focusing on developing enterprise applications in the cloud. At their core, these environments provide specialized languages that reduce, and sometimes totally remove, the use of traditional programming languages . The continuing raising of abstraction via languages has led to estimates that most new applications developed by organizations will use these technologies within a couple of years .
Today companies can start using existing environments (for a review of 30 no-code/low-code environments see ) or create dedicated languages for themselves. The latter approach is important when there is a need to have full control of the development approach, the code produced and the execution environment. These domain-specific languages (DSLs) move away from plain text to be visual modeling languages and narrow down the application domains into just the type of products the companies develop. Thanks to advances in tooling, in recent years the effort to define specialized languages in-house has decreased drastically : a review of industry cases shows that with good tooling it can be less than 2 person-weeks .
The motivation for this study was the need to evaluate if the internally created DSM solution improved developer productivity.
Comparing development approaches in companies
Companies conduct comprehensive empirical comparison or evaluation rather rarely, because of the high costs in terms of time or resources: Building the same system twice with different development approaches, using several teams , evaluating dozens of developers , analyzing large numbers of development tasks , inspecting user effort on the tooling  or focusing on development activities in detail with video recording, speaking while working, or observing individual developers’ actions . Some of the characteristics of good empirical research, like a large number of participants, are not always even possible since there may only be a handful of developers using the particular language within the company. Many good scientific research methods are simply too expensive and time-consuming for practical use in a commercial setting. Companies instead perform limited studies  or base their evaluation on collected experiences and opinions .
In our case, the company interested in productivity decided to focus on the development time. Time works over different languages and is a variable that acts as a proxy for the cost too . Development time is also a measure used as a basis for budgeting and project planning, and is related to salaries. The precise way to measure time was set to be the same as the basis for the company’s project business: How much effort was needed to implement an application that met customer requirements . In other words, the differences in development time were measured by implementing the same application in two different ways: with traditional programming and with a DSL. Data was collected by recording the time used for application development, categorizing it according to the three main technical parts of enterprise applications: database, user interface and application logic.
The start and end points with both development approaches were the same. The start point was the requirements created together with the company personnel and the end point was reviewing the developed application to ensure that it met all requirements. Two persons working in the company were involved at the start: they selected the case application and its requirements so that it matched typical situations. The company also provided material to learn the technologies applied along with the tools and languages needed.
The programming approach was chosen to be tested before the modeling approach, to avoid the risk that the generated implementation would guide manual implementation. Similarly, this lower level of abstraction would not be visible at all in the following modeling approach, whereas the models would already assist the conceptual design for the programming approach. There were no specific requirements for the user interface, which was developed from scratch in both cases. The appearance of the generated application was based on the user interface framework built by the company, but the manually programmed application was not expected to look the same because no requirements were set for the appearance of the application. This was also a reason to implement the programmed application first.
The case application to be built focused on managing customer projects. Its basic functionality is like other applications developed by the company — albeit somewhat smaller, to reduce the development resources needed for making the comparison. The selection of the case application was confirmed by the company and requirements for the application were created together with two persons from the company. The requirements for the customer project management application included:
- Manage customer projects, their applications and usage environments.
- Manage test users for each usage environment along with related resources and their descriptions.
- Manage each user’s details such as accounts, passwords, AD groups, roles and other details.
- All the items mentioned above can be added, modified and deleted.
- The user interface provides two different views: one focusing on an overview and another on a selected usage environment and its details.
Development with two approaches
With the same requirements and same review phase for acceptance, the comparison focused on the differences between the approaches. Programming contained the following steps:
- Schema definition with SQL Server Management Studio using SQL.
- Business logic implementation in C# with Visual Studio.
The use of the DSL contained the following steps:
- Modeling the application (data, user data, user rights, roles, alerts etc.) with MetaEdit+ and generating the code.
- Adjusting the user interface with SQL.
Both approaches also included some additional steps related to creating projects and other tooling settings, as this was the first project in both tools. During development the time used at each step was recorded. The times for defining the requirements, checking that the final application fulfilled the requirements, studying the technologies or their installations were excluded. The focus was on development tasks alone. The development was done for both ways in one cycle each — without partial deliveries or milestones.
This model was created for developing the customer project management application. Each model element and their connections included further details relevant for data items, rights, roles, alerts etc. as defined in the DSL. By privacy request of the company, we do not show that extra information and its editing UI here. In terms of the level of abstraction, the language hides the details of implementing the database schema and related operations (CRUD), optimizations, user rights and roles, security, and load balancing. The user interface is also produced automatically based on the models and framework — all applications have similar widgets (lists, buttons etc.), which can be modified later. In other words, the whole application was generated from the model.
The resulting two applications, programmed and modeled, were not completely identical, but they covered the same functionality. The main differences were in the user interface, as the framework applied when generating the code from the models provided a user interface that is common for all applications the company makes. Figures 3 and 4 show the running applications: Fig. 3 shows the application made with manual programming and Fig. 4 the application made with modeling. The latter looks more extensive, as the generated code automatically makes fuller use of the user interface framework that the company had built as a part of their development solution.
Table 1 summarize the effort in hours of the various steps. Creating the application with the company’s own DSL was significantly faster, taking 6.5 hours compared to manual programming taking 36 hours. The productivity difference is over 500% and is thus in the familiar range of 300–1000% that many other companies have reported across different domains and languages .
|Modeling (and generation)||1.25|
|Other (project setup etc.)||1.50||3.00|
The main effort in programming went on implementing the user interface (16.5 h) and business logic (12.5 h) rather than defining the schema. This division of effort between the different steps was similar to other development projects within the company, although detailed data was not measured for them. When using the company’s own DSL the situation was very different: modeling took 1.25 hours and the code generation from the models took just some seconds. After running the initial application, 2.25 hours was used to adjust the generated user interface. The views of the user interface (see Fig. 4) were modified with SQL. This low-code approach meant that less than 1% of the application code was written traditionally and not produced from the model.
Interestingly, installation and project settings took less time when using programming tools than the company’s own modeling solution. One reason could be that programming tools were ready products with installations and wizards whereas of the tools developed in-house were not at the same level.
Data collection on the time used also reveals a change in the development work: rather than dividing the work into different steps with different languages, using a DSL integrated those steps into building a single model. Changes were made to this model and all checks were performed on this same model too — the only exception was adjustments made to the user interface with SQL.
In addition to the productivity difference, the discussions with company personnel identified other benefits derived from the modeling approach that were not addressed by data collection in this study:
- Faster delivery throughput times
- Uniform user interface functionality
- Participatory and motivating development process
- Labor-intensive tasks are removed by automation
- Application enhancements and maintenance are done in the same way
Limitations of the study and future research topics
Our study has limitations that are related to size: one developer and one small application. A better research setup would be to have a larger number of people with two groups (as in ) as well as then varying order among development approaches (as in ). As this increases the time and resources needed from the company, one solution could be to extend the number of participants modestly, asking another developer to conduct the same study (as in ) or conducting a small laboratory experiment with a group of several developers (as in ). Developing more applications of different kinds could refine the findings more but would again require greater investment from the company. Often a clearly visible difference, like in our study, is enough for decision making: companies rarely have the luxury of demanding a statistically significant result.
The application developed was also quite small. On the one hand It was purposefully chosen to be small, to keep the implementation effort down, but still cover the familiar functionality (database, user interface and business logic). One direction for extending the study would be building larger applications (more tables, logic and views) and then measure how the approaches scale. We estimate that the differences would then be larger as the higher abstraction achieved via domain-specific model scales better to much larger applications, causing less cognitive load than a manual programming approach. This could be a subject for future studies.
Another interesting research direction would be to increase the size of the development team per application. This would again call for more resources but would bring into play important factors of collaboration and teamwork that could not be covered here with a single developer. Tools would start to have a larger influence, as traditional IDE tools and modeling editors built on top of them (like DSL tools for Visual Studio and various EMF and GMF-based editors for Eclipse) expect developers working in parallel to clone the code and merge it with others’ changes afterwards. This is time-consuming and error prone compared to the approach of multi-user tools such as MetaEdit+, which enable multiple developers to work with the same models simultaneously — merging being continuous and fully automated .
Finally, our study focused on a greenfield project, developing an application from scratch, while often a much larger portion of work goes on maintenance. There are not many studies comparing maintenance steps, but studies like  indicates that DSLs provide better productivity for maintenance than general purpose languages. This is also a good topic for future research.
We presented a case study comparing development productivity between traditional programming languages and a domain-specific modeling language. The domain of the case study was web-based enterprise applications, an area targeted by many no-code and low-code environments. Rather than using an existing low-code development environment, the company had developed their own environment. This gives the company full control over its development tooling, allowing to change it as needed. For example, to update the generator to produce Azure functions when so desired.
The result of the comparison shows over 500% productivity for domain-specific modeling compared to manual programming: A week’s workload could thus be reduced to less than one working day. This result is similar to what other companies have reported across a variety of domains, showing that the same results apply to the domain of web-based enterprise applications.
- Britton, C., Jones, S., The untrained eye: how languages for software specification support understanding in untrained users. Hum.-Comput. Interact. 14, 1999
- Cao, L., Ramesh, B., Rossi, M., Are Domain-Specific Models Easier to Maintain Than UML Models?, IEEE Software, vol. 26, no. 4, pp. 19-21, July-Aug. 2009
- MR, Low Code Development Market Outlook (2022-2032), 2022, https://www.factmr.com/report/low-code-development-market
- Fadjukoff, L., Domain-Specific Modeling in application development (In Finnish: Toimialakohtainen mallinnus sovelluskehityksessä), Master’s thesis, University of Tampere, 2021, https://urn.fi/URN:NBN:fi:tuni-202111018071
- Farshidi, S., Jansen, S., Fortuin, S. Model-driven development platform selection: four industry case studies. Softw Syst Model 20, 2021
- Jaksic, A., France, R., Collet, P., Ghosh, S., Evaluating the Usability of a Visual Feature Modeling Notation, Proceedings of Software Language Engineering, LNCS 8706, Springer, 2014
- Gartner, https://www.gartner.com/en/newsroom/press-releases/2021-11-10gartner-says-cloud-will-be-the-centerpiece-of-new-digital-experiences, 2021
- Kieburtz, R. B., McKinney, L., Bell, J. M., Hook, J., Kotov, A., Lewis, J., Oliva, D. P., Sheard, T., Smith, I., Walton, L., A software engineering experiment in software component generation. In ICSE ’96: Proceedings of the 18th international conference on Software engineering, IEEE Computer Society, 1996
- Kelly, S., 2013. Empirical Comparison of Language Workbenches. In Proceedings of the 2013 ACM Workshop on Domain-Specific Modeling (pp. 33–38), 2013
- Kelly, S., Collaborative Modeling with Version Control. In Proceedings of BigMDE 2017, 5th BigMDE Workshop, STAF 2017 Software Technologies: Applications and Foundations, Marburg, Germany, 21 July, 2017
- Kärnä, J., Tolvanen, J.-P., Kelly, S. Evaluating the use of domain-specific modeling in practice. In Proceedings of the 9th OOPSLA workshop on Domain-Specific Modeling, http://www.dsmforum.org/events/DSM09/Papers/ Karna.pdf, 2009
- Leitner, A., Preschern, C., Kreiner, C., Effective development of automation systems through domain-specific modeling in a small enterprise context, Journal of Software and Systems Modeling, Springer, 2012
- Puolitaival, O.-P., Home automation DSL case, Presentation at Code Generation Conference (http://codegeneration.net/cg2011/), 2011
- Puolitaival, O.-P., Kanstrén, T., Rytky, V.-M, Saarela, A., Utilizing Domain-Specific Modelling for Software Testing, The 3rd International Conference on Advances in System Testing and Validation Lifecycle, October 23-29, Barcelona, Spain, 2011
- Di Ruscio, D., Kolovos, D., de Lara, J. et al. Low-code development and model-driven engineering: Two sides of the same coin?. Softw Syst Model, 2022
- Safa, L., The making of user-interface designer a proprietary DSM tool. In 7th OOPSLA workshop on domain-specific modelling (DSM), 2007
- Tolvanen, J-P. and Kelly, S. Model-Driven Development Challenges and Solutions — Experiences with Domain-Specific Modelling in Industry. In Proceedings of the 4th International Conference on Model-Driven Engineering and Software Development (MODELSWARD 2016), pages 711-71, 2016
- Tolvanen, J., Kelly, S., Effort Used to Create Domain-Specific Modeling Languages. In Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems. Association for Computing Machinery, New York, NY, USA, 235–244, 2018
- Wijers, G., Modeling Support in Information Systems Development, Thesis Publishers Amsterdam, 1991
- Wu, Y., Hernandez, F., Ortega, F., Clarke, P. France, R., Measuring the Effort for Creating and Using Domain-Specific Models. Proceedings of the 10th Workshop on Domain-Specific Modeling, DSM’10. 10.1145/2060329.2060360, 2010
Thanks for sharing this. It’s great to see this kind of comparisons coming along. Some thoughts:
For an adoption business case, a key part would be the initial investment. In this case, it’d be how much effort has taken to build their internal modelling tool. You’d also have to look at the maintenance cost for the internal modelling tool. Having that information, you’d then look at the point where you’d break even, meaning that the savings of using DSMs have been covered by the one-off effort required to set up/ mantain the DSMs. I didn’t find that info on the paper.
In terms of KPIs, Development Time is probably a good starting choice since, as it says in the paper, it’s directly related to money. Theory will tell you should rather look at delivered business value, but in my experience I’ve never seen a company with a well developed definition of value, beyond pure euros.
On the Limitations of the study and future research topics, I’d look also at increased benefits on modifiability and maintainability, not on the models (as in ) but on the application. I’d expect a more automated source code generation to decrease the number of defects. Fixing bugs it’s expensive and may result in reputational damage/ business impact.
For a nice complement to this discussion, see also this post on “MDD pays off in the mid-term” reporting another experiment: https://modeling-languages.com/mdd-pays-mid-term-industrial-experiment/
Thanks Abraham for the comments. Unfortunately, the company did not want to share details on investment side, but we analyzed 10 cases on the initial investment side with 2 detailed case studies and 8 public company cases from Panasonic, Polar, Elektrobit, etc. You may find it published at ACM Models paper at: https://metacase.com/papers/effort-create-domain-cameraReady.pdf.
I know that also others have addressed the same topic, like https://dl.acm.org/doi/abs/10.1007/s10270-012-0289-1 but I’m not aware of any public work on the maintenance part. If you know please let me know as my experiences on maintenance from our company contacts with customers having applied their own domain-specific languages for years, or decades as in case of Helvetia, Airbus (https://metacase.com/cases/).