The productivity of software development has improved each time the level of abstraction has been raised. Recently, significant advances have occurred through no-code and low-code development, where developers build applications with specialized languages rather than traditional programming languages. These domain-specific modeling languages are either provided by tool vendors or created in-house.

Together with Juha-Pekka Tolvanen, we report a study analyzing the productivity impact of a domain-specific modeling solution, created internally by a company, for developing web-based enterprise applications. The comparison with traditional programming shows that domain-specific modeling was over 500% as productive. It also changed the way how development was carried out by integrating the development steps.

Introduction

Raising the level of abstraction through the languages used by developers has improved the productivity and quality of the development work. Today this trend is perhaps most readily seen in interest in low-code, no-code and citizen developer environments focusing on developing enterprise applications in the cloud. At their core, these environments provide specialized languages that reduce, and sometimes totally remove, the use of traditional programming languages [5][15]. The continuing raising of abstraction via languages has led to estimates that most new applications developed by organizations will use these technologies within a couple of years [3][7].

Today companies can start using existing environments (for a review of 30 no-code/low-code environments see [5]) or create dedicated languages for themselves. The latter approach is important when there is a need to have full control of the development approach, the code produced and the execution environment. These domain-specific languages (DSLs) move away from plain text to be visual modeling languages and narrow down the application domains into just the type of products the companies develop. Thanks to advances in tooling, in recent years the effort to define specialized languages in-house has decreased drastically [12][18]: a review of industry cases shows that with good tooling it can be less than 2 person-weeks [18].

We report a study comparing developer productivity across two different ways to build web-based enterprise applications: traditional programming languages versus a domain-specific modeling (DSM) language. The first way is the familiar approach of defining a database schema with SQL, and logic and user interface with programming languages like C# and JavaScript. The second way is a low-code approach in which the specification is created using a DSM language that enables full code generation from the models.

The motivation for this study was the need to evaluate if the internally created DSM solution improved developer productivity.

Comparing development approaches in companies

Companies conduct comprehensive empirical comparison or evaluation rather rarely, because of the high costs in terms of time or resources: Building the same system twice with different development approaches, using several teams [1][8], evaluating dozens of developers [2], analyzing large numbers of development tasks [8], inspecting user effort on the tooling [20] or focusing on development activities in detail with video recording, speaking while working, or observing individual developers’ actions [19]. Some of the characteristics of good empirical research, like a large number of participants, are not always even possible since there may only be a handful of developers using the particular language within the company. Many good scientific research methods are simply too expensive and time-consuming for practical use in a commercial setting. Companies instead perform limited studies [11][16] or base their evaluation on collected experiences and opinions [13][14].

In our case, the company interested in productivity decided to focus on the development time. Time works over different languages and is a variable that acts as a proxy for the cost too [9]. Development time is also a measure used as a basis for budgeting and project planning, and is related to salaries. The precise way to measure time was set to be the same as the basis for the company’s project business: How much effort was needed to implement an application that met customer requirements [4]. In other words, the differences in development time were measured by implementing the same application in two different ways: with traditional programming and with a DSL. Data was collected by recording the time used for application development, categorizing it according to the three main technical parts of enterprise applications: database, user interface and application logic.

The start and end points with both development approaches were the same. The start point was the requirements created together with the company personnel and the end point was reviewing the developed application to ensure that it met all requirements. Two persons working in the company were involved at the start: they selected the case application and its requirements so that it matched typical situations. The company also provided material to learn the technologies applied along with the tools and languages needed.

The first author of this report (Laura Fadjukoff) conducted both development cases and had less experience on modeling than on programming and related languages. While this situation did not treat the approaches equally, it resembles the situation today in most companies. For the programming approach, the selected implementation technology was C#, JavaScript and SQL. The author conducting the cases had some experience with JavaScript and SQL but not with C#. For the modeling approach the work was done with a DSL, new to the authors. In case there were questions, an experienced developer from the company was available. Both modeling and programming related tools were installed locally on the same Windows laptop. The modeling part, and the MetaEdit+ modeling tool, could also have been run in a remote cloud server, but it was decided to use a single environment for the developer in this case.

The programming approach was chosen to be tested before the modeling approach, to avoid the risk that the generated implementation would guide manual implementation. Similarly, this lower level of abstraction would not be visible at all in the following modeling approach, whereas the models would already assist the conceptual design for the programming approach. There were no specific requirements for the user interface, which was developed from scratch in both cases. The appearance of the generated application was based on the user interface framework built by the company, but the manually programmed application was not expected to look the same because no requirements were set for the appearance of the application. This was also a reason to implement the programmed application first.

Case application

The case application to be built focused on managing customer projects. Its basic functionality is like other applications developed by the company — albeit somewhat smaller, to reduce the development resources needed for making the comparison. The selection of the case application was confirmed by the company and requirements for the application were created together with two persons from the company. The requirements for the customer project management application included:

• Manage customer projects, their applications and usage environments.
• Manage test users for each usage environment along with related resources and their descriptions.
• Manage each user’s details such as accounts, passwords, AD groups, roles and other details.
• All the items mentioned above can be added, modified and deleted.
• The user interface provides two different views: one focusing on an overview and another on a selected usage environment and its details.

The conceptual model of the application’s data structure is detailed in Fig. 1. Technically, the application has the three parts typical to traditional enterprise applications: Database, user interface and business logic. The database was implemented in SQL, the user interface in HTML and JavaScript, and the business logic and operations in C#. The application runs in Azure.

Development with two approaches

With the same requirements and same review phase for acceptance, the comparison focused on the differences between the approaches. Programming contained the following steps:

1. Schema definition with SQL Server Management Studio using SQL.
2. Business logic implementation in C# with Visual Studio.
3. User interface implementation in HTML and JavaScript.

The use of the DSL contained the following steps:

1. Modeling the application (data, user data, user rights, roles, alerts etc.) with MetaEdit+ and generating the code.
2. Adjusting the user interface with SQL.

Both approaches also included some additional steps related to creating projects and other tooling settings, as this was the first project in both tools. During development the time used at each step was recorded. The times for defining the requirements, checking that the final application fulfilled the requirements, studying the technologies or their installations were excluded. The focus was on development tasks alone. The development was done for both ways in one cycle each — without partial deliveries or milestones.

Since SQL, C#, HTML and JavaScript are widely documented languages they are not detailed here. For the modeling approach the company used its own DSL. Details of this language are not public, but Fig. 2 shows an example from using the language.

 

This model was created for developing the customer project management application. Each model element and their connections included further details relevant for data items, rights, roles, alerts etc. as defined in the DSL. By privacy request of the company, we do not show that extra information and its editing UI here. In terms of the level of abstraction, the language hides the details of implementing the database schema and related operations (CRUD), optimizations, user rights and roles, security, and load balancing. The user interface is also produced automatically based on the models and framework — all applications have similar widgets (lists, buttons etc.), which can be modified later. In other words, the whole application was generated from the model.

The resulting two applications, programmed and modeled, were not completely identical, but they covered the same functionality. The main differences were in the user interface, as the framework applied when generating the code from the models provided a user interface that is common for all applications the company makes. Figures 3 and 4 show the running applications: Fig. 3 shows the application made with manual programming and Fig. 4 the application made with modeling. The latter looks more extensive, as the generated code automatically makes fuller use of the user interface framework that the company had built as a part of their development solution.

Results

Table 1 summarize the effort in hours of the various steps. Creating the application with the company’s own DSL was significantly faster, taking 6.5 hours compared to manual programming taking 36 hours. The productivity difference is over 500% and is thus in the familiar range of 300–1000% that many other companies have reported across different domains and languages [11][13][14][16][17].

Defined parts	Programming (hours)	Modeling (hours)
Modeling (and generation)		1.25
Database schema	5.50
Business logic	12.50
User interface	16.50	2.25
Other (project setup etc.)	1.50	3.00
Total	36.00	6.50
	Productivity ratio:	554%

 

The main effort in programming went on implementing the user interface (16.5 h) and business logic (12.5 h) rather than defining the schema. This division of effort between the different steps was similar to other development projects within the company, although detailed data was not measured for them. When using the company’s own DSL the situation was very different: modeling took 1.25 hours and the code generation from the models took just some seconds. After running the initial application, 2.25 hours was used to adjust the generated user interface. The views of the user interface (see Fig. 4) were modified with SQL. This low-code approach meant that less than 1% of the application code was written traditionally and not produced from the model.

Interestingly, installation and project settings took less time when using programming tools than the company’s own modeling solution. One reason could be that programming tools were ready products with installations and wizards whereas of the tools developed in-house were not at the same level.

Data collection on the time used also reveals a change in the development work: rather than dividing the work into different steps with different languages, using a DSL integrated those steps into building a single model. Changes were made to this model and all checks were performed on this same model too — the only exception was adjustments made to the user interface with SQL.

In addition to the productivity difference, the discussions with company personnel identified other benefits derived from the modeling approach that were not addressed by data collection in this study:

• Faster delivery throughput times
• Uniform user interface functionality
• Participatory and motivating development process
• Labor-intensive tasks are removed by automation
• Application enhancements and maintenance are done in the same way

Limitations of the study and future research topics

Our study has limitations that are related to size: one developer and one small application. A better research setup would be to have a larger number of people with two groups (as in [6]) as well as then varying order among development approaches (as in [8]). As this increases the time and resources needed from the company, one solution could be to extend the number of participants modestly, asking another developer to conduct the same study (as in [16]) or conducting a small laboratory experiment with a group of several developers (as in [11]). Developing more applications of different kinds could refine the findings more but would again require greater investment from the company. Often a clearly visible difference, like in our study, is enough for decision making: companies rarely have the luxury of demanding a statistically significant result.

The application developed was also quite small. On the one hand It was purposefully chosen to be small, to keep the implementation effort down, but still cover the familiar functionality (database, user interface and business logic). One direction for extending the study would be building larger applications (more tables, logic and views) and then measure how the approaches scale. We estimate that the differences would then be larger as the higher abstraction achieved via domain-specific model scales better to much larger applications, causing less cognitive load than a manual programming approach. This could be a subject for future studies.

Another interesting research direction would be to increase the size of the development team per application. This would again call for more resources but would bring into play important factors of collaboration and teamwork that could not be covered here with a single developer. Tools would start to have a larger influence, as traditional IDE tools and modeling editors built on top of them (like DSL tools for Visual Studio and various EMF and GMF-based editors for Eclipse) expect developers working in parallel to clone the code and merge it with others’ changes afterwards. This is time-consuming and error prone compared to the approach of multi-user tools such as MetaEdit+, which enable multiple developers to work with the same models simultaneously — merging being continuous and fully automated [10].

Finally, our study focused on a greenfield project, developing an application from scratch, while often a much larger portion of work goes on maintenance. There are not many studies comparing maintenance steps, but studies like [2] indicates that DSLs provide better productivity for maintenance than general purpose languages. This is also a good topic for future research.

Conclusions

We presented a case study comparing development productivity between traditional programming languages and a domain-specific modeling language. The domain of the case study was web-based enterprise applications, an area targeted by many no-code and low-code environments. Rather than using an existing low-code development environment, the company had developed their own environment. This gives the company full control over its development tooling, allowing to change it as needed. For example, to update the generator to produce Azure functions when so desired.

The result of the comparison shows over 500% productivity for domain-specific modeling compared to manual programming: A week’s workload could thus be reduced to less than one working day. This result is similar to what other companies have reported across a variety of domains, showing that the same results apply to the domain of web-based enterprise applications.

Another major change was how development work was divided into different steps and different languages. When using domain-specific modeling the development work focused on creating a single model. Changes were made to this model and all checks were performed on this same model too. This is in contrast to traditional programming with different languages, such as in this case study where the developer had to integrate SQL, JavaScript and C#.

References

1. Britton, C., Jones, S., The untrained eye: how languages for software specification support understanding in untrained users. Hum.-Comput. Interact. 14, 1999
2. Cao, L., Ramesh, B., Rossi, M., Are Domain-Specific Models Easier to Maintain Than UML Models?, IEEE Software, vol. 26, no. 4, pp. 19-21, July-Aug. 2009
3. MR, Low Code Development Market Outlook (2022-2032), 2022, https://www.factmr.com/report/low-code-development-market
4. Fadjukoff, L., Domain-Specific Modeling in application development (In Finnish: Toimialakohtainen mallinnus sovelluskehityksessä), Master’s thesis, University of Tampere, 2021, https://urn.fi/URN:NBN:fi:tuni-202111018071
5. Farshidi, S., Jansen, S., Fortuin, S. Model-driven development platform selection: four industry case studies. Softw Syst Model 20, 2021
6. Jaksic, A., France, R., Collet, P., Ghosh, S., Evaluating the Usability of a Visual Feature Modeling Notation, Proceedings of Software Language Engineering, LNCS 8706, Springer, 2014
7. Gartner, https://www.gartner.com/en/newsroom/press-releases/2021-11-10gartner-says-cloud-will-be-the-centerpiece-of-new-digital-experiences, 2021
8. Kieburtz, R. B., McKinney, L., Bell, J. M., Hook, J., Kotov, A., Lewis, J., Oliva, D. P., Sheard, T., Smith, I., Walton, L., A software engineering experiment in software component generation. In ICSE ’96: Proceedings of the 18th international conference on Software engineering, IEEE Computer Society, 1996
9. Kelly, S., 2013. Empirical Comparison of Language Workbenches. In Proceedings of the 2013 ACM Workshop on Domain-Specific Modeling (pp. 33–38), 2013
10. Kelly, S., Collaborative Modeling with Version Control. In Proceedings of BigMDE 2017, 5th BigMDE Workshop, STAF 2017 Software Technologies: Applications and Foundations, Marburg, Germany, 21 July, 2017
11. Kärnä, J., Tolvanen, J.-P., Kelly, S. Evaluating the use of domain-specific modeling in practice. In Proceedings of the 9th OOPSLA workshop on Domain-Specific Modeling, http://www.dsmforum.org/events/DSM09/Papers/ Karna.pdf, 2009
12. Leitner, A., Preschern, C., Kreiner, C., Effective development of automation systems through domain-specific modeling in a small enterprise context, Journal of Software and Systems Modeling, Springer, 2012
13. Puolitaival, O.-P., Home automation DSL case, Presentation at Code Generation Conference (http://codegeneration.net/cg2011/), 2011
14. Puolitaival, O.-P., Kanstrén, T., Rytky, V.-M, Saarela, A., Utilizing Domain-Specific Modelling for Software Testing, The 3rd International Conference on Advances in System Testing and Validation Lifecycle, October 23-29, Barcelona, Spain, 2011
15. Di Ruscio, D., Kolovos, D., de Lara, J. et al. Low-code development and model-driven engineering: Two sides of the same coin?. Softw Syst Model, 2022
16. Safa, L., The making of user-interface designer a proprietary DSM tool. In 7th OOPSLA workshop on domain-specific modelling (DSM), 2007
17. Tolvanen, J-P. and Kelly, S. Model-Driven Development Challenges and Solutions — Experiences with Domain-Specific Modelling in Industry. In Proceedings of the 4th International Conference on Model-Driven Engineering and Software Development (MODELSWARD 2016), pages 711-71, 2016
18. Tolvanen, J., Kelly, S., Effort Used to Create Domain-Specific Modeling Languages. In Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems. Association for Computing Machinery, New York, NY, USA, 235–244, 2018
19. Wijers, G., Modeling Support in Information Systems Development, Thesis Publishers Amsterdam, 1991
20. Wu, Y., Hernandez, F., Ortega, F., Clarke, P. France, R., Measuring the Effort for Creating and Using Domain-Specific Models. Proceedings of the 10th Workshop on Domain-Specific Modeling, DSM’10. 10.1145/2060329.2060360, 2010