Recommender systems (RSs) are information filtering systems used in many online applications such as music streaming, video broadcasting and e-commerce platforms, with the aim of proactively suggesting items (e.g., songs, videos, and products) that may be of interest for a target user . In the context of software engineering, RSs are being developed to facilitate a variety of activities , like assisting in the selection of appropriate third-party program libraries  or recommending a particular refactoring  for source code.
Following the same trend as in software engineering, recommender systems have also been proposed to support modelling and other model-driven engineering (MDE) tasks [5, 6]. To get a picture of the current panorama of RSs in modelling (e.g., most typical recommendation tasks and recommendation methods), as well as to identify open research opportunities, we have analyzed the state of the art. The results have been reported on a systematic mapping review  on the topic. We selected 66 papers for our analysis, from an initial set of over 1,450 papers published from 2004 to 2020.
The results of this study have been published in the paper “Recommender systems in model-driven engineering” by L. Almonte, E. Guerra, I. Cantador and J. de Lara, published in 2022 in the volume 21(1) of the journal Software and Systems Modeling (pages 249–280). Read on for the summary and highlights of our work! You can also follow more updates on this line of work on the Droid website.
We collected the articles from the Scopus, ACM and Web of Science digital libraries using a formal query comprising 23 terms. The query retrieved papers whose title, abstract or keywords contain at least one term related to RSs and at least one term related to modelling or MDE. We executed the query in September 2020, and only considered peer-reviewed papers written in English and published in journals, conferences, workshops, and book chapters.
Our report characterizes and categorizes the existing research work on RSs for MDE under four dimensions (see Figure 1):
- recommendation, and
The domain dimension
In the context of MDE, this dimension includes features such as the purpose of the recommendation, and the type of artefact that is the subject of the recommendation (see Figure 2).
We observed that existing RSs for MDE target five main purposes: complete, create, find, repair and reuse (Figure 2). When the purpose is ‘complete’, the RS analyses an artefact that already exists and suggests how to extend it. In the case of ‘create’, the RS helps in the construction of the initial version of a new artefact from scratch. When the purpose is ‘find’, the RS facilitates the discovery of elements or artefacts within a repository. In ‘repair’ tasks, the RS suggests solutions to fix errors in an existing artefact, such as adding or removing elements. Finally, for ‘reuse’, the RS helps in reusing an existing artefact, going beyond finding as it assists in integrating the artefact as well. For all these purposes, the recommendation task can be performed over models, meta-models, transformations, or code generators (see Figure 2). Additionally, in our mapping review, we reported if recommendation approaches were independent of the modelling language or tied to a particular one.
In our analysis, we observed that most recommendation approaches target completion and repair, adding up to 73.4% of all the approaches (see Figure 3). The proposed model completions are commonly computed using search-based techniques, for instance using solvers based on Alloy , Prolog or via rules such as DIAGEN  and DIG MDE . The remaining approaches focus on reuse (10.9%), find (7.8%) and create (3.2%).
Moreover, most recommenders work over models (76.5%) (see Figure 4), and the rest target meta-models (15.6%), transformations (6.2%) and code generators (1.6%).
We also distinguished whether the RS is language independent, or on the contrary, it is tied to a particular language for modelling (e.g., UML), meta-modelling (e.g., Ecore), model transformation (e.g., ATL), or code generation (e.g., Acceleo). While most works were designed for a specific language, a substantial amount of approaches applicable to models (e.g., ) are language-independent.
The tooling dimension
This dimension comprises aspects related to the recommendation tool, namely, its maturity, its ability to be integrated with other MDE tools, the recommender trigger, and the enactment of the recommendations (see Figure 5).
The maturity feature determines if a described recommendation tool is a proposal with no implementation, a prototype, or a full implementation. Fully implemented tools can be frameworks, plug-ins, or standalone systems. We also distinguish whether recommendations can be triggered proactively, offering recommendations to the modeler automatically, or if they are triggered on demand. Finally, we consider if the recommendation enactment is ‘manual’, meaning that the RS provides a list of recommendations and the user decides how to use them; ‘interactive’ if the RS permits the user to select a recommendation, which becomes applied to the given context; ‘automated’ if the recommendation is automatically applied without user intervention; or ‘semiautomated’ if the recommendation enactment is automated, but the user may be prompted during the process, e.g., to provide some input value or decide between alternative options.
We noted that among 51 approaches, 4 (7.8%) were proposals with no implementation, 19 (37.2%) presented prototypes as a proof-of-concept, and the remaining 28 (55%) provided full implementations. Most of the full implementations were either plug-ins (13) or systems/extensions of systems (13), while frameworks (3) were less pervasive.
Some tools were implemented atop EMF to achieve generality. Nevertheless, we only considered a tool as independent if it also provided explicit means to facilitate its integration with other tools. Under this perspective, only 4 (7.84%) approaches were truly independent of any modelling tool. For instance, the framework developed by Batot et al.  recommends how to co-evolve OCL invariants upon Ecore meta-model changes. This framework is not specific to a particular editor and can be extended with new heuristics to guide the search of recommendations. On the other hand, 84.31% of the approaches were built as complete systems or system extensions, and were tool dependent.
The triggers of most recommender tools were on demand, with 41 out 51 (80.39%) of the cases. Just 12 out of 51 of the approaches provided recommendations proactively without user intervention (23.53%), and few tools (5.88%) could trigger recommendations both on-demand and proactively. Regarding the recommendation enactment, most recommendations could be applied either manually (31.37%) or interactively (49.02%). Automated enactments typically occurred in model completion and model repair approaches. Only 3 approaches (5.88%) provided semi-automated enactment of recommendations.
The recommendation dimension
This dimension comprises the features typically considered in the RSs area to characterize how recommendations are generated, such as the applied recommendation method, the exploited user preferences, the recommended items, and the degree of recommendation (see Figure 6).
Most of the surveyed approaches applied one of the four major recommendation techniques in the area: content-based, collaborative filtering, knowledge-based and hybrid. Additionally, some works used ad-hoc techniques that do not fall into the previous categories.
We noted that most RSs for MDE are knowledge-based (47%), subsequently content-based (19.6%), hybrid (11.8%), based on collaborative filtering (7.8%) and ad-hoc (11.8%) (see Figure 7).
Regarding knowledge-based recommenders, they can be classified as constraint-based, if the recommendations are determined by the items that fulfil established domain-dependent rules (e.g., the recommender integrated in Kermeta ); or case-based, if the recommendations are provided by examining past solutions for alike cases (e.g., the approach by Chowdhury and collaborators ).
We also categorized the surveyed works distinguishing how the user preferences are collected (acquisition type) and the temporality of those preferences. All approaches but one collected the data implicitly, being the user’s previous interaction with the system the most common type of user preference data. Twelve approaches also collected the data explicitly through questionnaires, parameters, tags or requirement definitions, and somehow combined them with implicit user preferences. When it comes to the temporality of the data, 90% of the approaches collected the preferences for a brief period; and 3 approaches only used long-term preferences by storing the experience gained from a previous interaction, exploiting persistent social tags and enabling developers to persist user preferences as required by the recommendation method.
The last feature of the recommendation dimension is the recommendation degree. We found that 53% of the approaches proposed to the user all the recommendations generated. Since this can be overwhelming, some approaches (4%) presented just one recommendation, and others (39%) presented the top N recommendations.
The evaluation dimension
This dimension comprises the methodologies and metrics used to evaluate recommenders (see Figure 8). Only 32 out of 51 approaches have been evaluated. We first classified the evaluation methods into three types: offline experiments, online experiments, and user studies. Offline experimentation, the most popular evaluation methodology (21 out of 32 cases), considers analytical studies on datasets. Approaches like AMOR , anATLyzer  and Extremo  performed this type of evaluation. Online experimentation considers users in a real setting, but only 2 approaches followed this methodology (ASIMOV  and DoMoRe ). Finally, user studies are conducted by small groups of participants in controlled scenarios. Twelve approaches performed user studies.
When it comes to the recommendation goals and quality measures, we distinguish application-independent and application-dependent metrics. Among the application-independent ones, we found traditional metrics used to evaluate RSs regardless of the application, such as recommendation accuracy metrics —i.e., rating prediction metrics (e.g., MAE and RMSE) and ranking quality metrics (e.g., precision, recall, nDCG and MRR)—, and other measures that capture non-accuracy characteristics, such as recommendation diversity, coverage and novelty. Metrics related to the system performance included the time consumed and the resources required to perform a task, while metrics related to the usage satisfaction were user engagement, perceived usefulness, trust on the system, usability, responsiveness, security, and privacy. The application-dependent metrics, by contrast, were conceived for particular MDE applications and tasks. In this case, we found metrics such as the total number of valid matches in the recommendation of model transformation mappings.
Main opportunities in Recommender Systems for Model-Driven Engineering solutions
Most of the surveyed approaches focused on recommending models, commonly with the purpose of completing and repairing them. However, RSs for transformations and code generators would be very useful for the community, as well as approaches for finding relevant artefacts, reusing them in a given context, and creating artefacts from scratch.
We envision RSs that propose initial artefact templates out of higher-level descriptions, such as natural language. We see plenty of opportunities on exploring more collaborative filtering techniques for RSs in MDE, and on the development of methods for automating the construction of RSs for modelling languages. Another line of research is the formal definition and generalization of metrics for MDE tasks. Finally, we suggest disclosing replication packages to be able to reproduce published experiments, since one of the main impediments for RSs in MDE is the lack of data.
 Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge Data Engineering, 17(6), 734–749 (2005)
 Robillard, M.P., Walker, R.J., Zimmermann, T.: Recommendation systems for software engineering. IEEE Software 27(4), 80–86 (2010)
 Nguyen, P.T., Rocco, J.D., Ruscio, D.D., Penta, M.D.: CrossRec: Supporting software developers by recommending third-party libraries. Journal of Systems and Software 161, 110460 (17 pages) (2020)
 de Oliveira, M.C., Freitas, D., Bonifácio, R., Pinto, G., Lo, D.: Finding needles in a haystack: Leveraging co-change dependencies to recommend refactorings. Journal of Systems and Software 158, 110420 (2019)
 Agt-Rickauer, H., Kutsche, R., Sack, H.: Automated recommendation of related model elements for domain models. In: 6th International Conference on Model-Driven Engineering and Software Development (MODELSWARD’18), Revised Selected Papers, pp. 134–158. Springer (2018)
 Almonte, L., Pérez-Soler, S., Guerra, E., Cantador, I., de Lara, J.: Automating the synthesis of recommender systems for modelling languages. In: 14th ACM SIGPLAN International Conference on Software Language Engineering (SLE 2021), pp. 22–35. ACM (20201)
 Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: 12th International Conference on Evaluation and Assessment in Software Engineering (EASE’08). BCS (2008)
 Jackson, D.: Software Abstractions — Logic, Language, and Analysis. MIT Press (2006). http://alloytools.org/
 Mazanek, S., Minas., M.: Business process models as a showcase for syntax-based assistance in diagram editors. In: 12th International Conference on Model Driven Engineering Languages and Systems (MoDELS’09), pp. 322–336. Springer (2009)
 Nechypurenko, A., Wuchner, E., White, J., Schmidt, D.C.: Applying model intelligence frameworks for deployment problem in real-time and embedded systems. In: Workshops and Symposia at MoDELS’06, Reports and Revised Selected Papers, pp. 143–151. Springer (2006)
 Barriga, A., Rutle, A., Heldal, R.: Improving model repair through experience sharing. The Journal of Object Technology 19(2): 13:1-21 (2020)
 Batot, E., Kessentini, W., Sahraoui, H.A., Famelis, M.: Heuristic-based recommendation for metamodel – OCL coevolution. In: 20th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MoDELS’17), pp. 210–220. IEEE Computer Society (2017)
 Moha, N., Sen, S., Faucher, C., Barais, O., Jézéquel, J.: Evaluation of Kermeta for solving graph-based problems. Int. J. Softw. Tools Technol. Transfer 12(3–4), 273–285 (2010)
 Chowdhury, S.R., Daniel, F., Casati, F.: Recommendation and weaving of reusable mashup model patterns for assisted development. ACM Trans. Internet. Technol. 14(2–3), 21:1–21:23 (2014)
 Brosch, P., Seidl, M., Kappel, G.: A recommender for conflict resolution support in optimistic model versioning. In: ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, SPLASH/OOPSLA Companion, pp. 43–50. ACM (2010)
 Sánchez Cuadrado, J., Guerra, E., de Lara, J.: AnATLyzer: an advanced IDE for ATL model transformations. In: 40th International Conference on Software Engineering (ICSE), Companion Proceedings, pp. 85–88. ACM (2018)
 Segura, Á.M., de Lara, J.: Extremo: an eclipse plugin for modelling and meta-modelling assistance. Sci. Comput. Program. 180, 71–80 (2019)
 Florez, H., Sánchez, M. E., Villalobos, J., Vega, G.: Coevolution assistance for enterprise architecture models. In: 6th International Workshop on Models and Evolution ([email protected]), pp. 27–32. ACM (2012)
 Agt-Rickauer, H., Kutsche, R., Sack, H.: DoMoRe—a recommender system for domain modeling. In: 6th International Conference on Model-Driven Engineering and Software Development (MODELSWARD), pp. 71–82. SciTePress (2018)