I was wondering if any of you knows/uses any tool to automatically grade (UML) models created by students. I’m fully aware this far from easy (which probably explains why I don’t know of any such tool that is widely used, at least not among the people I know that teaches MDE). Clearly, a direct approach aiming to compare and match the model submitted by the student with the default solution model created by the teacher wouldn´t work. This would imply solving at least two big problems:
- differences in the names of the model elements (e.g. the concept “car” in the solution model could be called “vehicle” in the student’s one) and
- detecting semantic equivalence of models with a different structure (e.g. alternative ways to define a taxonomy or the use of association classes vs the equivalent model without them). We could automatically apply some predefined refactorings to both the instructor´s solution and the students’ models to try to get a normalized model easier to compare but still this won´t solve the problem in the general case.
If the models were executable models then one option would be to mark them same way we do with programs (and for that, many tools do exist): using test cases as input and checking the program output is the expected one. Nevertheless, this is not going to be typically the case.
Another option (a more feasible one) would be to build a tool that instead of trying to find a perfect match between the models just aims at checking (structurally) if the student model includes some key elements that the teacher believes are key to have a correct model. We would still have the same problems mentioned before but at least simplified.
Anyway, as I said above, I´d be interested in knowing more about you mark the modeling assignments you give to the students (feel free to also share how you do it even if you’re not using a tool to assist you with that).
FNR Pearl Chair. Head of the Software Engineering RDI Unit at LIST. Affiliate Professor at University of Luxembourg. More about me.
An interesting tool developed by the Universitat de Girona:
Soler, J.; Boada, I.; Prados, F.; Poch, J.; Fabregat, R., “A web-based e-learning tool for UML class diagrams,” Education Engineering (EDUCON), 2010 IEEE , vol., no., pp.973,979, 14-16 April 2010
doi: 10.1109/EDUCON.2010.5492473
URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5492473&isnumber=5492336
Yes, a great suggestion Ruth.
The work you comment is part of a bigger Project in which some researchers of the UdG (Universitat de Girona) created a website for automatically assessing exercises of any topic. In particular, they have automatic grading for databases, relational algebra, SQL, UML diagrams, calculus, logics, and etcetera. The project is called ACME, its website is http://acme.udg.edu/es/equip.php . The tool has been used by this university for many years and has been integrated in the teaching methodology of the UdG teaching staff.
I was present in the thesis defense of Josep Soler (http://ima.udg.edu/~soler/ ), who is member of the ACME group and did his PhD thesis about the application of the ACME tool to Databases. At that time the UML assessment part was very practical and tried to find out whether the main elements of the student diagram where modeled and the quality of the diagram. To do so, if I do not remember wrongly, the models of students where compared with a set of proposed solutions facilitated by the authors of the exercises. Also the names of the key elements to represent where fixed in the exercises in order to facilitate the detection of relevant elements in the schema. However, I read about the project three years ago and it is a tool that has been used since then. Then, it is possible that the system has evolved and more functionality are added. I recommend checking it if anyone is interested.
One of the interesting points of the ACME tool was also the environment created to analyse the information generated from students response. Such environment allow the teacher to analyse how the student evolves and identify easily the problems a given student may have with any kind of problem. Allowing the teacher to react quickly and therefore to adapt teaching activities to each student.
If anyone is interested more information about the project and publications are in the project website.
Hi Ruth,
You better cite your paper in this special issue 😉
http://www.ijee.ie/contents/c270111.html
It’s a really Interesting work that I was lucky to co-edit.
I’m also interested in that. That’s why I leave this comment to get notified 😉
A possible approach to detect semantic equivalence of model (to tackle the 2nd issue) is creating a backtracking strategy that attempts with trial pivoting elements (either classes and/or links), extends the trial model with selected elements, picked up from the proposed solution, compares it with the ‘official’ model, and extends the trial forward as long as it keeps equivalent, or backtrack if it does not. If eventually it becomes to an structurally equivalent solution (independently of the names of the selected elements), pose it as a potentially semantic equivalent. The bigger the model, the bigger the chance that they are actually equivalent, without having to know a bit of their ‘semantics’.
I think the problem is bigger than just checking semantic equivalence. The real problem is that there is more than one way to model the same thing. Modeling is a creative activity and there could be more than one good solution.
A good approach I have used for in-class understanding is this:
(i) Get students to create a model individually
(ii) Then team the students into pairs; they compare each other’s models and come up with an integrated version that resolves inconsistencies
(iii) Do the same again but now groups of 4
This is a really good way to demonstrate to students ‘what makes a good model’.
It’s harder to use this approach for assessment, of course (unless you believe in peer assessment). For assessment, I use 2 rules of thumb:
– Is it understandable? Can I quickly and easily read it and know what it means?
– Is it complete? Are there obvious issues that haven’t been covered? (Don’t worry too much about the non-obvious things).
This is a qualitative rather than quantitative approach to marking, but I don’t believe in an ‘official’ solution against which to compare (except perhaps as a checklist for the 2nd criterion; and to be used with caution).
Although automatic tools might be able to support some of this, I’m afraid AI is a long way off being able to assess models with any kind of reliability.
We have created a metric-based tool for assessing the quality of UML models:
http://www.win.tue.nl/empanada/metricview/
We have used it both for classroom projects and industrial cases.
It works if your models are not too small.
And: you have to make an interpretation of the metrics youself – the tool can not assess which values indicate a good or bad design.