In numerous programming and software engineering courses, students are asked to program on paper, which has supporters and detractors. Supporters claim that, among its advantages, programming on paper allows students to focus on functionality, avoiding the distractions caused by syntax and without limiting their thinking to a specific programming language or paradigm. Detractors claim that this method lacks advanced capabilities provided by IDEs such as syntax check and auto-completion. More importantly, it does not give the opportunity to execute and test the code, which prevents students from discovering bugs.
The benefits and disadvantages of programming on paper versus computer for general-purpose languages like Java and C with students of initial courses have been studied repeatedly. Nevertheless, to the best of our knowledge, no study has been performed targeting formal languages like OCL, which are taught in advanced courses.
Here, we aim to present our experience after introducing a modeling tool for the specification of OCL constraints in a Requirements Engineering course.
In recent years, the professors of the Requirements Engineering course of the Computer Engineering Bachelor Degree offered at the Open University of Catalonia (Universitat Oberta de Catalunya – UOC for short) have noticed that students showed some disappointment with the formative assessment. As part of this formative assessment, they were asked to define a series of restrictions using the declarative language OCL. No digital support was recommended or provided and the students used to do this exercise on paper. The complaints of some of the students, added to the feeling that the lack of a software tool could be affecting the development and results of this exercise, made us consider the introduction of a modeling tool. The idea behind this change is that the tool could allow the students to execute and test such OCL constraints and therefore we could be preventing a potential learning obstacle.
There is no consensus on the use of paper in Programming and Software Engineering courses. On the one hand, many educators endorse the benefits that programming on paper has for Computer Engineering students or for students of any other higher education program. They claim that programming on paper allows students to focus on the logic of the program they are writing, avoids distractions caused by syntax errors, and does not limit the students’ thinking to a specific programming language, platform or paradigm. These statements are also explicitly supported by many international companies. For example, during the hiring process, one of the several interviews that their interviewees must pass is the so-called whiteboard interview. During these sessions, the potential employees need to solve a problem on a whiteboard using the (pseudo-)language they prefer. This way interviewers assess the knowledge, competences and skills of a potential hire.
At universities, pencil-and-paper programming is frequently used not only as a means of learning but also when assessing the students’ knowledge in both formative and summative assessments [1, 2, 3].
Despite its benefits and adoption in specific contexts, paper programming also has its detractors and some widely recognized drawbacks. For example, students often complain that they cannot execute, test, and debug the code they are writing. This method prevents students from verifying correctness of their code and hinders the detection of those errors that could be easily and early discovered with the simple execution of the program.
Apart from the studies that address programming using general-purpose languages, to the best of our knowledge, there is no study that focuses on the use of paper as opposed to a tool when learning formal languages and/or standard languages for the definition of rules (such as OCL). Since modeling languages such as UML/OCL are extensively used in the academic environment, not only in our university but in many others, we decided to document and publish our experience so that anyone in our situation can benefit from it.
Our study shows that the use of tools for learning rule-definition languages such as OCL is perceived positively by both students and faculty members. Comparing to the results obtained in the fall of 2019—when no modeling tool was provided to our students—and the fall of 2020—semester in which we introduced a modeling tool and performed this study—, we have observed that the students’ grades have improved.
The experience described in this paper takes place in the Requirement Engineering course of the Computer Engineering Bachelor Degree offered at the Open University of Catalonia (UOC). Our Bachelor programmes have 240 ECTS credits and are planned to take four years of full-time study. This is an elective course that students can take during their third or fourth academic year. The course assumes a basic knowledge of software engineering and delves into the first stage of the software development life cycle. The contents of the course are organized into five modules: (1) introduction to requirements engineering; (2) requirements elicitation; (3) requirements analysis and management; (4) requirements documentation; and (5) requirements validation and verification. The OCL language is introduced during the fourth module, as a method to formally document requirements.
The progress of our students in this course is evaluated using a continuous assessment model. Concretely, the model of the course is composed of four Continuous Assessment Tests (CAT) scheduled throughout the semester, all of them formative. The course does not have any summative test. For each CAT, students are provided with detailed feedback consisting on their grade accompanied with individual comments. A few days after the feedback is given, we publish the solution to the CAT.
All CAT activities are built on top of the same case study, which allows the students to have a complete and more realistic vision of all the phases of requirements engineering lifecycle. In particular: the first CAT focuses on requirement elicitation given the textual description of the case study; the second CAT addresses the analysis and management of requirements; during the thrid CAT, the students document the requirements in an agile way through use cases and user stories; and, finally, in the fourth CAT, the students document the requirements in a formal way (using UML/OCL) and apply validation and verification techniques. The fourth CAT is the target of our experiment.
The faculty involved in this course is formed by two assistant professors and three teaching assistants. The course has around 150 students enrolled each semester, which are assigned to virtual classrooms with about 70 students each. Each classroom is energized by one of the teaching assistants, who guides and accompanies the students during their learning and assesses them and returns the correspondent feedback for each activity.
During the semester, all communication is carried out asynchronously and online, through the virtual classroom forums (where the messages are public to all students in the classroom). Less often, students communicate through email to ask questions directly and privately to the teaching assistants.
We built an experiment to address the following research question: What is the impact in the student learning process of a modeling tool with support for defining and executing OCL constraints?
We propose to measure the learning process by relying on three indirect measurements, namely: the student’s self-assessment, the teaching assistants’ assessment and the student’s academic performance.
Our hypothesis is that using the tool will have a positive impact in the understanding and learning of the OCL language, as the ability of experimenting during the learning process allows understanding the syntax and semantics of the language, detecting errors (i.e., enabling the test-and-error behavior); thus, promoting the definition of correct constraints.
Our research context is a case of Action Research. We employed empirical research methods  to answer our research question. In particular, we have performed a mixed-methods study combining qualitative and quantitative research approaches to address our question from different perspectives with the goal to mitigate the weaknesses of the different empirical methods used.
We defined a controlled experiment to study the cause-effect relationship between using the modeling tool (i.e., binary independent variable) and its consequences. We identified the following dependent variables: dedicated time to do the experiment, student’s perception on the usefulness of the modeling tool, student’s perception on the difficulty of use of the tool, and student’s academic performance.
The experiment was done during the fourth CAT of the course, which is composed of three exercises. One of the exercises assesses the knowledge about OCL and weights 40% of the total mark of the CAT. This exercise presents a class diagram and includes four question tests which ask the student to define OCL constraints. To facilitate the analysis and comparison of the results, the structure of the CAT is similar to those used in CATs from previous semesters. Thus, the OCL constraints to define in the exercise evaluate the same OCL features (e.g., variables, operators, traversals, etc.).
Prior to start the CAT, students were informed about the experiment and were given the opportunity to choose between using or not the modeling tool to solve the CAT’s exercises. The final decision to use the modeling tool is therefore up to the student. The modeling tool proposed was MagicDraw, as it is one of the most popular UML modeling tools with support for the definition and execution of OCL constraints. Besides, MagicDraw is used in the Software Engineering course, which is a prerequisite for Requirements Engineering course.
Once started the CAT, students had four weeks to work and deliver their solution proposal. During this development time, students could ask questions either in the virtual classroom’s forum or contacting the teaching assistants of the course via email. After this period of time, we collected the experiment’s data from three sources, namely: (1) students, via an online form; (2) teaching assistants, via online structured interviews; and (3) messages and grades, via the university virtual campus.
More details about the data collected from each source as well as the detailed results of the analysis can be consulted in our paper  (preprint available here). Next, we discuss these results.
Discussion about the results of our empirical study
- Modeling tools help with the definition of OCL constraints. We have observed that students preferred to use modeling tools to address the CAT. Our results show that student’s perception about the difficulty to solve the exercises is lower when they employ the tool.
- Modeling tools foster experimentation and self-learning. Students that decided to use MagicDraw admitted that the tool gave them the freedom to experiment with OCL constraints and practice self-learning. Besides, they would also recommend using this kind of tools in other courses.
- Student’s performance is slightly better. Although our form was anonymous and there was no relationship between the grades and the student’s identity, the final grade of those students who used MagicDraw was better. However, we believe it is necessary to perform a more exhaustive study, with additional experiments in future courses, to confirm that this difference is significant, as current results may be linked to external factors not considered in our study.
- No technical barrier to entry. In our experiment with students, we did not detect troubles related to the installation or use of MagicDraw other than the issues reported when installing the tool in the latest macOS version, which we easily addressed by providing a virtual machine. Note that MagidDraw has never experienced any problem in macOS before, however, the last version (codenamed BigSur) has caused several issues with, not only MagicDraw but a number of software applications. The teaching assistant also supported the use of modeling tools and recommended its inclusion in future courses.
- Teaching assistant’s workload was not affected by the use of modeling tools. Although the number of messages between students and teaching assistants was higher in 2020, the latter did not detect any increase in the workload. We believe that it may be due to the fact that students’ questions were more concise and easy to address.
- Low impact in classroom organization. As the use of MagicDraw was optionally, it did not imply any structural change in the organization of the course. However, as a result of our experiment. we may explore the modification of the course to make the use of modeling tools compulsory, thus allowing us to expand the course objectives.
- Introducing new tools require extra effort from coordinating professors. To introduce a new modelling tool coordinating professors had to perform an explorative study of existing tools, prepare manuals and documentation. The latter is specially important in MagicDraw, as its features vary according to the version and license provided.
- Limited number of tools supporting OCL validation and verification. Flexibility to propose and use different modeling tools is limited as there are very few with full support for OCL (i.e., including validation and verification). In our case, we decided to use MagicDraw due to its friendly interface and the fact that students were already familiar with it, but it is a proprietary tool. We also considered USE, whose use we are currently exploring for the future. Another problem was the lack of reference examples (and their implementation in the corresponding modeling tool).
We have presented a study that compares the impact of using of a modeling tool to specify OCL constraints versus the definition of the constraints on paper in a Requirements Engineering course. Our study uses different empirical methods (e.g., interviews and questionnaires) of different natures (quantitative and qualitative) and involves several subjects (students and teaching assistants) to cross-validate and confirm the validity of the results.
The study concludes that students have a positive perception towards the use of modeling tools to specify OCL constraints. Besides, teaching assistants also approve and advocate the use of such tools in subsequent editions of the course. Furthermore, the results seem to indicate that the use of MagicDraw has to some extent a positive impact on the students’ final grades, which are slightly higher when they use the tool.
This study is a first attempt towards improving the learning of formal and standard languages, such as OCL, by undergraduate students as well as their satisfaction with the learning process. After this experiment, we have chosen to maintain the optional use of MagicDraw in our Requirements Engineering course and we plan to continue analyzing the evolution of our students’ achievements and satisfaction.
In the future, we plan to study whether other instruments (such as the learning resources provided or other modeling tools) can help to improve the performance and satisfaction of the students. We also plan to measure how the usability of such tools can impact the perception of students and their learning experience. We will also explore solutions to be able to relate the student questionnaires with the grade obtained in the CATs at the same time that we ensure their anonymity. In the long term, the success in the use of these tools will allow us as professors to set more ambitious learning objectives both in this course and in other software engineering courses in our university.
 Jony Adkins and Diana Linville. Testing frequency in an introductory computer programming course. Information Systems Education Journal, 15:22–28, 2017.
 Michael de Raadt Anne Philpott Judy Sheard Mikko-Jussi Laakso Daryl D’Souza James Skene Angela Carbone Tony Clear Raymond Lister Geoff Warburton Simon, Donald Chinn. Introductory programming: Examining the exams. In Proc. of ACE’12, volume 123, page 61–70, 2012.
 Arto Vihavainen, Matti Paksula, and Matti Luukkainen. Extreme apprenticeship method in teaching programming for beginners. In Proc. of SIGCSE ’11, page 93–98, 2011.
 Steve Easterbrook, Janice Singer, Margaret-Anne D. Storey, and Daniela E. Damian. Selecting empirical methods for software engineering research. In Guide to Advanced Empirical Software Engineering, pages 285–311. 2008.
 Loli Burgueño, Javier Luis Cánovas Izquierdo, Elena Planas. An empirical study on the impact of introducing a modeling tool in a Requirement Engineering course. MODELS Companion 2021. (To appear)
Lola Burgueño is a postdoctoral researcher in the SOM Research Lab at the Internet Interdisciplinary Institute (IN3) of the Open University of Catalonia (UOC), in Barcelona (Spain) and CEA List in Paris (France). Her research interests focus on Software Engineering (SE) and Model-Driven Engineering (MDE). She has and is contributing to the fields of capturing and operating with uncertainty in software models for its use in the Industry 4.0, AI-enhanced software systems, testing models and transformations, the distribution of very large models and the parallelization of their manipulation to boost performance, among others.