Collaborative Software Modeling in Virtual Reality

Modeling is a key activity in conceptual design and system design. Through collaborative modeling, end-users, stakeholders, experts, and entrepreneurs are able to create a shared understanding of a system representation. While the Unified Modeling Language (UML) is one of the major conceptual modeling languages in object-oriented software engineering, more and more concerns arise from the modeling quality of UML and its tool-support. Among them, the limitation of the two-dimensional presentation of its notations and lack of natural collaborative modeling tools are reported to be significant.

Therefore, we have explored the potential of using Virtual Reality (VR) technology for collaborative UML software design. VR has been a topic of intense research in the last decades. In the past few years, massive advances in affordable consumer hardware and accessible software frameworks are now bringing these technologies to the masses. As VR is a technology that allows a user to immerse in a computer-generated 3D virtual world, our aim was to use such a technology as a new alternative for supporting collaborative software modeling. For this purpose, we have developed a VR modeling environment that offers a natural collaborative modeling experience for UML Class Diagrams. Until now, we had seen an augmented reality toolkit for modeling but not a virtual reality one!

The system overview of our VR-based collaborative modeling environment VmodlR is shown in the Figure 1 below. The top half of this figure represents User A and the Virtual Environment this user accesses through a VR Device. The bottom half symmetrically shows a remote collaborator, User B, and the VR Device this user employs to access the same shared Virtual Environment. Through the VR Device, each User is represented in the Virtual Environment as a Virtual Character. The View Orientation and Position of the VR Device together with the Controller Input controls the Virtual Character while the Virtual Environment with its content is displayed from the Virtual Character‘s perspective inside the VR Device.

As a target implementation platform for our collaborative VR modeling environment, we have chosen the Oculus Quest 2 which is a cable-free, mobile VR HMD. For the implementation of the VR environment, we have used the Unity 3D Game Engine developed by Unity Technologies as it includes a fully featured Graphics, Sound and Physics Engine and has easy-to-use APIs for many different aspects like controller input. Furthermore, Unity also provides support for the Oculus Quest 2 among other Oculus VR headsets through an Oculus VR SDK.

To enable remote collaboration, a networking system is needed that synchronizes aspects like model element- and user avatar positions, user hand gestures, etc. For the implementation of the networking system, we have used the PUN plug-in for Unity. This is a third-party system developed by Exit Games specifically for use in Unity multiplayer projects and offers easy-to-use high-level components. PUN realizes the network architecture by providing its own Cloud servers that automatically work with PUN without the need for any custom server-side development. The Oculus SDK, used to realize the interaction between the VR Device and the Virtual Environment, comes with a variety of assets that can be used to quickly implement common functionalities inside a VR application. Most aspects that were needed to realize the Virtual Character are covered by those assets provided by Oculus. Therefore, to save development time on those basics, we used Oculus’ OVRAvatar and OVRCameraRig assets and adjusted them slightly for our application. The OVRCameraRig deals with tracking the position and rotation of the head and controllers and rendering the world accordingly into the headset. It does not include animated hands by default but there are separate assets for that which can simply be placed under the empty GameObjects that track the controller positions to enable users to see their hands. The gestures of the hands are tracked locally by default through an animator so we only had to add an animator synchronizer provided by PUN to synchronize the hand’s animation states across the network. Since all Virtual Characters are represented equally in this prototype, we chose to display every user’s name above their character.

An example of how a remote user’s Virtual Character looks inside the VR app is shown in the following figure.

Based on a user study with 24 participants, we have compared collaborative VR modeling with conventional modeling with regard to efficiency, effectiveness, and user satisfaction. Results show that the use of VR has some disadvantages concerning efficiency and effectiveness, but the user’s fun, the feeling of being in the same room with a remote collaborator, and the naturalness of collaboration were increased.

More details on solution approach and the evaluation conducted is explained in the following papers:

• Yigitbas, E., Gorissen, S., Weidmann, N., & Engels, G. (2021, October). Collaborative software modeling in virtual reality. In 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS) (pp. 261-272).
• Yigitbas, E., Gorissen, S., Weidmann, N. et al. Design and evaluation of a collaborative UML modeling environment in virtual reality. Softw Syst Model (2022). https://doi.org/10.1007/s10270-022-01065-2

In ongoing work, we are developing a gamification-based UML learning environment in VR  that is based on the approach presented here. The VR environment provides minigames and multi-viewpoint modeling features to learn creating class diagrams. Furthermore, this environment supports a multi-modal interaction and a more realistic representation of the virtual characters. More information about this ongoing work can be found in the following paper:

• Yigitbas, E., Schmidt, M., Bucchiarone, A., Gottschalk, S., & Engels, G. (2022, October). Gamification-based UML learning environment in virtual reality. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (pp. 27-31).

Besides these improvements, we further plan to implement features like automatic alignment of model elements, and tracking of which collaborator created which parts of the model. Apart from that, scalability and performance are important technical aspects that should be investigated in future work, especially if larger groups of people use such a VR modeling tool for collaboration purposes. In addition, further usability evaluation studies with larger groups of heterogeneous participants and more complex modeling tasks should be conducted to analyze in more detail the benefit of collaborative modeling in VR. Finally, we believe that a cross-device mixed reality collaborative modeling approach is a promising way to support modeling across different AR/VR capable and conventional devices.

The code repository of our developed collaborative VR-based UML modeling tool VmodlR as well as details about the user study and corresponding evaluation data can be found at IRB Git (https://git.cs.uni-paderborn.de/scg/vmodlr).