From Mock-Up to IFML-like GUI Models: Using Large Language Models in Web Engineering

In modern web development, the design process is often initiated with the creation of mock-ups, which are visual representations of the proposed application’s aesthetic. However, the process of translating these designs into functional applications is a time-consuming and technical task.

On the one hand, tools such as Figma and Sketch can facilitate the process to a certain extent, but they are generally limited offering only prototyping capabilities for user interface elements.There are also a bunch of of GUI-to-code tools that promise a more complete implementation but at the cost of having then to check the final code if we want to actually use it beyond prototyping.

In line with the concept of vibe modeling, we propose an alternative approach where the power of LLMs is used to generate UI models, not direclty the code. This way you can review the models before actually generating the code following a “traditional” deterministic approach.

Indeed, our work (accepted at the ICWE 25 conference)  proposes a novel pipeline that uses Large Language Models (LLMs) to convert UI mock-ups – either hand-drawn or digitally designed – into IFML-like GUI models. These models can then be transformed into fully functional web applications using a low-code framework. In particular, this method enhances the BESSER low-code platform, a Python-based framework supporting both structural (UML) and GUI (IFML) modeling. Download the full paper or keep reading for a summary.

The Approach: Sketch-to-Model with LLMs

As illustrated in the figure headlining this post, the pipeline automates two fundamental tasks:

• Structural Model Extraction: Using LLMs to infer UML class diagrams from mock-ups.
• GUI Model Extraction: Generating IFML-like models that represent UI layout and user interactions.

Note that we cannot just generate the IFML ones alones as the IFML UI elements (such as a dropdown list) are connected to the data model elements from which we get the data. So, we are forced to infer not only the UI model but also the data model from the mock-up!

Once the models are prepared, they can be leveraged to automatically generate a complete web application using a Django-based code generator. This tool creates both the front-end interface and back-end logic, enabling a seamless transition from a hand-drawn or digital UI sketch to a functional app. This approach significantly reduces or eliminates the need for manual coding, accelerating the development process.

What makes our approach unique?

The present work integrates several powerful ideas:

• Prompt engineering: It is imperative that prompt engineering is implemented for LLMs to ensure the accurate conversion of UI images into modelling artefacts.
• Self-improvement prompting: The process of self-improvement involves the iterative refinement of models by the LLM, with the objective of aligning them with the structure and logic of the mock-up.
• Multi-page support: The system has been developed to support multi-page applications, with the capacity to combine individual pages into a cohesive application with complete navigation.

The entire system has been constructed on the open-source BESSER platform, which utilises B-UML, a modelling language that captures both structural and GUI (illustrated in Figure 2) aspects in a consolidated format. The code is generated using Jinja templates, a process which facilitates extensibility and ease of customisation.

 

Running Example: From Mock-Up to Web App

To demonstrate the nature of the pipeline, the approach was applied to a simplified Library Management application, which featured entities such as Books, Authors, and Libraries.

We began with the creation of hand-drawn mock-ups for three UI pages, with the Book page selected as a representative example (see Figure 3). These sketches are intended to capture layout ideas, including lists, navigation buttons and data forms.

During the Structural Model Extraction phase, a UML-based model was generated describing the domain logic, including classes, attributes, and relationships (see the Listing 1). This formed the basis for the application’s data layer.

 

In the GUI Model Extraction step, each UI mock-up was transformed into an IFML-like model. These models are used to represent GUI components, including screens, buttons and lists. To illustrate this point, consider the Book page model, which incorporates a BookListScreen complete with an add button, a back button, and a list of books, all of which are methodically organised within a module designated as MyModule in the GUI model gui_model (see Listing 2).

 

Next, the individual GUI models were combined by integrating navigation flow and page order, thereby forming a unified GUI model. In conjunction with the Structural model, this approach was employed to generate Django code that implements the full application, encompassing all three pages and utilising Django’s default styling. As illustrated in Figure 4, the generated web page is visible.

 

To evaluate fidelity, we used an image comparison algorithm based on OpenAI’s CLIP model. This method focuses on structural and layout similarity, ignoring aesthetic differences (like colors or fonts), to measure how well the generated app reflects the intended mock-up.

The results show high layout similarity:

• Book Page: 87%
• Library Page: 86%
• Author Page: 72%

This confirms the system’s ability to generate usable and consistent CRUD-ready applications from simple sketches.

Evaluation & Results

We tested our pipeline on 100 real-world UI mock-ups from CodePen (from 13 different categories), measuring accuracy, flexibility, and run-time performance.

According to the evaluation results, it is evident that there is a high degree of structural similarity with the original designs, with rapid model generation and seamless integration between the UI and data layers.

This approach has the potential to lower the barrier to web development and to showcase the potential of combining AI and MDD to create intelligent developer tools.

Try It Yourself

The entire infrastructure is available as an open-source project. If you’re interested in trying our tool, check out the GitHub repository.