Tree of Thoughts Framework for LLM-Based Domain Modeling

Domain modeling requires a deep understanding of the domain to carefully abstract the important elements that create a relevant conceptual model of it. This process is complex and involves iterative steps, requiring constant interaction and close collaboration between domain experts and modeling experts. Recent advancements in artificial intelligence, specifically Large Language Models (LLMs), are promising in assisting the different parts of this process.

But this assistance is not trivial as previous naïve attempts have shown us. To overcome the limitations of previous methods, this article explores how we adapted the Tree of Thoughts (ToT) framework to LLM-based domain modeling with the development of a new Domain Specific Language (DSL) that enables modelers to configure the ToT process for optimal results.

This work will be presented at the upcoming ER2024 conference. Continue reading for a summary of the paper or go to the end to browse the slideshare presentation and/or download the full paper.

LLM-based Domain Modeling challenges

Domain modeling requires both understanding the nuances of the domain and abstracting them into a model. Traditional methods rely heavily on human modelers to refine models iteratively. When LLMs are introduced into the domain modeling process, they typically struggle to recognize relationships, abstract constructs, and incorporate best practices. Early attempts at using LLMs for this task, such as GPT-3.5 and GPT-4, showed promise in identifying elements like classes and attributes, but fell short when handling relationships and complex modeling constructs like association classes.

Tree of Thoughts framework

We propose the application of the Tree of Thoughts (ToT) framework to incorporate exploration of multiple thoughts and self-assessment to generate domain models with intermediate steps. The ToT framework allows LLMs to break down a problem into smaller steps, providing a more structured way to generate and evaluate intermediate thoughts to solve a complex problem. This, in fact, aligns well with the modeling problem where we typically create models in an iterative process focusing on a submodeling problem every time.

The ToT framework requires the definition of four components:

1. Design of intermediate steps that guide the solution process with diverse alternatives. The steps should be neither too small nor too large, ensuring that they maintain diversity and are easy to evaluate.
2. Generating thoughts using either the ‘Sample’ strategy to generate multiple outputs from a single prompt or the ‘Propose’ strategy to create several thoughts within one output, ideal for shorter ideas like words or phrases.
3. Evaluating thoughts by selecting promising thoughts choosing the ‘Value’ strategy, which scores thoughts, or the ‘Vote’ strategy, where different options are compared, and the most promising one is selected.
4. Use a search algorithm between Breadth-first Search (BFS) to evaluate and explore the best thoughts at each level, or Depth-first Search (DFS) to fully explore one promising thought until a solution is reached. If the solution fails another path is explored.

These are the steps we need to adapt to successfully apply a ToT approach to modeling.

A definition of ToT for LLM-based Domain Modeling

Implementing the ToT for domain modeling is non-trivial. It starts with designing the intermediate steps to solve the complex problem of creating a domain model. To reduce the complexity of it, we decided to focus on UML class diagrams, a structural representation assessed in the literature with different LLMs. Typically, class diagrams consist of classes and different types of relationships. However, to push the boundaries of this approach, we incorporated association classes, which are a complex abstraction that LLMs often struggle to correctly identify.

Based on this, we decided to dedicate an intermediate step (in ToT terminology) to each element, asking the LLM to focus on individual types of concepts at every step.For instance, we separate the creation of classes and relationships into two separate steps. This is important as it has been identified that LLMs struggle in correctly identifying relationships so we want a dedicated step for them. Moreover, and for this same reason, we further divided the identification of relationships into three separate tasks: association, containment, and inheritance. The final step in our proposal is to identify the association classes, leveraging the insights gained from earlier steps involving classes and associations.

 

To choose the generator prompt strategy, we considered that LLMs generate multiple responses with the same prompt. So, we adopted the ‘Sample’ strategy that uses a single prompt to generate diverse model element proposals. For the evaluator prompt, we selected the ‘Vote’ strategy, by providing LLMs with specific criteria to select the best proposal using five votes, which minimizes the chance of ties. Finally, we selected the BFS algorithm, as it enables to continuously build on the best thought at each level. This approach selects the best proposal at each step to continue the creation of the domain models with more model elements in subsequent steps.

A DSL to configure ToT for Domain Modeling

To simplify the design and execution of ToT framework, we developed a DSL that captures the intermediate steps needed for solving domain modeling problems. Although our approach focuses on suggesting elements like classes, associations, containment, inheritance, and association classes, it can easily be adapted to suit the specific requirements for other modeling problems.

The DSL metamodel is divided into two key packages: the problem space, which defines the domain modeling challenge, and the solution space, which configures the ToT framework. In the problem space, we need to define the modeling with its corresponding sequence of tasks, and each task requires some assessment criteria to evaluate the LLM suggestions. Finally, the modeling notation is specified with detailed descriptions or examples (few-shot prompting), to guide how the final domain model should be represented.

In the solution space, our DSL configures the ToT levels based on the tasks defined in the problem space. During execution, the generator, and the evaluator prompts are created dynamically, incorporating the thoughts generated at each level. Ultimately, one thought is selected at the final level of the tree, which is then used to produce the domain model in the desired notation.

Our tool includes the DSL as an open source tool on GitHub. The concrete syntax was designed with the grammar of TextX for the metamodel described above and allows it to be executed in Python using the classes of the ToT framework library to execute the generator and evaluator strategy with the BFS algorithm. To use the2 DSL, a subscription to the OpenAI API or Azure OpenAI API is required; however, it is possible to extend the DSL to use it with other LLMs.

Experiments

We validated our approach running experiments using GPT-4. The results showed that using the ToT framework allows the LLM to identify complex modeling constructs, such as association classes. In particular, the framework demonstrates strong performance in recognizing classes and association classes; however, attribute recommendations show a significant disparity, with low precision compared to high recall, suggesting that many recommended attributes were not in the reference model. The major challenge faced was with the relationships, having the lowest performance in precision and recall.

One of the strengths of ToT in domain modeling is that it allows for continuous iteration and refinement of domain models. Our experiments indicate that ToT can handle complex constructs because the intermediate results from earlier steps, particularly the identification of classes and associations, offer insights that extend beyond the initial domain description. This promising outcome encourages further exploration of different configurations and task decompositions, particularly for other modeling problems like behavioral diagrams, in future research.

Adapt the DSL to your needs

It is possible to customize the configuration of ToT for different modeling problems with the proposed DSL. For that reason, we have included examples with other domain models such as BPMN, Entity-Relationship, and UML activity diagrams. We suggest experimenting with complex modeling tasks where prompting techniques do not achieve good results.

To know more 

Check also the presentation below or download the full paper)