PVG – IN the graph transformation DOMAIN, you have SOME similar benchmarks AND the nice thing IS that these benchmarks have actually inspired CASE submitters FOR TTC already! FOR example, see the AntWorld CASE FROM Zuendorf ( GraBaTs/TTC 2008 ): (AS a benchmark, the AntWorld simulation tries TO complement the Szierpinsiky triangles benchmark studied IN AGTiVE 2007. The Sierpinsiky triangles benchmark employed very few AND very simple rules. It was possible TO organize the rule execution such that ONLY LOCAL search was necessary FOR rule matching. Thus, the Szierpinsiky triangles benchmark actually measured the memory footprint employed by the various tools.)

LR – I think this would be extremly useful FOR TTC, particularly AS we consider automating SOME parts OF the scoring system.

SM – Interesting observation. Actually, AT the moment a student OF mine IS developing a so-called online judge FOR model transformations that we also want TO use FOR future editions OF TTC. This system will be similar TO the popular online judge FOR the ACM programming contests. WITH this judge it will be possible TO specify cases AND upload a lot OF accompanying reference INPUT/OUTPUT models. The solutions will THEN be black-box-tested against the reference models. Thereby, also performance DATA will be gathered, so that the tool comparison hopefully will become even MORE transparent.

PVG – Steffen touches an important issue: before thinking about performance, we need TO tackle the correctness checking properly.

JC – You put a lot OF emphasis ON the fact that people can submit their cases. Obviously, this gives MORE flexibility TO the contest but: aren’t you worried that people ONLY submit cases that benefit their own tools?

PVG – Indeed every advantage has its dark side THEN again, we have SOME control mechanisms TO tackle the threat that you DESCRIBE: FIRST OF ALL, we actively contact “problem owners” during our call FOR cases phase; secondly, we have a so-called CASE committee TO review the submitted cases; finally, we have a so-called steering committee TO assist us IN the selection OF the final cases.

SM – We try very hard TO also GET cases FROM other domains. But indeed many cases have been suggested by tools developers so far. This has the advantage that the specification IS very precise AND often there even IS a reference solution already. Regarding the bias you suspect: I think there often have been surprises AT the contests, WHERE a quite UNKNOWN tool ALL OF a sudden excells AT a particular problem

PVG – AS an example OF an active call, we have ongoing contacts WITH healthcare organizations (a hospital, a hospital information system provider, AND a Dutch standardization organization) AND hope that this leads TO a non-tool-biased CASE FOR TTC 2012.

LR – LAST YEAR, I suggested a CASE FOR TTC, AND was concerned about this. I tried TO minimize the chance OF this threat TO validity by selecting an example FROM UML, which IS obviously well-known, etc

JC – Along the same line, what’s the role of companies in TTC? or better said, what’s the role you would LIKE TO companies TO play IN TTC?

SM – AS Pieter mentioned we ARE looking quite actively FOR a commercial CASE. This would probably also make different criteria MORE relevant (the soft criteria such AS usability you asked FOR already). OF course, companies ARE also very welcome AS sponsors OF awards etc.

JC – Before you said that the tool contest could also help companies trying TO choose a transformation tool. IS this a realistic scenario?

PVG – Personally, I think TTC aims MORE AT facilitating design/LANGUAGE/tool selection FOR individual developers. I don’t think we should expect companies to use TTC for standardizing on one particular transformation tool.

LR – Yes, for some cases. For example, last year Tom Morris (the lead developer of Argo UML) expressed a keen interest in the results of some of the cases. He’s eager TO use SOME OF our WORK TO ADD functionality TO his tool. TTC would help Tom TO SELECT the RIGHT transformation tool FOR his problem.

SM – It might be easier TO suggest a tool FOR a specific (maybe industrial) problem if you have seen this tool AND SOME others IN ACTION already.

PVG – Continuing Louis’s remark about ArgoUML: in that case, I think TTC is useful for companies that want to use (and then in the future adapt/extend) an already complete (and highly ranked) solution, as developed by a TTC participant. I think that is indeed realistic (as illustrated by TTC 2010), but this is on another level than what I had in mind in my previous reply.

SM – Therefore, we give the solution submitters the possibility to demonstrate their solutions in front of the critical audience.

JC – Do you see a fast evolution in the transformation tools sector? Is still room for new tools? A lot of things to improve in the existing ones? Or, let’s rephrase it a little BIT. FROM YEAR TO YEAR, do tools clearly achieve better results IN the cases? (I know the cases change every YEAR but …)

LR – Maybe the live contest provides SOME answers FOR this question? (I don’t know I only participated in one live contest so far)

PVG – For sure, there is room for new tools. The Berlin team (http://www.tu-berlin.de/tfs) has demonstrated this with Henshin (EMF Tiger). Speaking from their example, a competitive area is support for statespace generation. As another example, the GrGen.NET tool shocked us some years back in the area of performance. Then again, TTC has also shown that some other tools (whom’s NAME I will NOT mention ) were perhaps NOT so innovating. I consider this very valuable, AND I would recommend everyone WITH a new idea TO participate ASAP WITH a prototype: either you shock the community WITH a very promising new approach, OR you learn that perhaps you ARE better OF implementing your ideas ON top OF another tool.

LR – It’s definitely a good opportunity to try out new prototypes, or use existing tools in a weird way! We used one of the Epsilon languages to do something we didn’t know was possible LAST YEAR

SM – I think the community should aim AT improving tool interoperability. It would be desirable that you could use the most appropriate tool FOR each OF your sub-problems AND develop an overall solution FROM those. I.e. use the best pattern matcher, the best tool FOR rewriting AND simulation, the best graphical editor etc. But I do NOT know how realistic this IS

PVG – Good point Steffen, IN that sense I would ADD TO my previous reply that IN CASE you learn via TTC that another tool IS much better IN things that ARE NOT the core focus OF your prototype, you may consider TO WRITE an integration WITH that tool after TTC (whereas otherwise you would spend effort ON implementing an inferior substitute IN your prototype AND users may dislike your tool because OF that)

If you enjoyed this post you can subscribe TO this Software Modeling blog , TO the portal’s mailing list , follow me on twitter, participate in the forums and/or check the most popular posts . And if you really liked it help me pass it on to others by bookmarking and sharing the post using the links below: