Quite regularly, I get questions about what empirical evidence supports my “belief” that models are good (what kind of models, how much modeling is necessary, whether models should be used as blueprints or as executable models,… are not today’s topic).
Until now, I used to point to the (true, few) scientific empirical studies on the effectiveness of software modeling but now I have an even answer to give you: “Empirical Evidence of the Value of Version Control”.
No, I haven’t lost my mind. The point of this link is to show you that there’s no proof that version control is better for software development, and yet, I don’t think any of you would argue against it.
Same for modeling and model-driven engineering. It would be great to have more proof but the absence of proof alone should not be used against it unless you want to start also abandoning other unproven things like version control.
The difference between the two (version control and MDE) is that, as we have discussed before, MDE is not a cool technology and therefore people needs a stronger argument to adopt it. You may not believe that MDE is useful and that’s fine but at least be consistent and stop whining about the lack of proof.
FNR Pearl Chair. Head of the Software Engineering RDI Unit at LIST. Affiliate Professor at University of Luxembourg. More about me.
The fact that there aren’t studies demonstrating the value of version control does not imply that other things should be accepted as valuable without studies. After all, if I use your logic, we should all adopt functional programming languages, reverse Polish notation, and variable names that only contain consonants…
True. But my point that absence of studies should then not be used as the only argument to reject something.
Why can we accept VCS but not modeling if none of them have enough empirical evidence?
I guess that many would say that VCS benefits are obvious while modeling ones are not. Well, to me modeling benefits are also obvious. But maybe I´m just weird 🙂
@jordi: why don’t you conduct more (not-only-scienficic) empirical studies on Model-Driven Engineering, any way?
This is easier said than done 🙂
To be valid, empirical studies need to be very well designed (otherwise they end up just being anecdotal evidence). I don’t have enough experience (yet) to do a good empirical study of this type.
While it’s certainly hard to draw conclusions when there is an absence of evidence, another equally challenging question is: “what do you do when the evidence doesn’t match your ‘gut'”. I know Greg has pointed to studies that refute the advantages of TDD, although I know others that swear by this method of development. I’m sure there are other studies that show MDE isn’t worth the effort; but what practical advice should we take from these results?
I agree that gathering the evidence is a great first step, but we need to be careful what conclusions we draw or how we use the results in our own work. Even if there were more studies on the effectiveness of modeling, would these results be applicable to software developers in general, or are there too many other variables in the equation?
We are certainly learning a lot from other disciplines about the advantage of evidence based science. The next step will be understanding how to apply the results of this work.
The point is that software engineering heavily depends on context. This is why empirical studies are always inconclusive or conflicting. What works in one company/domain/industry/team doesn’t necessarily work in another.
All empirical software engineering can do therefore is to highlight factors that may lead to success or failure but be sure to comment on the context in which those factors may be accelerated. We can never make statements like “MDD (or TDD) increases productivity” because it fundamentally depends on people and organizational context. All we can do is say: “Look, MDD (or TDD) seems to work well in these kinds of situations.” It is then up to each particular company to compare those situations to its own context and make a decision as to how or if MDD (or TDD) should be applied.
[This may also be the reason for the lack of empirical studies on VCS. It has been shown to work through thousands of hands-on experiences — which are empirical studies in lots of different contexts and the experience has been communicated.]
“How many empirical studies are there proving that 5 is a prime number?”
Hi Jordi,
I think some properties are so obvious that the “proofs” for it can easily be overlooked. E.g. VC: everyone who lost a project because his hard drive failed (happened in the 80ies) or everyone who tried to synchronize a set of files with a backup version and some copies on different people’s machines has by far enough proof that VC is a good thing, and not only good but unavoidable. You might even draw numbers from this and prepare management slides (“…hours spent on banging highly paid engineering heads against the wall after the disk crash…”) – but who cares? It is totally obvious.
The same holds for MDD, just compare the possibilities to make errors or measure the time to remember the way of doing something (the same reason why scripts and programming in general have been invented). There are many, many more.
On the other hand side: there are equally easy discoverable properties of MDD that are disadvantages, at least in the past and to some degree still at present. Say, if introducing technology XYZ implies learning 7 new languages, 14 configuration systems, 5 new tools in the chain and a few additional inhouse projects to puzzle that all together I need no additional proof that I don’t really want to use that technology in my project.
I think these disadvantages are the reason why MDD success presently depends much on project context: only in specific scenarios it makes sense. If we overcome these disadvantages, I’m sure there will also be empirical studies showing it’s “general” usefulness.
Andreas
“How many empirical studies are there proving that 5 is a prime number?”
Sorry, but “5 is a prime number” and “models are good” are completely unrelated sentences. You don’t need an empirical study to *prove* the first affirmative, while no empirical study will *prove* the second one. Providing an *evidence* to something is not *proving* something is true!
” version control is better for software development”
As everything in software engineering, no, that is not an absolute truth. VCS are a tool, and just as any other tools, they are good at some situations, and overkill in others.
I would never setup a VCS for a one-off short-term small sized project.
Hi Marcos,
to clarify the philosophical details 😉
1. absolute truths…
On the one end of the scale, we certainly cannot be sure of anything. It is actually possible that my brain is floating in a tank in an alien space ship and these blog debates are just a game the aliens play with me for there amusement.
2. evidence…
Accepting the assumption that I’m not in the alien space ship, I can try to do my best by e.g. using my limited intellect and live with the constructs I learned and the few I invented. Evidence is what I can use to convince likeminded ones.
3. proof…
Is for those who accepted a certain, small set of basic assumptions they considered very evident in the beginning; a proof to those is strong, deduced evidence.
4. postmodernism…
Taught us to always stay alerted and keep in mind that everything is still and will always be only a construct. In a sense, it is like “go back to 1 and repeat, but go back 1 one very soon again”.
5. agilism…
Braught postmodernism to IT.
That said, surely I could have written:
“In a certain situation on a blue (as far as we can talk about colors at all) planet in a time we named “year 1985” (remember: names are just names and not reality), I worked (wait, this “I”, isn’t that an illusion?)….etc. etc.”
But I prefer to write
“…everyone who lost a project because his hard drive failed…has by far enough proof that VC is a good thing…”
We can consider the latter just being a short form of the former.
I feel your pain, Jordi. 🙂
We do a lot of teaching on conceptual modelling to non-IT specialists, mainly from the fields of archaeology, architecture and related fields. The question of “why modelling” or “what is this good for” is always there, and I often give a two-fold answer. On the one hand, modelling helps you understand the reality that you observe; you can easily explore, validate and reason about that reality. On the other hand, modelling is useful to communicate our thoughts; you can document, convey and let others reuse the ideas that you have come up with with great ease.
All this is far easier said than done, i.e. I explain this at the beginning of each course and students nod away. Then, I still need to struggle throughout the course to make sure that they see the practical benefits of modelling to their daily work. Some arwe convinced after one week, but some are not even after one month.
Check also http://www.neverworkintheory.org/?p=457 for more on the value of evidence