Chapter 15. Appendix A: Eiffel vs C++
Throughout my study of the PSP, I have endured a fair amount of pain implementing each program in two different languages: first in C++, an industry-standard language which mixes the imperative, procedural style of C with some object-oriented features; and second in Eiffel, an imperative, strictly object-oriented language with many features not found in other languages (garbage collection, constrained genericity, contracts for classes, etc.)
Comparisons between the features of C++ and Eiffel have been done many times before; arguably the most accessible is Ian Joyner's essay, C++??, available online at www.elj.com. I don't think that a repeat of this comparison is needed, when Mr. Joyner has done an excellent job of describing it.
However, I have not seen a comparison of actual development between C++ and Eiffel, and the PSP projects have given me a significant opportunity to examine the two under actual development conditions-- I can now compare how C++ and Eiffel stack up in development time, defect count, compile time, performance, etc. The tools in use are gcc/g++ 2.95 for the C++ code, and SmallEiffel version -0.78 (rpm package 1.22.7), using gcc 2.95 as the C backend.
Of course, such a comparison is somewhat unfair-- in order to keep my PSP data constant, I have implemented each design in C++ first, and Eiffel second. It is certainly easier to write a program once you have implemented it first! On the other hand, a number of things work to keep the comparison more equitable:
Common design. Both the C++ and Eiffel versions of a given program were written from a common design; the languages have features which enabled me to use the same design (and in general the same object hierarchies and feature names, etc) for both programs. As a result, there should be minimal differences between the implementations.
Defect tracking. The defect tracking employed in the study has allowed me to do a fair amount to identify where defects lie-- for this section, this is significant because I can extract the time spent on either implementation which dealt with design issues, helping to focus on language differences entirely. If, during the C++ implementation, I spent a fair amount of time fixing design problems, that should not affect the comparison of C++ vs Eiffel-- the design fix time will simply be "erased from the record."
Objective comparisons. It's true that some of the comparisons, such as development time and defect count, could be affected by the development process. On the other hand, there are some comparisons (such as compile time and execution time) which are independent of the construction process and can provide a valid comparison of the two languages.
So, with the knowledge that the comparison might be flawed, let's see how the two languages stack up.
Development Time
Removing all time spent on finding and fixing design defects, we can generate the following data based on the results of the 10 programs. The linear regression was done using the PSP programs; according to [Humphrey95], a correlation of greater than 0.9 is "predictive, and you can use it with high confidence"; a correlation between 0.7 and 0.9 is "a strong correlation" and is "adequate for planning purposes"; between 0.5 and 0.7, "there is an adequate correlation for many purposes" but it should be used "with caution"; and below 0.5 "the relationship is not reliable for planning purposes":
Linear regression shows a Beta1 of about 0.71, showing that the Eiffel programs took about 71% of the time the C++ programs took to code, with a correlation of about 0.846 and a 2*(1-p) significance of 0.002, which is a strong correlation with less than 0.2% likelihood of finding this correlation by chance.
Linear regression shows a Beta1 of about 0.37, showing that the Eiffel programs took about 37% of the time the C++ programs took to compile, with a correlation of about 0.528 and a 2*(1-p) significance of 0.116, which is a low-to-moderate correlation with less than 11% likelihood of finding this correlation by chance. This is not particularly significant, and as compile time is a fairly small portion of overall process time, not a very useful observation.
Linear regression shows a Beta1 of about 0.37, showing that the Eiffel programs took about 37% of the time the C++ programs took to test, with a correlation of about 0.66 and a 2*(1-p) significance of 0.036, which is a moderate correlation with less than 3.6% likelihood of finding this correlation by chance.
Linear regression shows a Beta1 of about 0.63, showing that the Eiffel programs took about 63% of the time the C++ programs took to develop, with a correlation of about 0.79 and a 2*(1-p) significance of 0.006, which is a strong correlation with less than 0.6% likelihood of finding this correlation by chance. This, to me, is a fairly significant finding in favor of Eiffel to C++.
Of course, it's certainly easy to find fault with an experiment in which data was taken from only one developer and with only ten data points for each series, but it is what I can offer. Furthermore, from a developer's standpoint, the Eiffel errors were significantly easier to detect, even those injected in the design phase. In the area of application development time, Eiffel wins.