Benchmarking Freeware

When reading these, remember that all benchmarks are to be taken with a pinch of salt, since they often depend also on factors that are not necessarily made explicit, or controlled by the experimenter. The dates for these benchmarks and comparative studies is also important, since some software evolve fast in quality or performance.

If you are aware of other analyses/benchmarks/studies of freeware, concerning performance, reliability, usability, maintainability, portability or any other quality factor, please forward this information to the maintainer of this page Bernard.Lang@inria.fr. Note that all types of freeware are relevant, though what I have been able to find so far is mostly concerned with operating systems, and compilers.

Papers about freeware dissemination, or any other relevant statistical data are welcome too, though that will be kept on another page. Thank you for helping.

Bibliography

1. Making Programs Explode: Using Simple Random Testing on Real Programs, Barton P. Miller, University of Wisconsin.
2. Fuzz Revisited: A re-examination of the Reliability of UNIX Utilities and Services, B.P. Miller et al., University of Wisconsin, 1995. ftp://grilled.cs.wisc.edu/technical_papers/fuzz-revisited.ps.Z. (Version A4 http://pauillac.inria.fr/~lang/hotlist/free/bench/fuzz-revisited-a4.ps)
3. A Performance Comparison of UNIX Operating Systems on the Pentium, Kevin Lai et Mary Baker, C.R. 1996 Usenix Technical Conference, Janvier 1996. http://mosquitonet.Stanford.EDU/~laik/benchmarks/index.html
4. Benchmark Linux vs. Solaris, Stéphane Casset, 29-Apr-98. [BYTE UNIX Benchmarks éalisé sur UltraSparc].
5. Linux ou FreeBSD, Louis Léon et Patrick Rougeau, Le Micro Bulletin, No 68, pp.54-57, mars/avril 1997. (avec une note de Ge'rard Gachelin).
6. lmbench: Portable tools for performance analysis, Larry McVoy, and Carl Staelin, 1996 USENIX Annual Technical Conference, San Diego (CA), Janvier 1996. http://reality.sgi.com/lm_engr/lmbench/lmbench.html http://reality.sgi.com/lm_engr/lmbench/lmbench-usenix.ps
7. The Measured Performance of Personal Computer Operating Systems, J. Bradley Chen, Yasuhiro Endo, Kee Chan, David Mazieres, Antonio Dias, Margo Seltzer, and Michael Smith, Harvard Univsersity, August 1995 http://www.eecs.harvard.edu/~yaz/p5.html - 15th ACM Symposium on Operating System Principles http://www.eecs.harvard.edu/~yaz/p5/sosp95.ps - ACM Transaction of Computer Systems, Fe'vrier 1996. http://www.eecs.harvard.edu/~yaz/p5/tocs96.ps
8. Linux is faster than SunOS and Solaris on Sun's own hardware http://www.cs.uml.edu/~acahalan/ and http://www.caip.rutgers.edu/~davem/scoreboard.html
9. lmbench on Linux/alpha, Linus Torvalds, Octobre 1995 transaction re'sumant les re'sultats de Larry McVoy. http://www.uwsg.indiana.edu/hypermail/linux/alpha/9510/0129.html
10. Unices are created equal, but ..., Discussion de Forum a` propos des bancs d'essai d'Unix, http://www.uwsg.indiana.edu/hypermail/linux/kernel/9604/index.html#1264
11. Native Linux on the PowerPC - Spreading the Disease, Cort Dougan, Fe'vrier 1997, Linux Journal, Mai 1997. http://www.cs.nmt.edu/~cort/linuxppc/article.dvi (postscript http://pauillac.inria.fr/~lang/hotlist/free/bench/powerpc.ps), Mesures plus comple`tes: http://www.cs.nmt.edu/~cort/linuxppc/numbers.html.
12. cc and gcc Performance Comparison on DEC Alpha, Björn Tiemann, Aou^t 1995, http://www.ife.ee.ethz.ch/~tiemann/performance.html avec commentaires http://www.uwsg.indiana.edu/hypermail/linux/alpha/9508.2/0018.html et contexte (see HPCN Europe 1995 Paper) http://www.ife.ee.ethz.ch/music/alpha/alpha.html
13. The CamlStone benchmark: a benchmark of the CAML code performance on various plateforms, using different compilers. This may also be seen as a comparative benchmark for the compilers used on the same plateform, or for the different plateforms. http://pauillac.inria.fr/~doligez/camlstone.txt
14. Porting UNIX to Windows NT David G. Korn, USENIX 1997 Conference Anaheim, CA, January 1997. Contains various remarks comparing performance and usability of Linux and NT, and some benchmark data.
15. OS Holy Wars, Sean Fulton, InternetWeek, September 1, 1997. Comparing several x86 OS as Web server platforms
16. Serveurs réseau : Linux une bonne affaire ?, Christian Claveleira, CRU, septembre 1997.
17. Management of Information Systems, Jean-Paul Smets, 26 february 1998. cost analysis of various solutions, some based on free software.
18. Lessons Learned by Use of (C)OTS, Rainer Gerlich, DASIA 98', Data Systems in Aerospace, May 25 - 28, 1998 ... It also compares COTS with non-commercial Off-the-Shelf Software (called OTS here). For the reference project Linux as OTS turned out as rather stable and suitable for a commercial, fault-tolerant system while some COTS products caused some crashes. By introduction of clear interfaces and use of standards the project could survive. - zipped postscript
19. Samba up-tempo performer Pankaj Chowdhry, PC Week, March 14, 1999. - Samba 2.0 for the Irix platform
20. Web Server Comparisons, ACME Labs Webmaster, February 17, 2000. - Some comparisons between various web servers. We look at features, sizes, and performance.

Opinions comparing Unix and Windows NT

1. SWePIX - Swiss Web Performance Index, Sysformance, 03-oct-00. - IIS crashes 3 times more often than Apache for professional servers.
2. Robustness Testing of the Microsoft Win32 API, Charles Shelton, Philip Koopman & Kobey DeVale, Dependable Systems and Networks Conference 2000/FTCS-30, New York City, June 26-28 2000. - full paper in pdf - other publications from P. Koopman and Ballista project.
3. UNIX ou NT ? Vos responsables informatiques ont-ils fait leur travail ?, Lettre ouverte de l'AFUU aux dirigeants d'entreprises - UNIX or NT ? Did your IT departments do their job ?, AFUU open letter to enterprise managers, Alain Cohen, Révision 9/2/99.
4. Windows NT et Linux sur la machine Pentium Pro ( ascii, rtf -- best version is rtf, both html and ascii have to be improved ), Louis Léon et Patrick Rougeau, Le Micro Bulletin, No 68, pp.8-18, mars/avril 1997.
   Voir également : http://www.dsi.cnrs.fr/~lmb/ServeurLeon/HomePage.html.
   Re'sultats repris dans http://www.mygale.org/00/prozion/dedmusi.htm.
5. Tests and Comparisons by Eric Carr, Sm@rt Reseller, April 5, 1999. -
   • The New King Of The Hill Our main point, though, is that Linux is as valid a choice as NetWare for your customers' servers. What it lacks in features and support is more than made up in speed.
   • Our Test Environment
6. Samba 2.0: A License To Kill NT? Steven J. Vaughan-Nichols and Eric Carr, Sm@rt Reseller, March 22, 1999. -
7. Tests and Comparisons by Steven J. Vaughan-Nichols and Eric Carr, Sm@rt Reseller, January 25, 1999. -
   • Linux Up Close: Time To Switch there are significant differences among the three commercial Linux releases: Caldera Systems Inc.'s OpenLinux 1.3, Red Hat Software Inc.'s Red Hat Linux 5.2 and SuSE Inc.'s SuSE Linux 5.3.
   • The Best Windows File Server: Linux!
   • Linux Is The Web Server's Choice
   • How They Stack Up
   • Linux remplaçant Windows NT How to ... replace NT by Linux+Samba
8. Microsoft Windows NT Server 4.0 versus UNIX, John Kirch, 18 March 1998 [ other source 5 April 1998]
9. Comparaison Microsoft Windows NT serveur 4.0 --- Unix, John Kirch, traduit par Sébastien Blondeel, version avec commentaires et abondante bibliographie.
10. Replacing Windows NT Server with Linux, Quinn P. Coldiron, 1997 (mars 1998).
11. The Great Linux-vs-NT Debate Jim Mohr, Jimmo Doc Services, 06 May 1998. ... a long comparison between Linux and Windows NT, encompassing a variety of technical aspects: configurability, compatibility, security, MM interface, ...
12. HP hedges its bets with Windows NT, Uwe Harms, Primeur, August 1997.
13. A Mainframe on Your Desktop... Today, Jesse Berst, ZDNet AnchorDesk, February 02, 1998. [NT as the ultimate workstation ... the interesting part is the reactions from readers.]
14. The fog of war Peter Fabris, CIO, July 15 1998. - News from the Unix-NT front ... ignoring the open-source troops, as usual
15. Integrated System Design: Linux and NT for EDA
    • Linux Wins the Shootout with Windows NT Jonah McLeod, Integrated System Design, August 1998. - 'Linux bigots' slam NT and Microsoft and criticize EDA vendors. The vendors say that they see no demand for Linux and tell them to take their fight to their management.
    • Linux vs. Windows NT: Engineers Speak Out, Part 2 Murry Shohat, Integrated System Design, August 1998. - Why Most Engineers Insist on Unix." Here are some representative responses from readers. - Readers complain of the poorer reliability of NT compared with Unix
    • Linux Vs. NT Showdown ISD, Integrated System Design, June 16, 1998 The panel on Tuesday June 17 at the Design Automation Conference in San Francisco explored the benefits of Linux, NT, and Solaris.
      Linux vs. NT Shootout Transcript
    • Why Most Engineers Insist on Unix Jonah McLeod, ISD, Integrated System Design, April 1998. - Unlike many managers, engineers generally oppose designing on Windows NT. Their reasons are ease of use and, more important, reliability. Is Linux a reasonable alternative?
    • More on Unix vs. Windows Dan Pinvidic, FEEDBACK, ISD March 1998.
    • FEEDBACK Brad Martin, FEEDBACK, ISD, December 1997. A design house details why it's found that Windows NT isn't as effective as Unix as an EDA platform.
    • Focus Report: Windows EDA Tools Carolyn Mathas, Integrated System Design, Focus Report, October 1997. - EDA software for Windows NT is moving from point tools to full solutions. Soon, some say, it will win out over Unix.
16. Windows NT face à Linux/Unix... et à lui-même Jerome Kalifa, July 31 1998. - Survey of Unix vs NT articles, with a good bibliography.
17. Linux vs. The Other Operating Systems, Definite Linux Systems, July 21, 1998 - Features comparison for Windows/NT 4, OS/2, NetWare 4.1, Linux
18. Operating Systems Comparison Chart Caldera OpenLinux, SCO Unixware, Windows NT 4.0
19. NT vs. Unix: Same old arguments arise at Comdex Enterprise Tom Diederich, Computerworld, Online News, September 11/98. - Can Windows NT really scale? It depends on whom you talk to... Microsoft Corp. and Sun Microsystems, Inc.
20. VAR Resuscitates NT Server -- Microserver Solution Solves Heart Clinic's Network Overload Herman Mehling, Computer Reseller News, section Small Business (CMP net) September 21, 1998, Issue: 808. - When Florida Heart Group got tired of very slow E-mail and Internet performance on its Windows NT machines, the clinic sought help. The VAR recommended the Qube 2700WG microserver, a thin server from Cobalt Networks Inc., Mountain View, Calif. It comes preconfigured with Internet and intranet applications for communication and collaboration, such as Web publishing, E-mail, cross-platform file services, discussion groups and HTML page generation and editing. It also features document indexing, searching, archival and retrieval. The box uses the Linux 2.0 operating system. Its suggested price ranges from $1,449 to $2,449, depending on configuration. Installation was done in less than an hour ...
21. The HOTMAIL story
    • Microsoft's Next Ambition: The Net's Backend James Glave, Wired News, 16.Jan.98. - Sources say that Microsoft is mandating Web technology companies it has purchased - such as Hotmail - and those it partners with, such as Inktomi, which will build a search engine for MSN - to abandon their Unix-based Web servers and operating systems. In their place, Microsoft is reportedly suggesting those sites install Intel-based hardware running the company's Microsoft Internet Information Server (IIS) or the Windows NT Server.
    • Hotmail dans la hotte de Bill Gates PIERRE GRUMBERG , Le Monde Informatique, n°748 - 9 janvier 199 Cette mise sous tutelle ne devrait pas pour autant présager d'une réforme fondamentale du service, tout au plus d'un passage d'Unix à NT, qui, selon Microsoft, était de toute manière programmé.
    • Solaris calls Hotmail shots for Microsoft , Network News, (VNU Business Publications Ltd., vnu/net.com), 22-April -1998. - The software giant has attempted to exchange the Sun/Solaris infrastructure of Hotmail with NT since buying it in December 1997. However, the demands of supporting 10 million users reportedly proved too great for NT, and Solaris was reinstated. (other link)
    • Microsoft Windows NT Server 4.0 versus UNIX John Kirch, 24 September 1998. - This free Web-based e-mail service runs a mixture of Sun Solaris and FreeBSD. Apache 1.2.1 is the Web server software. After Microsoft purchased the company in December 1997, they tried to migrate to NT, but ". . . the demands of supporting 10 million users reportedly proved too great for NT, and Solaris was reinstated." Get the full story: Solaris calls Hotmail shots for Microsoft.
      "The software giant has attempted to exchange the Sun/Solaris infrastructure of Hotmail with NT since buying it in December 1997. However, the demands of supporting 10 million users reportedly proved too great for NT, and Solaris was reinstated."
    • Engineering Opportunities, hotmail, seen on September 24, 1998. - Job Description: Analyze, write, and maintain code used by operations to manage the Hotmail Web site - - Education/Experience: Seven years experience with Perl and C++, experience and knowledge with UNIX and UNIX-based servers.
    • No Windows But Lots of Sun for Microsoft's Hotmail Tom Abate, Digital Bay, San Francisco Chronicle, Wednesday, May 20, 1998, page B3. - Microsoft will have to swallow its pride for a year and rely on Sun software. - printer friendly
22. Unix trounces Windows NT in testing , Stephen Shankland , CNET News.com, December 1, 1998. - Microsoft's Windows NT finished dead last overall in a comparison with five different versions of the Unix operating system, concluded a market research firm that assessed the latest versions of these operating system - printer friendly