[AFUL] Livre: La bataille du logiciel libre Livre: Libres enfants du savoir numérique Livre: Liberté Egalité Business [No Software Patents] [Free Speech Online] [Privacy Golden Key]

Benchmarking Freeware

Variuos people and organizations have developed benchmarks and other measurement techniques, and applied them to freeware, and often to the comparison of freeware and payware. This page is a compendium of all such studies we have been abble to find on the internet and elsewhere, or of papers reporting indirectly the results of these studies.

When reading these, remember that all benchmarks are to be taken with a pinch of salt, since they often depend also on factors that are not necessarily made explicit, or controlled by the experimenter. The dates for these benchmarks and comparative studies is also important, since some software evolve fast in quality or performance.

If you are aware of other analyses/benchmarks/studies of freeware, concerning performance, reliability, usability, maintainability, portability or any other quality factor, please forward this information to the maintainer of this page Bernard.Lang@inria.fr. Note that all types of freeware are relevant, though what I have been able to find so far is mostly concerned with operating systems, and compilers.

Papers about freeware dissemination, or any other relevant statistical data are welcome too, though that will be kept on another page. Thank you for helping.


  1. * Making Programs Explode: Using Simple Random Testing on Real Programs, Barton P. Miller, University of Wisconsin.
  2. Fuzz Revisited: A re-examination of the Reliability of UNIX Utilities and Services, B.P. Miller et al., University of Wisconsin, 1995. ftp://grilled.cs.wisc.edu/technical_papers/fuzz-revisited.ps.Z. (Version A4 http://pauillac.inria.fr/~lang/hotlist/free/bench/fuzz-revisited-a4.ps)
  3. A Performance Comparison of UNIX Operating Systems on the Pentium, Kevin Lai et Mary Baker, C.R. 1996 Usenix Technical Conference, Janvier 1996. http://mosquitonet.Stanford.EDU/~laik/benchmarks/index.html
  4. * Benchmark Linux vs. Solaris, Stéphane Casset, 29-Apr-98. [BYTE UNIX Benchmarks éalisé sur UltraSparc].
  5. Linux ou FreeBSD, Louis Léon et Patrick Rougeau, Le Micro Bulletin, No 68, pp.54-57, mars/avril 1997. (avec une note de Ge'rard Gachelin).
  6. lmbench: Portable tools for performance analysis, Larry McVoy, and Carl Staelin, 1996 USENIX Annual Technical Conference, San Diego (CA), Janvier 1996. http://reality.sgi.com/lm_engr/lmbench/lmbench.html http://reality.sgi.com/lm_engr/lmbench/lmbench-usenix.ps
  7. The Measured Performance of Personal Computer Operating Systems, J. Bradley Chen, Yasuhiro Endo, Kee Chan, David Mazieres, Antonio Dias, Margo Seltzer, and Michael Smith, Harvard Univsersity, August 1995 http://www.eecs.harvard.edu/~yaz/p5.html - 15th ACM Symposium on Operating System Principles http://www.eecs.harvard.edu/~yaz/p5/sosp95.ps - ACM Transaction of Computer Systems, Fe'vrier 1996. http://www.eecs.harvard.edu/~yaz/p5/tocs96.ps
  8. Linux is faster than SunOS and Solaris on Sun's own hardware http://www.cs.uml.edu/~acahalan/ and http://www.caip.rutgers.edu/~davem/scoreboard.html
  9. lmbench on Linux/alpha, Linus Torvalds, Octobre 1995 transaction re'sumant les re'sultats de Larry McVoy. http://www.uwsg.indiana.edu/hypermail/linux/alpha/9510/0129.html
  10. Unices are created equal, but ..., Discussion de Forum a` propos des bancs d'essai d'Unix, http://www.uwsg.indiana.edu/hypermail/linux/kernel/9604/index.html#1264
  11. Native Linux on the PowerPC - Spreading the Disease, Cort Dougan, Fe'vrier 1997, Linux Journal, Mai 1997. http://www.cs.nmt.edu/~cort/linuxppc/article.dvi (postscript http://pauillac.inria.fr/~lang/hotlist/free/bench/powerpc.ps), Mesures plus comple`tes: http://www.cs.nmt.edu/~cort/linuxppc/numbers.html.
  12. cc and gcc Performance Comparison on DEC Alpha, Björn Tiemann, Aou^t 1995, http://www.ife.ee.ethz.ch/~tiemann/performance.html avec commentaires http://www.uwsg.indiana.edu/hypermail/linux/alpha/9508.2/0018.html et contexte (see HPCN Europe 1995 Paper) http://www.ife.ee.ethz.ch/music/alpha/alpha.html
  13. The CamlStone benchmark: a benchmark of the CAML code performance on various plateforms, using different compilers. This may also be seen as a comparative benchmark for the compilers used on the same plateform, or for the different plateforms. http://pauillac.inria.fr/~doligez/camlstone.txt
  14. Porting UNIX to Windows NT David G. Korn, USENIX 1997 Conference Anaheim, CA, January 1997. Contains various remarks comparing performance and usability of Linux and NT, and some benchmark data.
  15. * OS Holy Wars, Sean Fulton, InternetWeek, September 1, 1997. Comparing several x86 OS as Web server platforms
  16. * Serveurs réseau : Linux une bonne affaire ?, Christian Claveleira, CRU, septembre 1997.
  17. Management of Information Systems, Jean-Paul Smets, 26 february 1998. cost analysis of various solutions, some based on free software.
  18. Lessons Learned by Use of (C)OTS, Rainer Gerlich, DASIA 98', Data Systems in Aerospace, May 25 - 28, 1998 ... It also compares COTS with non-commercial Off-the-Shelf Software (called OTS here). For the reference project Linux as OTS turned out as rather stable and suitable for a commercial, fault-tolerant system while some COTS products caused some crashes. By introduction of clear interfaces and use of standards the project could survive. - zipped postscript
  19. * Samba up-tempo performer Pankaj Chowdhry, PC Week, March 14, 1999. - Samba 2.0 for the Irix platform
  20. Web Server Comparisons, ACME Labs Webmaster, February 17, 2000. - Some comparisons between various web servers. We look at features, sizes, and performance.

Opinions comparing Unix and Windows NT

  1. SWePIX - Swiss Web Performance Index, Sysformance, 03-oct-00. - IIS crashes 3 times more often than Apache for professional servers.
  2. Robustness Testing of the Microsoft Win32 API, Charles Shelton, Philip Koopman & Kobey DeVale, Dependable Systems and Networks Conference 2000/FTCS-30, New York City, June 26-28 2000. - full paper in pdf - other publications from P. Koopman and Ballista project.
  3. UNIX ou NT ? Vos responsables informatiques ont-ils fait leur travail ?, Lettre ouverte de l'AFUU aux dirigeants d'entreprises - UNIX or NT ? Did your IT departments do their job ?, AFUU open letter to enterprise managers, Alain Cohen, Révision 9/2/99.
  4. * Windows NT et Linux sur la machine Pentium Pro (* ascii, * rtf -- best version is rtf, both html and ascii have to be improved ), Louis Léon et Patrick Rougeau, Le Micro Bulletin, No 68, pp.8-18, mars/avril 1997.
    Voir également : http://www.dsi.cnrs.fr/~lmb/ServeurLeon/HomePage.html.
    Re'sultats repris dans http://www.mygale.org/00/prozion/dedmusi.htm.
  5. Tests and Comparisons by Eric Carr, Sm@rt Reseller, April 5, 1999. -
  6. * Samba 2.0: A License To Kill NT? Steven J. Vaughan-Nichols and Eric Carr, Sm@rt Reseller, March 22, 1999. -
  7. Tests and Comparisons by Steven J. Vaughan-Nichols and Eric Carr, Sm@rt Reseller, January 25, 1999. -
  8. * Microsoft Windows NT Server 4.0 versus UNIX, John Kirch, 18 March 1998 [* other source 5 April 1998]
  9. Comparaison Microsoft Windows NT serveur 4.0 --- Unix, John Kirch, traduit par Sébastien Blondeel, version avec commentaires et abondante bibliographie.
  10. * Replacing Windows NT Server with Linux, Quinn P. Coldiron, 1997 (mars 1998).
  11. The Great Linux-vs-NT Debate Jim Mohr, Jimmo Doc Services, 06 May 1998. ... a long comparison between Linux and Windows NT, encompassing a variety of technical aspects: configurability, compatibility, security, MM interface, ...
  12. * HP hedges its bets with Windows NT, Uwe Harms, Primeur, August 1997.
  13. * A Mainframe on Your Desktop... Today, Jesse Berst, ZDNet AnchorDesk, February 02, 1998. [NT as the ultimate workstation ... the interesting part is the reactions from readers.]
  14. The fog of war Peter Fabris, CIO, July 15 1998. - News from the Unix-NT front ... ignoring the open-source troops, as usual
  15. Integrated System Design: Linux and NT for EDA
  16. Windows NT face à Linux/Unix... et à lui-même Jerome Kalifa, July 31 1998. - Survey of Unix vs NT articles, with a good bibliography.
  17. Linux vs. The Other Operating Systems, Definite Linux Systems, July 21, 1998 - Features comparison for Windows/NT 4, OS/2, NetWare 4.1, Linux
  18. Operating Systems Comparison Chart Caldera OpenLinux, SCO Unixware, Windows NT 4.0
  19. NT vs. Unix: Same old arguments arise at Comdex Enterprise Tom Diederich, Computerworld, Online News, September 11/98. - Can Windows NT really scale? It depends on whom you talk to... Microsoft Corp. and Sun Microsystems, Inc.
  20. VAR Resuscitates NT Server -- Microserver Solution Solves Heart Clinic's Network Overload Herman Mehling, Computer Reseller News, section Small Business (CMP net) September 21, 1998, Issue: 808. - When Florida Heart Group got tired of very slow E-mail and Internet performance on its Windows NT machines, the clinic sought help. The VAR recommended the Qube 2700WG microserver, a thin server from Cobalt Networks Inc., Mountain View, Calif. It comes preconfigured with Internet and intranet applications for communication and collaboration, such as Web publishing, E-mail, cross-platform file services, discussion groups and HTML page generation and editing. It also features document indexing, searching, archival and retrieval. The box uses the Linux 2.0 operating system. Its suggested price ranges from $1,449 to $2,449, depending on configuration. Installation was done in less than an hour ...
  21. The HOTMAIL story
    • Microsoft's Next Ambition: The Net's Backend James Glave, Wired News, 16.Jan.98. - Sources say that Microsoft is mandating Web technology companies it has purchased - such as Hotmail - and those it partners with, such as Inktomi, which will build a search engine for MSN - to abandon their Unix-based Web servers and operating systems. In their place, Microsoft is reportedly suggesting those sites install Intel-based hardware running the company's Microsoft Internet Information Server (IIS) or the Windows NT Server.
    • Hotmail dans la hotte de Bill Gates PIERRE GRUMBERG , Le Monde Informatique, n°748 - 9 janvier 199 Cette mise sous tutelle ne devrait pas pour autant présager d'une réforme fondamentale du service, tout au plus d'un passage d'Unix à NT, qui, selon Microsoft, était de toute manière programmé.
    • Solaris calls Hotmail shots for Microsoft , Network News, (VNU Business Publications Ltd., vnu/net.com), 22-April -1998. - The software giant has attempted to exchange the Sun/Solaris infrastructure of Hotmail with NT since buying it in December 1997. However, the demands of supporting 10 million users reportedly proved too great for NT, and Solaris was reinstated. (other link)
    • Microsoft Windows NT Server 4.0 versus UNIX John Kirch, 24 September 1998. - This free Web-based e-mail service runs a mixture of Sun Solaris and FreeBSD. Apache 1.2.1 is the Web server software. After Microsoft purchased the company in December 1997, they tried to migrate to NT, but ". . . the demands of supporting 10 million users reportedly proved too great for NT, and Solaris was reinstated." Get the full story: Solaris calls Hotmail shots for Microsoft.
      "The software giant has attempted to exchange the Sun/Solaris infrastructure of Hotmail with NT since buying it in December 1997. However, the demands of supporting 10 million users reportedly proved too great for NT, and Solaris was reinstated."
    • Engineering Opportunities, hotmail, seen on September 24, 1998. - Job Description: Analyze, write, and maintain code used by operations to manage the Hotmail Web site - - Education/Experience: Seven years experience with Perl and C++, experience and knowledge with UNIX and UNIX-based servers.
    • No Windows But Lots of Sun for Microsoft's Hotmail Tom Abate, Digital Bay, San Francisco Chronicle, Wednesday, May 20, 1998, page B3. - Microsoft will have to swallow its pride for a year and rely on Sun software. - printer friendly
  22. Unix trounces Windows NT in testing , Stephen Shankland , CNET News.com, December 1, 1998. - Microsoft's Windows NT finished dead last overall in a comparison with five different versions of the Unix operating system, concluded a market research firm that assessed the latest versions of these operating system - printer friendly

Maintenance and support

  1. Microsoft says email patch flawed "J Edgar Hoover" , Bugtraq List , 29 Jul 1998. - Microsoft has warned that a patch posted yesterday intended to fix a security hole in its Outlook Express email program does not fix a related problem.

Business Figures

  • NT leads OS market by units, Unix by dollars By William Fellows, Computergram, 14 Jan 1999.
    benchmarks - licences - usage - abuse - text - advocacy - shareware - freeware sources - main - français - papers

    [HOME] Modifié le: Friday, February 9, 2001 : Last modified
    S.V.P. Veuillez envoyer commentaires & contributions à Bernard.Lang@inria.fr,
    Please: report bugs, send comments & contributions to Bernard.Lang@inria.fr,
    Original URL of this page: http://bat8.inria.fr/~lang/hotlist/free/bench/index.html

    Le contenu de ces pages relève de la seule responsabilité de leurs auteurs, et ne représente pas le point de vue officiel de l'INRIA. Les informations sont présentées de bonne foi, mais leur exactitude ne saurait être garantie.
    All documents are under the responsibility of their authors, and do not represent the official views of INRIA. The information is given in good faith, but its accuracy cannot be guaranteed.

    Powered by Apache