First Monday

Open Source Software Development as a Special Type of Academic Research

Eric Raymond's bazaar model provides a too simplistic view of the open source software (OSS) development process. This paper tries to explore links between open source software development and academic research as a better paradigm for OSS development. Open source software development should better be viewed as a special case of academic research. Viewing OSS this way probably can lead to a better understanding of open source phenomena.

Contents

Introduction
"Cathedral and Bazaar" Postulates
Open Source Development as a Special Case of Applied Science
Open Source Problems and Limitations
Conflicts Among Open Source Developers
Pseudoproblems
Conclusion

Introduction

"[Open Source] programming is like sex, one mistake and you have to support it for the rest of your life."
M. Sinz, CBM Inc.

The nature of open source development fascinates me - that's why I have become an observer and researcher on the open source process in addition to being a participant. At the same time during my tenure as an editor-in-chief of the Softpanorama Bulletin and my teaching career, I have become painfully aware of the overly optimistic and unrealistic views on open source held by students and partially even participants like Eric Raymond.

There are several different reasons for this distortion of reality. Open source development is now fashionable and it makes big news. The news too often emphasize achievements and successful projects, but fails to address difficulties, failures and aborted projects. Failures are not news, in most cases. That leaves many with the impression that open source is a panacea and a "magic bullet" that will solve all, or almost all, difficulties. This paper is a skeptical treatment of the subject. I respect open source and believe that it is optimal in teaching, but I see a lot of problems. I am not concerned that the skepticism and frankness of this paper will discourage many open source developers. The need to program is an inherent feature of some, much like a cow needs to give milk or authors need to write. One critical paper will not force a true open source enthusiast to give it up. This paper will, however, forewarn open source developers about the problems of working on open source projects, and reduce some of the frustrations.

I believe that there are serious problems that are inherent to the open source development model. They need to be discussed and understood and the best way to understand them is to use an analogy between open source and scientific communities (actually these communities overlap). In addition, this paper touches on several different problems of OSS that have intrigued me for the last couple of years.

Now about skeptical part of the paper. Starting with his famous paper "Cathedral and Bazaar" Eric Raymond published a series of articles (see especially his comments on the so-called Halloween documents) he promoted an overoptimistic and simplistic view of open source, as a variant of socialist (or, to be more exact, vulgar Marxist) interpretation of software development.

This paper is to a certain extent a reaction to Raymond's "Embrace and Extend" policy towards Linux and the software that runs on it, his attempt to represent it as a single (and simple) phenomenon and call it "open source". Especially unpleasant is his deriding of the Free Software Foundation (FSF). Without FSF Linux would have been just impossible. As for the Open Source Initiative - "open source" is a nice term and I prefer it - but it's just a nice term. The recent interest in open source happened because of Linux, not because the "open source" moniker became fashionable. See, for example, the comments at "Shut Up And Show Them The Code" and especially the remarks by Leandro Dutra.

"Cathedral and Bazaar" Postulates

"For every complex problem, there is an answer that is short, simple and wrong."
H. L. Mencken

"Cathedral and Bazaar" series of papers (and especially ESR comments on so-called Halloween documents) has some implicit postulates (here we use the term "postulate" to mean an underlying basic assumption like in Euclid geometry) that present "open source" as a magic solution:

Open Source Development as a Special Case of Applied Science

"If you keep proving stuff that others have done, getting confidence, increasing the complexities of your solutions - for the fun of it - then one day you'll turn around and discover that nobody actually did that one! And that's the way to become a computer scientist."
Richard Feynmann.

First of all I would like to stress that the Internet can significantly reduce the costs of providing some types of software like OS, compilers or utilities. The Internet makes it possible to produce an infinite number of remotely accessible perfect copies of a computer program, multimedia presentations, or interesting e-mail discussions.

It is generally accepted that the fact that the cost of duplicating a computer program is close to zero creates important differences between computer programs and other consumer products. These differences are to a certain extent ignored or suppressed in the conventional "shrink-wrap" software distribution model. I think that the creation of a program is similar to the creation of applied theory. I would like to classify programming as a special kind, or at least a close relative, of scientific activities. The second important difference between software and other consumer products is that minor modifications are easy. Software can be more adaptable. Here again the important analogy with science holds.

In both science and programming,those involved aren't in it for the money. Most of the OSS developers are doing it to chase a dream, not to build up their bank balances. They can be motivated for a variety of reasons but simply there are many with great programming abilities that often are underutilized in their current (often corporate) environments. Some want a particular tool and found that is either unavailable or too expensive so they try to build one themselves. Again they are chasing a dream, not the competition.

Due to the Internet, it is now possible to create a virtual team for software development, parallel to the interests in science where several researchers interested in a particular phenomenon create a virtual scientific community to resolve a question or problem.

Team structure and responsibilities can be a dynamic process. Software will be produced not by an isolated person or a group of persons, but by a deeply interwoven network of actors. I would like to stress that virtual teams probably resemble historical informal communities of scientists that used hand-written letters to share achievements, ideas and criticism.

Yet little is known about how Internet-based virtual teams (IVT) really operate and what problems develop in that sort of cooperation. Some evident problems are:

There are several other visible problems with open source projects that were pointed out by others. Understanding these problems probably can help current and future open source projects. Again, I strongly believe that the problems of open source are by and large the same as those that confront academic culture and are better understood in this context. From a theoretical point of view, participants in an open source project should probably be considered as a special kind of academic workers. Solutions for typical problems developed in academic community are directly applicable to open source and its use can probably provide important benefits to the open source development model.

The financing of open source projects is very similar to the financing of applied science. Sometimes scientific research is funded indirectly, because individuals employed in a particular institution become interested in a particular phenomenon and research with existing funds. A considerable number of open source software developers work either in academic institutions or large corporations. Large corporations usually provide a very good environment for open source projects as they often contain niches where gifted developers are partially unemployed or underemployed due to various internal issues. The initial development of Unix at Bell Labs is a good example of such possibilities.

Recently several large corporations used a second avenue of financing of open source projects. When a given open source project can promote a strategically important hardware component, programmers can be assigned to it. This activity becomes a standard "loss leader". Digital contributions to Linux seems to belong to this scheme. Current Intel and IBM interest in Linux seems to fall into this category.

Linux distributors and IBM activity on Apache are examples of a third way. In these cases, companies try to improve on a product that they sell or improve their position against Microsoft. This is also very close to applied science paradigm where businesses take research results and commercialize them.

Open Source Problems and Limitations

"Experiences is a wonderful thing, it enables you to recognize a mistake when you make it again."
jdstone@ingr.com

Open source is interesting and important phenomenon, but like science it has its own set of problems that one needs to understand. For each free/open sourced software package we have a developer or group of developers committed to making that package the best it can be. If the product has a bug, the developer responsible for it wants it to not have bugs. But the reality is not that simple. Below is my a limited and incomplete attempt at classifying open source problems.

Does open source really provide a rapid development environment?

"Open source does work, but it is most definitely not a panacea ... Software is hard. The issues aren't that simple."
Jamie Zawinski

Suspicions were voiced in the "Halloween I" memo indicating that the re-creation of existing systems and protocols could be achieved faster by using the Internet. For other kinds of projects it could be slower than traditional development processes. A known target could be used as a testbed for checking progress; it could also provide parallelism that could be beneficial for Internet-connected communities of developers.

When programming work requires everyone to develop on a single path the pace of open source development can became as slow or even slower than traditional projects. In these cases there is no implicit advantage in using the Internet. A key factor often is the personality of the leading developer. For example, TeX was created without the medium of the Internet; I doubt that the Internet or an Internet-based community of developers could have made any difference.

At the same time, a large community inhibits innovation, so a given project may stagnate. As Jamie Zawinski put it:

"There exist counterexamples to this, but in general, great things are accomplished by small groups of people who are driven, who have unity of purpose. The more people involved, the slower and stupider their union is."

But there is one more problem with speed and open source development. I feel that if speed is really important then authoritarian methods have distinct advantages over democratic methods. "Speed kills" and the first victim is democracy (first noted by Frederick Brooks in his analysis of OS/360 development). A purely authoritarian style will lead to creation of project elite and strict hierarchy that sooner or later will kill any given open source project. Therefore any attempt to speed up an existing OSS project beyond certain limits could lead to unforeseen consequences including changes in the project social structure. Competition with Microsoft (as promoted by some Linux evangelists) is a dangerous threat to the open source movement as a whole. In order to compete with an authoritarian organization speed of delivery will be a matter of survival.

Small projects do not need explicit coordination structures. Coordination is usually automatically performed by the leader of a project or a group of the leaders. This approach does not scale well. For large projects an authoritarian model is the only choice if speed of development is a major issue. If speed is perceived as an important project goal the community of developers consciously or unconsciously will act by increasing the authoritarian tendencies of their leader. Eventually this can lead to problems.

As Jordan Hubbard put it "Despite what some free-software advocates may erroneously claim from time to time, centralized development models like the FreeBSD Project's are hardly obsolete or ineffective in the world of free software. A careful examination of the success of reputedly anarchistic or "bazaar" development models often reveals some fairly significant degrees of centralization that are still very much a part of their development process."

Also it depends whether a project has a single, clear direction (e.g. recreate UNIX, implement HTTP protocol) that can be efficiently communicated and acted upon by a group.

Generally for any rapidly evolving OSS project the level of centralization is high. This need of centralization may be perceived as a "cult of personality", where one developer has a tremendous amount of authority (for example, Linus Torvalds and Linux). This in turn can create a high level of discontent sometimes splitting a project into two of more competing development tracks. Examples are NetBSD/OpenBSD, Emacs/Xemacs, and gcc/egcs. If Linus Torvalds would drop too many important patches and be perceived as a bottleneck for all Linux kernel development, you could see a splitting Linux kernel development in the future.

Open source generally emphasizes quality and simplicity, not speed of development. An emphasis on quality as a project goal actually improve the chances of a project surviving in the long run.

The Town council effect or the effect of "The committee for the administration of the structural planning of the Linux kernel"

"Show me the source."
Linus Torvalds

Developers are not creating programs in isolation. Users have the final word and users with e-mail can play an important part in the process. E-mail can often take the form of flames or operate to improve status in the movement by creating some benefit to a particular bureaucratic superstructure, the "town council effect." Alan Cox coined both the term "town council effect" and the phrase "The committee for the administration of the structural planning of the Linux kernel." In his paper "Cathedrals, Bazaars and the Town Council" Alan Cox wrote (italics are mine):

"The problem that started to arise was the arrival of a lot of (mostly well meaning) and dangerously half clued people with opinions -- not code, opinions. They knew enough to know how it should be written but most of them couldn't write "hello world" in C. So they argue for weeks about it and they vote about what compiler to use and whether to write one - a year after the project started using a perfectly adequate compiler. They were busy debating how to generate large model binaries while ignoring the kernel swapper design.

Linux 8086 went on, the real developers have many of the other list members in their kill files so they can communicate via the list and there are simply too many half clued people milling around. It ceased to be a bazaar model and turns into a core team, which to a lot of people is a polite word for a clique. It is an inevitable defensive position in the circumstances.

In the Linux case the user/programmer base grew slowly and it grew from a background group of people who did contribute code and either had a basis in the original Minix hacking community or learned a few things the hard way reboot by reboot. As the project grew people who would have turned into "The committee for the administration of the structural planning of the Linux kernel" instead got dropped in an environment where they were expected to deliver and where failure wasn't seen as a problem. To quote Linus "show me the source"."

Signal to noise ratio of e-mail conferences and supersized ego problems

"To succeed in the world, it is not enough to be stupid, you must also be well-mannered."
Voltaire

The content of messages on discussion groups might lead one to the wrong impression that the Internet is filled with junk and jerks. It is common for Internet inhabitants to complain bitterly about the lack of cooperation, decorum, and useful information. This is not completely true, but the signal-to-noise ratio is bad and getting worse.

A casual trip through cyberspace will turn up evidence of hostility, selfishness, and simple nonsense (much like a random walk in the real world will yield evidence of hostility, selfishness and nonsense). The recent developments over the open source trademark provide an excellent example. For many, it is much easier to be hostile in an e-mail discussion than face-to-face. That characteristic partially explains many flame wars that consume useful bandwidth of otherwise technical discussions.

Let me provide some examples. Fred Moody wrote:

"When I contacted one resolute Microsoft-hater, who works at what is probably the Internet's busiest Linux-using commercial site, he replied immediately via e-mail that he was willing to detail Linux's numerous shortcomings, but only anonymously. "I work with all these linux zealots who have nearly fired me over my pokings at linux."

Most of Linux's failings, according to my expert witness, are about what you would expect from a free operating system that exists in almost as many versions as there are people who use it. (Like Unix, Linux keeps mutating because the source code is available to anyone who wants to tinker with it.)

"Linux isn't secure and it isn't stable," my informant writes, with his usual bracing disdain for grammar and punctuation. "its a moving target that never really gets out of beta. sure people run production sites on linux. I know a lot of these people. they don't get much sleep and have grown opaque from the lack of sunlight. I have admin'd large Linux shops. they require huge amounts of admin overhead, and if you want shit to really work you are going to spend alot of time manually fixing things. the number of outstanding security holes and lack of stable functionality is monumental."

... Discussions of Linux's weaknesses can be found on several Web sites, along with programs used by hackers to attack Linux sites. To outsiders, many of the exchanges between devotees of BSD, Solaris or other Unix variations sound opaque or shrill: ("It will be a cold day at the equator before L. Torvalds sets aside his ego for the sake of someone else's better ideas.")."

There are diatribes against Free BSD; Larry McVoy assessed the situation this way:

"I'd like to point out that some BSD bigots (like myself) have abandoned BSD for Linux for reasons other than the purely technical ones. In particular, the BSD world is elitist, antagonistic, and uncooperative. You are either part of the in crowd or you are not and it makes a big difference. I'm someone that certainly has the credentials to be part of the "in crowd" but has rejected the invitations because I don't like organizations that play that game. Linux is a much nicer place to be.

Finally, Linux is covered by the GPL - BSD is not. If BSD ever gets popular again, the people running the show can, and will, take the source access away (try and get all the BSDi source, for example)."

Conflicts Among Open Source Developers

The open source model is a close relative to the scientific development model, with code treated as research results, and published for peer review and for the benefit of mankind. Problems are very similar in both disciplines over priority and the correctness of a vision that are as vital for developers of OSS as they are for scientific researchers. I've seen equivalent behavior in the scientific community. Normally such disputes are played out in a more mature manner, although no less vicious. Here is an anonymous example:

"What "normally" happens to an open source development project when people writing code for it don't get along? How do open source authors deal with conflicts with other open source authors? Let me relate the story about my first experience maintaining open source software, as I think it's a pretty good illustration of this. I used to work on a program (for the sake of this discussion, let's call it P, for Program) that I volunteered to take over development work on around two years ago, after the original author produced version 1.0, and had no more time to work on P.

At the time I took over maintenance and development for the package, I had lots of time free at the business where I worked. My boss was understanding enough to allow me to devote a few hours a week of paid time to P, which I believe doesn't happen often for open source authors.

As time passed, our business grew, and I had less and less time to work on P. I managed to release an alpha version with some improvements, but some people weren't pleased with the pace of development. One of them took it upon himself to start work on his own version of P. For brevity, I'll call him Mr. J, for Jerk.

Now, normally I'd cheer this, since it's what open source is all about. However, it seemed what Mr. J really wanted was acclaim. I had released version 2.0 beta of P, based on the 1.0 version from the original author. Mr. J released a version he called P 1.2, also based on the original 1.0 code, with his own modifications. This generated a lot of confusion, at least in my opinion, since he had the same name for his package as mine, and similar version numbers. So, I asked him to please change the name of the package to something else to clear things up.

His response was to publicly declare that he should be named the official maintainer of P, since I was taking too long. I should note that at this point I would have happily turned over development to him, if I'd thought he could do the job well. I didn't think he could (putting it mildly). I'm going to skip over the discussion we had on the subject on the P mailing list (on my mail server) since it includes a lot of childishness on the part of Mr. J. Suffice it to say that things ended up with Mr. J calling me a few names, and vowing to take over P as part of a larger project he was working on.

Mr. J proceeded to take a copy of the names on my mailing list, and create his own list. He declared his version to be the official version of P, and ceased to take part in the original list I had set up. After all the argument on the original list, I was glad he was gone.

That is, until someone posted a question about my version of P to his list (which he had subscribed me to as well).

He took the opportunity to publicly call me a few more names, and make some comments about my lack of progress. I had enough at this point, and mailed him telling him that I didn't want him to mention my name at all, ever again. I was hoping he'd start completely ignoring me, leaving me free to work on P quietly, without his interruptions.

Boy, was I wrong. Mr. J proceeded to mail me back and tell me that he would do whatever he pleased (again, putting it mildly). He also added a text description offering to let me perform an obscene act on him to his .sig file, which he used publicly on mailing lists and whatnot.

Completely appalled at this point, I e-mailed his providers for web space and connectivity, threatening them and him with lawsuits if he didn't remove my name from his postings. This got another nasty response from Mr. J, but eventually did get him to remove my name from his sig file.

This brings me to present time. I now have such bad feelings associated with the whole affair that I don't like to think about P, much less work on it. I've stopped working publicly on it, in fact, and I only do development on in-house versions that will never see the light of day."

The Problem of the "Lowest Hanging Fruit"

"Blessed are those who have no expectations, for they will never be disappointed."
Buddha

It seems that open source projects are more successful in domains that are directly or indirectly interesting to developers themselves. The growth of the Internet is making a larger spectrum of projects available. The initial period of an OSS project - building a more or less complete prototype by a single individual - tends to be more developer-oriented, which means more complex, but no less useful software may be thought of duller and less interesting. Those who can program naturally tend to work on programs they find personally interesting or programs that looks cool (editors, themes in Gnome), as opposed to applications considered dull. Without other incentives other than the joy of hacking and "vanity fair" a lot of worthwhile projects die because the initial author lost interest and nobody pick up the tag.

That tendency is probably positive. It does not make sense to develop OSS software outside a commercial context without fun. Programming without joy is a kind of slavery. OSS - viewed as a kind of research project - should concentrate on more personally interesting aspects of software and leave mundane tools development to commercial entities.

Not all projects are created equal. The most important feedback in a project directly deals with those parts of the code important to the developers, dealing with operating system, GUI and development tools. As for non-development-related software, the incentives can be much less and the quality and amount of feedback could be insufficient to keep a given project afloat.

Fragmentation and NIH syndrome

Let's start by quoting from the recent interview with Dennis M. Ritchie:

"LF: I cannot avoid making a comparison between you and all the people that is currently working on non-profit projects for free, just because they like it - although I am sure they wouldn't refuse money for the work they do for free. Can you see yourself involved in projects like Linux, or similar, if you were not at Bell Labs? How do you see all these people from inside an innovative research lab with many years of experience on your shoulders? Since our magazine is mainly for Linux users we cannot forget to ask you a questions about Linux. First of all, what is your opinion about all the Linux momentum, and the decision of many companies to start developing software for it (Bell Labs, for example: Inferno has its own port to Linux)?

Dennis: Let me put these questions together. I think the Linux phenomenon is quite delightful, because it draws so strongly on the basis that Unix provided. Linux seems to be the among the healthiest of the direct Unix derivatives, though there are also the various BSD systems as well as the more official offerings from the workstation and mainframe manufacturers. I can't help observing, of course, the "free source" Unix-derived world seems to be suffering from exactly the same kind of fragmentation and strife that occurred and is still occurring in the commercial world."

And this observation that "...the "free source" Unix-derived world seems to be suffering from exactly the same kind of fragmentation and strife that occurred and is still occurring in the commercial world" is a symptom of the fundamental problem - "the tragedy of commons". We already have a dozen of slightly incompatible Linux distributions with Debian and Red Hat as two major players with strong supporters and several second level players (Suse, Caldera, Mandrake, Slackware, etc.). I would like to say that in a sense Linux suffers from a multiple personality disorder. That means that an application written for Red Hat Linux might not run under Caldera's version because, for example, of different versions of libraries used.

As Jordan Hubbard put it "... it is a perfectly natural human desire to form clans and proudly wear the clan colors on Robert the Bruce Day, or whatever, diversity and competition being good things which encourage innovation and inspire people to greater heights of productivity. When you add Rampaging Egos to the mix, however, things get messy very quickly as the various clans decide that they want to be *the* clan, the only ones in line when the awards for best clan colors or most unique sporran are being handed out. Before long, rival clans are firing flaming arrows at one another and launching raids into each other's camps, generally making life difficult all around for a lot of folks who would really rather just get on with the business of living."

The NIH problem is connected to this phenomenon. Everything in the software world is artificial. As in an academic community at some point of maturity problems with adding new features or some kinds of modifications or some license limitations suddenly become ideological.In turn the software community splits into "more permissive", "oppressed" and "orthodox" groups. If one of these groups is successful it's only a matter of time before it becomes "orthodox" and a new split eventually occurs.

It's obvious that Linux wants to be distinct from FreeBSD and FreeBSD needs to be different from Linux. These two camps fought for the same ecological niche, over a matter of personal prestige. Being less PC-friendly then Linux and due to legal problems with AT&T, at some point the FreeBSD movement lost momentum and later suffered also from the additional competition from OpenBSD (after the latter emerged as a viable competitor on Intel platform in the internal split NetBSD/OpenBSD). As a result Linux now looks like a king of the hill on PC platform, cannibalizing available developer resources.

But that does not mean that the problem cannot reoccur at some point. For example, if a sufficient part of kernel developers would feel that the Linus management is unsatisfactory and is harming the Red Hat future, Linux can split into "orthodox" Linux and, say, Redux. It would be interesting to see if Linux kernel development splits if, for example, Alan Cox grow dissatisfied with Linus Torvalds' management. A change of name would probably be mandatory as Linus Torvalds is the owner of the Linux trademark.

In fact, a major "religious war" is currently raging over the relative merits of KDE and GNOME. That in itself is nothing amazing. Like scientific communities, the free software movement is constantly driven by factional disputes over both ideological and technological issues. Mostly, they're meaningless to outsiders. But the KDE-GNOME conflict helps illustrate both the essential strength of the Linux community and its potential Achilles' heel. The conflict is ideological because KDE is built around a piece of software that is not protected by the GPL license; it's not fully "free" in the eyes of Linux purists.

Although KDE is today the most fully developed user-friendly Linux desktop, the main Linux distributor Red Hat devoted programming resources and money to GNOME. As Andrew Leonard wrote:

"We are sad about the fact that the GNOME project was founded," says Bernd Johannes Wuebben, a KDE developer. "We feel that what the free software community needs is unity and focus, not competing standards and the repeated hostility that members of the GNOME project and the more radical elements of the GNU [free software] movement have brought forth against KDE ... We at KDE do believe in free software but we believe that radical views ... are utopian and in the end hurt the free software community severely. In our view there is clearly a place for both: commercial as well as free software development."

Larry Augustin believes that GNOME is ultimately the better desktop on pure technological grounds. But he regrets the fact that Red Hat, the market leader, will not ship the most advanced desktop interface for Linux, thus inhibiting Linux's wider market penetration. Red Hat's Bob Young argues that Red Hat is the market leader because the "technical community trusts us," and that trust depends on Red Hat choosing the appropriate technology. He also suggested that the next release of Red Hat might include an optional version of KDE, packaged separately from the core Red Hat distribution.

Linux outsiders may watch this whole spectacle with some perplexity. Microsoft's great advantage is that it offers software developers a single standard to write their code to, and provides users with a guarantee of software compatibility. Linux, on the other hand, has at least five major distributions -- in addition to Red Hat and Caldera, there is also S.u.S.E. (from Germany), Slackware and Debian (a completely noncommercial version). Each distribution differs from all the others, has different setup procedures and requires a different approach when installing new software.

Corporate America, not to mention the individual consumer, shies away from such variety, with its potential for confusion. Yet at the same time, the diversity is a major source of Linux's strength. "It's the beauty of anarchy. It's the beauty of freedom," says Red Hat's Bob Young. "The distributions that do not do a good job keeping up ... will not survive in the long run."

"It's an advantage because you have more choices and competition," says Augustin. "The distribution vendors are fighting to make distributions better, and that competition helps a lot, it really drives them. There is very little proprietary work in a distribution. It's very easy for someone to come out with a free version of Red Hat, so the distribution vendors have to be constantly making themselves better. If they don't, people will migrate to whoever is best technically.""

Bias toward power users

In the absence of customer support and marketing organizations, feedback and thus development direction (especially requests for new features) are dominated by the most technically savvy users. This factor probably makes open source products more attractive to the top tiers of the user community. Unix traditionally is most popular among professional programmers and in academic, educational and research environments that can be classified as advanced users groups.

Critical mass problem - does winning means stagnation?

"Nothing in the world can replace persistence. Talent will not; nothing more common than the unsuccessful man with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence and Determination are omnipotent."
Calvin Coolidge.

As in corporate software environment, the most viable projects negatively affect competitive OSS projects (Linux vs Hurd) by depleting developer resources (see Halloween I memo):

"As soon as there is no critical mass of developers users project became a history. Now Linux is killing BSD Unix development by absorbing most of its core ideas. This limit competition in a way very similar to PC world ...Possession of the lion's share of the market provides extremely powerful control over the market's evolution."

Large marketshare has risks, such as inhibiting innovation. As Jamie Zawinski put it:

"... Because the company stopped innovating. The company got big, and big companies just aren't creative. There exist counterexamples to this, but in general, great things are accomplished by small groups of people who are driven, who have unity of purpose. The more people involved, the slower and stupider their union is.

And there's another factor involved, which is that you can divide our industry into two kinds of people: those who want to go work for a company to make it successful, and those who want to go work for a successful company. Netscape's early success and rapid growth caused us to stop getting the former and start getting the latter."

Cult of Personality, burnout of the leader

"Never in the field of human conflict was so much owed by so many to so few."
Winston Churchill

Open source may sound democratic, but it isn't. Leaders of the best-known open source development efforts often explicitly stated that they function as dictators. If Linus Torvalds does not like your patch, no matter how technically good it might be that's it. He controls what's in the kernel. Malcolm Maclachlan wrote:

"The ultimate example is Linux itself. Creator Linus Torvalds has final say over all changes to the kernel of the popular open source clone of Unix. Because the Linux development community has grown so large, most software patches are reviewed by many different people before they reach him, Torvalds said.

If he rejects a patch, it can mean a lot of other people threw a lot of effort down the drain, he said. However, it enables him to keep Linux organized without spending all of his time on it, he added.

"My workload is lower because I don't have to see the crazy ideas," Torvalds said. "I see the end point of work done for a few months or even a year by other people.""

If speed of development is an important issue (as is the case with Linux), a single leader must exercise almost absolute power over the code of others (and to less extent over the direction of the project). This situation tend to be more and more of a problem as a project becomes successful.

A determined and dedicated leader is essential for the survival of a project in its early stages. Often the qualities of the leader and the difficulties of the early stages lead to a very high level of centralization. To a certain extent they objectively lead to a "cult of personality" in the best socialist fashion.

But the same qualities of the leader that ensured success of the project in its early stages can and often became a liability at the mature stages, when qualities other than dedication count and the project scales beyond the capacities of a single person to coordinate it. The same qualities that make projects success in their early states lead to their undoing later on. The latest situation with Linux as well as the earlier examples of Emacs and GCC are typical.

I would like to stress that if a project becomes a success, then its success creates a tremendous amount of work for the leader and eventually leads to burnout. Burnout is the price of becoming a media darling.

The pace of work on an OSS project should be determined by the developer himself and here the analogy with academic research is appropriate. I would like to reiterate that "speed kills." Pressing the developer often leads to the opposite results. Linux is not business as usual. There is a strong connection between Linux and FSF communities, there is a common culture of "GNU generation" to it. That culture is in many ways the same as Unix's culture in the early to mid-1980s. Until recently before mass hiring by Red Hat and other Linux distributors, developers worked on Linux for a variety of reasons, but mostly because for fun and a challenge (for many from Minix days). These programmers received recognition, and they have a sense of control and belonging. Take away that fun with a tight schedule and unless they are converted into regular employees or kept by discounted IPO shares many developers will walk away. So the fact they many Linux developers are now hired is to certain extent a very positive development that probably prevented a revolt of the "old guard".

Microsoft as a vital part of the OSS movement: ABM/BTM as a political agenda

If we use the ESR metaphor in a different context, Linux can be looked as a cathedral as it was build to express passion about Unix by a large group that cared about it very much. The ability to attract a geographically distributed community is characteristic of political movements and religions. Although people are physically separated, they all are working toward a common and important goal. That fuels the Linux movement.

Along with a positive agenda, every powerful social movement requires an enemy, a target that can be used as a powerful unifying force. Therefore it is not accidental that a large proportion of the Linux and open source community really hate the 'Evil Empire'. The history of religious and political movements suggests that the OSS is not simply a loose collection of Internet-connected developers and users motivated only by mutual recognition. The OSS really operates on a larger stage as a political movement and as such has its own political agenda.

Probably the most important part of OSS political agenda is its negative aspect, taking the form of a revolt against Microsoft. I call it the ABM/BTM ("anything but Microsoft/Better than Microsoft") agenda. Both statements are simplifications and both presuppose that Microsoft software is not good enough for any task in hand and view Microsoft in black and white in the best of "we against them" war mentality. That's why the OSS movement and Linux have powerful allies in OS/2 and Macintosh communities. Both OS/2 and Macintosh are based on closed proprietary models (actually not much different from the Windows model), but with powerful communities that also has ABM/BTM as their political agendas.

The ABM/BTM agenda, like any political agenda, is simple yet attractive to some. More importantly, it appeals to certain elements of corporate management with support from a wide spectrum of executives at Intel, IBM and almost all major PC vendors. Given the licensing fees paid by Compaq, Gateway and other PC vendors to Microsoft, you begin to understand why there are programmers working on Linux development in these companies with full (and, if necessary, covert) support of management. Politically, some compare Microsoft to IBM in 70's; that attitude fuels reasonable attempts to develop opposition to Microsoft.

Among Internet providers the same agenda is fueled by a fear that Microsoft can usurp the Internet by using well-funded top corporate technocracy in large manufacturing companies. Large and influential Internet providers are resentful about expensive and restrictive licenses; they perceive Microsoft as a threat ("dark force") that tries to slow them down

A cWare White Paper was one of the first documents to describe Microsoft as an important organizing force in the OSS movement:

"In the early days of the Internet, it was IBM and mainframe hegemony. Today it is Microsoft. Just as the German Reformation enfranchised specific groups previously disaffected (specifically, Luther and the German princes), the Internet empowered individuals and groups previously outside the traditionally well funded technocracy that supported and in turn was nurtured by IBM. Linux has been propelled by the same forces. Currently, a major share of commercial software resources is concentrated around Microsoft products like a large low pressure area. However, such a coalescence of power and influence disenfranchises many for whom high cost and restrictive licenses (lack of freedom really) prevent full and easy access to computing resources. So alternative paths are sought. Like the weather, alternatives may appear randomly and then dissipate. Typically, an additional sustaining force, an opposing low pressure area, is required. For Luther this pressure was provided by the German princes, for the early days of the Internet it was provided by ARPA, and for Linux, it has been provided by the Internet community itself. In the case of Linux, the Internet community desperately needed a competent OS platform. AT&T had shut out many Unix users with restrictive licenses and high fees. UC Berkeley had crippled BSD by removing all vendor proprietary code which adapted it to the underlying hardware: you could study it but not run it! Many saw a potential in Andy Tanenbaum's Minix to counterbalance an increasingly unfree Unix. But Minix was incomplete, did not have critical mass and its source distribution became too restrictive. These conditions inspired the community OS effort, initially derived from Minix, which produced Linux. Linux became readily available and increasingly capable. When it aligned with FSF licensing and could support the powerful GNU tools as well as run on a wide range of inexpensive hardware, a truly useful operating system platform was born. The Internet community finally had a way to run a fully networked Unix cheaply and reliably with no strings attached.

Linux appeared almost randomly on the scene but quickly gathered into a well organized storm because it had a powerful force to react against. It also had a sponsor.

Therefore, the Linux "Bazaar" is not simply a loose collection of vendors and other proponents, motivated only by mutual recognition. The "Bazaar" really operates on a larger stage. When forces of the larger stage organize around a dominant restrictive group, a reactionary force is generated in the remaining community. Over time, this reactive force propels various alternatives. If one or more of these alternatives can find support (the Internet community in the case of Linux), then a new "movement" is born which is sustained and even enriched by the powerful forces of the larger stage. Ironically the more dominant Microsoft becomes, the more powerful the reactive forces become, and the more fuel is fed to movements such as Linux. If an unencumbered BSD had been available earlier running on inexpensive Intel hardware, BSD might have become the seed for this storm. But the same drama would have unfolded: thesis and antithesis on a dialectic stage whose imperative will persist until Microsoft runs out of energy or dissipates its focus. Microsoft has only to look over its shoulder at the cycle of hegemony and superannuation revealed by a once almost omnipotent old technocrat: IBM."

Later this ideas were developed in the Halloween papers. In one way, IBM in the past could be considered as a "bazaar" style organization; it was not much of an advantage because this style does not encourage development at great speeds (see Jonathan Eunice's comments).

In a sense, with Linux, Torvalds "switched camps" and start playing the role of a political ABM/BTM-style leader rather than a technical leader in the course of promoting his idea of "world domination". That means that Linux started losing focus and drifted to tactically beneficial but potentially dangerous in a long run ABM/BTM ideology with the underling requirement to compete with Microsoft both on desktop and server, not to an academic community model, with its focus and mantra "chase the dream, not the competition."

This political coloring increase stress and workload (encouraging quicker burnout) on the main developers by demanding unrealistic goals and schedules. It enforces a (autocratic by definition) war camp mentality which logically should be avoided as much as possible.

How an Open Source Project Die

Belonging to the OSS community guarantees nothing. OSS is not a magic bullet ensuring the success of a project. The mortality of open source projects is pretty high; a number of projects never make it to the magic version 1.0. When you see a Web page that was not updated for a year or so it is usually a message indicating that something is wrong. There are several explanations for these extinctions:

I would also to suggest that success does not equate to a superiority of a given technical vision. As in science, shifts of dominant paradigms are difficult and painful. Small and less dramatic changes are more attractive than radical shifts, in software or science. Inertia in software development like inertia in the academy. It means that talented individuals need to overcome more than technical obstacles; they need to overcome the resistance of the community to accept their ideas. As in an academic community, direct contacts with an influential community members and access to major centers of development are very important and can significantly reduce the barriers to the acceptance of a given idea. This phenomenon - "the tragedy of provincial talent" - was first investigated by Nobel Prize winner Pyotr Leonidovich Kapitsa. His analysis of the work of several Russian physicists, never popularized or accepted by the mainstream Western scientific community, is directly applicable to OSS software.

Kapitsa understood that the proximity to leading centers of research and to members working at these centers greatly contributes to the acceptance of a discovery. Once a discovery is integrated into the corpus of scientific knowledge, a paradigm shift can start.

Pseudoproblems

Open source is important to programmers only

While it is true that the vast majority of users of Linux do not have the skill, nor the motivation, to add anything to the OS, other things being equal, the suitability of a program for a particular user is higher if the source code is available, as the source code is the ultimate documentation to the program. The possibility of making changes is less important especially for a large program. It is simplistic to assume that all open source users are C programmers that are both capable and willing to make changes and/or correct errors in the OS and tools. Most users of Linux use it the same way they use Windows -- they install the ready-made distribution and just apply patches until a new more attractive version arrives. Then the cycle is repeated.

The value of the source code availability extends far beyond the simple ability of a particular user to change something the code. Source availability is an additional form of consumer protection; the user benefits even if he or she does not program in C. It essentially ensures much better survivability of a product in a changing OS environment.

Is open source ideology "utopian balderdash" like communism?

Bob Metcalfe - who expressed this sentiment most clearly - is dead wrong. Bryan Pfaffenberger described the errors in Metcalfe's reasoning in this way:

"For Bob, the Open Source movement resembles communism and he sees it headed for the same, final meltdown at the hands of capitalism. Admittedly, if you listen to Richard Stallman long enough, you start hearing John Lennon's "Imagine". But I don't see any resemblance between open source and communism. Open source reminds me of the university, or more to the point, of the long-standing traditions of open knowledge-creating and sharing that are responsible for the impressive successes of Western science.

When you become a scientist, you give up the quest for great worldly wealth. You get a decent wage, to be sure. But you don't capitalize on your discoveries - you give them away. You publish, and reveal all. You don't get a penny from the journals, either. In fact, some of them make the author pay the typesetting charges!

What's the payoff? There are probably as many motivations as there are scientists. Some are curious; they just can't stop thinking about why the edge of a waterfall curls, or why the Milky Way's arms form those big, elegant spirals. Others are big kids who can't wait to get into the lab for another rewarding day of exquisite, exploratory fun. Still others care very deeply about the value and meaning of science. In a recent series of lectures, physicist Freeman Dyson reveals the source of his commitment to science: a quest for ways that science and technology can contribute to social justice, the elimination of poverty, and the preservation of the Earth's environment.

I've never heard anyone call science communism. It has absolutely nothing to do with communism. In fact, communists don't like scientists very much. Scientists are too hard to control. They care too much about truth.

University scientists aren't the only people doing research, of course. For-profit corporations engage in research and development. But the consequences illustrate precisely whats at stake. Proprietary knowledge doesn't get disseminated unless doing so enhances the corporation's bottom line. That's understandable, but we need an alternative. In that most capitalist of all countries, the U.S., a bipartisan Congressional consensus supports public investment in university research and the creation of knowledge for everyone, including for-profit companies. This isn't communism; it's common sense.

So what's at stake with computer software? Plenty. Computer technology is partly responsible for the longest sustained economic boom in U.S. history. It has helped establish U.S. economic, technological, and political dominance in a fractious and dangerous world. It promises to help solve the most vexing problems now facing us: the demand to create efficient, non-polluting energy sources, to deal with the ravages of world hunger, to unlock the mysteries of cancer. It's vital that computer knowledge remains an open public possession, but that's not what's happening. Right now, for-profit computer firms are falling all over themselves trying to patent even the silliest snippets of code, and an unbelievably myopic U.S. Patent Office is giving away the store. ... Computer systems that are vital to public safety and welfare are operating with closed, commercial code, which is loaded with unknown (and unknowable) bugs. Consider the USS Norfolk lying dead in the water for two hours after the failure of its onboard NT systems, and you'll get the idea.

The Open Source movement won't wipe out commercial software, any more than it will create an important and valuable alternative. Computer software is too important to leave to for-profit corporations. There needs to be a balance between publicly accessible knowledge and proprietary, for-profit knowledge, and the Open Source movement is lighting the way.

If you still think this is pie-in-the-sky utopianism, maybe you should go talk to kids in a Mexican school. Without computer literacy, they don't stand a chance in today's global economy. Thanks to GNOME, an open source desktop system for Linux, the Mexican government is saving $124 million that would have otherwise lined Microsoft's coffers, and it's spending the money on computers instead. Call it communism, if you like. I call it progress."

Essentially, open source provides an alternative that one can take or reject.

Moreover even if we (incorrectly) assume that there is a strong connection between open source and Marxism, things are not that bad because Marxism encompasses a very wide spectrum of political movements that includes western social-democratic parties. Let's assume that the political agenda of the open source movement is a kind of Marxism applied to software. This is not true; Metcalfe is definitely wrong equating Marxism and the open source philosophies. However there are definitely some overtones to the open source political agenda that are analogous to Marxist views. So let's for a moment assume that Bob Metcalfe is right. What does this mean? First of all, there is a big difference between the theory of communism (called Marxism) and the practice of communism. Western social democracies, Russian bolshevism (Leninism) and Chinese Maoism are all different. Marxism was a utopian vision created by Karl Marx and his followers. The key component of this vision was that the companies should belong to those who work in them; the simplest way of eliminating unnecessary overhead represented by owners and managers is an equal distribution of wealth between workers. That should be achieved by mean of revolution that should "downgrade" owners into workers, a kind of streamlining of the system to eliminate all middleware. Modified and moderate versions of this theoretical vision are now part of most developed states; radical branches of this version are represented by Russian bolshevism and Chinese Maoism, for example. Each is different from the theoretical model suggested by Marx and each is quite different from each other. Some are successful, some are not. Similar differentiation is happening in the open source movement. Like in the social-democratic movement, the moderate branch, represented by Perl creator Larry Wall and supported by O'Reilly, Red Hat and several other companies, has the most momentum. Much like western social democratic parties, the moderate branch will probably win. But the future is unpredictable by definition.

Technical support for Open Source is inferior in comparison to commercial software

The first problem with this statement is that it assumes that the quality of commercial support is good. Nothing is further from the truth. Here is a more or less typical judgment about the quality of commercial technical support:

"Have you tried to get technical support lately? I have. I can sum up the quality in two words: It sucks. I don't care if you pay for it or if you get it for free as part of your purchase - it sucks."

Jesse Berst describes the situation in this way:

"Many leading companies are reducing what they spend on customer support, as Gateway did recently (while claiming the cuts would help customers. Yeah. Right). Click for more.

The situation will only get worse, I'm sorry to say. In today's overheated Internet economy, vendors rush products to market with only a minimum of quality control, increasing the need for help. And, since they are often forced to give away products for free, they have no budget for tech support. The situation has gotten so bad that companies are springing up to supply the support for a fee. Even Wal-Mart now offers tech support in some of its stores."

Essentially, we have situation very similar to the open source support model. Probably the argument can be made even stronger because if we view the outsourcing of technical support as a progressive business decision then open source has an advantage.

OSS projects are limited to projects that are based on existing protocols

The existence of a standard or prototype can be an important advantage for distributed Internet-based project. It seems that Vinod Valloppillil singled out this important factor in his famous Halloween I memo:

"... commodity protocols actually become the means of integration for open source projects. There is a large amount of IQ being expended in various IETF working groups which are quickly creating the architectural model for integration for these OSS projects."

OSS projects are not completely dependent on standards and prototypes. Corporations have the advantage of being able to collect developers in one place and provide them with management, funds and tools but they have one important disadvantage, too. OSS projects can afford to be simpler than traditional commercial projects because complexity is a commercial advantage. A corporation will usually try to produce a more complex solution than necessary to secure this advantage. This idea was probably first introduced in the decommoditization of protocols:

"For proprietary software to be profitable, it must create a proprietary advantage. Thus, simple software that just gets the job done has one serious disadvantage: a competitor can duplicate it. Given the extremely low marginal cost of software, this inevitably drives the price to near-zero. Thus, all truly profitable software has built-in barriers to competition."

Conclusion

The open source movement is playing an important and vital role in software development at the end of the 20th century, and open source will continue to be an important center for creativity in the next century. But despite the technical skill and imagination of the members of the open source community there are limitations to the open source model that should be carefully studied and compensated for. These limitations correlate to those all too human characteristics that make science in general and applied science in particular difficult activities. This paper stresses the important advantage of OSS over commercial development - the inherent possibility of creating simpler products that are superior to commercial products in terms of functionality and user interface. The famous KISS principle is more applicable to open source projects than to the commercial software development. The second important idea is that speed of development is not necessary an advantage to the open source development model and sometime slowing project development can be a viable strategy that should be considered among other alternatives. A rat race against commercial developers can be actually self-destructive. It is dangerous to consider open source as a completely flawless phenomenon, ignoring the overall history of applied science and more than 50 years of software development. Unrealistic expectations and ignoring the important lessons of history as well as limitation of the model may make some otherwise viable open source projects just an interesting footnote in the overall history of computing a century from now.

About the Author

Nikolai Bezroukov is a Senior Internet Security Analyst at BASF Corporation, Professor of Computer Science at Fairleigh Dickinson University (NJ) and Webmaster of www.softpanorama.org -- Open Source Software University - a volunteer technical site for the United Nations SDNP program that helps with Internet connectivity and distributes Linux to developing countries. He authored one of the first classification system for computer viruses and an influential Russian language book on the subject -- Computer Virology in 1991 (see also the author current views on the subject). Right now is more interested in e-commerce security, Perl and so called Orthodox File Managers (Midnight commander, etc.).
E-mail: postmaster@softpanorama.org

Contents Index

Revised October 8, 1999; October 11, 1999; October 12, 1999 - 13:00; October 12, 1999 - 17:30.
Copyright © 1999, First Monday

Open Source Software Development as a Special Type of Academic Research (Critique of Vulgar Raymondism) by Nikolai Bezroukov
First Monday, volume 4, number 10 (October 1999),
URL: http://firstmonday.org/issues/issue4_10/bezroukov/index.html