Chapter 3: From So Simple a Beginning

Another excerpt to peruse. As always, hack away at it with your sharpest blades. I'm leading off with the beginning of Chapter 3, but so as not to disturb continuity too much, here's a synopsis of the end of Chapter 2:

Following where we left off, I go on to argue that our current rise in amateur activity can be attributed, in part, to the "over-education of the middle class," a winking characterization of the enormous increase in undergraduate enrollments that started in the wake of WW II. The argument, to wit, is that as a society we train people to think broadly and critically, and yet we employ them to follow directions and perform a highly limited range of tasks, a vestige of the assembly-line mentality (Fordism) that still dominates our approach to production, be it automobiles or information.

The rest of the chapter goes back to examples, but doesn't make any new points except to show that "amateur" is a designation requiring much latitude these days (am I an "amateur" blogger? I don't get paid for it, though I'm a "professional" writer), and to show that the same rise of amateurism is affecting the sciences as profoundly as the arts. Here's how I summarize the amateurism argument at the end of Chapter 2:

And crowdsourcing has no more regard for professional qualifications than it does nationality. InnoCentive and iStock are pure meritocracies—all that matters is the final product. This is one of its greatest strengths. One revealing MIT study into InnoCentive revealed that solvers were more successful when they had less experience in the relevant discipline. In other words, chemists were better suited to solving life biology problems, and vice-versa. This is less surprising than it seems at first blush. If a P&G chemist could have solved a stubborn predicament in his own field, it would have never wound up posted to InnoCentive’s Website. This is powerful mojo: The untrained are also untainted. Their greatest asset is a fresh set of eyes, which is simply a restatement of the truism that with many eyes, all flaws become evident, and easily corrected. But that concept wasn’t always clear. It took a handful of renegade computer programmers show just how powerful the law of large numbers could be.

Which brings us to that most fundamental building block of crowdsourcing, the open source software movement. The first bits of Chapter 3 after the jump.

Chapter 3: From So Simple a Beginning

“There is grandeur in this view of life. From so simple a beginning, endless forms most beautiful and most wonderful have been, and are being evolved.”     — Charles Darwin

In the beginning, all source code was open source code. This was due more to circumstance than design, but it would nonetheless have considerable consequences far beyond the realm of computer programming. Source code is the English-language commands that, once translated into zeros and ones, tell a computer what to do. Open source code is pretty much what it sounds like—open for anyone to see, copy, tweak and use for whatever purpose they see fit. Because it was open, a spirit of collaboration and free exchange of information developed in computer programming. And because it had been open once, a small group of principled programmers determined that it should stay open. They couldn’t force Microsoft or Sun Microsystems or Apple to reveal their code, but they could create a free and open alternative.

In order to do so, the founders of the open source software movement had to invent a new way to get things done. They couldn’t offer anyone money, and the task before them—to write an entire operating system, requiring millions of programmer-hours—was daunting. Would highly skilled people contribute their spare time to a project that seemed doomed to fail? Actually yes, they would. Many, many people would. And because many people came forward, the burden did not fall heavily on the few. By the early 1990s the crowd had produced its first substantial work—Linux, an operating system superior, in many respects, to the best products of any corporation.

Open source provided a precedent—a proof of concept. If people working in their spare time—stove-top chemists, basement musicians, Sunday photographers—provide the crowdsourcing engine with fuel, it’s the open source software movement that provided it with a blueprint to build from.

No one knew any of this in 1969, of course, when Ken Thompson, a computer programmer with Bell Telephone Laboratories suddenly found himself at loose ends. Thompson had been working on an ambitious, five-year collaboration between MIT, General Electric and Bell. The project was supposed to create a more efficient operating system for the mainframe computers of the era, one that would be capable of performing more than a single task at a time, a limitation that had greatly slowed the processing speed of even the most advanced computers. But instead of efficiency the effort had created chaos. The manual for the operating system ran over 3,000 pages. By that spring Bell had lost faith and pulled out.

Facing a four-week vacation, Thompson decided to start from scratch, this time with only himself to answer to. Instead of aiming high, he would aim low. He devoted one week each to writing four components of a program. As Steven Weber, a political scientist and author writes in The Success of Open Source , “With just one man-month and very basic hardware Thompson had to leave behind big-system mentality and do something simple.” Or as one of Thompson’s collaborators put it at the time, “build small neat things instead of grandiose ones.”

By the end of the month, Thompson had written the elementary outlines of Unix, destined to become the most successful and long-lasting operating system ever created. But it wasn’t just Unix’s popularity that made Thompson’s actions historic; it was his decision to make Unix out of small, discrete programs meant to do one thing but do it well. Such an architecture would eventually allow hundreds of programmers to work together in a totally decentralized fashion—in much the same way thousands of contributors work together today in a decentralized fashion to form a single reference work—Wikipedia. Breaking labor into little units—what the business writer and consultant Umair Haque calls the “microchunk”—is one of the hallmarks of crowdsourcing. In this case, it facilitated the come-one, come-all approach of open source programming.

Not that there were many interested parties at the time. Computer science developed out of academic and quasi-academic research labs like those at Bell and MIT. Sharing computer code conformed the general academic tradition of free exchange of information, but it was also a simple expedient: It was the only way to get anything done. One of the first commercial computers, the IBM 705, cost $1.6 million in 1953, the year it was released. (That’s more than $12 million in 2008 dollars.) Beyond being expensive and requiring the larger part of a room to put it in, it required an extraordinary amount of time to write the code that told it what to do. In order to maximize their resources, the few people qualified to perform such a task pooled their time and talents.

A programmer culture emerged during these formative years, which were characterized by playful, highly competitive and yet highly collaborative interactions between devoted specialists. The resulting hacker ethos prized originality and creativity and sharing. You couldn’t separate the users from the programmers, because the only people who used computers were the people who programmed them. It wasn’t until the introduction of the personal computer a need for proprietary software developed.

This created a conflict. In 1976, Bill Gates and Paul Allen—co-founders and sole employees of what was still called “Micro-Soft”—wrote an “open letter to hobbyists.” It did not mince words: “As the majority of hobbyists must be aware, most of you steal your software.” The letter was an unequivocal condemnation of what had become a hacker habit—the free sharing of software and the source code it was based on. Gates pointed out that the royalties he and Allen had been paid for Micro-Soft’s first commercial release averaged out to $2 an hour. Who would write software with such an incentive, Gates wondhered. The hobbyists needed professional programmers because, after all, “What hobbyist can put 3-man years into programming, finding all bugs, documenting his product and distribute for free?" Gates could have never anticipated the answer to his question, which was that no single hobbyist could put 3-man years into such a daunting task, but 3,000 hobbyists easily could, and soon would.

Fighting the Good Fight

In 1983, an MIT computer scientist named Richard Stallman decided to wage a one-man war against the software industry that Gates had helped create. In doing so, he gave a name and an unstinting ethos to what had previously been a vague predilection on the part of the hacker community. Stallman had arrived in Cambridge to attend Harvard University in 1970, a product of a stridently liberal upbringing in Manhattan’s Upper West Side. An aggressively idiosyncratic and precocious youth, Stallman has said he didn’t have friends within his peer group until he arrived at MIT’s Artificial Intelligence Lab. He would spend the next 12 years of his life there, sleeping in his office and living to write code.

In 1980 Stallman watched as the hacker community that had thrived in MIT’s lab fell apart. Most of his friends and colleagues left academia to create companies dedicated to developing proprietary software that would serve the booming computer market. At this point even UNIX—that stalwart of hacker culture—had become proprietary. In protest, Stallman founded the GNU Project, an effort to create an operating system based on “open” or freely available source code. (GNU is a recursive acronym for “GNU-Not-Unix”—a fine example of humorous hacker wordplay.)

It was the first shot in a remarkable revolution, but few heard it at the time. Stallman severed all affiliations with MIT (although the university continued to let him use the lab and sleep in his office), and began to write an operating system based on UNIX, but in a form that would allow other users to steal, copy, cut, paste, modify and most importantly, add their own contributions to the source code Stallman was writing. Attracted to the easy access of Stallman’s system, other programmers began working with him on the GNU Project. “People started asking for and writing improvements,” Stallman says, “and it ended up much better than I’d originally planned on its being.” And because the GNU OS was based on Unix, with its thousands of small files, it was easy for other programmers to pick and choose what individual bits they could work on, according to how much time they could devote to the project.

By making his operating system freely accessible, Stallman was almost single-handedly keeping the hacker culture alive. By 1986, Stallman had created a C compiler, arguably the most important part of an operating system, in completely free code. In order to insure that no enterprising hacker took the code and incorporated it into a piece of software that he or she would then put on the market, Stallman created what is a much greater contribution to the open source movement, and to culture at large: the GNU/General Public License. The GNU/GPL required not only that anything released under the license be made freely available, but that any software that incorporated it employ the same license. “The GNU/GPL ‘converted’ software it was used with to its own license, an extraordinarily clever approach to propagating freedom,” notes Glyn Moody in his history of open source movement, Rebel Code. This little trick has come to be known as “copyleft,” as opposed to copyright.

Yet for almost a decade Stallman’s efforts remained of interest to few outside academia and a radical fringe of hackers. Then a young Finnish programmer named Linus Torvalds would send an email inviting anyone with the interest and the spare time to contribute to his own project – another operating system based on GNU, but adding some substantial improvements to it. Thousands responded to Torvalds’ email, and after several years of volunteer labor, Linux emerged. Today, Linux powers everything from supercomputers to corporate servers to cell phones to digital video recorders like TiVo. In fact, about 70 percent of Web server software runs on the Apache Web server, which was developed using open source methods, and more than half of all large-scale e-mail programs also use open source software . Working outside of any organizing agency like a company or academic institution, the open source community proved that the most intelligent networks were self-organized. Who authored Linux? The crowd.

The open source software movement has always been as much about advancing a philosophy as it has been about developing new software. Proponents of the open source model value transparency for its own sake, not simply because opening up the development process to outsiders happens to produce better code. But it is the efficacy of the open source model, not the egalitarian principles underlying it, that drove companies like IBM, and more recently even Microsoft, to begin adopting it as a way to save money and develop better products. The open source evangelist Eric Raymond wrote that, “given enough eyeballs, all bugs are shallow”—which is to say that a large and diverse labor pool will come up with better solutions than the most talented, specialized workforce. This is as true in fields like corporate science, product design and content creation as it is in software.

This is one of the central epiphanies behind crowdsourcing. It appears in an essay of Raymond’s entitled, “The Cathedral and the Bazaar” that played a seminal role in the migration of open source ideas into the mainstream. Raymond contrasts two methods of software development. The "cathedral" approach, in Raymond’s terms the standard corporate mode of software development, in which managers established goals and timelines and assigned specific tasks to individual teams of programmers. But he could as well have been describing the production process of any industry at the time. Raymond contrasts this with Linux, “a world-class operating system [that] coalesced as if by magic … by several thousand developers scattered all over the planet, connected only by the tenuous strands of the Internet?”

“Linus Torvalds's style of development - release early and often, delegate everything you can, be open to the point of promiscuity—came as a surprise. No quiet, reverent cathedral-building here—rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches ... out of which a coherent and stable system could seemingly emerge only by a succession of miracles.”

Raymond makes a persuasive case for open source's raw, hurly-burly horsepower, which not only produced a fairly bug-free operating system, but did so at a “speed barely imaginable to the cathedral builders.” When Raymond first presented his paper to the Linux Kongress of 1997, its significance was only recognized by a few of his peers in the insular world of computer programming. But a much larger audience would soon take note.

Comments

[international professors project]

I've read Jeff's entire chapter and will order this thoughtful book for my library and some universities in poor, almost poor, developing and un-developing counries.

There are some connections of Jeff's work to various fields, if you want to google "crowd behavior" and several related phrases. Applied Social Psychology may be another avenue to elaborate further.

It may be a stretch for many or most terms you search,and set google Alerts for, but its a wonderful way to sit quitly in a room to think, esp. about interconnections.... and, the wisdom, and the stupidity, of crowds.

Kind regards,
Minh,
International Professors Project

[Tabish Bhimani]

Hey Jeff,

Now I'm no expert at this, but here are my two cents. I really suggest taking a look at it even if it may look like crap. Thanks.

first paragraph: used the word "open" too much.

second paragraph: should talk about unix as well, sort of to give context to linux in terms of modding.

third paragraph: provided the crowdsourcing engine... not provide the crowdsourcing engine.

fourth paragraph: but instead of efficiency, (comma) the effort had created chaos. *just a thought*

eight: lastline, "...ethos prized orginality, (comma) creativity, (comma) and sharing..." "..you could not..." not couldn't. Professional text not informal. "...It was not until the introduction of the personal computer THAT a need for proprietary software developed..."

ninth: Gates wondhered? spelling. wondered, correct?

Paragraph about "the open source software movement has always been as much..." I believe instead of using the statement made by Eric Raymond, you should use the one made by Jamie De Guerre "Much has been said about the wisdom of crowds. The idea that many people can achieve results more effectively than individuals has gained credence, especially as the internet has bought those people together and allowed them to cooperate in innovative ways. These techniques apply to everything from online encyclopedias to citizen journalism – and even anti-spam technologies." Although it was used in a paper about Anti Spam technology called the Vipul's Razor, it still holds importance in the world of crowd-sourcing. The article is: "The mechanics of Vipul’s Razor technology" published in Network Security in September 2007. If not replace, maybe you should use it. It would be a great sum up.

[alan booker]

Jeff, the excerpt from chapter 3 follows a thought stream that as dense as it is still breathes beautifully.

The exceedingly powerful imagery used by ER “The Cathedral and the Bazaar” to describe the two diametrically apposed processes provide us with a deep mystery to ponder.

The cathedral provides possible entry, in a fold who adheres to a common faith, to higher worlds; the bazaar provides unadulterated life forces that might live closer to that higher world albeit unrestrained by any dogma. A simple shepherd might after all be closer to god than all the upper echelons of the cathedrals hierarchies. Therein lies the mystery of those images.

The shackles that fetter process and offer conformity for advancements sake ultimately rob a process of any truly organic evolution whereas individual biography, as it unfolds, whether utilitarian or not, is the manifestation of idea and life paths that are birthed, realized and ultimately form, from the inside out, vessels that work directly back upon society and culture.

“Social behavior is never entirely predictable, but neither are crowds irrational, crowd behavior reflects the desires of it’s participants, but it is also guided by norms that emerge as the situation unfolds” and where do those norms originate?

I would suggest that those norms are the result of deeply organic impulses, which are not yet fettered, and originate from individual biographies. These individual impulses are intrinsically tied to personal destiny. Not predetermined in any way yet central to any ones life’s path and ultimately released to shape the world.

As your excerpt clearly articulates, seeds once planted might take years to show fruit yet the hand of an individual sowed each seed.

Warm regards, Alan

[Geeth2008G]

I would suggest that those norms are the result of deeply organic impulses, which are not yet fettered, and originate from individual biographies.

..............

Geeth

Wow, check out this site called http://www.fluc.com
Free SMS and free mobile ads!! Its fantastic

[Priyendu]

This is a great site for IT students, please visit http://www.itprojectsforyou.com

[frank chaplin]

wow. one of the worst interpretations i've read.