The lessons of Xanadu

post by jasoncrawford · 2022-08-07T17:59:57.839Z · LW · GW · 20 comments

This is a link post for https://jasoncrawford.org/the-lessons-of-xanadu

Contents

20 comments

One of my all-time favorite articles is “The Curse of Xanadu,” by Gary Wolf, which ran in WIRED Magazine in 1995. On the surface, it’s a piece of tech history, a story of a dramatic failure. But look closer, and you can find deep philosophical insight.

Xanadu was a grand vision of a hypertext system, conceived long before the Web, that at the time of this article had been “under development” for three decades without launching. The visionary behind it was Ted Nelson, one of the originators of the concept of hypertext. Here’s how the article describes him and the project:

Nelson’s life is so full of unfinished projects that it might fairly be said to be built from them, much as lace is built from holes or Philip Johnson’s glass house from windows. He has written an unfinished autobiography and produced an unfinished film. His houseboat in the San Francisco Bay is full of incomplete notes and unsigned letters. He founded a video-editing business, but has not yet seen it through to profitability. He has been at work on an overarching philosophy of everything called General Schematics, but the text remains in thousands of pieces, scattered on sheets of paper, file cards, and sticky notes.

All the children of Nelson’s imagination do not have equal stature. Each is derived from the one, great, unfinished project for which he has finally achieved the fame he has pursued since his boyhood. During one of our many conversations, Nelson explained that he never succeeded as a filmmaker or businessman because “the first step to anything I ever wanted to do was Xanadu.”

Xanadu, a global hypertext publishing system, is the longest-running vaporware story in the history of the computer industry. It has been in development for more than 30 years. This long gestation period may not put it in the same category as the Great Wall of China, which was under construction for most of the 16th century and still failed to foil invaders, but, given the relative youth of commercial computing, Xanadu has set a record of futility that will be difficult for other companies to surpass.

The project had many of the earmarks of other failed or long-overdue efforts. As a product, it was over-designed:

Xanadu was meant to be a universal library, a worldwide hypertext publishing tool, a system to resolve copyright disputes, and a meritocratic forum for discussion and debate. By putting all information within reach of all people, Xanadu was meant to eliminate scientific ignorance and cure political misunderstandings. And, on the very hackerish assumption that global catastrophes are caused by ignorance, stupidity, and communication failures, Xanadu was supposed to save the world.

In contrast to the later Web, links in Xanadu did not point to entire documents, but to any arbitrary range of characters within any document. Links were to be bi-directional, so they could not be broken. And there was an advanced feature in which “parts of documents could be quoted in other documents without copying”:

The idea of quoting without copying was called transclusion, and it was the heart of Xanadu’s most innovative commercial feature—a royalty and copyright scheme. Whenever an author wished to quote, he or she would use transclusion to “virtually include” the passage in his or her own document.…

The key to the Xanadu copyright and royalty scheme was that literal copying was forbidden in the Xanadu system. When a user wanted to quote a portion of document, that portion was transcluded. With fee for every reading.

Transclusion was extremely challenging to the programmers, for it meant that there could be no redundancy in the grand Xanadu library. Every text could exist only as an original. Every user in the world would have to have instant access to the same underlying collection of documents.

The vision for the application of this technology was nothing short of utopian, based on delusions of technological solutions to social and epistemic problems:

… the Xanadu architects became obsessed with developing the widest possible applications of hypertext technology. A universal democratic library, they decided, was only the beginning. Xanadu could also provide a tool for rational discussion and decision making among very large groups. In the Xanadu docuverse, an assertion could always be followed back to its original source. An idea would never become detached from its author. Public discussion on important issues would move forward logically, rather than merely swirling ineffectively through eddies of rhetoric. In fact, any reader could, by creating and following links, freeze the chaotic flow of knowledge and grasp the lines of connection and influence.

The design also blithely ignored the realities of computer performance, developing as they were on minicomputers and early workstations:

The Onyx also had 128 Kbytes of RAM, which they later doubled to a screaming 256 Kbytes. Looking back at the specifics of the endeavor, the approach of the Xanadu programmers seems quixotic. [Xanadu collaborator Roger] Gregory and his colleagues were trying to build a universal library on machines that could barely manage to edit and search a book’s worth of text.

The project suffered from infighting and a lack of good management:

“It was not rapid prototyping—it was rabid prototyping,” said one of [Xanadu programmer Michael] McClary’s friends who watched the project closely.

There was never a realistic schedule: the team perpetually believed they were six months away from completion. The project was so badly conceived and managed that it couldn’t ship even after being acquired by Autodesk and given a full budget:

[Autodesk founder] John Walker, Xanadu’s most powerful protector, later wrote that during the Autodesk years, the Xanadu team had “hyper-warped into the techno-hubris zone.” Walker marveled at the programmers’ apparent belief that they could create “in its entirety, a system that can store all the information in every form, present and future, for quadrillions of individuals over billions of years.” Rather than push their product into the marketplace quickly, where it could compete, adapt, or die, the Xanadu programmers intended to produce their revolution ab initio.

“When this process fails,” wrote Walker in his collection of documents from and about Autodesk, “and it always does, that doesn’t seem to weaken the belief in a design process which, in reality, is as bogus as astrology. It’s always a bad manager, problems with tools, etc.—precisely the unpredictable factors which make a priori design impossible in the first place.”

There are too many good quotes in the article to include them all here—read the whole thing.

---

What struck me most deeply, however, was the response of some of the Xanadu team to the rise of the World Wide Web. You would think that the web would be an object lesson for them—a slap in the face hard enough to wake them from their pie-in-the-sky reverie and bring them back to Earth.

Indeed, one junior programmer on the later team, Rob Jellinghaus—who was born after the Xanadu project had begun (!)—did have such an awakening:

While the Xanaduers paid lip service to libertarian ideals, they imagined a more traditional revolution in which all users would be linked to a single, large, utopian system. But in their quest for a 21st-century model, they created a Byzantine maze.

“There were links, you could do versions, you could compare versions, all that was true,” Jellinghaus reports, “provided you were a rocket scientist. I mean, just the code to get a piece of text out of the Xanadu back end was something like 20 lines of very, very hairy C++, and it was not easy to use in any sense of the word. Not only was it not easy to use, it wasn’t anything even remotely resembling fast. The more I worked at it, the more pessimistic I got.”

The young programmer’s doubts were magnified by his dawning realization that a grand, centralized system was no longer the solution to anything. He had grown up with the Internet—a redundant, ever-multiplying and increasingly chaotic mass of documents. He had observed that users wanted and needed ever more clever interfaces to deal with the wealth of information, but they showed little inclination to obey the dictates of a single company.…

Although he sympathized with the fanaticism of his colleagues, Jellinghaus also began to question whether a hypertext revolution required the perfect preservation of all knowledge. He saw the beauty of the Xanadu dream—“How do you codify all the information in the world in a way that is infinitely scalable?”—but he suspected that human society might not benefit from a perfect technological memory. Thinking is based on selection and weeding out; remembering everything is strangely similar to forgetting everything. “Maybe most things that people do shouldn’t be remembered,” Jellinghaus says. “Maybe forgetting is good.” …

After a couple of months, he began to come to his senses. “What was I doing?” he remembers saying to himself. “This is silly. This was silly all along.”

But here was the reaction of Mark Miller, one of the original developers:

I asked Miller if the Internet was accomplishing his dreams for hypertext. “What the Web is doing is easy,” Miller answered. He pointed out that the Web still lacks nearly every one of the advanced features he and his colleagues were trying to realize. There is no transclusion. There is no way to create links inside other writers’ documents. There is no way to follow all the references to a specific document. Most importantly, the World Wide Web is no friend to logic. Rather, it permits infinite redundancy and encourages maximum confusion. With Xanadu—that is, with tranclusion and freedom to link—users would have had a consistent, easily navigable forum for universal debate.

“This is really hard,” Miller said.

And what about Nelson himself?

Nelson’s response to the Web was “nice try.” He said it is a trivial simplification of his hypertext ideas, though cleverly implemented. And he has not entirely given up hope for the old Xanadu code. “I’d like to stress that everyone involved in Xanadu believes that the software is valid and can be finished,” he asserted.

“It will be finished,” Nelson added. “The only question is which decade.”

---

Miller, Nelson, and the rest of the Xanadu team might have benefitted from reading another one of my all-time favorite articles: Clay Shirky’s “In Praise of Evolvable Systems”.

Shirky begins by pointing out several ways in which the fundamental standards of web technology have seemingly absurd limitations and inefficiencies: HTTP doesn’t use persistent connections and incurs the entire overhead of a new session for each file transferred, web servers have no built-in load balancing, HTML uses one-directional hypertext links that are easily broken, etc.:

HTTP and HTML are the Whoopee Cushion and Joy Buzzer of Internet protocols, only comprehensible as elaborate practical jokes. For anyone who has tried to accomplish anything serious on the Web, it’s pretty obvious that of the various implementations of a worldwide hypertext protocol, we have the worst one possible.

Except, of course, for all the others.…

The problem with that list of deficiencies is that it is also a list of necessities—the Web has flourished in a way that no other networking protocol has except e-mail, not despite many of these qualities but because of them. The very weaknesses that make the Web so infuriating to serious practitioners also make it possible in the first place. In fact, had the Web been a strong and well-designed entity from its inception, it would have gone nowhere.

Contrasting the “evolvable system” of the web with centrally designed protocols such as Gopher and WAIS, he concludes:

Centrally designed protocols start out strong and improve logarithmically. Evolvable protocols start out weak and improve exponentially. It’s dinosaurs vs. mammals, and the mammals win every time.

---

The Xanadu project still exists. I was able to quickly learn its current status, because it has a homepage on a global hypertext-based information system: https://xanadu.com.

“With ideas which are still radical, WE FIGHT ON. We hope for vindication, the last laugh, and recognition as an additional standard…” It complains that “everyone is hypnotized by the Web browser,” which is “basically crippled.”

The project is no longer entirely vaporware. There are two demo viewers of Xanadocs: “XanaduSpace is our best-looking viewer, our flagship demo—but alas, it’s a stuck demo and can’t go further.” The “new working viewer”, xanaviewer3, makes it “possible (but not easy) for anyone who is determined enough to create a xanadoc, and send it to others, who may open and use it.” One supposes that, in order to view it, the recipients of the document must be equally determined.

“With our limited resources,” they explain, “we can only go slowly, unlike today’s Red Bull–fueled young teams.”

The WIRED article describing Xanadu as running for over 30 years is now 27 years old, meaning Xanadu itself is nearing 60. If its “record of futility” was difficult to surpass back then, it is doubly so now.

---

The lessons of Xanadu can be learned at multiple levels.

On one level, the lesson is to scope projects realistically and to strive for simplicity of design.

On a deeper level, the lesson is to ship continually. Doing so keeps the schedule honest, forces difficult scope decisions, and allows for feedback from real users.

On a still deeper level, the lesson is to learn from failure, which the vast majority of the Xanadu team does not appear to have done: thirty years of missed deadlines did not cause them to fundamentally question their schedule, project management, or design scope.

But the deepest lesson, I think, is to value real-world results. Nelson and Miller didn’t fail to notice the Web, they failed to care about its success or even to recognize it as a success. Its epic, world-changing status in the history of technology is meaningless to them beside the fantasy system they had dreamed up.

In the end, despite the title of the WIRED article, Xanadu was not, in fact, cursed. It achieved exactly what its originators wanted: theoretical perfection in a Platonic realm of forms so idealized that it can never quite be brought to Earth.

20 comments

Comments sorted by top scores.