Being too popular on the Web can be costly

Wired on Friday: Glenn Fleishmann is a well-known author, online and off

Wired on Friday: Glenn Fleishmann is a well-known author, online and off. In the real world, he writes books and articles on technical topics close to the more techie Net user's hearts

On the Net, he runs several websites where he offers opinions, utilities and advice for free. This March, he blended the two worlds, and offered a complete book he had co-written - a 900-page guide to the graphics program Adobe GoLive - for free. You just needed to download the 12 MB document from his website.

It was a generous gesture. Quite how generous, Fleishmann did not realise until it was almost too late. His book was downloaded so many times and so quickly from his website (over 10,000 times in the first 36 hours) that he overran his website host company's contractual bandwidth allotment. Like many companies who proved connectivity, Fleishmann's provider charges extra when websites reach a certain level of popularity, switching from a fixed charge of a few dollars a month to a fee that varies based on the level of excess Internet traffic. The book was very popular. Had Fleishmann not pulled the plug on his offer as soon as he did the calculations the monthly bill could have been close to $15,000.

It's a familiar story online. Small sites built by enthusiasts and generally visited by a few passersby a day are suddenly selected for fame, receiving hundreds of thousands of hits overnight. But unlike any other field of endeavour, being popular on the Web rarely brings in any cash, and can cost the star dearly. Everyone gets to download their goodies for free, while the subject of their fascination, at best lives with crashing webservers fighting to keep up with demand, at worse ends up thousands of dollars out of pocket.

READ MORE

The economics of content on the Net are enough to drive any analyst - or businessman - quite mad. Popularity and profit seem to sit at opposite ends of the table.

A great deal of the problem - if it is a problem - comes from what the underlying infrastructure is optimised for. The Internet is excellent at sending small messages between many equals: the so-called many-to-many model. It's terrible at sending one message from one place to thousands or millions of others: the broadcast model. The costs of being popular mount up because distributing the same message from one computer to thousands of others goes against how the Net was designed. It's like trying to broadcast a radio show by phoning up all the listeners, and playing the playlist down the phone at them. The bills add up.

What has been needed for some time was a way to be popular online, that fitted how the Net worked.

Last year, Bram Cohen, a San Francisco programmer, set out to do just that. Working in his spare time, he developed a small program that takes the pressure off centralised servers, and lets anyone with a large file take the hit when popularity knocks. It manages this by riding with how the Net works, rather than against it.

Here's how the program, called BitTorrent, works. Instead of putting your masterwork on your website, you put up a much, much smaller "torrent" file instead. Torrent files can only be understood by Cohen's client program, itself a small (and free) download.

When a visitor clicks on the torrent file, the client program on their computer locates other visitors who are currently downloading the same file. Like eager fans at a swapmeet, those other downloaders can trade data with you. So, if another "bittorrenter" has the last few megabytes of the file, they can send you that chunk. And, in return, you send whatever chunks you've located from other downloaders. After the initial "seeding" by the original author, BitTorrent downloads rarely need to return to the main provider to get their data. Instead, they mostly trade the data between themselves on the edges of the Net.

The bandwidth cost of providing the file is, effectively, spread out among all those who are attempting to download it. And because most home and business bandwidth is free below certain limits, the distributed cost of the file drops to zero (for the downloaders and provider, at least).

Cohen's utility gives other advantages, too. Because you can connect to more than one other bittorrenter simultaneously, downloads can actually be faster than transfers from even the most well-connected dedicated server. Because torrenters only share the file for as long as they are downloading it, the longer a file takes, the more clients you should find to download from.

Similarly, the more popular a file is, the more likely you are to find fellow downloaders. The economy of BitTorrent, in other words, is the very opposite of the economies of a traditional download. The more people, and the bigger the file, the faster it will transfer.

In a pleasant mirroring of form and function, distributing and encouraging the use of BitTorrent has largely passed out of Cohen's centralised hands into the wider community of the Net. Hundreds of sites now use it to serve their heaviest content - some legal, like releases of Red Hat Linux CD images to eager developers, or the Grateful Dead tape traders at http://www.etree.org/, some not (such as the underground torrenters who distribute bootleg films and TV episodes).

With the minimum of publicity, Cohen's program has provided the maximum utility for the minimum cost by sharing the burden across the Net. It may not be the best way to make a buck, but it's how the Net works. And it's a great way to be popular.