Cuil has become the latest search engine to take on the might of Google - and the latest to be wounded in the fight, writes Sean O'Driscollin New York
THE BIGGEST DAY in Tom Costello's life began with an ominous jolt. At 4.30am last Monday, his wife called to say that he had better come into work quick.
Earlier that morning, Tom had launched the couple's new search engine, Cuil, an Irish name meaning "knowledge" and a reference to Finn McCool, who, according to folklore, ate the salmon of knowledge near Costello's home town of Drogheda.
The new site claimed to index three times more web pages than Google, and Wall Street was waiting anxiously to see if Cuil could cut into Google's colossal hold on search engine advertising.
Costello, Cuil's chief executive, had got to bed late while putting the final touches to the launch at the company's California headquarters. Early on Monday morning, he put on his clothes, left the couple's four children in the care of his visiting mother-in-law, and headed into a maelstrom.
When he got to the office, the €20 million Cuil was already beginning to creak. Users were complaining that the site was crashing as they clicked from one page to the next.
"There were issues. There were bugs. There are always bugs in programmes," says Costello, who postponed an interview with The Irish Times by three hours while he continued to sort out problems.
The early, technology-driven crashes were only the start of the problems. By 3pm, the system crashed completely under the weight of heavy traffic. Costello closed down the servers and restarted, but for financial analysts and the media, Cuil was a dud.
"A disaster," says Susan Feldman, and search engine analyst with consultants IDC. Within hours, everyone from Time magazine to the Wall Street Journal had weighed in, denouncing the site's lack of focus and skewered search results. Perhaps most condemning of all, a search for Cuil did not return the company's own website in the first page of results. Costello said the company's web crawlers can only search active sites. As Cuil had not yet been launched, it was not recognised.
But there was a bigger problem, and it cut to the very core of Cuil's mission. It claims to have indexed about 120 billion web pages and says Google has indexed about 35 billion.
However, Google's strength has been to weed out sites it does not think worth indexing, claiming its technology is aware of over one trillion pages.
In Cuil's case, that lack of selection may be leading to confusion. A search for Barack Obama on the first day of Cuil's launch placed him in a 'Latino Politicians' category, while another search failed to provide a phone number for Cuil's own publicist.
A number of media analysts have said that Cuil's aggressive indexing may be collecting irrelevant and repetitive sites, dead links and, they'll love this at the parent-teacher meetings, lots of unwanted porn.
Costello, however, has put in long hours this week trying to refine the search. "Most people fail to compete against Google because they try to exactly match what they do. We're going out of our way to produce different results," he says.
A Trinity graduate, Costello applied for a postgraduate course in Stanford University, California after hearing a line in the film Ghostbusters, in which one of the characters notes that the team's latest screw-up has cost him the chance of a Stanford teaching post. "At the time, I didn't even know which side of the country the university was on," he says.
While at Stanford, he began working on a US military programme that sought foreign policy answers from a computer. "It was really blue sky stuff but it deepened my search knowledge and, with the Department of Defense, the budget is generous."
In the heady days of 90s internet start ups, Stanford was buzzing with energy. On Fridays he would hang out in the bar for computer socials with Sergey Brin, who would go on to make billions as the co-founder of Google.
"He was a very gregarious and very funny guy," recalls Costello. "I don't think Sergey's lost any of that with the years."
Costello took a job at IBM while his wife, Anne Patterson, worked on a new system for internet archiving. Brin and his cohorts loved her work and snapped up her technology. She went on to design some of Google's most important search programmes.
As her career soared, Costello quit IBM to become a house husband. While minding the young family, he took over the garage, turning it into a makeshift research lab in his attempts to take on Google.
"A lot of the great internet stories come out of garages, but they are big US garages, not like the ones at home. There's room to play around," he says.
The result was Cuil.
All very beautiful, say analysts, but many complain that the public don't know how to pronounce the company's name.
Yet, IDC's Susan Feldman recommends keeping the Cuil brand in the hope that the public absorbs it.
She believes that Cuil's much-debated indexing is a clever move that caught the public's attention.
She is also impressed with the calibre of people behind the project, many of whom Patterson and Costello have enticed from Google and from former search engine king AltaVista.
"In the late 90s, AltaVista was the darling of the web search world," Feldman says. "The problem was not its technology, but the lack of funding for extreme requirements in hardware, software and research." Which leaves the question: can anyone possibly challenge Google's position as the entry point to the largest collection of knowledge ever assembled by humanity? An image search for "Tiananmen Square" on Google China makes the importance of the question all too stark — the first pages feature smiling tourists, while the English language version features tanks rolling over unarmed students.
If Google is willing to censor itself to appease the Chinese government, should we trust it as our online librarian? Feldman does not see a serious threat to Google from some of the search world's favourite young Turks, such as Powerset, a San Francisco-based company.
Powerset could not afford to invest in Cuil's massive indexing and has suffered as a result, Feldman said. "Web scale search requires sub-second response time to millions of queries a day. That's not an easy requirement to meet. In terms of hardware and network required . . . it's a massive undertaking."
Costello remains defiant, despite this week's backlash, believing that it may take a few years before Cuil is ready to take on Google. He has learnt his lessons, he says, from the sudden rise and collapse of the dot.com era. "I learned a lot more on my way down than my way up. Success teaches you nothing. Getting hurt is what will see me through."