News Web Sites Propose System For Search Engines

Thursday, November 29th 2007, 10:51 am

By: News On 6

NEW YORK (AP) - The desire for greater control over how search engines index and display Web sites is driving an effort launched Thursday by leading news organizations and other publishers to revise a 13-year-old technology for restricting access.

Currently, Google Inc., Yahoo Inc. and other top search companies voluntarily respect a Web site's wishes as declared in a text file known as ``robots.txt,'' which a search engine's indexing software, called a crawler, knows to look for on a site. The formal rules allow a site to block indexing of individual Web pages, specific directories or the entire site, though some search engines have added their own commands.

The proposal, unveiled by a consortium of publishers at the global headquarters of The Associated Press, seeks to have those extra commands, and more, apply across the board. Sites, for instance, could try to limit how long search engines may retain copies in their indexes, or tell the crawler not to follow any of the links that appear within a Web page.

The current system doesn't give sites ``enough flexibility to express our terms and conditions on access and use of content,'' said Angela Mills Wade, executive director of the European Publishers Council, one of the organizations behind the proposal. ``That is not surprising. It was invented in the 1990s and things move on.''

Robots.txt was developed in 1994 following concerns that some crawlers were taxing Web sites by visiting them repeatedly or rapidly. Although the system has never been sanctioned by any standards body, major search engines have voluntarily complied.

As search engines expanded to offer services for displaying news and scanning printed books, news organizations and book publishers began to complain. The proposed extensions, known as Automated Content Access Protocol, partly grew out of those disputes. Leading the ACAP effort were groups representing publishers of newspapers, magazines, online databases, books and journals. The AP is one of dozens of organizations that have joined ACAP.

``ACAP was born, in part at least, against a growing backdrop of mistrust,'' said Gavin O'Reilly, president of the World Association of Newspapers.

News publishers complained that Google was posting their news summaries, headlines and photos without permission. Google claimed that ``fair use'' provisions of copyright laws applied, though it eventually settled a lawsuit with Agence France-Presse and agreed to pay the AP without a lawsuit filed. Financial terms haven't been disclosed.

O'Reilly said ACAP should ``enhance the availability and accessibility of all kinds of content online'' by making publishers more comfortable their rights would be respected.

AP Chief Executive Tom Curley said the news cooperative spends hundreds of millions of dollars annually covering the world ; and in many cases its employees risk their lives doing so. Technologies such as ACAP, he said, are important to protect the AP's original news reports from sites that distribute them without permission.

``The free riding deprives AP of economic returns on its investments,'' he said.

The new ACAP commands will use the same robots.txt file that search engines now recognize. Web sites can start using them Thursday alongside the existing commands.

Like the current robots.txt, ACAP's use would be voluntary, so search engines ultimately would have to agree to recognize the new commands. Search engines also could ignore them and leave it to courts to rule on any disputes over fair use.

Google spokeswoman Jessica Powell said the company supports all efforts to bring Web sites and search engines together but needed to evaluate ACAP to ensure it can meet the needs of millions of Web sites; not just those of a single community.

``Before you go and take something entirely on board, you need to make sure it works for everyone,'' Powell said.

ACAP organizers tested their system with French search engine Exalead Inc. but had only informal discussions with others. Google, Yahoo and Microsoft Corp. sent representatives to the announcement, and O'Reilly said their ``lack of public endorsement has not meant any lack of involvement by them.''

Danny Sullivan, editor in chief of the industry Web site Search Engine Land, said robots.txt ``certainly is long overdue for some improvements.'' But he questioned whether ACAP would do much to prevent future legal battles.

And being an initiative of news publishers, he said, it might lack attributes that blogs, online retailers and other Web sites might need in an updated robots.txt.

Francis Cave, ACAP's technical project manager, said Thursday's plan was only ``a first stab. ... We full expect we will need to add to that.'' Already contemplated is support for video files, not just text and still images. Cave said online archives such as the British Library and the Internet Archive might also need special commands.