From AP via globeandmail.com:

The desire for greater control over how search engines index and display websites is driving an effort by leading news organizations and other publishers to revise a 13-year-old technology for restricting access.

Currently, Google Inc., Yahoo Inc. and other top search companies voluntarily respect a website's wishes as declared in a text file known as “robots.txt,” which a search engine's indexing software, called a crawler, knows to look for on a site.

The formal rules allow a site to block indexing of individual web pages, specific directories or the entire site, though some search engines have added their own commands.

The new proposal, to be unveiled Thursday by a consortium of publishers at the global headquarters of The Associated Press, seeks to have those extra commands — and more — apply across the board. Sites, for instance, could try to limit how long search engines may retain copies in their indexes, or tell the crawler not to follow any of the links that appear within a webpage.