Robots.txt, again

At news.sky:

[Rupert Murdoch] cited the Wall Street Journal as an example of where only the first paragraph comes up on search engines and is free. Anything after that is subscription-based.
He is planning to make newspapers like The Times and Sunday Times chargeable online.
Using the robots.txt protocol on a site indicates to automated web spiders such as Google's not to index that particular page or to serve up links to it in users' search results.
As well as Google, he criticised other sites like Microsoft and Ask.com for also taking a free ride on its content - "the people who just simply pick up everything and run with it - steal our stories ... without payment", he claimed.

see also


Robots.txt at the USPTO