# # robots.txt for http://www.salon.com # # check out http://info.webcrawler.com/mak/projects/robots/norobots.html # for more info on what this is all about. # User-agent: * Crawl-delay: 5 Disallow: /cgi-bin/ Disallow: /RealMedia Disallow: /demo Disallow: /politics/feature/2002/08/29/white Disallow: /news/opinion/feature/2006/08/25/eichenwald Disallow: /sex/feature/2000/07/31/mother_1 Sitemap: http://www.salon.com/sitemap.xml User-agent: Googlebot-News Crawl-delay: 5 Disallow: /cgi-bin/ Disallow: /RealMedia Disallow: /demo Sitemap: http://www.salon.com/google_news.xml