# www.ok4s.com # 按照robots.txt的标准写法,规定一些不允许爬虫爬的页面或目录。 # robots.txt 的写法参照 # Format is: # User-agent: # Disallow: | # ----------------------------------------------------------------------------- User-agent: * Disallow: /bin/ Disallow: /upload/ Disallow: /common/ Disallow: /editor/ Disallow: /img/ Disallow: /IsapiRewrite/ Disallow: /post.asp Disallow: /postedit.asp Disallow: /reply.asp Disallow: /a_dealdata.asp Disallow: /a_manage.asp Disallow: /a_recycle.asp Disallow: /ajax.asp Disallow: /redirect.asp