User-agent: * Disallow: /search.html?* User-agent: Bytespider Crawl-delay: 50 User-agent: MJ12bot Crawl-delay: 20