User-Agent: Google-WHOI-Internal-Searcher #Majority of users directory will not be crawled by internal crawler, but needed to add an exception. #Will control this with search engine. Disallow: /usage User-agent: * Disallow: /users/ Disallow: /usage/