# Allow all search engine bots to crawl your site User-agent: * Disallow: # Disallow sensitive or irrelevant directories or pages Disallow: /login/ Disallow: /settings/