User-agent: * Allow: / Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
What is robots.txt?
A robots.txt file lives at https://example.com/robots.txt and tells crawlers which pages to index and which to skip. Always include your sitemap URL at the bottom.
Pair with other SEO tools
After generating your robots.txt, build matching XML with the Sitemap Generator, audit on-page tags using the Meta Tag Analyzer, and verify headings with the Heading Checker.
Explore more free tools
Everything below runs in your browser — private, fast and free.
Text EditorNotepadWord CounterKeyword DensityPomodoro TimerPassword GeneratorString GeneratorBase64 EncoderCase ConverterLorem IpsumURL EncoderJSON FormatterText ReverserDuplicate RemoverMarkdown PreviewNumber to WordsMorse CodeText SorterQR Code GeneratorColor PickerRegex TesterSlug GeneratorJWT DecoderMeta Tag GeneratorMeta Tag AnalyzerHeading CheckerSitemap GeneratorHTML ↔ TextKeyword SuggestionBroken Link CheckerFamous BooksBlogTemporary Email (Free)Love Calculator