A robots.txt file tells search engine crawlers which parts of your website they may crawl and index. Podpage automatically creates one for your site that welcomes the major search engines and blocks some common unwanted bots — no setup required.
If you'd like to take control of the file yourself, you can replace it with your own custom content.
Why robots.txt matters
- Control over crawling: you can tell search engines to skip pages you don't want appearing in search results.
- Efficiency: steering crawlers away from unimportant pages keeps them focused on your most valuable content.
- Server load: blocking aggressive bots preserves bandwidth for real visitors.
How to customize your robots.txt
- In your dashboard, go to Search Optimization → Robots.txt. You'll see the file your site currently serves.
- Click Settings.
- Enter your directives in the Custom Robots.txt Content field. For example, to allow all robots to crawl your entire website:
User-agent: * Disallow: - Click Save. Your custom content now completely replaces Podpage's automatic file at yourdomain.com/robots.txt.
To confirm everything looks right, use the robots.txt report in Google Search Console.
Tip: robots.txt guides crawlers, but it isn't a security tool — pages blocked from crawling can still be found by people. Use it to shape indexing, not to hide sensitive content.