ToolBento guide
How to create a robots.txt file for a small website
Generate a simple robots.txt file with a crawl rule and sitemap location so search engines can discover your important pages.
Recommended tool
Open the matching ToolBento utility →What robots.txt does
A robots.txt file is a small text file placed at the root of a website, such as example.com/robots.txt. It gives crawler instructions before search engines request other pages. For many small websites, the most useful starting point is simple: allow normal crawling and point crawlers to the sitemap so important URLs are easier to find.
When a simple file is enough
If you run a brochure site, blog, landing page, portfolio, or small tool site, you often do not need a complex set of rules. A basic file that allows all crawlers and lists the sitemap is usually clearer than copying a long template full of blocked folders you may not actually have. Keep the file easy to review before uploading it.
What to enter in ToolBento
Open ToolBento's Robots.txt Generator and enter the Website URL. This should be the public homepage address for the site, including https://, such as https://example.com. The tool uses that address to build the sitemap line, so make sure it matches the canonical domain you want search engines to use.
Use Generate to create the rules
After entering the website URL, press Generate. The output creates a simple robots.txt body with User-agent: *, Allow: /, and a Sitemap line that points to your site's sitemap.xml file. Copy the result, save it as robots.txt, and upload it to the root of your website so it is available at /robots.txt.
Check the sitemap URL before publishing
The generated Sitemap value assumes your sitemap is located at /sitemap.xml on the same domain. That is common for static sites and many content systems, but it is still worth opening the sitemap URL in a browser before you publish the robots.txt file. If your sitemap lives somewhere else, edit that line before uploading.
Mistakes to avoid
Do not add Disallow rules unless you know exactly what they block. A typo such as Disallow: / can tell cooperative crawlers not to crawl the whole site. Also remember that robots.txt is crawler guidance, not security; never rely on it to hide private pages, admin screens, tokens, customer files, or anything that requires real access control.
Test after deployment
After the file is live, visit yourdomain.com/robots.txt and confirm the text matches what you intended. Then submit or refresh the sitemap in your search console if you use one. Recheck robots.txt after redesigns, domain changes, or static-site rebuilds because an old sitemap URL can quietly send crawlers to the wrong place.