Robots.txt file for WordPress.
What is a Robots.txt File
A robots.txt file tells search engines which parts of your WordPress website they can crawl and index. This file should be placed in your root directory.
It helps you control what search engine bots can crawl and index on your website. For example, you can block access to sensitive folders like wp-admin or wp-includes.
This keeps your backend safe and prevents duplicate content from showing up in search engine results. A well-structured robots.txt can also improve crawl efficiency.
If search engines waste time crawling and indexing unnecessary files, they may miss your important pages. Make sure you don’t accidentally block content you want to rank.
Here’s a common, default example you can copy:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap_index.xml
Here’s a custom Robots.txt file you can also use to enhance your website is crawled and indexed efficiently by search engine.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /cgi-bin/
Disallow: /xmlrpc.php
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/feed/
Allow: /wp-content/uploads/
Sitemap: https://yourwebsite.com/sitemap_index.xmlIt’s a good practice to review your robots.txt regularly. You can easily do that visiting https://yourwebsite.com/robots.txt
Ask yourself: Are you hiding the right things while allowing search engines to access your valuable content?
Testimonials form Our Clients
Read how our clients boosted growth, improved security, and strengthened their brand with CBN Tek
Amazing Experience with My Website Redesign
“ I’m so happy with how everything turned out! They completely transformed my website, created an amazing new logo, and handled all my marketing stuff. The CBN Tek team was really professional and knew exactly what they were doing. They were always there when I needed help and made the whole process so much easier. I’ll definitely be telling everyone I know about them! ”
The service was awesome from Dr. Brenda
“ The service was awesome from rebranding the website outlook, logo and designing my marketing materials was top notch. Very professional, reliable, knowledgeable and genuinely helpful. Will recommend this business anytime.”
Very professional and helpful
“ Very professional and helpful. Bless was very knowledgeable about building my website and giving me information on my business. I really recommend this company especially for small business owner.”
My SEO is finally working
“ I hired CBN Tek to fix my website’s SEO, and I started seeing real traffic within weeks. They explained everything in plain language and helped optimize every page. I now rank on Google and get more leads without running ads.”
Brand Identity Assured
“ As a small business, having a cohesive brand identity is crucial. CBN Tek designed a complete comprehensive brand kit that made our marketing materials shine,and our website designed by them laid a solid foundation for our growth. Thanks to Bless Che for the excellent work, more referrals definitely coming your way.”