If you received this message this morning from Google Search Console (formerly Google Webmaster Tools), you’re not alone. Google has recently been pushing harder for webmasters to allow crawlers full access to all Javascript and CSS so that they can render a site to determine whether or not it meets mobile standards (among other things). Many WordPress sites (along with the other major CMSs) received an ominous warning that their site was blocking assets, and it could affect rankings. While technically true, many of the sites were only blocking /wp-admin/, where absolutely no public-facing assets should live. For those of you that are in that situation, relax.

However, a number of sites are also blocking /wp-includes/. While it doesn’t seem obvious, there are a number of things that live here which would need to be accessed by users (i.e., crawlers) to render pages properly. For example, Dashicons, the small icons you generally associate with the admin side of WordPress, can often be called by themes for front-end usage. Another major thing that can hinder proper rendering by crawlers is jQuery. Sometimes, themes will enqueue a different version, but by default, it lives in the /wp-includes/ folder. If we dive even further into the issue, we’d see that the built-in emojis and comment reply handling would also be affected.

So, what can be safely blocked? At this point, here’s what a “compliant” WordPress robots.txt would look like (as far as what is safe to block). Of course, you’d want to add in your own sitemap directives and other special cases, but this is a good starting point.


User-agent: *
Disallow: /cgi-bin/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /xmlrpc.php


One big word of warning: DO NOT block anything under /wp-content/. You can assume anything blocked here is going to hit you pretty hard eventually, since everything to do with site rendering exists here; plugins, themes, images, and so on. It was popular at one point to block crawlers (like Googlebot) from plugin folders, but given the amount of Javascript and CSS many plugins employ now, it’s a dangerous proposition. If you’re blocking anything here, open it back up immediately.

Questions or comments? Leave them below!