How to Grant Googlebot Permissions in WordPress

Posted on by Prakash Tiwari
Googlebot Permissions

Granting Googlebot permission on a website is a must, as it helps your website be crawled by Google to index your pages on Google search. If Googlebot is not set up correctly, this process can block Googlebot from crawling your WordPress website. Once Google’s crawler stops crawling, it will cause a significant drop in the website’s traffic and keyword ranking.

In this article, you will learn how to grant Googlebot permissions so your WordPress site is easily accessible for crawling and indexing.

What is Googlebot?

Googlebot is Google’s web crawler that visits websites, reads their content, and adds them to Google’s search index. If Googlebot can’t crawl your website correctly, your pages might not appear in search results or may rank poorly.

Steps to Grant Googlebot Permissions in WordPress

Let’s break down the process of granting Googlebot permissions step-by-step.

Step 1: Check Your WordPress Reading Settings

WordPress has a built-in setting that can block search engines from crawling your site.

  1. Log in to your WordPress admin dashboard.
  2. Go to Settings > Reading.
  3. Scroll down to the Search Engine Visibility section.
  4. Make sure the ‘Discourage search engines from indexing this site’ box is unchecked.
  5. Click Save Changes.

If this box is checked, Googlebot won’t be able to crawl your site. Unchecking it allows search engines to access your content.

Step 2: Evaluate and Edit the robots.txt File

The robots.txt file tells crawlers which parts of your website they can and cannot access.

How to Access the robots.txt File:

  1. Go to your WordPress dashboard.
  2. Install and activate the Yoast SEO or Rank Math SEO plugin if you haven’t installed it yet.
  3. In the plugin settings, find the File Editor or Tools section.
  4. Open the robots.txt file.

Check for Blocking Rules:

Check lines such as this one:

User-agent: Googlebot

Disallow: /

The Disallow: / line blocks Googlebot from accessing your site. To grant permissions, replace it with:

User-agent: Googlebot

Allow: /

If you don’t see any blocking lines, your site is likely already accessible to Googlebot.

Save the File:

After editing, save your changes. You can manually upload the updated robots.txt file via FTP if necessary.

Step 3: Verify with Google Search Console

Google Search Console is a free tool that helps you monitor how Google crawls and indexes your site.

  1. Go to Google Search Console.
  2. Log in with a Google account.
  3. Get your WordPress website added in case you haven’t already.
  4. Open the URL Inspection Tool and enter your homepage URL.
  5. Click Test Live URL to see if Googlebot can access your site.

If you see a message that says, ‘URL is available to Google,’ everything will work fine. If you see ‘Blocked by robots.txt’ or ‘Crawled, but not indexed,’ you may need to review your robots.txt or sitemap.

Step 4: Submit an XML Sitemap

An XML sitemap helps Googlebot find all the pages on your site quickly.

  1. In your WordPress dashboard, open your SEO plugin (like Yoast or Rank Math).
  2. Go to the Sitemap settings.
  3. Copy your sitemap URL (it looks something like this: https://yourwebsite.com/sitemap.xml).
  4. Open Google Search Console.
  5. Go to the Sitemaps section.
  6. Paste your sitemap URL and click Submit.

This helps Googlebot discover and index your pages faster.

Step 5: Ensure Noindex Tags Are Removed

Sometimes, pages might have a noindex tag, which tells search engines not to index them.

  1. Open your WordPress post or page editor.
  2. Scroll down to the SEO plugin settings (Yoast or Rank Math).
  3. Check if the Allow Search Engines to Index option is ‘Yes.’
  4. If it’s set to ‘No,’ change it to ‘Yes’ to allow indexing.

This ensures that Googlebot can include the page in search results.

Step 6: Disable Firewall or Security Plugin Blocks

Some security plugins, like Googlebot, may block crawlers by mistake.

  1. Go to Plugins > Installed Plugins.
  2. Open your security plugin’s settings (e.g., Wordfence or Sucuri).
  3. Check if there is an option to block bots or crawlers.
  4. Ensure Googlebot isn’t listed in the blocked user-agents section.
  5. Save your settings.

Turning off these restrictions will allow Googlebot to crawl your site freely.

Step 7: Check Hosting Settings

Some web hosts block Googlebot by default for security reasons.

  1. Log in to your hosting control panel (like cPanel or Plesk).
  2. Check for any settings related to bot blocking or crawler restrictions.
  3. Turn off any option that blocks Googlebot.
  4. Save and exit.

If you’re unsure, contact your hosting provider for help.

Step 8: Monitor Crawl Activity

Keep an eye on how often Googlebot visits your site.

  1. Go to Google Search Console.
  2. Click on Settings > Crawl Stats Report.
  3. Review the chart to see how frequently Googlebot crawls your pages.

If you notice gaps, check for any recent changes that may have blocked Googlebot and resolve them.

These practical steps ensure that Googlebot can access and index your WordPress site.