WiscWeb - Using the UW No Index / Remove Pages from Search Crawlers plugin

This document will walk you through how to prevent a page or post from appearing in search results for services like Google.

Important considerations

Recommended use

It is strongly recommended that this option only be used on 1-2 pages of your content. It is not tested for large-scale use across all pages of a website.

For previously published pages where you've recently added the 'noindex' meta tag:

  • If a page was created and already indexed by Google, it won’t become un-indexed until Google re-crawls and finds the ‘noindex’ meta tag for that page.

    • For these instances, consider running the URL Inspection Tool to let Google know to recrawl your content. To complete this step, you’ll need to have a Google Search Console account setup and linked to your site: [Link for document 90062 is unavailable at this time].

    • Tip: If you know you have a page that should not be indexed, consider adding the meta tag upon first page publish.

Interior page links

A page with ‘noindex’ on won't show up in search results. However, the page will still be crawled and links found there will be followed and possibly indexed (unless those pages also have ‘noindex’ tags).

'noindex' versus 'nofollow':

  • Where 'noindex' is telling Google not to index the page, 'nofollow' is asking Google to ignore hyperlinks. For utilizing ‘nofollow’, use the instructions on the WiscWeb - Sharing pages to social media and search engines doc under Custom Meta Tag Settings but use ‘robots’ as the property and ‘nofollow’ as the content.

  • ‘nofollow’ may be treated differently by Google. Google will only see this as a strong recommendation. In other words, Google may ignore 'nofollow' tags for hyperlinks. 

  • If you want to dive super deep, here's an article that breaks down all of the differences.

Temporarily blocking a page from appearing in Google search results:

In-site search results

Please note that your non-indexed page may still appear in search results within that website: 

  • If your site uses the native WordPress search, the non-indexed page WILL appear in search results. 
  • If your site uses Google Programmable Search, the non-indexed page SHOULD NOT appear in search results. However, this may only be true if this setting was added to a brand new page. As a reminder, existing pages may have already been crawled by Google and therefore could appear in search results.

If you're not sure which one it is using, here's a simple test:

  1. Type a word into your website's search box
  2. If the results look styled like Google, it's probably Google Programmable Search
  3. If the results look styled to look like a UW Theme website, it's likely the WordPress native search

If your site does not have a search feature activated, then this will not be an issue for you.

Plugin activation

The UW No Index / Remove Pages from Search Crawlers plugin can be activated by Administrators using the instructions in WiscWeb - Self service plugin activation / deactivation.

Steps

  1. Once the plugin is activated in the project, navigate to the page you wish to restrict from being crawled

  2. In the right Publish box, check the box that says “Discourage search engines from indexing this page”

    Publish box in WiscWeb with green box around the new checkbox for setting 'noindex' for a page

  3. Publish/update the page (Reminder: If the page has already been published before you add this setting, there is a chance it has already been indexed by Google. You’ll have to request the site to be re-crawled to ensure that this setting is honored by Google).