Jean Paul's Blog

There are 2 types of People in the World, One who Likes SharePoint and..

    Advertisements
  • Microsoft MVP

  • MindCracker MVP

  • CodeProject MVP

  • eBook on SharePoint 2010

  • eBook on Design Patterns

  • eBook on Windows Azure

  • NLayers Framework @ CodePlex

  • MSDN Forums

  • .Net vs. Java

    Due to Public Demand
  • Advertisements

Posts Tagged ‘search error’

Search Crawling Error : The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl.

Posted by Paul on May 15, 2014

While setting Content Sources in SharePoint 2013, especially External Web content sources, one may encounter following errors.

I had faced the error while configuring web content source for my blog:

http://www.jeanpaulva.com

I had configured Crawl Rules to include domain/*.  But still the error happened.

The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl.

error blog

As the content is able to parse inner URLs, we can discard the chances of Firewall & Antivirus blockages.

Solution

The error essentially may not depend on the Source URL; It can be caused by the Crawling Machine configuration as well.

Following are the fixes you can try one by one:

Solution 1 Check robots.txt

Open your URL/robots.txt & Ensure there are no blocking verbs.

For example a fully-liberal robots.txt looks like below:

image

Solution 2 Check meta tags

View the source of the HTML page & ensure no blocking tags like below are there:

image

You can even discard the META tags Or create new tag with INDEX or FOLLOW values.

Solution 3 Check Crawling Machine & Crawl Credentials

Open your Crawling Machine (If in Farm) using the Crawl Credentials.  You can find it from Search Service Application

image

Try accessing the URL in the browser. If any extra login prompts appears, you need to increase the permissions for Crawling Account.

Solution 4 Turn Off IE Enhanced Security Configuration

image

Restart your machine after this. (If still unresolved, restart all servers in farm & retry)

Solution 5 Lower Internet Explore Security Configuration

Open your Crawling Machine & Lower the Internet Explorer Security settings.

image image image

Restart your machine after this. (If still unresolved, restart all servers in farm & retry)

Solution 6 Re-create Search Service Application

Go to Central Administration, Delete & Re-create the new Search Service Application.

Solution 7 Run Fiddler & Check ULS Logs

You can experiment with Fiddler in examining the underlying request/response.

http://www.telerik.com/fiddler

You can also check ULS Logs with Verbose mode turned on.

Solution 8 Run Cumulative Updates

Find the latest cumulative updates for SharePoint 2013

Solution 9 Update SharePoint with Service Pack

The latest service pack for SharePoint Server 2013 is available at:

http://www.microsoft.com/en-us/download/details.aspx?id=42544

Solution 10 Contact Microsoft Support

If you have Paid Support, Contact Microsoft for fxing this.

References

http://googlewebmastercentral.blogspot.com/2007/03/using-robots-meta-tag.html

Summary

In this post we have explored a weird content source crawling issue & possible solutions to fix that.

Advertisements

Posted in SharePoint, SharePoint 2013 | Tagged: , , , | Leave a Comment »