How to resolve "Indexed, though blocked by robots.txt".



Yesterday I got the above email from Google Search Console. There was an issue on my blog.
I want to talk about the way to resolve it.



How to resolve Indexed, though blocked by robots.txt


I resolved the issue : Indexed, though blocked by robots.txt
This is sort of warnings. It is not a critical error. Anyway it bothered me.
As you know it is related with "robots.txt".


This is my "robots.txt" file before editing.
1
2
3
4
5
6
7
8
User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://jcpark79.blogspot.com/sitemap.xml
cs

There were 5 URLs indexed, though blocked by robots.txt. In my case the main reason was line 5. All URLs included "/search" path. It blocked all path below "/search".
For example, "https://jcpark79.blogspot.com/search/label/".  The path "/label" will be blocked.


So I modified "robots.txt". I added "Allow: /search/label/".
1
2
3
4
5
6
7
8
9
User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Disallow: /search
Allow: /
Allow: /search/label/
  
Sitemap: https://jcpark79.blogspot.com/sitemap.xml
cs
Then Googlebot can crawle paths below "/search/label/". Your case may be different from my case. 
You need to add different paths. 



Check your URLs with Google Search Console


You can check which URLs make issues. 

1. Go to "Google Search Console" and login.
2. Click "Index -> Coverage", then you can see the below screen.

3. Click a warning in the Details. 
4. Then you can see URL lists.



Test your "robots.txt" with "The robots.txt Tester"


You can also test your "robots.txt" blocks URLs. 

1. Click "help" icon. It is in the upper right corner.
2. Search "test robots.txt". 
3. Select "Test your robots.txt with the robots.txt Tester".
4. Then you can test your URLs.



How to modify your "robots.txt"?


If you want to modify your "robots.txt", you have to do on your blog settings. 
I changed my "robots.txt" and submitted it on Google Search Console, but unfortunately it did not work.


How to resolve "Indexed, though blocked by robots.txt". How to resolve "Indexed, though blocked by robots.txt". Reviewed by John.P on 12월 06, 2018 Rating: 5

댓글 없음:

Powered by Blogger.