Yesterday I got the above email from Google Search Console. There was an issue on my blog.
I want to talk about the way to resolve it.
How to resolve Indexed, though blocked by robots.txt
I resolved the issue : Indexed, though blocked by robots.txt
This is sort of warnings. It is not a critical error. Anyway it bothered me.
As you know it is related with "robots.txt".
This is my "robots.txt" file before editing.
1
2
3
4
5
6
7
8
|
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://jcpark79.blogspot.com/sitemap.xml
| cs |
There were 5 URLs indexed, though blocked by robots.txt. In my case the main reason was line 5. All URLs included "/search" path. It blocked all path below "/search".
For example, "https://jcpark79.blogspot.com/search/label/". The path "/label" will be blocked.
So I modified "robots.txt". I added "Allow: /search/label/".
1
2
3
4
5
6
7
8
9
|
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Allow: /search/label/
Sitemap: https://jcpark79.blogspot.com/sitemap.xml
| cs |
Then Googlebot can crawle paths below "/search/label/". Your case may be different from my case.
You need to add different paths.
Check your URLs with Google Search Console
You can check which URLs make issues.
2. Click "Index -> Coverage", then you can see the below screen.
3. Click a warning in the Details.
4. Then you can see URL lists.
Test your "robots.txt" with "The robots.txt Tester"
You can also test your "robots.txt" blocks URLs.
1. Click "help" icon. It is in the upper right corner.
2. Search "test robots.txt".
3. Select "Test your robots.txt with the robots.txt Tester".
4. Then you can test your URLs.
How to modify your "robots.txt"?
If you want to modify your "robots.txt", you have to do on your blog settings.
I changed my "robots.txt" and submitted it on Google Search Console, but unfortunately it did not work.
How to resolve "Indexed, though blocked by robots.txt".
Reviewed by John.P
on
12월 06, 2018
Rating:
댓글 없음: