How to Fix Twitter Card Validator Error: Robots Denied

If you encountered this Twitter ERROR: Fetching the page failed because it’s denied by robots.txt and you want to know how to resolve it, you are in the right place.

Many of you might be wondering, what is a technology-related-topic doing on a poetry website? I will tell you why:

Simply because I like to write poems.
I know something about computer stuff.
I like to help other people, as well.



If I have a technical problem—whether on my website or while accessing an application—and it took me long to find a solution online, often, I try to figure out the answer myself. When issues tend to be repetitive, either I jot down notes or write about them and share through this blog.

I am grateful to have Siteground, a fairly reliable partner for the seamless hosting of my website.

Back to the Twitter error.

When I shared the URL of my post via Twitter, I wondered why the image was not showing. Only the link is visible.

I quite remember how to debug sharing error on Facebook. Normally, I go to their Facebook debug link to identify the cause.

I thought that Twitter would have a very similar tool and true enough, searching online led me to the Twitter Card Validator.

To use, just enter the post’s URL and hit Preview Card:

How to Fix Twitter Card Validator Error Robots Denied

As you can see above, “Fetching the page failed because it’s denied by robots.txt”.

Yes, I have robots.txt on my website and I also use Yoast SEO plugin. But nowhere in my robots text file I can see anything that possibly blocks Twitter bots from crawling.

YOAST SEO & TWITTER METADATA
I found out that my Yoast SEO setting for Twitter was incomplete so I completed that first.

On your WordPress dashboard:
Head over to SEO > Social and add the Twitter username on the field. Still on the SEO > Social page, go to Twitter tab, under Twitter Settings:

Enable Add Twitter card meta data. Doing this will add meta data to the section of your website.

Select the default card type to use. In my case, I picked Summary with large image.

Save the changes.

Using the Google structured data testing tool, I confirmed that the Twitter meta data is there. However, the Twitter Card Validation still have the same error.



ALLOW TWITTERBOT
The next thing to do: Make sure Twitterbot is able to crawl your posts.

Open your robots.txt file via File Editor. You can edit it directly from your website’s File Manager, or if you want to access via Yoast: head over to SEO > Tool > File Editor:

If anyone asks you how to add Twitterbot user agent on your robots.txt file, grab your keyboard and type,

LIKE THIS:
User-agent: Twitterbot
Disallow:

DO NOT DO LIKE THIS: It won’t work.
User-agent: Twitterbot
Allow: *

Go back to the Card Validator link and attempt to Preview Card. Here’s the expected output:

How to Fix Twitter Card Validator Error Robots Denied

No more error! The content is showing but there is no image.

Keep calm. As long as you completed the Yoast SEO plugin setting and allowed Twitterbot on the robots.txt file, you are good. Wait for an at least an hour then try to validate the URL one more time. See below:

How to Fix Twitter Card Validator Error Robots Denied

SUMMARY
It is recommended to use the following popular validations tools to fix social media sharing error:

Facebook: https://developers.facebook.com/tools/debug/sharing/
Google: https://search.google.com/structured-data/testing-tool
Twitter: https://cards-dev.twitter.com/validator


If you think this post helps, please share or leave your comment. Small gesture can have meaningful impact.

Ymatruz is the author of poetry book: The Coffee Cries Foul. She founded PoetsGig.com hoping to accommodate works of other poets and writers, anywhere in the world. She also writes about her blogging and migration experience on PoetsGig. Her motto: "By experience you learn. By embracing mistakes, you become a master of perfection."
Follow on: Twitter | Facebook | Instagram


Thank you for reading. Tell us what you think.