2nd October, 2012
As Twitter opens up to more crawling, the question remains – do search engines actually want its search results shown in theirs? Twitter recently updated its robots.txt file and millions of pages will be opened up to crawling as a result, so Google, Bing, Yahoo and other bots will be able to crawl through Twitter’s search results.
By opening up its content search results in the form of tweets and hashtags, Twitter is allowing Google to search these results, while still preventing bots from crawling its search results for users, videos and images.
There is no guarantee however that the major search engines will want what Twitter is offering. Google remains adamant that it does not wish to show search results pages in its own search results.
In the meantime, of course, online users will be able to locate popular and informative hashtag pages such as that created for the 2012 Olympics etc…
Twitter already has sufficient domain authority to rank well for certain queries, but there is clearly SEO value to them in having their pages show up in Google and Bing’s index systems during a big event.
Twitters tweet and hashtag search results often contain interesting content. Their Olympic hashtag page was the result of a formal partnership with NBC combining human and algorithmically-curated content about the Games.
Whether or not search engines will want Twitter Search Results showing in their search results remains to be seen. It will be interesting to witness developments in the oncoming months.
Kerry McPhail