Get Twitter Feed as JSON without authentication

Question

I wrote a small JavaScript a couple of years ago that grabbed a users (mine) most recent tweet and then parsed it out for display including links, date etc.

It used this json call to retrieve the tweets and it no longer works.

http://twitter.com/statuses/user_timeline/radfan.json

It now returns the error:

{"errors":[{"message":"Sorry, that page does not exist","code":34}]}

I have looked at using the api version (code below) but this requires authentication which I would rather avoid having to do as it is just to display my latest tweet on my website which is public anyway on my profile page:

http://api.twitter.com/1/statuses/radfan.json

I haven't kept up with Twitter's API changes as I no longer really work with it, is there a way round this problem or is it no longer possible?

Answer 1

Previously the Search API was the only Twitter API that didn't require some form of OAuth. Now it does require auth.

Twitter's Search API is acquired from a third party acquisition - they rarely support it and are seemingly unenthused that it even exists. On top of that, there are many limitations to the payload, including but not limited to a severely reduced set of key:value pairs in the JSON or XML file you get back.

When I heard this, I was shocked. I spent a LONG time figuring out how to use the least amount of code to do a simple GET request (like displaying a timeline).

I decided to go the OAuth route to be able to ensure a relevant payload. You need a server-side language to do this. JavaScript is visible to end users, and thus it's a bad idea to include the necessary keys and secrets in a .js file.

I didn't want to use a big library so the answer for me was PHP and help from @Rivers' answer here. The answer below it by @lackovic10 describes how to include queries in your authentication.

I hope this helps others save time thinking about how to go about using Twitter's API with the new OAuth requirement.

Answer 2

You can access and scrape Twitter via advanced search without being logged in:

• https://twitter.com/search-advanced

GET request

When performing a basic search request you get:

https://twitter.com/search?q=Babylon%205&src=typd

• q (our query encoded)
• src (assumed to be the source of the query, i.e. typed)

by default, Twitter returns top 25 results, but if you click on all you can get the realtime tweets:

https://twitter.com/search?f=realtime&q=Babylon%205&src=typd

JSON contents

More Tweets are loaded on the page via AJAX:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd&include_available_features=1&include_entities=1&last_note_ts=85&max_position=TWEET-553069642609344512-553159310448918528-BD1UO2FFu9QAAAAAAAAETAAAAAcAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Use max_position to request the next tweets

The following json array returns all you need to scrape the contents:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd

• has_more_items (bool)
• items_html (html)
• max_position (key)
• refresh_cursor (key)

DOM elements

Here comes a list of DOM elements you can use to extract

The authors twitter handle

div.original-tweet[data-tweet-id]   

The name of the author

div.original-tweet[data-name]

The user ID of the author

div.original-tweet[data-user-id]    

Timestamp of the post

span._timestamp[data-time]  

Timestamp of the post in ms

span._timestamp[data-time-ms]

Text of Tweet

p.tweet-text
 

Number of Retweets

span.ProfileTweet-action–retweet > span.ProfileTweet-actionCount[data-tweet-stat-count] 

Number of Favo

span.ProfileTweet-action–favorite > span.ProfileTweet-actionCount[data-tweet-stat-count]    

Resources

• https://code.recuweb.com/2015/scraping-tweets-directly-from-twitter-without-authentication/

Answer 3

If you're still looking for unauthenticated tweets in JSON, this should work: https://github.com/cosmocatalano/tweet-2-json