Twint: CRITICAL:root:twint.feed:Follow:IndexError for any call

Question

import twint
import os, requests, re, time

c = twint.Config()
c.Username = <anyusername> #Replace with an actual uname in quotes
c.Store_object = True
c.Limit = 10
try:
    twint.run.Followers(c)
except:
    print("Unexpected error:", sys.exc_info()[0])
f = twint.output.follows_list
print(f)

Output

CRITICAL:root:twint.feed:Follow:IndexError
[]

Have done

pip install twint
pip install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint  

Googling, a lot of people have faced this error - but I cannot really find a solution

Running the twint command line also gives the same error

twint -u <uname> --followers
CRITICAL:root:twint.feed:Follow:IndexError

This is not only followers. Anything I try, I get a similar error.

Running Python 3.8.1 on Windows 10
Twint - latest version - 2.1.21

Answer 1

Legacy mobile Twitter version will shut down on December 15th 2020. (M2 mobile web)

That's the headline of a reddit thread talking about the recent shutdown of the M2 Mobile Web ("Legacy") Twitter version; from this date on Twitter will only support these browsers.

If you take a look at the GitHub repository of Twint:

_usr.followers = int(ur['data']['user']['legacy']['followers_count'])

... it uses the legacy attribute and therefore won't work from the 15th of December 2020.

A contributor to the GitHub repo, namely @himanshudabas, also mentioned this in a reply to an identical issue:

That's because twitter killed the no js (mobile version) version of twitter on 15th December. Followers were scraped using that version.

---

You're not the only one facing this problem (obviously), and unfortunately with the Twint module there is no workaround (currently?).

Speaking of the current situation you would need to either use the official Twitter API or alternative wrappers / unofficial APIs.

Answer 2

Looks like it could be answered here.

That's because twitter killed the no js (mobile version) version of twitter on 15th December. Followers were scraped using that version.

In fact, it looks like the following twitter parser uses html processing (BeautifulSoup as we see).

Also there is a workaround, which described there. Probably, it is dead too, due we see nojs version of site in the description.

Looks like, you have to use Twitter API.

Also next time you can just trace your requests instead of asking quesions here, and see what actually happens. Probably Wireshark can be a little bit difficult with https, but you can just hook requests directly from python. Here is an example: Log all requests from the python-requests module.