Selenium is not loading TikTok pages

Question

I'm implementing a TikTok crawler using selenium and scrapy

start_urls = ['https://www.tiktok.com/trending']
....
def parse(self, response):
    options = webdriver.ChromeOptions()
    from fake_useragent import UserAgent
    ua = UserAgent()
    user_agent = ua.random
    options.add_argument(f'user-agent={user_agent}')
    options.add_argument('window-size=800x841')
    driver = webdriver.Chrome(chrome_options=options)
    driver.get(response.url)

The crawler open Chrome but it does not load videos. Image loading

The same problem happens also using Firefox No loading page using Firefox

The same problem using a simple script using Selenium

from selenium import webdriver
import time


driver = webdriver.Firefox()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

driver = webdriver.Chrome()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

Answer 1

Did u try to navigate further within the selenium browser window? If an error 404 appears on following sites, I have a solution that worked for me:

I simply changed my User-Agent to "Naverbot" which is "allowed" by the robots.txt file from Tik Tok

(Robots.txt)

After changing that all sites and videos loaded properly.

Other user-agents that are listed under the "allow" segment should work too, if you want to add a rotation.