Scraping a json request `ak` parameter value

Question

I'm currently trying to scrape embedded m3u8 url paths for self-study.

So far, I managed to pin down the request produces a json response with the m3u8 information.

For example, the https://headlines.yahoo.co.jp/videonews/ann?a=20190527-00000051-ann-soci page would make the following request:

https://feapi-yvpub.yahooapis.jp/v1/content/1579522?appid=dj0zaiZpPVZMTVFJR0FwZWpiMyZzPWNvbnN1bWVyc2VjcmV0Jng9YjU-&output=json&space_id=2078710307&domain=headlines.yahoo.co.jp&ak=e25b66ca8b37b7a383feecf4e084fe95&device_type=1100&thumb_width=1204&thumb_height=676&thumb_priority=l&thumb_bd=0

In order to make a successful request, the contentid (1579522), appid, space_id, device_type, and ak parameters are required. I managed to scrape the id and device_type values but have no idea where to get the ak value, which I guess stands for access key. Any idea on how to get this value?

Side note: I accessed the page through different browsers and they all produced the same ak parameter value in the request, so I'm guessing the value is not uniquely linked to each session, but I could be wrong.

Thanks in advance!

Answer 1

The mystery argument ak is created by first combining space_id and domain into a single string such as "2078710307_headlines.yahoo.co.jp". Then it is run through an obfuscator which is located in player.script.js.

The obfuscator looks a bit complicated and seems to be made hard to understand on purpose. Below are debugger screenshots from the obfuscator main function, and a subroutine that it calls.

While a bit daunting, it doesn't seem impossible to convert this to a Python program so that you could then get ak by obfuscating any space_id and domain you needed in youtube-dl. Good luck.