beautifulsoup get URL from javascript variable

Question

I'm trying to get URL from a javascript through beautifulsoup. I have the following javascript source

<script type="text/javascript">
            var abc_url = "http://www2.example.com/ar/send/0?tk=13_s&id=12345678&l=9";
            var etc = [
                    'http://xyz.example.com/content/1.png',
                     'http://xyz.example.com/content/2,png' ];
</script>

I've tried the following statement in python but "print m" returns None.

soup = BeautifulSoup(page)

p = re.compile('/var abc_url = (.*);/')        
all_script  = soup.find_all("script", {"src":False})
for individual_script in all_script:
    all_value =  individual_script.string        
    if all_value:            
        m = p.match(all_value)
        print m           

Using RegExr it seems to be able to get the entire line of "var abc_url..." based on above regex but in my code it doesn't work. Wondering how can I get the URL value of this?

Thank you

Answer 1

You can't parse Javascript with BeautifulSoup. Essentially you can get the contents of the script tag with BS and then start processing the Javascript as text with stock python. Like simple string processing with str.split or more complex processing with regular expressions. The following code prints the string you are looking for:

p = re.compile('var abc_url = (.*);')        
for script in soup.find_all("script", {"src":False}):
    if script:            
       m = p.search(script.string)
       print m.group(1)

Be sure to use re.search instead of re.match, because re.match only matches at the start of the string, but you have leading blanks in your string. And remove the slashes from your regex string.

Finally the return type of both re.search and re.match are so called Match objects that evaluate to a boolean value. When there's a match the Match object the group method returns the match groups. group(0) returns the whole match, group(1) the first parenthesized subgroup and so on.