How to check if a web element is visible

Question

I am using Python with BeautifulSoup4 and I need to retrieve visible links on the page. Given this code:

soup = BeautifulSoup(html)
links = soup('a')

I would like to create a method is_visible that checks whether or not a link is displayed on the page.

Solution Using Selenium

Since I am working also with Selenium I know that there exist the following solution:

from selenium.webdriver import Firefox

firefox = Firefox()
firefox.get('https://google.com')
links = firefox.find_elements_by_tag_name('a')

for link in links:
    if link.is_displayed():
        print('{} => Visible'.format(link.text))
    else:
        print('{} => Hidden'.format(link.text))

firefox.quit()

Performance Issue

Unfortunately the is_displayed method and getting the text attribute perform a http request to retrieve such informations. Therefore things can get really slow when there are many links on a page or when you have to do this multiple times.

On the other hand BeautifulSoup can perform these parsing operations in zero time once you get the page source. But I can't figure out how to do this.

Answer 1

AFAIK, BeautifulSoup will only help you parse the actual markup of the HTML document anyway. If that's all you need, then you can do it in a manner like so (yes, I already know it's not perfect):

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc)


def is_visible_1(link):
    #do whatever in this function you can to determine your markup is correct
    try:
        style = link.get('style')
        if 'display' in style and 'none' in style:#or use a regular expression
            return False
    except Exception:
        return False
    return True

def is_visible_2(**kwargs):
    try:
        soup = kwargs.get('soup', None)
        del kwargs['soup']
        #Exception thrown if element can't be found using kwargs
        link = soup.find_all(**kwargs)[0]
        style = link.get('style')
        if 'display' in style and 'none' in style:#or use a regular expression
            return False
    except Exception:
        return False
    return True


#checks links that already exist, not *if* they exist
for link in soup.find_all('a'):
    print(str(is_visible_1(link)))

#checks if an element exists
print(str(is_visible_2(soup=soup,id='someID')))

BeautifulSoup doesn't take into account other parties that will tell you that the element is_visible or not, like: CSS, Scripts, and dynamic DOM changes. Selenium, on the other hand, does tell you that an element is actually being rendered or not and generally does so through accessibility APIs in the given browser. You must decide if sacrificing accuracy for speed is worth pursuing. Good luck! :-)