Need to dump entire DOM tree with element id from selenium server

Question

I have been using python selenium for web automation testing. The key part of automation is to find the right element for a user-visible object in a HTML page. The following API will work most of the time, but not all the time.

find_element_by_xxx,  xxx can be id, name, xpath, tag_name etc. 

When HTML page is too complicated, I would like to search the dom tree. Wonder if it's possible to ask the selenium server to serialize the entire DOM (with the element id that can be used to perform action on through webdriver server). Client side (python script) can do its own search algorithm to find the right element.

Note that python selenium can get the entire html page by

drv.page_source

However, parsing this doesn't give the internal element id from selenium server's point of view, hence not useful.

EDIT1: Paraphrase it to make it more clear (thanks @alecxe): what's needed here is a serialized representation of all the DOM elements (with their DOM structure preserved) in the selenium server, this serialized representation can be sent to the client side (a python selenium test app) which can do its own search.

Answer 1

See my other answer for the issues regarding any attempts at getting Selenium's identifiers.

Again, the problem is to reduce a bunch of find_element calls so as to avoid the round-trips associated with them.

A different method from my other answer is to use execute_script to perform the search on the browser and then return all the elements needed. For instance, this code would require three round-trips but can be reduced to just one round-trip:

el, parent, text = driver.execute_script("""
var el = document.querySelector(arguments[0]);
return [el, el.parentNode, el.textContent];
""", selector)

This returns an element, the element's parent and the element's textual contents on the basis of whatever CSS selector I wish to pass. In a case where the page has jQuery loaded, I could use jQuery to perform the search. And the logic can get as complicated as needed.

This method takes care of the vast majority of cases where reducing round-trips is desirable but it does not take care of a scenario like the one I've given in illustration in my other answer.

Answer 2

Try:

find_elements_by_xpath("//*")

That should match all elements in the document.

UPDATE (to match question refinements):

Use javascript and return the DOM as a string:

execute_script("return document.documentElement.outerHTML")