Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Bash diff body text of html file only

Tags:

html

shell

diff

I'm writing a shell script which tracks the changes of a website and emails me with the contents of the change if one occurs. The idea is to use wget to grab a copy of the html and compare it to the version from the last time the script ran. Wget works fine to save the html file but I'm having trouble comparing the files. The trouble is that I'm only interested in changes in the html file's plain text, not the code, links, etc.

Diff works to find all the changes in the two files but it ALWAYS returns changes even when the plain text is identical. This is because each link on the site has a corresponding authenticity token that differs each time the page is accessed. In order to diff only the lines that include plain text I'm attempting to filter it to exclude any line that begins with "<" OR "(any_amount_of_spaces)<". I've looked at the diff man page but I can't seem to find an operator that will do what I need. I don't know much about REGEX but would that work with diff -I for this?

Thanks!

like image 615
James_M Avatar asked Oct 13 '26 07:10

James_M


1 Answers

You could use lynx -dump to render the pages and feed those to diff, but since you are not interested in links you would need to get rid of the References section that this yields (with e.g. awk) rendering this a not-so-robust solution (but maybe good enough for your use case).

If you don't mind using something 3rd-party go for html2text:

diff <(html2text before.html) <(html2text after.html)

PS: There are two different programs called html2text.

like image 142
Adrian Frühwirth Avatar answered Oct 14 '26 20:10

Adrian Frühwirth