I need a tool to find duplicates or similar blocks of text in a singular text file or set of text files

Question

I want to automate moving duplicate or similar C code into functions.

This must work under Linux.

Answer 1

A subset of your problem: Detecting duplicate code:

Try: PMD

Duplicate code can be hard to find, especially in a large project. But PMD's Copy/Paste Detector (CPD) can find it for you! CPD has been through three major incarnations:

• First we wrote it using a variant of Michael Wise's Greedy String Tiling algorithm (our variant is described here)

• Then it was completely rewritten by Brian Ewins using the Burrows-Wheeler transform
• Finally, it was rewritten by Steve Hawkins to use the Karp-Rabin string matching algorithm.

...

Note that CPD works with Java, JSP, C, C++, Fortran and PHP code.

Answer 2

Simian (noted earlier) is a good tool for this. I have been using CloneDetective on my project and it works great. CloneDetective is free, so it can't hurt to give it a try.