Automatically parsing PHP to separate PHP code from HTML

Question

I'm working on a large PHP code base; I'd like to separate the PHP code from the HTML and JavaScript. (I need to do several automatic search-and-replaces on the PHP code, and different ones on the HTML, and different on the JS). Is there a good parser engine that could separate out the PHP for me? I could do this using regular expressions, but they're not perfect. I could build something in ANTLR, perhaps, but a good already existing solution would be best.

I should make clear: I don't want or need a full PHP parser. Just need to know if a given token is: - PHP code - PHP single quote string - PHP double quote string - PHP Comment - Not PHP, but rather HTML/JavaScript

Answer 1

How about the tokenizer built right into PHP itself?

The tokenizer functions provide an interface to the PHP tokenizer embedded in the Zend Engine. Using these functions you may write your own PHP source analyzing or modification tools without having to deal with the language specification at the lexical level.

You ask in the comments whether you can regenerate the code from the tokenized output - yet you can, all whitespace is preserved as T_WHITESPACE tokens. Here's how you might turn the tokenized output back into code:

$regenerated='';

$tokens = token_get_all($code);
foreach($tokens as $idx=>$t)
{
    if (is_array($t))
    {

         //do something with string and comments here?
         switch($t[0])
         {
             case T_CONSTANT_ENCAPSED_STRING:
                  break;
             case T_COMMENT:
             case T_DOC_COMMENT:
                 break;

         }
         $regenerated.=$t[1];


    }
    else
    {
         $regenerated.=$t;
    }
}

Answer 2

To separate the PHP from the rest, PHP's inbuilt tokenizer is your best choice: See token_get_all()

For the rest, you might be best off with a DOM parser. Isolating the <script> parts (and external scripts, and even onXXXX events) is easy that way.

It might be tough to re-build the identical document from a parsed DOM tree, though - I guess it depends on what you need to do with the results and how clean the original HTML is. A regular expression (yuck!) could work better for that part.