How to build a tokenizer in PHP?

Question

I'm building a site to learn basic programming, I'm going to use a pseudolanguage in which users can submit their code and I need to interpret it. However I'm not sure how to build a tokenizer in PHP.

Having a snippet such as this one:

a = 1
b = 2
c = a - b

if(a > b) {
    buy(a)
    } else {
    buy(b)
    }

How would I go about separating this code into tokens?

--

This is what I'm trying now:

$tokens = array();

// First token (define string)
$token  = strtok($botCode, '=');
$tokens[] = $token;

// Loop
while($token) {
    $token  = strtok('=');
    $tokens[] = $token;
}

However I haven't been able to figure out how to use strtok with a list of regular expresions... I could do something similar to strtok but that accepts arrays as needles with substr and strrpos but it seems to me that it should be possible to do it with strtok as it's designed just for this. Any info or pointing in the right direction will be thanked

Answer 1

Do not wait some magic from strtok. It is similar to preg_split.

I think that you want to build your own lexer. So you could use article Writing a simple lexer in PHP or something else.