PHP Tokens From a String

Question

Let's say you have a string that looks like this: token1 token2 tok3

And you want to get all of the tokens (specifically the strings between the spaces), AND ALSO their position (offset) and length).

So I would want a result that looks something like this:

array(
    array(
        'value'=>'token1'
        'offset'=>0
        'length'=>6
    ),
    array(
        'value'=>'token2'
        'offset'=>7
        'length'=>6
    ),
    array(
        'value'=>'tok3'
        'offset'=>14
        'length'=>4
    ),
)

I know that this can be done by simply looping through the characters of the string and I can simply write a function to do this.

I am wondering, does PHP have anything built-in that will do this efficiently or at least help with part of this?

I am looking for suggestions and appreciate any help given. Thanks

Answer 1

You can use preg_match_all with the PREG_OFFSET_CAPTURE flag:

$str = 'token1 token2 tok3';
preg_match_all('/\S+/', $str, $matches, PREG_OFFSET_CAPTURE);
var_dump($matches);

Then you just need to replace the items in $matches[0] like this:

function update($match) {
    return array( 'value' => $value[0], 'offset' => $value[1], 'length' => strlen($value[0]));
}   
array_map('update', $matches[0]);
var_dump($matches[0]);

Answer 2

There's a simpler way, in most respects. You'll have a more basic result, but with much less work put in.

Assuming you have tokena tokenb tokenc stored in $data

$tokens = explode(' ', $data);

Now you have an array of tokens separated by spaces. They will be in order, so $tokens[0] = tokena, $tokens[1] = tokenb, etc. You can very easily get the length of any given item by doing strlen($tokens[$index]); If you need to know how many tokens you were passed, use $token_count = count($tokens);

Not as sophisticated, but next to no work to get it.