Splitting a multibyte string in Lua

Question

I have a multibyte string in Lua.

local s = "あいうえお"

How do I take the string and split it into a table of strings?

In English texts, then I can use this code. But this does not work with the multibyte.

local s = "foo bar 123"
local words = {}
for word in s:gmatch("%w+") do
    table.insert( words, word )
end

Answer 1

As others have noted, it's hard to tell what you want to do: where do you want to split for non-ASCII characters, if splitting at spaces doesn't suffice?

If you just want to split between individual characters for non-ASCII characters, something like the following may suffice:

s = "oink barf 頑張っています"
for word in s:gmatch("[\33-\127\192-\255]+[\128-\191]*") do
   print (word)
end

produces:

oink
barf
頑
張
っ
て
い
ま
す

The trick here is that in UTF-8, multi-byte characters each consist of a "lead byte" with the top two bits equal to 11 (so \192–\255 in Lua—remember, character escapes in Lua are decimal), followed by zero or more "trailing bytes" with the top two bits equal to 10 (\128–\191 in Lua).