PHP-5.6 compile tokenizer error Ubuntu 14.04

Question

I'm setup my new notebook with multiple PHP versions (7 and 5.6), when after compile PHP-7.0 perfectly I get tokenizer error in PHP-5.6 compile process.

Steps

• sudo mkdir /opt/source
• cd /opt/source
• sudo git clone https://github.com/php/php-src.git
• cd php-src
• sudo git checkout PHP-5.6.17
• sudo ./configure --prefix=/opt/php-5.6 --with-pdo-pgsql --with-zlib-dir --with-freetype-dir --enable-mbstring --with-libxml-dir=/usr --enable-soap --enable-calendar --with-curl --with-mcrypt --with-zlib --with-gd --with-pgsql --disable-rpath --enable-inline-optimization --with-bz2 --with-zlib --enable-sockets --enable-sysvsem --enable-sysvshm --enable-pcntl --enable-mbregex --enable-exif --enable-bcmath --with-mhash --enable-zip --with-pcre-regex --with-mysql --with-pdo-mysql --with-mysqli --with-png-dir=/usr --enable-gd-native-ttf --with-openssl --with-fpm-user=nginx --with-fpm-group=nginx --with-libdir=/lib/x86_64-linux-gnu --enable-ftp --with-imap --with-imap-ssl --with-kerberos --with-gettext --with-gd --with-jpeg-dir=/usr/lib/ --enable-fpm

The error

/bin/bash /opt/source/php-src/libtool --silent --preserve-dup-deps --mode=compile cc  -Iext/standard/ -I/opt/source/php-src/ext/standard/ -DPHP_ATOM_INC -I/opt/source/php-src/include -I/opt/source/php-src/main -I/opt/source/php-src -I/opt/source/php-src/ext/date/lib -I/opt/source/php-src/ext/ereg/regex -I/usr/include/libxml2 -I/usr/include/freetype2 -I/usr/include/c-client -I/opt/source/php-src/ext/mbstring/oniguruma -I/opt/source/php-src/ext/mbstring/libmbfl -I/opt/source/php-src/ext/mbstring/libmbfl/mbfl -I/usr/include/postgresql -I/opt/source/php-src/ext/sqlite3/libsqlite -I/opt/source/php-src/ext/zip/lib -I/opt/source/php-src/TSRM -I/opt/source/php-src/Zend    -I/usr/include -g -O2 -fvisibility=hidden  -c /opt/source/php-src/ext/standard/info.c -o ext/standard/info.lo 
/bin/bash /opt/source/php-src/libtool --silent --preserve-dup-deps --mode=compile cc  -Iext/tokenizer/ -I/opt/source/php-src/ext/tokenizer/ -DPHP_ATOM_INC -I/opt/source/php-src/include -I/opt/source/php-src/main -I/opt/source/php-src -I/opt/source/php-src/ext/date/lib -I/opt/source/php-src/ext/ereg/regex -I/usr/include/libxml2 -I/usr/include/freetype2 -I/usr/include/c-client -I/opt/source/php-src/ext/mbstring/oniguruma -I/opt/source/php-src/ext/mbstring/libmbfl -I/opt/source/php-src/ext/mbstring/libmbfl/mbfl -I/usr/include/postgresql -I/opt/source/php-src/ext/sqlite3/libsqlite -I/opt/source/php-src/ext/zip/lib -I/opt/source/php-src/TSRM -I/opt/source/php-src/Zend    -I/usr/include -g -O2 -fvisibility=hidden  -c /opt/source/php-src/ext/tokenizer/tokenizer.c -o ext/tokenizer/tokenizer.lo 
/bin/bash /opt/source/php-src/libtool --silent --preserve-dup-deps --mode=compile cc  -Iext/tokenizer/ -I/opt/source/php-src/ext/tokenizer/ -DPHP_ATOM_INC -I/opt/source/php-src/include -I/opt/source/php-src/main -I/opt/source/php-src -I/opt/source/php-src/ext/date/lib -I/opt/source/php-src/ext/ereg/regex -I/usr/include/libxml2 -I/usr/include/freetype2 -I/usr/include/c-client -I/opt/source/php-src/ext/mbstring/oniguruma -I/opt/source/php-src/ext/mbstring/libmbfl -I/opt/source/php-src/ext/mbstring/libmbfl/mbfl -I/usr/include/postgresql -I/opt/source/php-src/ext/sqlite3/libsqlite -I/opt/source/php-src/ext/zip/lib -I/opt/source/php-src/TSRM -I/opt/source/php-src/Zend    -I/usr/include -g -O2 -fvisibility=hidden  -c /opt/source/php-src/ext/tokenizer/tokenizer_data.c -o ext/tokenizer/tokenizer_data.lo 
In file included from /opt/source/php-src/main/php.h:406:0,
                 from /opt/source/php-src/ext/tokenizer/tokenizer_data.c:26:
/opt/source/php-src/ext/tokenizer/tokenizer_data.c: In function ‘tokenizer_register_constants’:
/opt/source/php-src/ext/tokenizer/tokenizer_data.c:89:40: error: ‘T_CHARACTER’ undeclared (first use in this function)
  REGISTER_LONG_CONSTANT("T_CHARACTER", T_CHARACTER, CONST_CS | CONST_PERSISTENT);
                                        ^
/opt/source/php-src/Zend/zend_constants.h:43:103: note: in definition of macro ‘REGISTER_LONG_CONSTANT’
 #define REGISTER_LONG_CONSTANT(name, lval, flags)  zend_register_long_constant((name), sizeof(name), (lval), (flags), module_number TSRMLS_CC)
                                                                                                       ^
/opt/source/php-src/ext/tokenizer/tokenizer_data.c:89:40: note: each undeclared identifier is reported only once for each function it appears in
  REGISTER_LONG_CONSTANT("T_CHARACTER", T_CHARACTER, CONST_CS | CONST_PERSISTENT);
                                        ^
/opt/source/php-src/Zend/zend_constants.h:43:103: note: in definition of macro ‘REGISTER_LONG_CONSTANT’
 #define REGISTER_LONG_CONSTANT(name, lval, flags)  zend_register_long_constant((name), sizeof(name), (lval), (flags), module_number TSRMLS_CC)
                                                                                                       ^
/opt/source/php-src/ext/tokenizer/tokenizer_data.c:90:44: error: ‘T_BAD_CHARACTER’ undeclared (first use in this function)
  REGISTER_LONG_CONSTANT("T_BAD_CHARACTER", T_BAD_CHARACTER, CONST_CS | CONST_PERSISTENT);
                                            ^
/opt/source/php-src/Zend/zend_constants.h:43:103: note: in definition of macro ‘REGISTER_LONG_CONSTANT’
 #define REGISTER_LONG_CONSTANT(name, lval, flags)  zend_register_long_constant((name), sizeof(name), (lval), (flags), module_number TSRMLS_CC)
                                                                                                       ^
/opt/source/php-src/ext/tokenizer/tokenizer_data.c: In function ‘get_token_type_name’:
/opt/source/php-src/ext/tokenizer/tokenizer_data.c:228:8: error: ‘T_CHARACTER’ undeclared (first use in this function)
   case T_CHARACTER: return "T_CHARACTER";
        ^
/opt/source/php-src/ext/tokenizer/tokenizer_data.c:229:8: error: ‘T_BAD_CHARACTER’ undeclared (first use in this function)
   case T_BAD_CHARACTER: return "T_BAD_CHARACTER";
        ^
make: *** [ext/tokenizer/tokenizer_data.lo] Error 1
root@patrick:/opt/source/php-src# 

Setup

• Ubuntu 14.04.3 x64
• 8gb RAM
• Intel i5

---

If you need more information, tell me before down vote.

Thanks.

Answer 1

I encountered this same error attempting to compile PHP 5.6.30 on Ubuntu 14 and decided to follow up some more. It turned out I was using an unsupported version of GNU Bison. Ubuntu 14.04 provides Bison version 3.x.x. It turns out that PHP5 supports only Bison 2.4 and above (but not greater-than or equal-to 3.x.x). I haven't found an exact source that backs up this claim. However I believe my conclusion is reasonable based on trial and error.

To compile PHP 5.6 on Ubuntu 14.04, I downloaded Bison 2.5 from Launchpad. Find the architecture of your machine and download the corresponding .deb archive. Extract the contents of the package's data section under some directory like so:

$ ar x bison_2.5.dfsg-2.1_ARCH.deb
$ tar xf data.tar.gz

Now you need to point the configure script to use that version of Bison when it configures the build. To do this, set the PATH variable to first point at the location of the custom Bison binary (note: The data archive expands to ./usr/bin/bison under the extraction directory):

$ PATH=/path/to/extraction/usr/bin:$PATH ./configure ...

You should see something like the following in the output:

checking for bison... bison -y
checking for bison version... 2.5 (ok)

Now try make. If the results are like mine it should build properly.

Interestingly, I've built PHP 7.0.9 before using Bison 3 and it worked fine. I'm inclined to think that this is only a limitation for PHP 5 builds.