Add a new cpython keyword

Question

I'm currently on a cpython guide: https://realpython.com/products/cpython-internals-book/

I will detail what i did and what the problem is. So as it says on the book, i cloned the github :

git clone directory https://github.com/python/cpython

Then i compiled:

./configure --with-pydebug --enable-shared
make -j2 -s

Then, on the part that talks about grammar, i modified the python.gram file and on the small_stmt line i made an addition:

| ('pass' | 'proceed') { _Py_Pass(EXTRA) }

Then i tried to run the command make regen-pegen. I had an error with the dataclass module so i decided to start from scratch.

After a new installation, i ran the command make all. I don't know if it's a mistake but i don't have python 3.9 anymore but python 3.10.

This time the command make regen-pegen works but when i try to use it, i get a proceed is not defined error.

I tried to remove the keyword pass from python.gram to see if the changes were taken into account.

While trying to recompile I got a pass is not defined error so the changes made to the file worked fine.

I then checked that the new keyword does appear on the generated C file and yes it is in it.

    (KeywordToken[]) {{NULL, -1}},
    (KeywordToken[]) {{NULL, -1}},
    (KeywordToken[]) {
        {"if", 509},
        {"in", 517},
        {"ace", 519},
        {"is", 526},
        {"gold", 532},
        {NULL, -1},
    },
    (KeywordToken[]) {
        {"del", 502},
        {"try", 510},
        {"for", 516},
        {"def", 522},
        {"not", 525},
        {"and", 533},
        {NULL, -1},
    },
    (KeywordToken[]) {
        {"from", 513},
        {"elif", 514},
        {"else", 515},
        {"with", 518},
        {"True", 527},
        {"None", 529},
        {"pass", 530},
        {NULL, -1},
    },
    (KeywordToken[]) {
        {"raise", 501},
        {"yield", 503},
        {"break", 505},
        {"while", 511},
        {"class", 523},
        {"False", 528},
        {NULL, -1},
    },
    (KeywordToken[]) {
        {"return", 500},
        {"assert", 504},
        {"global", 507},
        {"import", 512},
        {"except", 520},
        {"lambda", 524},
        {NULL, -1},
    },
    (KeywordToken[]) {
        {"finally", 521},
        {"proceed", 531},
        {NULL, -1},
    },
    (KeywordToken[]) {
        {"continue", 506},
        {"non-local", 508},
        {NULL, -1},
    },
};

I don't understand why the change is not taken into account. Thank you

Answer 1

I don't have the book and this is my first time editing Python's grammar, but I got it to work.

---

1. Build the "regular" Python 3.10 and install to some temporary directory, like cpython/build/INSTALL:

   git clone https://github.com/python/cpython
   mkdir cpython/build && cd cpython/build
   mkdir INSTALL
   ../configure --prefix=/full/path/to/cpython/build/INSTALL
   make -j4 install
   

   I think some earlier version like 3.9 might work (I got an error about the walrus operator on 3.6 and some other error on 3.8, so these seem to be "too old" for building 3.10), but Python builds quickly, so why not do it, just in case.

2. Modify python.gram:

   small_stmt[stmt_ty] (memo):
       | assignment
       ...
       | ('pass' | 'proceed') { _Py_Pass(EXTRA) }  # HERE
       | &'del' del_stmt
       ...
   

3. Use the newly built Python to regenerate the parser:

   $ make PYTHON_FOR_REGEN=INSTALL/bin/python3 regen-pegen
   PYTHONPATH=../Tools/peg_generator INSTALL/bin/python3 -m pegen -q c \
           ../Grammar/python.gram \
           ../Grammar/Tokens \
           -o ../Parser/parser.new.c
   python3 ../Tools/scripts/update_file.py ../Parser/parser.c ../Parser/parser.new.c
   $
   

4. Clean (might also want to reconfigure to install to a different path) and rebuild:

   make clean
   rm -rf INSTALL/*
   make -j4 install
   

   This will install to build/INSTALL, as earlier.

Now, the fun part. You get this error:

# huge huge huge traceback...
File "/var/folders/qv/bsx06c394yzb1dlrtcj_9gkc0000gn/T/tmpzt58rw8n/pip-20.2.3-py2.py3-none-any.whl/pip/_vendor/distlib/util.py", line 318
    def proceed(prompt, allowed_chars, error_prompt=None, default=None):
        ^
SyntaxError: invalid syntax

As you can see, the new keyword is there, and now you can't have a function called proceed, so some things break. In this case, pip broke, so this modified version of Python won't have pip, but you could as well go and edit this particular file, see if anything else breaks, edit that etc.

But who cares - Python itself was built successfully:

$ INSTALL/bin/python3 
Python 3.10.0a2+ (heads/master-dirty:f3cb814315, Nov  5 2020, xx:xx:xx) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> proceed # works!
>>> pass
>>>