What are "soft keywords"?

Question

According to the documentation of the keyword module, two new members have been added in Python 3.9:

• issoftkeyword
• softkwlist

However their documentation doesn't reveal anything about their purpose. This change is not even mentioned in the What's New article, where typically all API changes are documented. Digging a little further into the source code eventually leads to this pull request where it is mentioned that "this is essentially an internal tool" and that "soft keywords are still unused". So what's the purpose of Python's soft keywords?

Answer 1

Short: Soft keywords can still be used as variable or argument names.

PEP 622 sheds some light (Emphasis mine):

The difference between hard and soft keywords is that hard keywords are always reserved words, even in positions where they make no sense (e.g. x = class + 1), while soft keywords only get a special meaning in context.

[...] The match and case keywords are proposed to be soft keywords, so that they are recognized as keywords at the beginning of a match statement or case block respectively, but are allowed to be used in other places as variable or argument names.

Answer 2

I think this is best explained with a demo. async and await were soft keywords in Python 3.5 and 3.6, and thus they could be used as identifiers:

>>> async = "spam"
>>> async def foo():
...     pass
...
>>> await = "bar"
>>> async, await
('spam', 'bar')

But in Python 3.7 they became proper keywords and can only be used in specific contexts where they make sense:

>>> async = "123"
  File "<stdin>", line 1
    async = "123"
          ^
SyntaxError: invalid syntax
>>> async def foo():
...     pass
...
>>> await = "bar"
  File "<stdin>", line 1
    await = "bar"
          ^
SyntaxError: invalid syntax
>>> async, await
  File "<stdin>", line 1
    async, await
         ^
SyntaxError: invalid syntax

The idea in first introducing them as soft keywords was mainly to not break any existing code that uses them as identifiers. The same reasoning is with the upcoming match keyword which would completely break for example re.match and millions of projects.