Are Python3.5 tuple comprehension really this limited?

Question

I've been loving the tuple comprehensions added to Python3.5:

In [128]: *(x for x in range(5)),
Out[128]: (0, 1, 2, 3, 4)

However, when I try to return a tuple comprehension directly I get an error:

In [133]: def testFunc():
     ...:     return *(x for x in range(5)),
     ...: 
  File "<ipython-input-133-e6dd0ba638b7>", line 2
    return *(x for x in range(5)),
           ^
SyntaxError: invalid syntax    

This is just a slight inconvenience since I can simply assign the tuple comprehension to a variable and return the variable. However, if I try and put a tuple comprehension inside a dictionary comprehension I get the same error:

In [130]: {idx: *(x for x in range(5)), for idx in range(5)}
  File "<ipython-input-130-3e9a3eee879c>", line 1
    {idx: *(x for x in range(5)), for idx in range(5)}
          ^
SyntaxError: invalid syntax

I feel like this is a bit more of a problem since comprehsions can be important for performance in some situations.

I have no problem using dictionary and list comprehensions in these situations. How many other situations is the tuple comprehension not going to work when others do? Or perhaps I'm using it wrong?

It makes me wonder what the point was if it's use is so limited or perhaps I am doing something wrong? If I'm not doing something wrong then what is the fastest/most pythonic way to create a tuple that is versitile enough to be used in the same way as list and dictionary comprehensions?

Answer 1

TLDR: If you want a tuple, pass a generator expression to tuple:

{idx: tuple(x for x in range(5)) for idx in range(5)}

---

There are no "tuple comprehensions" in Python. This:

x for x in range(5)

is a generator expression. Adding parentheses around it is merely used to separate it from other elements. This is the same as in (a + b) * c, which does not involve a tuple either.

The * symbol is for iterator packing/unpacking. A generator expression happens to be an iterable, so it can be unpacked. However, there must be something to unpack the iterable into. For example, one can also unpack a list into the elements of an assignment:

*[1, 2]                         # illegal - nothing to unpack into
a, b, c, d = *[1, 2], 3, 4      # legal - unpack into assignment tuple

Now, doing *<iterable>, combines * unpacking with a , tuple literal. This is not useable in all situations, though - separating elements may take precedence over creating a tuple. For example, the last , in [*(1, 2), 3] separates, whereas in [(*(1, 2), 3)] it creates a tuple.

In a dictionary the , is ambiguous since it is used to separate elements. Compare {1: 1, 2: 2} and note that {1: 2,3} is illegal. For a return statement, it might be possible in the future.

If you want a tuple, you should use () whenever there might be ambiguity - even if Python can handle it, it is difficult to parse for humans otherwise.

When your source is a large statement such as a generator expression, I suggest to convert to a tuple explicitly. Compare the following two valid versions of your code for readability:

{idx: tuple(x for x in range(5)) for idx in range(5)}
{idx: (*(x for x in range(5)),) for idx in range(5)}

Note that list and dict comprehensions also work similar - they are practically like passing a generator expression to list, set or dict. They mostly serve to avoid looking up list, set or dict in the global namespace.

---

I feel like this is a bit more of a problem since comprehsions can be important for performance in some situations.

Under the covers, both generator expressions and list/dict/set comprehensions create a short-lived function. You should not rely on comprehensions for performance optimisation unless you have profiled and tested them. By default, use whatever is most readable for your use case.

dis.dis("""[a for a in (1, 2, 3)]""")
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x10f730ed0, file "<dis>", line 1>)
              2 LOAD_CONST               1 ('<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_CONST               5 ((1, 2, 3))
              8 GET_ITER
             10 CALL_FUNCTION            1
             12 RETURN_VALUE