Understanding Python syntax in lists vs series

Question

I am new to Python (no computer science background) for data science. I keep hearing that Python is easy, but I am making incremental progress. As an example, I understand:

len(titles[(titles.year >= 1950) & (titles.year <=1959)])

"In the titles dataframe, create a series and take from the year column of the titles dataframe anything greater than or equal to 1950 AND anything less than or equal to 1959. The take the length of it."

But when I encounter the following, I don't understand the logic of:

t = titles
(t.year // 10 * 10).value_counts().sort_index().plot(kind='bar')

or

titles.title.value_counts().head(10)

In both these cases, I can piece it together obviously. But it is not clear. In the second, why does Python not allow me to use square brackets and regular brackets like in the first example?

Answer 1

This is not about lists vs pd.Series, but rather about the function of parentheses (()) vs brackets ([]) in Python.

Parentheses are used in two main cases: to modify the order of precedence of operations, and to delimit arguments when calling functions.

The difference between 1 + 2 * 3 and (1 + 2) * 3 is obvious, and if you want to pass a and b to a function f, f a b will not work, unlike in, say, Haskell.

We are concerned mostly with the first use here; for example, in this line:

(t.year // 10 * 10).value_counts().sort_index().plot(kind='bar')

Without the parentheses, you would be calling that chain of methods on 10, which wouldn't make sense. Clearly, you want to call them on the result of the parenthesised expression.

Now, in mathematics, brackets can also be used to denote precedence, in conjunction with parentheses, in a case where multiple nested parentheses would be confusing. For example, the two may be equivalent in mathematics:

[(1 + 2) * 3] ** 4
((1 + 2) * 3) ** 4

However, that is not the case in Python: ((1 + 2) * 3) ** 4 can be evaluated, whereas [(1 + 2) * 3] ** 4 is a TypeError, since the part within brackets resolves to a list, and you can't perform exponentiation on lists.

Rather, what happens in something like titles[titles.year >= 1950] is not directly relevant to precedence (though of course anything outside the brackets will not be part of the inner expression).

Instead, the brackets represent indexing; in some way, the value of titles.year >= 1950 is used to get elements from titles (this is done using overloading of the __getitem__ dunder method).

The exact nature of this indexing may differ; lists take integers, dicts take any hashable object and pd.Series take, among other things, boolean pd.Series (that is what is happening here), but they ultimately represent some way to subset the indexed object.

Semantically, therefore, we can see that brackets mean something different from parentheses, and are not interchangeable.

For completeness, using brackets as opposed to parentheses has one tangible benefit: it permits reassignment, because it automatically delegates to either __setitem__ or __getitem__, depending on whether assignment is being performed.

Therefore, you could do something like titles[titles.year >= 1950] = 'Nothing' if you wanted. However, in all cases, titles(titles.year >= 1950) = 'Nothing' delegates to __call__, and therefore will fail in the following way:

SyntaxError: can't assign to function call

Answer 2

Square brackets are used for indexes on lists and dictionaries (and things that act like these). On the other hand, parentheses are used for a variety of reasons. In this case, they are used for grouping in (t.year // 10 * 10) or as a function call in value_counts() and other places.

In the case of a library like pandas, whether you use indexing notation with [] or a function call is entirely determined by the implementation of the library. You can learn these details through tutorials and the library's documentation.

Before digging deeper into the pandas library, I suggest that you study the basics of Python syntax. The official tutorial is a good place to start.

On a side note, when you write code, do not make each line as complex as what you see in these examples. You should instead break things into smaller pieces and assign intermediate parts to variables. For example, you can take

(t.year // 10 * 10).value_counts().sort_index().plot(kind='bar')

and turn it into

decade = (t.year // 10 * 10)
counts = decated.value_counts()
sorted = counts.sort_index()
sorted.plot(kind='bar')

Answer 3

t = titles
(t.year // 10 * 10).value_counts().sort_index().plot(kind='bar')

titles is a data frame. year is a column in that frame. In order, the operations are

• Divide the year by 10 (integer division) and multiply by 10. This truncates the last digit to 0, so that each year is the beginning of its decade. The result of this is another column, the same length as the original.
• Count the values; this will produce a new table with an entry (year, frequency) for each decade-year.
• Sort this table by the default index
• Make a bar plot of the result.

Does that get you going?