Python generic type annotation for pandas series

Question

I am encountering a type checking error when running my code through typeguard.

I the following file in my package:

from __future__ import annotations
import pandas as pd

def col_sum(x: pd.Series[float]) -> pd.Series[float]:
   return x*(x.sum()-1)

Originally, I was using pd.Series for these type-hints, but when I run my code through mypy it requires that I explicitly subscript a type on the generic type, so I changed it to pd.Series[float] (this makes sense as it is more explicit and allows mypy to be certain that x.sum() is a float).

I also have a test script which performs some unit tests on this function.

However, my test-suite also includes running pytest --typeguard-packages={MyPackage}, but this is failing due to this function. Here is the error:

>   def col_sum(x: pd.Series[float]) -> pd.Series[float]:
E   TypeError: 'type' object is not subscriptable

From my understanding, this is because mypy is importing the pd.Series from the pandas-stubs package which is a generic type (thus the requirement for the subscript). But pytest passes my code through the install_import_hook() function from typeguard which imports pd.Series as a class/type from the regular pandas package (which cannot be subscripted).

How can I make my code pass both of these tests?

For clarity, I am using a trimmed down version of the hypermodern cookiecutter template. I am using nox to manage my test-suite, and it runs the following Sessions:

* pre-commit -> Lint using pre-commit.
* safety -> Scan dependencies for insecure packages.
* mypy-3.10 -> Type-check using mypy.
* mypy-3.11 -> Type-check using mypy.
* tests-3.10 -> Run the test suite.
* tests-3.11 -> Run the test suite.
* coverage -> Produce the coverage report.
* typeguard -> Runtime type checking using Typeguard.
* xdoctest-3.10 -> Run examples with xdoctest.
* xdoctest-3.11 -> Run examples with xdoctest.
* docs-build -> Build the documentation.

All of which are passing, except for typeguard Relevant package versions:

python = "3.10.11"
pandas = "2.1.4"
pandas-stubs = "2.1.4.231218"
typeguard = "4.1.5"
pytest = "7.4.3"

Answer 1

I had a similar problem with pd.Series: mypy gave me error: Missing type parameters for generic type "Series" [type-arg], but when I added the type parameter in brackets, pylint would tell me TypeError: 'type' object is not subscriptable. I concluded, like you did, that mypy was reading the declaration of pd.Series from the pandas-stubs package, whereas pylint was reading the definition from pandas.

I got the errors to vanish by following dsz's answer to a related question How to specify the type of pandas series elements in type hints. By adding:

from __future__ import annotations

at the top of my file, it changed the timing of when the annotations were evaluated (see PEP 563), and now pylint appears to be happy with the pd.Series[Any] syntax.