Pandas .loc and PEP8

Question

I've tried to search this a number of times but I don't see it answered so here goes...

I often use pandas to clean up a dataframe and conform it to my needs. With this comes a lot of .loc accessing to query it and return values. Depending on what I am doing (and column lengths), this can get pretty lengthy. Given PEP8 constrains to 79 characters a line, are there any best practices? Some examples below (these are simplified and for explanatory purposes):

missing_address_df = address_df.loc[address_df['address'].notnull()].copy()

or multiple query points:

nc_drive_df = address.loc[(address_df['address'].str.contains('drive')) & (address_df['state'] == 'NC')]

Answer 1

I'd advise two things

• Ignore PEP 8's 80 char advice, but try to keep to 120 or 150 lines
  Keeping some line length requirement makes sense to aid readability, but if you're trying to keep to 80 chars in (for example) a class method, it will lead to worse and less-readable code

  PEP 8 actually has a section on this, A Foolish Consistency is the Hobgoblin of Little Minds, which describes cases you should deviate from its other advice, for example

  1. When applying the guideline would make the code less readable, even for someone who is used to reading code that follows this PEP

• split the .loc contents onto multiple lines

  nc_drive_df = address.loc[
      (address_df['address'].str.contains('drive')) & \
      (address_df['state'] == 'NC')
  ]
  

It's hard to be objective about when code "looks bad", despite being valid syntax, but you will experience it. Practically, PEP 8 and Cyclomatic Complexity checkers are tools which will help you fight against and defend and propose code styles in a scientific way.

---

If you have a great many boolean statements, you (often must) break them up with parentheses to clarify their order

nc_drive_df = address.loc[
    (
        (address_df['address'].str.contains('drive')) & \
        (address_df['state'] == 'NC')
    ) || (
        address_df['zip'] == "00000"
    )
]

This is somewhat in conflict with conventional Python operators, which are suggested to preceed lines (PEP8), but I challenge this when forming a Pandas boolean array because the dataframes must be the same to get a good result and it's likely easier to observe this when working with many dataframes when they are first.

Finally, often when doing scientific Python, you should absolutely try many possibilities against a partial and full data if possible to draw good performance conclusions, consider their readability to be second, and provide excellent comments about and linking to your research, etc. over any particular style.