Anonymize specific columns with pii in pandas dataframe python

Question

I have loaded an s3 bucket with json files and parsed/flattened it in to a pandas dataframe. Now i have a dataframe with 175 columns with 4 columns containing personally identifiable information.

I am looking for a quick solution anonymising those columns (name & adress). I need to keep information for multiples so that if names or adresses of the same person occuring multiple times have the same hash.

Is there existing functionality in pandas or some other package i can utilize for this?

Answer 1

Using a Categorical would be an efficient way to do this - the main caveat is that the numbering will be based solely on the ordering in the data, so some care will be needed if this numbering scheme needs to be used across multiple columns / datasets.

df = pd.DataFrame({'ssn': [1, 2, 3, 999, 10, 1]})

df['ssn_anon'] = df['ssn'].astype('category').cat.codes

df
Out[38]: 
   ssn  ssn_anon
0    1         0
1    2         1
2    3         2
3  999         4
4   10         3
5    1         0

Answer 2

You can using ngroup or factorize from pandas

df.groupby('ssn').ngroup()
Out[25]: 
0    0
1    1
2    2
3    4
4    3
5    0
dtype: int64

---

pd.factorize(df.ssn)[0]
Out[26]: array([0, 1, 2, 3, 4, 0], dtype=int64)

---

In sklearn, if you are doing ML , I will recommend this approach

from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le.fit(df.ssn).transform(df.ssn)

Out[31]: array([0, 1, 2, 4, 3, 0], dtype=int64)