Faster way to change dataframe's row value based upon condition

Question

I have a large dataframe containing millions of records,

---

---

Lists that I am using in my code are,

image_jpg= ['image/jpeg','image/jpg','image/pjpeg']
image_png = ['image/png','image/x-png','application/png']
image_gif = ['image/gif']

---

I want to make a new column named name such that, for example:

Index 0 has content_type value image/jpeg that is in the list image_jpg, so, name column get value of 5efc61356f85e500694bcbbbbb3ee4c2.jpg ( sys_id column + .jpg)

---

Right now I am achieving this via:

file_name = []
for index, row in df.iterrows():
    if row['content_type'] in image_jpg:
        file_name.append(str(row['sys_id'])+'.jpg')
    elif row['content_type'] in image_png:
        file_name.append(str(row['sys_id'])+'.png')
    elif row['content_type'] in image_png:
        file_name.append(str(row['sys_id'])+'.gif')
    else:
        file_name.append(str(row['sys_id']))

df['name'] =  file_name

Output:

Problem is, it takes quite a long time, since dataframe is quite large.

Is there a faster way to accomplish this task ?

Answer 1

Use a dictionary and column-wise operations:

d = {'image_jpg': ['image/jpeg','image/jpg','image/pjpeg'],
     'image_png': ['image/png','image/x-png','application/png'],
     'image_gif': ['image/gif']}

d_rev = {w: k for k, v in d.items() for w in v}

for k, v in d_rev.items():
    mask = df['content_type'].str.contains(v, regex=False)
    df.loc[mask, 'name'] = df.loc[mask, 'sys_id'] + '.' + k.split('/')[-1]

Or, if equality is required:

for k, v in d_rev.items():
    mask = df['content_type'].eq(v)
    df.loc[mask, 'name'] = df.loc[mask, 'sys_id'] + '.' + k.split('/')[-1]

For the equality case, @AntonvBR's pd.Series.map solution is better.

Explanation

d_rev maps each list value to a key:-

print(d_rev)

{'application/png': 'image_png', 'image/gif': 'image_gif',
 'image/jpeg': 'image_jpg', 'image/jpg': 'image_jpg',
 'image/pjpeg': 'image_jpg', 'image/png': 'image_png',
 'image/x-png': 'image_png'}

Given there are very few categories and a large number of rows, it is more efficient to iterate the dictionary and use optimized column-wise operations. Remember iterrows is just a slow row-wise loop, it will always be inefficient for a large number of rows.