Efficient way to access a mapping of identifiers in Python

Question

I am writing an app to do a file conversion and part of that is replacing old account numbers with a new account numbers.

Right now I have a CSV file mapping the old and new account numbers with around 30K records. I read this in and store it as dict and when writing the new file grab the new account from the dict by key.

My question is what is the best way to do this if the CSV file increases to 100K+ records?

Would it be more efficient to convert the account mappings from a CSV to a sqlite database rather than storing them as a dict in memory?

Answer 1

As long as they will all fit in memory, a dict will be the most efficient solution. It's also a lot easier to code. 100k records should be no problem on a modern computer.

You are right that switching to an SQLite database is a good choice when the number of records gets very large.