I have a Python datetime timestamp and a large dict (index) where keys are timestamps and the values are some other information I'm interested in.
I need to find the datetime (the key) in index that is closest to timestamp, as efficiently as possible.
At the moment I'm doing something like:
for timestamp in timestamps:
closestTimestamp = min(index,key=lambda datetime : abs(timestamp - datetime))
which works, but takes too long - my index dict has millions of values, and I'm doing the search thousands of times. I'm flexible with data structures and so on - the timestamps are roughly sequential, so that I'm iterating from the first to the last timestamps. Likewise the timestamps in the text file that I load into the dict are sequential.
Any ideas for optimisation would be greatly appreciated.
Dictionaries aren't organized for efficient near miss searches. They are designed for exact matches (using a hash table).
You may be better-off maintaining a separate, fast-searchable ordered structure.
A simple way to start off is to use the bisect module for fast O(log N) searches but slower O(n) insertions:
def nearest(ts):
# Given a presorted list of timestamps: s = sorted(index)
i = bisect_left(s, ts)
return min(s[max(0, i-1): i+2], key=lambda t: abs(ts - t))
A more sophisticated approach suitable for non-static, dynamically updated dicts, would be to use blist which employs a tree structure for fast O(log N) insertions and lookups. You only need this if the dict is going to change over time.
If you want to stay with a dictionary based approach, consider a dict-of-lists that clusters entries with nearby timestamps:
def get_closest_stamp(ts):
'Speed-up timestamp search by looking only at entries in the same hour'
hour = round_to_nearest_hour(ts)
cluster = daydict[hour] # return a list of entries
return min(cluster, key=lambda t: abs(ts - t))
Note, for exact results near cluster boundaries, store close-to-the-boundary timestamps in both the primary cluster and the adjacent cluster.
datetime objects are comparable to each other, so make a sorted list of your key/value pairs like this:
myPairs = list(dict.iteritems())
myPairs.sort()
For each element myPairs[i]
, myPairs[i][0]
is the datetime
key and myPairs[i][1]
is the value.
You can search this list efficiently using bisect_left
:
import bisect
i = bisect.bisect_left(myPairs, targetDatetime)
The element myPairs[i]
is the element with the lowest datetime no earlier than targetDatetime
. But the prior element (if there is one) might be closer in time to targetDatetime
. Or targetDatetime
might be later than any time in myPairs
. So you need to check:
if i > 0 and i == len(myPairs):
i -= 1
elif i > 0 and targetDatetime - myPairs[i-1][0] < myPairs[i][0]- targetDatetime:
i -= 1
If your list is truly sorted and not just "roughly sequential", you can use a binary search. Have a look at the bisect
module documentation for more information.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With