How can I implement decrease-key functionality in Python's heapq?

Question

To implement "decrease-key" effectively, you'd need to access the functionality "decrement this element AND swap this element with a child until heap condition is restore". In heapq.py, that's called _siftdown (and similarly _siftup for INcrementing). So the good news is that the functions are there... the bad news is that their names start with an underscore, indicating they're considered "internal implementation details" and should not be accessed directly by application code (the next release of the standard library might change things around and break code using such "internals").

It's up to you to decide whether you want to ignore the warning leading-_, use O(N) heapify instead of O(log N) sifting, or reimplement some or all of heapq's functionality to make the sifting primitives "exposed as public parts of the interface". Since heapq's data structure is documented and public (just a list), I think the best choice is probably a partial-reimplementation -- copy the sifting functions from heapq.py into your application code, essentially.

Decrease-key is a must-have operation for many algorithms (Dijkstra's Algorithm, A*, OPTICS), i wonder why Python's built-in priority queue does not support it.

Unfortunately, i wasn't able to download math4tots's package.

But, i was able to find this implementation by Daniel Stutzbach. Worked perfectly for me with Python 3.5.

hd = heapdict()
hd[obj1] = priority
hd[obj1] = lower_priority
# ...
obj = hd.pop()

The heapq documentation has an entry on exactly how to do this.

However, I have written a heap package that does exactly this (it is a wrapper around heapq). So if you have pip or easy_install you could do something like

pip install heap

Then in your code write

from heap.heap import heap

h = heap()

h['hello'] = 4 # Insert item with priority 4.

h['hello'] = 2 # Update priority/decrease-key has same syntax as insert. 

It is pretty new though, so might be full of bugs.

Imagine you are using a heap as a priority queue, where you have a bunch of tasks represented by strings and each task has a key. For concreteness, look at: task_list = [[7,"do laundry"], [3, "clean room"], [6, "call parents"]] where each task in task_list is a list with a priority and description. If you run heapq.heapify(task_list), you get your array to maintain the heap invariant. However, if you want to change the priority of "do laundry" to 1, you have no way of knowing where "do laundry" is in the heap without a linear scan through the heap (hence can't do decrease_key in logarithmic time). Note decrease_key(heap, i, new_key) requires you to know the index of the value to change in the heap.

Even if you maintain a reference to each sublist and actually change the key, you still can't do it in log time. Since a list is just a reference to a bunch of mutable objects, you could try doing something like remember the original order of the task: (in this case that "do laundry" was the 0th task in your original task_list):

task_list = [[7, "do laundry"], [3, "clean room"], [6, "call parents"]]
task_list_heap = task_list[:] # make a non-deep copy
heapq.heapify(task_list_heap)
# at this point:
# task_list = [[7, 'do laundry'], [3, 'clean room'], [6, 'call parents']]
# task_list_heap = [3, 'clean room'], [7, 'do laundry'], [6, 'call parents']]
# Change key of first item of task_list (which was "do laundry") from 7 to 1.
task_list[0][0] = 1
# Now:
# task_list = [[1, 'do laundry'], [3, 'clean room'], [6, 'call parents']]
# task_list_heap = [3, 'clean room'], [1, 'do laundry'], [6, 'call parents']]
# task_list_heap violates heap invariant at the moment

However, you now need to call heapq._siftdown(task_list_heap, 1) to maintain the heap invariant in log time (heapq.heapify is linear time), but unfortunately we don't know the index of "do laundry" in task_list_heap (the heap_index in this case is 1).

So we need to implement our heap keeps track of the heap_index of each object; e.g., have an list (for the heap) and a dict mapping each object to its index in the heap/list (that gets updated as the heap positions get swapped adding a constant factor to each swap). You could read through heapq.py and implement yourself as the procedure is straightforward; however, others have implement this sort of HeapDict already.

It might be unnecessary to have the decrease_key function (albeit it's nice to have it).

You can just push your (priority, item) into the priority queue anyway, and use a set to check whether you have seen it. For example:

pq = []  # heapq is a min heap
seen = set()
heappush(pq, (2, "item1"))
heappush(pq, (3, "item2"))
heappush(pq, (1, "item3"))
heappush(pq, (4, "item4"))
heappush(pq, (2, "item2"))

while pq:
    p, item = heappop(pq)
    if item not in seen:
        seen.add(item)
        print(item, p)
    else:
        print(item, "is already handled with a higher priority!")

The output is:

item3 1
item1 2
item2 2
item2 is already handled with a higher priority!
item4 4