Python, calculating the status of computation slows down the computation itself

Question

This is a basic example of what I'm talking about:

Count from 0 to 10000000

import time

k = 0

beginning = time.time()

while k < 10000000:

    k = k+1

elapsed = round(time.time()-beginning, 5)

print (elapsed)

Now with a sort of "status" of the computation (displays percentage every 1 second):

import time

k = 0

beginning = time.time()
interval = time.time()

while k < 10000000:

    if time.time() - interval > 1:
        print (round((k/100000), 5))
        interval = time.time()

    k = k+1

elapsed = round(time.time()-beginning, 5)

print (elapsed)

The first example takes 3.67188 seconds, the second example takes 12.62541 seconds. I guess that's because the scripts has to continuously check if 1 second has elapsed. Is there a way to solve this issue? I've found something about threads and multiprocess but i can't figure out how to implement it. Thanks

Answer 1

Benchmarking

I wrote different solutions and compare them. I needed to multiply your value by 10 to get measurable results. First without any progress measurement to see how fast it runs on my machine.

def process_without_measuring(self):
    self._start = time.time()

    self._k = 0
    while self._k < 100000000:
        self._k = self._k+1

    print (time.time() - self._start)

I get a duration of 13.8 seconds.

Let's start with your implementation:

def process_with_original_measuring(self):
    self._start = time.time()
    next = self._start
    self._k = 0
    while self._k < 100000000:
        self._k = self._k+1
        if time.time() - next > 1:
            print (round((self._k / 1000000), 5))
            next = time.time()

    print("duration:")
    print (time.time() - self._start)

If I run this, i get a duration of 30.31 seconds and around 3 percents up per second. The problem is that it has to compare time every loop rund and do a arithmetic operation. You can reduce the time by changing the loop to:

def process_with_manual_measuring(self):
    self._start = time.time()
    next = self._start + 1
    self._k = 0
    while self._k < 100000000:
        self._k = self._k+1
        if time.time() > next:
            print (round((self._k / 1000000), 5))
            next = time.time() + 1

    print("duration:")
    print (time.time() - self._start)

Instead of subtracting the timestamps every loop I calculate the next timestamp only once and compare to it. This is of course not really fast, but faster than before. It gets me to 22.0 seconds, so saving 8 seconds only with removing this one operation.

With a timer object as thread you get a much better result and it is the preferable way:

def show_progress(self):
    print (round((self._k / 1000000), 5))
    self._timer = Timer(1, self.show_progress)
    self._timer.start()

def process_with_timer(self):
    self._start = time.time()
    self._timer = Timer(1, self.show_progress)
    self._timer.start()
    self._k = 0
    while self._k < 100000000:
        self._k = self._k+1
    self._timer.cancel()

    print("duration:")
    print (time.time() - self._start)

Running this I get a output of 7 percent more every second and it is done after 13.8 seconds. As you can see, no difference. There are only few more calls to make and these are done in nearly no time.

How to use timer class

The constructor of Timer expects a time duration in seconds and a method to call after time elapsed. You can use class method, a function or a lambda expression. After construction you need to start the timer with start().

The first timer is started by the process itself. After this with each timer call a new timer is started to get a interval of one second. When the process finishes remember to call cancel() on timer. Otherwise it will run endless because it will restart itself every second.

How to run the examples

Please notice that the above methods are class methods so watch indentation.

import time
from threading import Timer

class ProgressMeter:
    def __init__(self):
        self._k = 0
        self._timer = 0

    # insert above methods with indentation as same level as __init__(self):

To run them you only need to create a instance of ProgressMeter and call the method you want.

meter = ProgressMeter()
meter.process_with_timer()