Average excluding specific values from very large list

Question

I am working with a very large list (~1GB) of travel times and am trying to average them, but there is a quirk where if the trip is not possible the value is set to the highest possible integer value, which destroys the entire calculation. The travel times are stored in a list, and the lists are in a dictionary.

From point A to B and B to C would look like:

{'AB':[3,5,10],'BC':[2,3,5,10,2147483647]}

The average between AB should be 6 and BC should be 5 (not 429496733.4).

How can I exclude rogue values from the average calculation?

Answer 1

The statistics module provides a mean() function which can take an iterator as input, so you don't have to make a copy of the list to filter out the values you want to discard.

Here's a mocked up example of your data, where 90% of the 1 million elements are in the range 1 to 9 inclusive, and 10% are your rogue value:

from random import randint, random

data = [randint(1, 9) if random() < 0.9 else 2147483647 for _ in range(1000000)]

Here's how to use statistics.mean() to get the mean including rogue values:

>>> from statistics import mean

>>> mean(data)
215405499.193486

… and here's how to do so iterating over it excluding rogue values:

>>> mean(x for x in data if x != 2147483647)
4.998926301609214

Wrapping that up in a dictionary comprehension:

>>> travel_times = {'AB':[3,5,10],'BC':[2,3,5,10,2147483647]}
>>> {k: mean(x for x in v if x != 2147483647) for k, v in travel_times.items()}
{'BC': 5, 'AB': 6}