Generating a KML heatmap from given data set of [lat, lon, density]

Question

I am looking to build a static KML (Google Earth markup) file which displays a heatmap-style rendering of a few given data sets in the form of [lat, lon, density] tuples.

A very straightforward data set I have is for population density.

My requirements are:

• Must be able to feed in data for a given lat, lon
• Must be able to specify the density of the data at that lat, lon
• Must export to KML

The requirements are language agnostic for this project as I will be generating these files offline in order to build the KML used elsewhere.

I have looked at a few projects, most notably heatmap.py, which is a port of gheat in Python with KML export. I have hit a brick wall in the sense that the projects I have found to date all rely on building the heatmap from the density of [lat, lon] points fed into the algorithm.

If I am missing an obvious way to adapt my data set to feed in just the [lat, lon] tuples but adjusting how I feed them using the density values I have, I would love to know!

Answer 1

Hey Will, heatmap.py is me. Your request is a common-enough one and is on my list of things to address. I'm not quite sure yet how to do so in a general fashion; in heatmap.py parlance, it would be straightforward to have a per-point dotsize instead of a global dotsize as it is now, but I'm not sure that will address the true need. I'm aiming for a summer 2010 release, but you could probably make this mod yourself.

You may try searching for Kernel Density Estimator tools; that's what the statisticians call heatmaps. R has some good built-in tools you can use that might satisfy your need more quickly.

good luck!

Answer 2

I updated the heatmap.py script so you can specify a density for each point. I uploaded my changes to my blog. Not sure if it'll do exactly what you want though!

Cheers, Alex

Update [13 Nov 2020] I archived my blog a while back, so the link no longer works, so for reference here are the changes:

the diff file:

--- __init__.py 2010-09-14 08:40:35.829079482 +0100
+++ __init__.py.mynew   2010-09-06 14:50:10.394447647 +0100
@@ -1,5 +1,5 @@
 #heatmap.py v1.0 20091004
-from PIL import Image,ImageChops
+from PIL import Image,ImageChops,ImageDraw
 import os
 import random
 import math
@@ -43,10 +43,13 @@
     Most of the magic starts in heatmap(), see below for description of that function.
     """
     def __init__(self):
+        self.minIntensity = 0
+        self.maxIntensity = 0
         self.minXY = ()
         self.maxXY = ()
+       
 
-    def heatmap(self, points, fout, dotsize=150, opacity=128, size=(1024,1024), scheme="classic"):
+    def heatmap(self, points, fout, dotsize=150, opacity=128, size=(4048,1024), scheme="classic", area=(-180,180,-90,90)):
         """
         points  -> an iterable list of tuples, where the contents are the 
                    x,y coordinates to plot. e.g., [(1, 1), (2, 2), (3, 3)]
@@ -59,33 +62,41 @@
         size    -> tuple with the width, height in pixels of the output PNG 
         scheme  -> Name of color scheme to use to color the output image.
                    Use schemes() to get list.  (images are in source distro)
+        area    -> specify the coordinates covered by the resulting image 
+                   (could create an image to cover area larger than the max/
+                   min values given in the points list) 
         """
-        
+        print("Starting heatmap")
         self.dotsize = dotsize
         self.opacity = opacity
         self.size = size
         self.imageFile = fout
- 
+
         if scheme not in self.schemes():
             tmp = "Unknown color scheme: %s.  Available schemes: %s"  % (scheme, self.schemes())                           
             raise Exception(tmp)
 
-        self.minXY, self.maxXY = self._ranges(points)
-        dot = self._buildDot(self.dotsize)
+        self.minXY = (area[0],area[2])
+        self.maxXY = (area[1],area[3])
 
+        self.minIntensity, self.maxIntensity = self._intensityRange(points)
+        
         img = Image.new('RGBA', self.size, 'white')
-        for x,y in points:
+        for x,y,z in points:
+            dot = self._buildDot(self.dotsize,z)
             tmp = Image.new('RGBA', self.size, 'white')
             tmp.paste( dot, self._translate([x,y]) )
             img = ImageChops.multiply(img, tmp)
 
-
+        print("All dots built")
         colors = colorschemes.schemes[scheme]
         img.save("bw.png", "PNG")
+        print("Saved temp b/w image")
+        print("Colourising")
         self._colorize(img, colors)
 
         img.save(fout, "PNG")
-
+        print("Completed colourising and saved final image %s" % fout)
     def saveKML(self, kmlFile):
         """ 
         Saves a KML template to use with google earth.  Assumes x/y coordinates 
@@ -110,17 +121,19 @@
         """
         return colorschemes.schemes.keys() 
 
-    def _buildDot(self, size):
+    def _buildDot(self, size,intensity):
         """ builds a temporary image that is plotted for 
             each point in the dataset"""
+        
+        intsty = self._calcIntensity(intensity)
+        print("building dot... %d: %f" % (intensity,intsty))
+
         img = Image.new("RGB", (size,size), 'white')
-        md = 0.5*math.sqrt( (size/2.0)**2 + (size/2.0)**2 )
-        for x in range(size):
-            for y in range(size):
-                d = math.sqrt( (x - size/2.0)**2 + (y - size/2.0)**2 )
-                rgbVal = int(200*d/md + 50)
-                rgb = (rgbVal, rgbVal, rgbVal)
-                img.putpixel((x,y), rgb)
+        draw  =  ImageDraw.Draw(img)
+        shade = 256/(size/2)
+        for x in range (int(size/2)):
+            colour = int(256-(x*shade*intsty))
+            draw.ellipse((x,x,size-x,size-x),(colour,colour,colour))
         return img
 
     def _colorize(self, img, colors):
@@ -139,7 +152,7 @@
                 rgba.append(alpha) 
 
                 img.putpixel((x,y), tuple(rgba))
-            
+     
     def _ranges(self, points):
         """ walks the list of points and finds the 
         max/min x & y values in the set """
@@ -153,6 +166,23 @@
             
         return ((minX, minY), (maxX, maxY))
 
+    def _calcIntensity(self,z):
+        return (z/self.maxIntensity)        
+               
+    def _intensityRange(self, points):
+        """ walks the list of points and finds the 
+        max/min points of intensity
+        """   
+        minZ = points[0][2]
+        maxZ = minZ
+        
+        for x,y,z in points:
+            minZ = min(z, minZ)
+            maxZ = max(z, maxZ)
+        
+        print("(minZ, maxZ):(%d, %d)" % (minZ,maxZ))
+        return (minZ, maxZ)
+        
     def _translate(self, point):
         """ translates x,y coordinates from data set into 
         pixel offsets."""

and a demo script:

import heatmap
import random
import MySQLdb
import math

print "starting script..."

db = MySQLdb.connect(host="localhost", # your host, usually localhost
                     user="username", # your username
                      passwd="password", # your password
                      db="database") # name of the data base
cur = db.cursor() 

minLng = -180
maxLng = 180
minLat = -90
maxLat = 90

# create and execute the query
query = "SELECT lat, lng, intensity FROM mytable \
            WHERE %f<=tllat AND tllat<=%f \
            AND %f<=tllng AND tllng<=%f" % (minLat,maxLat,minLng,maxLng)

cur.execute(query)

pts = []
# print all the first cell of all the rows
for row in cur.fetchall() :
    print (row[1],row[0],row[2])
    # work out the mercator projection for latitute x = asinh(tan(x1))
    proj = math.degrees(math.asinh(math.tan(math.radians(row[0]))))
    print (row[1],proj,row[2])
    print "-"*15
    if (minLat < proj and proj < maxLat):
        pts.append((row[1],proj,row[2]))

print "Processing %d points..." % len(pts)

hm = heatmap.Heatmap()
hm.heatmap(pts, "bandwidth2.png",30,155,(1024,512),'fire',(minLng,maxLng,minLat,maxLat))

Answer 3

I think one way to do this is to create a (larger) list of tuples with each point repeated according to the density at that point. A point with a high density is represented by lots of points on top of each other while a point with a low density has few points. So instead of: [(120.7, 82.5, 2), (130.6, 81.5, 1)] you would use [(120.7, 82.5), (120.7, 82.5), (130.6, 81.5)] (a fairly dull dataset).

One possible issue is that your densities may well be floats, not integers, so you should normalize and round the data. One way to do the conversion is something like this:

def dens2points (dens_tups):
    min_dens = dens_tups[0][2]
    for tup in dens_tups:
        if (min_dens > tup[2]):
           min_dens = tup[2]
    print min_dens

    result = []
    for tup in dens_tups:
        for i in range(int(tup[2]/min_dens)):
            result.append((tup[0],tup[1]))
    return result

if __name__ == "__main__":
    input = [(10, 10, 20.0),(5, 5, 10.0),(10,10,0.9)]
    output = dens2points(input)
    print input
    print output

(which isn't very pythonic, but seems to work for the simple test case). This subroutine should convert your data into a form that is accepted by heatmap.py. With a little effort I think the subroutine can be reduced to two lines.