Loading 100-200K markers on google map

Question

At the moment I'm using Google Maps v.3 API for drawing markers on the map. I have around 500 markers in total.

For displaying purposes I use markerCluster and group markers using this tool on the client side in the browser.

However, I plan to expand the locations number and assume it can grow to 100K or even 200K quickly.

I did some stress tests, and realized that current solution basically kills the browser and about 10-20K markers.

So my question what is the best approach to draw that many markers (not necessary google maps)?

I've read posts with similar questions, e.g.:

Showing many markers in Google Maps

Best solution for too many pins on google maps

Basically people suggest to use some clusterer for display purposes, which I already use.

Or to use fusion tables for retrieving data, which is not an option, as the data has to stay on my server. Also I assume the display functionality is limited with fusion tables.

I'm thinking about implementing the following scenario:

• on every page zoom / load - send ajax request with bounds of the display view, add about 30% on all sides and retrieve markers, which fall into this geo area only. 30% is added in case user zooms out, so that I can display other markers around quickly and then retreive further in background the rest around (broader territory)

• When the number of markers is more than 50 - then I plan to apply clustering for displaying purposes. But as the markerCluster in javascript is quite slow, namely not markerCluster but google itself, as it still applies locations of all the markers, I plan to do clustering on the server side by spliting the bounds of the displayed map in about 15*15 grid and drop markers into particular cells and then basically send to the client clusters with number of markers inside (e.g. like for heatmap). And then to display clusters as markers.

Could you please give some insight whoever did similar. Does it makes sense in general. Or is it a stupid approach as ajax requests will be sent to the server on every map zoom and shift and basically overload server with redundant requests?

What I want to achieve is a nice user experience on big datasets of markers (to load in less than 2 secs).

Answer 1

Your approach is solid. If at all possible, you'll want to precompute the clusters and cache them server-side, with their update strategy determined by how often the underlying dataset changes.

Google maps has ~20 zoom levels, depending on where you are on the planet. Depending on how clustered your data is, if you have 200,000 markers total and are willing to show about 500 on the map at a given time, counting all the cluster locations and original markers you'll only end up storing roughly 2n = 400,000 locations server-side with all your zoom levels combined.

Possible cluster updating strategies:

• Update on every new marker added. Possible for a read-heavy application with few writes, if you need a high degree of data timeliness.
• Update on a schedule
• Kick off an update if ((there are any new markers since the last clustering pass && the cache is older than X) || there are more than Y new markers since the last clustering pass)

Storing these markers in a database that supports geo-data natively may be beneficial. This allows SQL-like statements querying locations.

Client-side, I would consider fetching a 50% margin on either side, not 30%. Google zooms in powers of 2. This will allow you to display one full zoom level.

Next, if this application will get heavy use and optimization is worthwhile, I would log server-side when a client zooms in. Try to profile your usage, so you can determine if users zoom in and out often. With solid numbers (like "70% of users zoom in after retrieving initial results, and 20% zoom out"), you can determine if it would be worthwhile to preload the next layer of zoomed data for your users, in order to gain UI responsiveness.