Error importing spatial data in GeoDjango - KeyError for mpoly field

Question

I was following the tutorial on https://docs.djangoproject.com/en/1.8/ref/contrib/gis/tutorial/#importing-spatial-data for setting up GeoDjango on my machine. But it seems like there is some issue there. While importing data using LayerMapping by running load.run(), I get the following error:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ubuntu/src/django/world/load.py", line 23, in run
    lm = LayerMapping(WorldBorder, world_shp, world_mapping, transform=False, encoding='iso-8859-1')
  File "/home/ubuntu/Envs/vir-env/local/lib/python2.7/site-packages/django/contrib/gis/utils/layermapping.py", line 105, in __init__
    self.check_layer()
  File "/home/ubuntu/Envs/vir-env/local/lib/python2.7/site-packages/django/contrib/gis/utils/layermapping.py", line 178, in check_layer
    ogr_field_types = self.layer.field_types
  File "/home/ubuntu/Envs/vir-env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/layer.py", line 153, in field_types
    for i in range(self.num_fields)]
KeyError: 12

Then I found out that, there is no 'MULTIPOLYGON' field in the .shp file:

>>> from django.contrib.gis.gdal import DataSource
>>> ds = DataSource('world/data/TM_WORLD_BORDERS-0.3.shp')
>>> layer = ds[0]
>>> layer.fields
[u'FIPS', u'ISO2', u'ISO3', u'UN', u'NAME', u'AREA', u'POP2005', u'REGION', u'SUBREGION', u'LON', u'LAT']

But it's there in the WorldBorder model, as type MultiPolygonField. So, definitely in the world_mapping file, importing will fail for the 'mpoly': 'MULTIPOLYGON' mapping. Has anyone else faced this issue? I hope so, as I've followed the tutorial step-by-step. But it doesn't say anything about such issue. What effect it will have, if I load data by removing mpoly mapping?

Here's my load.py file:

  1 import os
  2 from django.contrib.gis.utils import LayerMapping
  3 from models import WorldBorder
  4
  5 world_mapping = {
  6     'fips' : 'FIPS',
  7     'iso2' : 'ISO2',
  8     'iso3' : 'ISO3',
  9     'un' : 'UN',
 10     'name' : 'NAME',
 11     'area' : 'AREA',
 12     'pop2005' : 'POP2005',
 13     'region' : 'REGION',
 14     'subregion' : 'SUBREGION',
 15     'lon' : 'LON',
 16     'lat' : 'LAT',
 17     'mpoly' : 'MULTIPOLYGON',
 18 }
 19
 20 world_shp = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data/TM_WORLD_BORDERS-0.3.shp'))
 21
 22 def run(verbose=True):
 23     lm = LayerMapping(WorldBorder, world_shp, world_mapping, transform=False, encoding='iso-8859-1')
 24
 25     lm.save(strict=True, verbose=verbose)

---

Just an update: After going through source code, via stack trace, I figured that I'm unable to access field_types propery of layer module. So, from python shell, when I access that property, I get the same error:

>>> from django.contrib.gis.gdal import DataSource
>>> ds = DataSource(wshp)
>>> layer = ds[0]
>>> layer.fields
[u'FIPS', u'ISO2', u'ISO3', u'UN', u'NAME', u'AREA', u'POP2005', u'REGION', u'SUBREGION', u'LON', u'LAT']
>>> layer.field_types
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ubuntu/Envs/rj-venv/local/lib/python2.7/site-packages/django/contrib/gis/gdal/layer.py", line 153, in field_types
    for i in range(self.num_fields)]
KeyError: 12

Now, this is strange, because now I've also removed mpoly field from WorldBorder model.

---

Update 2:

After digging through the source code, I found out that, OGDFieldTypes in my version of gdal, might not have the key 12, as in the source code here: https://github.com/django/django/blob/master/django/contrib/gis/gdal/field.py. But it says that keys 12, and 13 will be available for GDAL 2, and that is what I've installed. Really seems to be some conflict among libraries now.

I've installed the following libraries:

• geos-3.4.2.tar.bz2
• proj-datumgrid-1.5.tar.gz
• proj-4.8.0.tar.gz
• gdal-2.0.0.tar.gz

And PostGIS version 2.1.5 is installed in Amazon RDS instance.

Answer 1

The problem here is that the version 1.8.2 of Django (latest release version at the time of writing) does not support GDAL 2.0.0 field types. The fix for this is in the Django master code branch - but not yet in a release version.

At time of writing you have two options - build Django from the latest source on Github (i.e. upgrade), or downgrade GDAL so that the native code called by this line cannot return the value 12 as a field type.

Because GDAL 2.0 can return 12 as a field type, the lookup into this dictionary fails with the KeyError you see using Django 1.8.2 code. On master, the dict contains the right fields for GDAL 2.0 - fixed by this commit (only 9 days old at time of writing!).

---

For completeness - the diagnosis of this fault followed a discussion in the SOPython chat room - a link to the bookmarked conversation is here

N.B. from the chat If you have multiple GDAL versions installed, you may have to add GDAL_LIBRARY_PATH in django settings file to point to the correct version of libgdal.so (or I guess the corresponding .dll if you're on Windows)