Using SQLITE for local Django development for Google App Engine?

Question

I am doing development in Django with Google App Engine. For production I plan to use Google Cloud SQL but for local development I want to use the simple SQLITE. The Google Tutorial (https://developers.google.com/appengine/docs/python/cloud-sql/django) suggests that I run the development server with

dev_appserver.py mysite

instead of the Django default

manage.py runserver

However, when I run the development server as Google suggested I get two strange errors (I removed the rest of the stack trace for clarity):

File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django1.5/django/db/backends/sqlite3/base.py", line 34, in <module>
raise ImproperlyConfigured("Error loading either pysqlite2 or sqlite3 modules (tried in that order): %s" % exc)ImproperlyConfigured: Error loading either pysqlite2 or sqlite3 modules (tried in that order): No module named _sqlite3


File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/core/handlers/base.py", line 53, in load_middleware
raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e)) ImproperlyConfigured: Error importing middleware django.contrib.auth.middleware: "cannot import name utils"

The strangest part is that when I just use manage.py runserver the site works fine. Also when I tested directly in the interactive python prompt for sqlite3 it also works:

Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> 

Can anyone here help me? I'm not sure what to do next. Thanks in advance!

Answer 1

The question is very old and the proposed solution is OK. However, I believe my answer can be interesting for App Engine users as well.

All the problems with sqlite arise from the fact that App Engine Environment is stateless. You cannot write or modify files of the local file system. Dev environment simulates this behaviour. There is, however, one folder which is writeable: /tmp/ This should be the case for Dev environment as well. A simple trick like the following can make sqlite usable for DEV purposes or even for PROD cases when one does not care preserving the data in dbsqlite. In settings.py on can do something like this:

DB_PATH = os.path.join(BASE_DIR, 'db.sqlite3')
try:
    from shutil import copyfile
    DB_PATH = "/tmp/db.sqlite3"
    copyfile(os.path.join(BASE_DIR, 'db.sqlite3'), DB_PATH)
except:
    pass
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': DB_PATH,
    }
}

This solution runs e.g. when db.sqlite3 is deployed together with the app, but it of course doesn't scale, since all the new instances start with the original file.

Answer 2

I stumbled on the same stone until I realized that GAE does not allow you to use SQLite, at least not right out-of-the-box...

Therefore their development server does not allow it neither in order to prevent any bad surprise at deployment time.

Eventhough there might be a way because the Google Appengine docs sometimes refer to some "SQlite Stub", I was not able to navigate safely throughout their doc jungle and found no recent, decent, working example.

So as for now, one solution is: use mysql if you require a relational database, BUT you will have to pay a subscription once you roll it out on GAE. Note that the pricing can be really cheap.

Here is an example declaration for your settings.py file:

if (os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine') or
    os.getenv('SETTINGS_MODE') == 'prod'):
    # Running on production App Engine, so use a Google Cloud SQL database.
    DATABASES = {
        'default': {
            'ENGINE': 'google.appengine.ext.django.backends.rdbms',
            'INSTANCE': '<project>-pts:ptsdb',
            'NAME': '<dbname>',
            'OPTIONS': {"init_command": "SET storage_engine=INNODB"},
        }
    }
else:
    # Running in development, so use a local MySQL database.
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': '<dbname>',
            'USER': '<username>',
            'PASSWORD': '',
            'HOST': 'localhost',
            'OPTIONS': {"init_command": "SET storage_engine=INNODB"},
        }
    }

Otherwise you can also use their NDB database for free, BUT you will not be able to define "strong" relationships (foreign keys) between your entities.