How am I getting 'InternalError: table "dev~guestbook!!Entities" already exists' when I just created datastore?

Question

I'm trying to figure out how to setup Test Driven Development for GAE.

I start the tests with:

nosetests -v --with-gae  

I keep getting the error:

InternalError: table "dev~guestbook!!Entities" already exists

The datastore doesn't exist until I create it in the setUp(), but I'm still getting an error that the entities already exists?

I'm using the code from the GAE tutorial.

Here is my testing code in functional_tests.py:

import sys, os, subprocess, time, unittest, shlex   
sys.path.append("/usr/local/google_appengine")    
sys.path.append("/usr/local/google_appengine/lib/yaml/lib")    
sys.path.append("/usr/local/google_appengine/lib/webapp2-2.5.2")    
sys.path.append("/usr/local/google_appengine/lib/django-1.5")    
sys.path.append("/usr/local/google_appengine/lib/cherrypy")    
sys.path.append("/usr/local/google_appengine/lib/concurrent")    
sys.path.append("/usr/local/google_appengine/lib/docker")    
sys.path.append("/usr/local/google_appengine/lib/requests")    
sys.path.append("/usr/local/google_appengine/lib/websocket")    
sys.path.append("/usr/local/google_appengine/lib/fancy_urllib")    
sys.path.append("/usr/local/google_appengine/lib/antlr3")    

from selenium import webdriver    
from google.appengine.api import memcache, apiproxy_stub, apiproxy_stub_map   
from google.appengine.ext import db    
from google.appengine.ext import testbed    
from google.appengine.datastore import datastore_stub_util     
from google.appengine.tools.devappserver2 import devappserver2    


class NewVisitorTest(unittest.TestCase):    

    def setUp(self):    
        # Start the dev server  
        cmd = "/usr/local/bin/dev_appserver.py /Users/Bryan/work/GoogleAppEngine/guestbook/app.yaml --port 8080 --storage_path /tmp/datastore --clear_datastore --skip_sdk_update_check"  
        self.dev_appserver = subprocess.Popen(shlex.split(cmd),   
                                              stdout=subprocess.PIPE)  
        time.sleep(2) # Important, let dev_appserver start up  

        self.testbed = testbed.Testbed()
        self.testbed.setup_env(app_id='dermal')    
        self.testbed.activate()    

        self.testbed.init_user_stub()    
        # Create a consistency policy that will simulate the High Replication consistency model.
        # with a probability of 1, the datastore should be available.
        self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=1)
        # Initialize the datastore stub with this policy.
        self.testbed.init_datastore_v3_stub(datastore_file="/tmp/datastore/datastore.db", use_sqlite=True, consistency_policy=self.policy)    
        self.testbed.init_memcache_stub()    
        self.datastore_stub = apiproxy_stub_map.apiproxy.GetStub('datastore_v3')            
        # setup the dev_appserver    
        APP_CONFIGS = ['app.yaml']    

        # setup client to make sure   
        from guestbook import Author, Greeting  
        if not ( Author.query( Author.email == "[email protected]").get()):  
            logging.info("create Admin")  
            client = Author(  
            email = "[email protected]",   
            ).put()
            Assert( Author.query( Author.email == "[email protected]").get() )
        self.browser = webdriver.Firefox()    
        self.browser.implicitly_wait(3)    

    def tearDown(self):    
        self.browser.quit()    
        self.testbed.deactivate()    
        self.dev_appserver.terminate()  

    def test_submit_anon_greeting(self):
          self.browser.get('http://localhost:8080')
          self.browser.find_element_by_name('content').send_keys('Anonymous test post')
          self.browser.find_element_by_name('submit').submit()
          Assert.assertEquals(driver.getPageSource().contains('Anonymous test post'))

Here is the traceback:

test_submit_anon_greeting (functional_tests.NewVisitorTest) ... INFO     2015-05-11 14:41:40,516 devappserver2.py:745] Skipping SDK update check.  
INFO     2015-05-11 14:41:40,594 api_server.py:190] Starting API server at: http://localhost:59656  
INFO     2015-05-11 14:41:40,598 dispatcher.py:192] Starting module "default" running at: http://localhost:8080  
INFO     2015-05-11 14:41:40,600 admin_server.py:118] Starting admin server at: http://localhost:8000  
WARNING  2015-05-11 14:41:45,008 tasklets.py:409] suspended generator _run_to_list(query.py:964) raised InternalError(table "dev~guestbook!!Entities" already exists)  
ERROR    2015-05-11 14:41:45,009 webapp2.py:1552] table "dev~guestbook!!Entities" already exists  
Traceback (most recent call last):  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__  
    rv = self.handle_exception(request, response, e)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__  
    rv = self.router.dispatch(request, response)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher  
    return route.handler_adapter(request, response)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__  
    return handler.dispatch()  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch  
    return self.handle_exception(e, self.app.debug)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch  
    return method(*args, **kwargs)  
  File "/Users/Bryan/work/GoogleAppEngine/guestbook/guestbook.py", line 50, in get  
    greetings = greetings_query.fetch(10)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/utils.py", line 142, in positional_wrapper  
    return wrapped(*args, **kwds)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/query.py", line 1187, in fetch  
    return self.fetch_async(limit, **q_options).get_result()  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 325, in get_result  
    self.check_success()  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along  
    value = gen.throw(exc.__class__, exc, tb)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/query.py", line 964, in _run_to_list  
    batch = yield rpc  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 454, in _on_rpc_completion  
    result = rpc.get_result()  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result  
    return self.__get_result_hook(self)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_query.py", line 2870, in __query_result_hook  
    self._batch_shared.conn.check_rpc_success(rpc)  
  File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1342, in check_rpc_success  
    raise _ToDatastoreError(err)  
InternalError: table "dev~guestbook!!Entities" already exists  

Answer 1

It looks like there are a couple of things happening here.

First, it looks like you are using NoseGAE --with-gae. The plugin handles setting up and tearing down your testbed so you don't have to. This means that you do not need any of the self.testbed code and actually it can cause conflicts internally. Either switch to doing it the NoseGAE way, or don't use the --with-gae flag. If you stick with NoseGAE, it has an option --gae-datastore that lets you set the path to the datastore that it will use for your tests. Then inside your test class, set the property nosegae_datastore_v3 = True to have it set up for you:

class NewVisitorTest(unittest.TestCase):
    # enable the datastore stub
    nosegae_datastore_v3 = True

Second, the way that dev_appserver / sqlite work together, the appserver loads the sqlite db file into memory and works with it there. When the app server exits, it flushes the database contents back to disk. Since you are using the same datastore for your tests as the dev_appserver.py process you are opening for selenium, they may or may not see the fixture data you set up inside your test.

Here is an example from https://github.com/Trii/NoseGAE/blob/master/nosegae.py#L124-L140

class MyTest(unittest.TestCase):
    nosegae_datastore_v3 = True
    nosegae_datastore_v3_kwargs = {
        'datastore_file': '/tmp/nosegae.sqlite3',
        'use_sqlite': True
    }
    def test_something(self):
        entity = MyModel(name='NoseGAE')
        entity.put()
        self.assertNotNone(entity.key.id())