Amazon DynamoDB Local - unknown error, exception or failure

Question

I have to test an application which relies heavily on Amazon's DynamoDB. I want the tests to be able to be run separately, which is why I opted for DynamoDB Local .jar. I am aware of the recent update, making us able to run this without an external bash command call. However, when I try to run the example, that was specified here, i get the following stacktrace:

Exception in thread "main" com.amazonaws.AmazonServiceException: The request processing has failed because of an unknown error, exception or failure. (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalFailure; Request ID: cab7a550-aaa6-4bfe-a591-0b255481cc14)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1275)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:873)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1805)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1223)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.listTables(AmazonDynamoDBClient.java:1235)

This is the code I am trying to run :

public static void main( String[] args ) throws Exception
    {
        AmazonDynamoDB dynamodb = null;

        DynamoDBProxyServer server = null;
        final String[] localArgs = { "-inMemory", "-port", "13005" };
        server = ServerRunner.createServerFromCommandLineArgs(localArgs);
        server.start();

        BasicAWSCredentials auth = new BasicAWSCredentials("key", "secret");
        dynamodb = new AmazonDynamoDBClient(auth);
        dynamodb.setEndpoint("http://127.0.0.1:13005");

        // use the DynamoDB API over HTTP
        System.out.println(dynamodb.listTables());

        // Stop the DynamoDB Local endpoint
        if(server != null) {
            server.stop();
        }
    }

I have observed that if I try to run it completely from the Java program itself, that's when the exception is thrown and the specified port is no longer available (An error is thrown stating that this port is taken). But if I start the DynamoDB Local from a command prompt and use the Java program as an access client only, then everthing runs okay. Any suggestions?

Answer 1

There are at least 2 problems with DynamoDBLocal. Resolve both, and you will be running embedded DynamoDB.

First, the -port parameter does not work correctly. So Jetty isn't setup on the port you expect. Instead something like port 51205 (or random?) is set as the Jetty listener as a default port.

Here is my code for how I start the server avoiding the built-in command-line parsing which is better anyway... After starting the server in this way, http://localhost:19444/shell works, so Jetty is fine. But then you might have another problem with Sqlite4java (see after the code block).

NOTE: code is Kotlin, but the Java would be very similar. Also I have SLF4j configured and don't let the ServerRunner classes break the logging, so this is doubly better way to start the server and what ServerRunner does internally.

class TestAccountManager {
    companion object {
        private val localDbPort = 19444

        private lateinit var localDb: DynamoDBProxyServer
        private lateinit var dbClient: AmazonDynamoDBClient

        @BeforeClass @JvmStatic fun setup() {
            System.setProperty("org.eclipse.jetty.util.log.class", "org.eclipse.jetty.util.log.Slf4jLog")
            localDb = DynamoDBProxyServer(localDbPort, LocalDynamoDBServerHandler(
                    LocalDynamoDBRequestHandler(0, true, null, true, true), null)
            )
            localDb.start()

            val auth = BasicAWSCredentials("fakeKey", "fakeSecret")
            dbClient = AmazonDynamoDBClient(auth)
            dbClient.signerRegionOverride = "us-east-1"
            dbClient.setEndpoint("http://localhost:$localDbPort")
        }

        @AfterClass @JvmStatic fun teardown() {
           localDb.stop()
        }
    }

    @Test fun testSomething() {
        dbClient.listTables().tableNames.forEach {
            println(it)
        }
    }
}

Once you have this code, you now are running on the expected port.

Now you likely have the 2nd error, such as Sqlite4java not finding its correct binary for your platform. For some versions of Mac OSX it will generate a binary filename that actually does not exist. And DynamoDBLocal hides all Sqlite4java logging forcibly (impossible to override it), so you will not see it.

You can test the sqlite library by downloading a distribution, unzipping and then running:

java -jar sqlite4java-1.0.392.jar -d

And it will report what it try to load, and how it failed or not. You just need to find that JAR on your system from wherever Gradle, Maven or you placed it. My output with error was:

sqlite4java 392
160212:002049.833 FINE [sqlite] Internal: loading library
160212:002049.834 FINE [sqlite] Internal: java.library.path=/Users/jminard/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
160212:002049.834 FINE [sqlite] Internal: sqlite4java.library.path=null
160212:002049.834 FINE [sqlite] Internal: cwd=/Users/jminard/DEV/Collokia/repos/collokia-web-back/.
160212:002049.834 FINE [sqlite] Internal: default path=/Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d
160212:002049.834 FINE [sqlite] Internal: forced path=null
160212:002049.834 FINE [sqlite] Internal: os.name=mac os x; os=osx
160212:002049.835 FINE [sqlite] Internal: os.arch=x86_64
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d.dylib
160212:002049.837 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-1.0.392
160212:002049.838 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
160212:002049.838 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-1.0.392
160212:002049.839 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-1.0.392 in java.library.path
160212:002049.839 FINE [sqlite] Internal: trying to load sqlite4java-osx-1.0.392
160212:002049.840 FINE [sqlite] Internal: cannot load sqlite4java-osx-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-1.0.392 in java.library.path
160212:002049.840 FINE [sqlite] Internal: trying to load sqlite4java-1.0.392
160212:002049.841 FINE [sqlite] Internal: cannot load sqlite4java-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-1.0.392 in java.library.path
160212:002049.841 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d-1.0.392 in java.library.path
160212:002049.842 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-osx-d-1.0.392
160212:002049.843 FINE [sqlite] Internal: cannot load sqlite4java-osx-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-d-1.0.392
160212:002049.844 FINE [sqlite] Internal: cannot load sqlite4java-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-d-1.0.392 in java.library.path
160212:002049.844 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx
160212:002049.846 FINE [sqlite] Internal: cannot load sqlite4java-osx: java.lang.UnsatisfiedLinkError: no sqlite4java-osx in java.library.path
160212:002049.846 FINE [sqlite] Internal: trying to load sqlite4java
160212:002049.847 FINE [sqlite] Internal: cannot load sqlite4java: java.lang.UnsatisfiedLinkError: no sqlite4java in java.library.path
160212:002049.847 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d
160212:002049.848 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d in java.library.path
160212:002049.848 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d in java.library.path
160212:002049.849 FINE [sqlite] Internal: trying to load sqlite4java-osx-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d in java.library.path
160212:002049.850 FINE [sqlite] Internal: trying to load sqlite4java-d
160212:002049.850 FINE [sqlite] Internal: cannot load sqlite4java-d: java.lang.UnsatisfiedLinkError: no sqlite4java-d in java.library.path
Error: cannot load SQLite
java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at com.almworks.sqlite4java.Internal.tryLoadFromSystemPath(Internal.java:352)
    at com.almworks.sqlite4java.Internal.loadLibraryX(Internal.java:124)
    at com.almworks.sqlite4java.SQLite.main(SQLite.java:368)

If you have an error about the dynamic library loading, read these for how to resolve:

• UnsatisfiedLinkError with sqlite4java Jar on Mac OS X NetBeans
• UnsatisfiedLinkError with sqlite4java Jar on Mac OS X
• and specifically related to this DynamoDBLocal Run DynamoDB Local with the java command on Mac OS X.
• https://groups.google.com/forum/#!topic/sqlite4java/9J1lmCuoKLA
• https://groups.google.com/forum/#!topic/sqlite4java/jhwt44nYGvw

The only answer that seems to work reliably is to either:

• add the libraries from a downloaded sqlite4java distribution to a known location and set the system property java.library.path
• add the libraries from a downloaded sqlite4java distribution to /Library/Java/Extensions (macOS only)

Of these I do something like the first option, and add the libraries into the project and make sure the build adds a -Djava.library.path=./lib/sqlite4java where the dynamic libraries are unzipped. For more resilient testing you can set the java.library.path programatically in your code using this trick (otherwise it is ignored when set in code): http://blog.cedarsoft.com/2010/11/setting-java-library-path-programmatically/

It is evil that DynamoDBLocal hides all Sqlite errors, so you have to debug to find silent failures. The library is full of things that adjust logging levels and make it hard to debug since they break your ability to see errors. For example every time when a SqlLite file is opened (class SQLiteDBAccess):

LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java");
LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java.Internal");
java.util.logging.Logger.getLogger("com.almworks.sqlite4java").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("com.almworks.sqlite4java.Internal").setLevel(Level.OFF);

Closing Notes: I am looking at alternatives, this is not the best built thing on the planet, looking at the code of DynamoDbLocal and decisions made for it give me very little confidence in it. Alternator or Jcabi - Dynamo Mock are next on my list.

Answer 2

In August 2018 Amazon announced new Docker image with Amazon DynamoDB Local onboard. It does not require downloading and running any JARs as well as adding using third-party OS-specific binaries (I'm talking about sqlite4java).

It is as simple as starting a Docker container before the tests:

docker run -p 8000:8000 amazon/dynamodb-local

You can do that manually for local development, as described above. IDEs usually provide a way to run arbitrary commands before executing a task, so you can make IDE to start the container for you.

Or you can use it in your CI pipeline. Many CI services provide an ability to start additional containers during the pipeline that can provide dependencies for your tests. Here is an example for Gitlab CI/CD:

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://dynamodb-local:8000 ./gradlew clean test

Or Bitbucket Pipelines:

definitions:
  services:
    dynamodb-local:
      image: amazon/dynamodb-local
…
step:
  name: test
  image:
    name: openjdk:8-alpine
  services:
    - dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://localhost:8000 ./gradlew clean test