Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to persist data in a dockerized DynamoDB using volumes

My docker compose file has two containers and looks like this

  version: '3'

  services:
     dynamodb:
       image: amazon/dynamodb-local
       ports: 
         - '8000:8000'
       networks:
         - testnetwork

     audit-server:
         image: audit-dynamo
         environment:
         DYNAMO_URL: 'http://0.0.0.0:8000'
         command: node app.js
         ports: 
           - '3000:3000'
         depends_on: 
           - dynamodb
         # restart: always

         networks:
           - testnetwork

  networks:
    testnetwork:

My goal is to mount local data to some volume. currently losing data on docker-compose down

like image 364
veeresh yh Avatar asked Oct 17 '18 10:10

veeresh yh


People also ask

Do Docker compose volumes persist?

Whichever you choose, once you have set up a volume to the folder where the data is stored in the container, if you do a docker-compose down , and then a docker-compose up , your data will not be erased and it will become persistent.

Which allows containers to run with persistent volume?

Docker has an option to allow specific folders in a container to be mapped to the normal filesystem on the host. This allows us to have data in the container without making the data part of the Docker image, and without being bound to AUFS.

How do I persist data in Docker image?

Volumes are the best way to persist data in Docker. Bind mounts may be stored anywhere on the host system. They may even be important system files or directories. Non-Docker processes on the Docker host or a Docker container can modify them at any time.

How do I run a docker container from a local DynamoDB table?

Once you stop the local dynamodb docker and restart it, you can still see the table created persists. Alternatively, we can use docker-compose to run the container instead of using the docker build and run command with the existing Dockerfile. Once we have this file, we can run the docker-compose up command.

How do I persist data in a docker container?

To persist data, the best option is to mount a volume to this. We can do this by using Dockerfile to create a local data folder in the container and map it to the volume on the local machine. There is already an example available for both Dockerfile.

Why is the cluster size in DynamoDB large?

The cluster is large when the data is large. This makes perfect sense when you’re playing to Spark’s strengths by operating on the data. However, when writing to DynamoDB we only need a few items at a time to batch writes efficiently. Reading a whole Dataset into large-scale distributed memory, for example, is wasteful.

Why is DynamoDB so slow to write?

First rule of thumb when trying to write lots of rows into DynamoDB — make sure the data is modeled so that you can batch insert, anything else is painfully slow. Even with Batch write item, latency matters. With my laptop in Singapore, writing to the AWS Singapore region, I sensed that latency was causing issues.


Video Answer


2 Answers

So that image uses by default in-memory dynamodb (what you can find by running docker inspect on that image)

"CMD [\"-jar\" \"DynamoDBLocal.jar\" \"-inMemory\"]"

So if you want to keep your data, you need to do something like this in your docker-compose:

version: '3'

volumes: 
  dynamodb_data:

services:
  dynamodb:
    image: amazon/dynamodb-local
    command: -jar DynamoDBLocal.jar -sharedDb -dbPath /home/dynamodblocal/data/
    volumes:
     - dynamodb_data:/home/dynamodblocal/data
    ports:
     - "8000:8000"
like image 65
Jakub Bujny Avatar answered Oct 04 '22 21:10

Jakub Bujny


To preserve data across docker installations create volume using docker.

docker volume create --driver local --opt type=none \
    --opt device=/var/opt/dynamodb_data --opt o=bind dynamodb_data

use external option:

version: "3"
volumes: 
  dynamodb_data:
    external: true
services:
  dynamodb-local:
    image: amazon/dynamodb-local
    command: ["-jar", "DynamoDBLocal.jar", "-sharedDb", "-dbPath", "/home/dynamodblocal/data"]
    volumes:
      - dynamodb_data:/home/dynamodblocal/data
like image 40
Vladimir Koltunov Avatar answered Oct 04 '22 21:10

Vladimir Koltunov