How to deploy TensorFlow Serving using Docker and DigitalOcean Spaces

Question

How do you configure TensorFlow Serving to use files stored in DigitalOcean Spaces?

It's important that the solution:

• provides access to both the configuration and model files
• provides non-public access to the data

I have configured a bucket named your_bucket_name in DigitalOcean Spaces with the following structure:

- your_bucket_name
  - config
    - batching_parameters.txt
    - monitoring_config.txt
    - models.config
  - models
    - model_1
      - version_1.1
        - variables
          - variables.data-00000-of-00001
          - variables.index
        - saved_model.pb
   - model_2
       - ...
   - model_3
       - ...

Answer 1

EDIT: Since TensorFlow 2.6, S3 support has been deprecated, and the Docker images also no longer support models stored on S3 [reference]. (Credit to @RonaldDas.)

---

HISTORY: TensorFlow Serving supports integration with Amazon S3 buckets. Since DigitalOcean Spaces provide a similar interface, it's possible to easily run TensorFlow Servings with DigitalOcean Spaces via Docker by piggybacking off the S3 interface.

To make it easier for others, I've detailed everything you need to know about running the server below:

1. Environment Variables (Optional)

Define the following variables in your environment:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

(This is not strictly necessary but defining these variables makes your deployment more secure than hard-coding the values into your docker-compose file, for example.)

You receive the values for these variables from DigitalOcean Spaces as part of configuring your cloud storage bucket.

2. Server

You can start the server using Docker or docker-compose:

2.1. Using Docker

Here's a minimal docker command for starting the server from a command prompt:

docker run \
    -p 8500:8500 \
    -p 8501:8501 \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_REGION=nyc3 \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e S3_ENDPOINT=nyc3.digitaloceanspaces.com \
    tensorflow/serving \
    --model_config_file=s3://your_bucket_name/config/models.config

(For running this on Windows, you may need to remove backtick-newlines to make this a single-line command.)

2.2. Using docker-compose

This docker-compose configuration is a bit more detailed in the way that the server is configured, but you can use these options with the simple docker command as well.

version: "3"
services:
  tensorflow-servings:
    image: tensorflow/serving:latest
    ports:
      - 8500:8500
      - 8501:8501
    command:
      - --batching_parameters_file=s3://your_bucket_name/config/batching_parameters.txt
      - --enable_batching=true
      - --model_config_file=s3://your_bucket_name/config/only_toxic.config
      - --model_config_file_poll_wait_seconds=300
      - --monitoring_config_file=s3://your_bucket_name/config/monitoring_config.txt
      - --rest_api_timeout_in_ms=30000
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_LOG_LEVEL=3
      - AWS_REGION=nyc3
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - S3_ENDPOINT=nyc3.digitaloceanspaces.com

The log levels are reduced here because there are a lot of "Connection released" and "No response body" messages that are not actual errors. (See GitHub Issue: AWS lib is verbose when using S3 for more details.)

3. Configuration Files:

The configuration files look like this, they are protocol buffers:

3.1. models.config

model_config_list {
  config {
    name: 'model_1'
    base_path: 's3://your_bucket_name/models/model_1/'
      model_platform: "tensorflow"
  },
  config {
    ...
  },
  config {
    ...
  }
}

3.2. batching_parameters.txt (Optional)

This file defines guidelines to TensorFlow Serving; shepherding the way it handles batching in the server.

    max_batch_size { value: 1024 }
    batch_timeout_micros { value: 100 }
    num_batch_threads { value: 4 }
    pad_variable_length_inputs: true

3.3. monitoring_config.txt (Optional)

This file makes various statistics available via the endpoint defined below.

prometheus_config {
  enable: true,
  path: "/monitoring/metrics"
}