`docker run` as Prefect task

Question

My actual workloads that should be run as tasks within a Prefect flow are all packaged as docker images. So a flow is basically just "run this container, then run that container".

However, I'm unable to find any examples of how I can easily start a docker container as task. Basically, I just need docker run from a flow.

I'm aware of https://docs.prefect.io/api/latest/tasks/docker.html and tried various combinations of CreateContainer and StartContainer, but without any luck.

Answer 1

Using the Docker tasks from Prefect's Task Library could look something like this for your use case:

from prefect import task, Flow
from prefect.tasks.docker import (
    CreateContainer,
    StartContainer,
    GetContainerLogs,
    WaitOnContainer,
)

create = CreateContainer(image_name="prefecthq/prefect", command="echo 12345")
start = StartContainer()
wait = WaitOnContainer()
logs = GetContainerLogs()


@task
def see_output(out):
    print(out)


with Flow("docker-flow") as flow:
    container_id = create()
    s = start(container_id=container_id)
    w = wait(container_id=container_id)

    l = logs(container_id=container_id)
    l.set_upstream(w)

    see_output(l)

flow.run()

This snippet above will create a container, start it, wait for completion, retrieve logs, and then print the output of echo 12345 to the command line.

Alternatively you could also use the Docker Python client directly in your own tasks https://docker-py.readthedocs.io/en/stable/api.html#module-docker.api.container