How does Prefect scale with thousands of workflows concurrently?

Question

I have a prefect server running locally (0.13 core version). I called flow.run() in a loop 1000 thousand times in a server machine with 64 GB of RAM with 32 cores of CPU. When it got up to ~300 runs, it started throwing connection refused errors from GraphQL.

I am still considering whether to use Prefect for my workflows, but it looks like it's using up way too much RAM. How does Prefect scale with thousands of workflows concurrently?

I am running the workflow with a simple example:

176 from flask import Flask
177 app = Flask(__name__)
178
179 import prefect
180 client = prefect.Client()
181
182 @app.route('/')
183 def hello_world():
184     client.create_flow_run("032275d0-6c31-4dc5-bf32-5b2afadbe531")
185     return 'Hello, World!'

Then I am calling the REST API to trigger the flow from 1 to 1000.

for i in {1..1000}; do curl localhost:5000/; done

I am getting that GraphQL is using a lot of memory (up to 10 GB RAM). Then the Prefect UI starts to hang around 100.

I am not sure if I am using Prefect workflow as its intended usage, but would like to work this out if possible.

Answer 1

The open source Prefect Server was not designed for that sort of scale; as described in this new doc, this is one of the reasons people migrate to Prefect Cloud, which is designed for scale and performance.