Performance is *way* too slow with pubsub push -> app engine

Question

I have a requirement to process ~20k calls per second. My system processes lists of ~1M entries and performs multiple jobs for each item. It is very “bursty” in nature as it isn’t always processing a list. I added a App Engine flex env (with Rails), using automatic scaling, with an test endpoint to wait 5 seconds and return. I push to the pubsub topic and a push subscription sends to App Engine. Running this hits a steady state of 20-30 requests per second.

I guessed that the problem was the interaction of the pubsub push volume algorithm interacting with the App Engine, but then I ran a second test where I just blasted curl requests as in a loop with multiple processes. This also ran at 20-30 rps.

I’m stuck at this point and wondering how to proceed. How can I configure the system for higher performance? I need a performance of three orders of magnitude from what I see.

Thanks so much for helping!

Answer 1

My experience with pub/sub is that is really slow for single messaging processing. I'd guess that there's an overhead for being http. And the average time I see is around 40ms outside google and 20ms if you are running your code on google servers. What worked for me is to batch messages instead, I was able to get up to 100k msgs/sec when batching them in 1k messages per publish.