UUID Generated randomly is having duplicates

Question

I'm using the below function to generate UUID

UUID.randomUUID().toString()

In production we have 50+ servers (application server - each is a JVM on its own) and for requests that land in these servers, as a first step we generate a UUID which essentially uniquely identifies a transaction.

What we are observing is that in Server 6 and Server 11, the UUIDs generated are matching at least for 10 to 15 messages per day which is strange because given the load i.e. about 1 million transactions a day, these UUIDs being duplicate within the same day is very odd.

This is what we have done so far

1. Verified the application logs - we didn't find anything fishy in there, all logs are as normal
2. Tried replicating this issue in the test environment with similar load in production and with 50+ servers - but this didn't happen in the test environment
3. Checked the application logic - this doesn't seem to be an issue because all other 48 servers except 6 and 11 which have a copy of the same code base is working perfectly fine and they are generating unique UUIDs per transaction.

So far we haven't been able to trace the issue, my question is basically if there is something at JVM level we are missing or UUID parameter that we need to set for this one off kind of an issue?

Answer 1

Given time, I'm sure you'll find the culprit. In the meantime, there was a comment that I think deserves to be promoted to answer:

You are generating pseudo random UUIDs at multiple locations. If you don't find other bugs, consider either generating all the pseudo random UUIDs at one location, or generate real random UUIDs

So create a UUID server. It is just a process that churns out blocks of UUIDs. Each block consists maybe 10,000 (or whatever is appropriate) UUIDs. The process writes each block to disk after the process verifies the block contains no duplicates.

Create another process to distribute the blocks of UUIDs. Maybe it is just an a web service that returns an unused block when it gets a request. The transaction server makes a request for a block and then consumes those UUIDs as it creates transactions. When the server has used most of its assigned UUIDs, it requests another block.