Best clock or number generator function for concurrency/scalability on Erlang OTP 18?

Question

Currently in the process of converting an erlang application from version 17 to 18. Scalability and performance are prime directives in the design. The program needs a way to differentiate and sort new input coming in, either with lots of unique monotonically increasing numbers (a continuous stream of them), or some other mechanism. The current version (17) did not use now() for this because it is a scalability bottleneck (global lock), so it made due with reading the clock and doing other things to generate the tags for the data coming in. I'm trying to figure out the best way to do this in 18 and have some interesting results from the tests I've run.

I expected erlang:unique_integer([monotonic]) to have poor results, because I expected it to have a global lock like now(). I expected one of the clock functions to have the best results, assuming the clock could be read in parallel. Instead, erlang:unique_integer([monotonic]) gets the best results out of all the functions I benchmarked, and the clock functions do worse.

Could someone explain the results, tell me which erlang functions SHOULD give the best results, and which things (clocks, number generators, etc) are or are not globally locked in 18? Also, if you see any issues with my test methodology, by all means point them out.

TEST PLATFORM/METHODOLOGY

windows 7 64 bit
erlang otp 18 (x64)
2 intel cores (celeron 1.8GHz)
2 erlang processes spawned to run each test function concurrently 500000 times
    for a total of 1000000 times, timed with timer:tc
each test run 10 times in succession and all results recorded

BASELINE TEST, SEQENTIAL

erlang:unique_integer([monotonic])
47000-94000

PARALLEL TIMES

erlang:unique_integer([monotonic])
~94000

ets:update_counter
450000-480000

erlang:monotonic_time
202000-218000

erlang:system_time
218000-234000

os:system_time
124000-141000

calendar:universal_time
453000-530000

Answer 1

If you ask about test methodology I would expect you also include your code because there can be a small mistake in benchmark code which could ruin the result. So I write one and made Gist so we can compare result using the same code. YMMV especially because I use Linux and timers strongly depends on underlying OS. There are mine results:

$ uname -a
Linux hynek-notebook 4.1.0-1-amd64 #1 SMP Debian 4.1.3-1 (2015-08-03) x86_64 GNU/Linux
$ grep 'model name' /proc/cpuinfo 
model name      : Intel(R) Core(TM) i5 CPU       M 520  @ 2.40GHz
model name      : Intel(R) Core(TM) i5 CPU       M 520  @ 2.40GHz
model name      : Intel(R) Core(TM) i5 CPU       M 520  @ 2.40GHz
model name      : Intel(R) Core(TM) i5 CPU       M 520  @ 2.40GHz
$ erl
Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V7.0  (abort with ^G)
1> c(test).
{ok,test}
2> test:bench_all(1).
[{unique_monotonic_integer,{38341,39804}},
 {update_counter,{158248,159319}},
 {monotonic_time,{217531,218272}},
 {system_time,{224630,226960}},
 {os_system_time,{53489,53691}},
 {universal_time,{114125,116324}}]
3> test:bench_all(2).
[{unique_monotonic_integer,{40109,40238}},
 {update_counter,{307393,338993}},
 {monotonic_time,{120024,121612}},
 {system_time,{123634,124928}},
 {os_system_time,{29606,29992}},
 {universal_time,{177544,178820}}]
4> test:bench_all(20).
[{unique_monotonic_integer,{23796,26364}},
 {update_counter,{514835,527087}},
 {monotonic_time,{91916,93662}},
 {system_time,{94615,96249}},
 {os_system_time,{27194,27598}},
 {universal_time,{317353,340187}}]
5>

The first thing what I should note, only erlang:unique_integer/0,1 and ets:update_counter/3,4,5 generate unique value. Even erlang:monotonic_time/0 can generate two same timestamps! So if you want unique number you don't have other option than use erlang:unique_integer/0,1. If you want unique monotonic timestamp you can use {erlang:monotonic_time(), erlang:unique_integer()} or if you don't need the time part you can use erlang:unique_integer([monotonic]). If you don't need monotonic and unique, you can use other options. So if you need unique monotonic number there is only one good option and it is erlang:unique_integer([monotonic]).

The second time I should note, spawning two processes is not enough to test scalability. As you can see, when I use os:timestamp/0 with 20 processes, they start catch up with erlang:unique_integer/0,1. And there is another problem. We both use HW with only two CPUs. It is far too few to test scalability. Imagine how would result look on HW with 64 and more cores.

Edit: Using {write_concurrency, true} will improve ets:update_counter but still far beyond erlang:unique_integer/0,1.

2> test:bench(test:update_counter(),1).
{203830,213657}
3> test:bench(test:update_counter(),2).
{129148,140627}
4> test:bench(test:update_counter(),20).
{471858,501198}

Answer 2

According to erlang code base, erlang:unique_integer([monotonic]) is just increasing atomic integer. This works fast. While this still creates memory barrier, atomic operation is still cheap comparing to conventional global lock approach.