Are erlang:send_after/3 and timer:send_after/3 intended to behave differently?

Question

I wanted to send a message to a process after a delay, and discovered erlang:send_after/4.

When looking at the docs it looked like this is exactly what I wanted:

erlang:send_after(Time, Dest, Msg, Options) -> TimerRef

Starts a timer. When the timer expires, the message Msg is sent to the process identified by Dest.

However, it doesn't seem to work when the destination is running on another node - it tells me one of the arguments are bad.

1> P = spawn('node@host', module, function, [Arg]).
<10585.83.0>
2> erlang:send_after(1000, P, {123}).
** exception error: bad argument
     in function  erlang:send_after/3
        called as erlang:send_after(1000,<10585.83.0>,{123})

Doing the same thing with timer:send_after/3 appears to work fine:

1> P = spawn('node@host', module, function, [Arg]).
<10101.10.0>
2> timer:send_after(1000, P, {123}).
{ok,{-576458842589535,#Ref<0.1843049418.1937244161.31646>}}

And, the docs for timer:send_after/3 state almost the same thing as the erlang version:

send_after(Time, Pid, Message) -> {ok, TRef} | {error, Reason}

Evaluates Pid ! Message after Time milliseconds.

So the question is, why do these two functions, which on the face of it do the same thing, behave differently? Is erlang:send_after broken, or mis-advertised? Or maybe timer:send_after isn't doing what I think it is?

Answer 1

TL;DR

Your assumption is correct: these are intended to do the same thing, but are implemented differently.

Discussion

Things in the timer module such as timer:send_after/2,3 work through the gen_server that defines that as a service. Like any other service, this one can get overloaded if you assign a really huge number of tasks (timers to track) to it.

erlang:send_after/3,4, on the other hand, is a BIF implemented directly within the runtime and therefore have access to system primitives like the hardware timer. If you have a ton of timers this is definitely the way to go. In most programs you won't notice the difference, though.

There is actually a note about this in the Erlang Efficiency Guide:

3.1 Timer Module

Creating timers using erlang:send_after/3 and erlang:start_timer/3 , is much more efficient than using the timers provided by the timer module in STDLIB. The timer module uses a separate process to manage the timers. That process can easily become overloaded if many processes create and cancel timers frequently (especially when using the SMP emulator).

The functions in the timer module that do not manage timers (such as timer:tc/3 or timer:sleep/1), do not call the timer-server process and are therefore harmless.

A workaround

A workaround to gain the efficiency of the BIF without the same-node restriction is to have a process of your own that does nothing but wait for a message to forward to another node:

-module(foo_forward).
-export([send_after/3, cancel/1]).
% Obviously this is an example only. You would want to write this to
% be compliant with proc_lib, write a proper init/N and integrate with
% OTP. Note that this snippet is missing the OTP service functions.

start() ->
    spawn(fun() -> loop(self(), [], none) end).

send_after(Time, Dest, Message) ->
    erlang:send_after(Time, self(), {forward, Dest, Message}).

loop(Parent, Debug, State) ->
    receive
        {forward, Dest, Message} ->
            Dest ! Message,
            loop(Parent, Debug, State);
        {system, From, Request} ->
            sys:handle_msg(Request, From, Parent, ?MODULE, Debug, State);
        Unexpected ->
            ok = log(warning, "Received message: ~tp", [Unexpected]),
            loop(Parent, Debug, State)
    end.

The above example is a bit shallow, but hopefully it expresses the point. It should be possible to get the efficiency of the BIF erlang:send_after/3,4 but still manage to send messages across nodes as well as give you the freedom to cancel a message using erlang:cancel_timer/1

But why?

The puzzle (and bug) is why erlang:send_after/3,4 does not want to work across nodes. The example you provided above looks a bit odd as the first assignment to P was the Pid <10101.10.0>, but the crashed call was reported as <10585.83.0> -- clearly not the same.

For the moment I do not know why erlang:send_after/3,4 doesn't work, but I can say with confidence that the mechanism of operation between the two is not the same. I'll look into it, but I imagine that the BIF version is actually doing some funny business within the runtime to gain efficiency and as a result signalling the target process by directly updating its mailbox instead of actually sending an Erlang message on the higher Erlang-to-Erlang level.

Maybe it is good that we have both, but this should definitely be clearly marked in the docs, and it evidently is not (I just checked).

Answer 2

There is some difference in timeout order if you have many timers. The example below shows erlang:send_after does not guarantee order, but timer:send_after does.

1> A = lists:seq(1,10).
[1,2,3,4,5,6,7,8,9,10]
2> [erlang:send_after(100, self(), X) || X <- A].
...
3> flush().
Shell got 2
Shell got 3
Shell got 4
Shell got 5
Shell got 6
Shell got 7
Shell got 8
Shell got 9
Shell got 10
Shell got 1
ok
4> [timer:send_after(100, self(), X) || X <- A]. 
...
5> flush().
Shell got 1
Shell got 2
Shell got 3
Shell got 4
Shell got 5
Shell got 6
Shell got 7
Shell got 8
Shell got 9
Shell got 10
ok