How to start a gen_server or gen_fsm on demand in Erlang without race conditions?

Question

I need to spawn several independent instances of the same gen_fsm on demand, and then be able to route calls to the correct instance.

Gproc library seems to be a great way of registering processes with arbitrary names. It has a function gproc:reg_or_locate/3 for spawning things on demand without race conditions. This way I don't even need a supervisor - if they crash they'll just get spawned on demand again. But I can't figure out how to apply gproc:reg_or_locate/3 to spawning a gen_fsm or gen_server.

What I've tried so far:

I just call gen_server:start() through that function, it will create an intermediate process, give the name to the intermediate process, the intermediate process will spawn a gen_server and terminate, and I end up with a nameless gen_server.

Both gen_server and gen_fsm export an enter_loop function that seems to do what I need if I feed it to gproc:reg_or_locate/3, but the documentation reads:

The process must have been started using one of the start functions in proc_lib, see proc_lib(3).

And the docs for gproc:reg_or_locate/3 do not mention that they do anything through proc_lib.

Alternatively I could make the intermediate process acquire the name and then atomically transfer it to the gen_server or gen_fsm that it spawned, but that creates a race condition: the intermediate process would have gen_fsm's name and any messages intended for the gen_fsm would go to the intermediate process and get lost.

I feel like I'm missing something simple here. It's not an uncommon pattern, so there should be a good way to do this. What did I miss?

Answer 1

For your purposes I don't think gproc:reg_or_locate/3 really gives you anything useful. If it returns a PID (due to spawning a new process, or locating an existing one) the process could still die before you send a message to it, so unless you have a mechanism on top of the basic Erlang messaging you'll never know this didn't happen. The server could also die before receiving the message or die processing it even if it was alive when you sent the message, so given you express some concern for messages getting lost, one component of the solution must be a reliable message mechanism. The sensible and ready available solution to that is gen_server:call and gen_fsm:sync_send_event in your case, rather than just sending a message.

This eliminates the problem of messages getting lost from any spawning solution you care to implement. That is, you will know the message was lost, or failed, and then you can take whatever action is appropriate.

Now, for the actual spawning of your servers, there is always going to be a race condition where multiple processes might try and spawn the same server (a server with a given name) no matter how you implement it; anything you to to lookup the name (e.g. erlang:whereis/1) could be out of date before you do anything else (it could return a PID, but the PID could die before you message it, or it could return undefined but some other process could register the name before you try to) so the only point at which the race is won (or lost) is when erlang:register/2 is called.

You know then that there might be a race, but that there can be at most one winner. It might not be you, some other process might beat you to the spawning, but since you are naming your processes that doesn't matter, you can simply spawn your gen_server, giving it the name to register itself as, and then message it by name:

gen_server:start({local, Name}, ?MODULE, [], []),
gen_server:call(Name, Message)

It doesn't matter who won the race (the gen_server:start/4 call might return {error,{already_started, Pid}}) but so what, all that's important is someone should have won, and the call to gen_server:call has every chance of success thereafter.

You obviously do need to make sure the call returned a suitable success result, technically you could check for a noproc exception and try to spawn it again, but then you'd have to make sure this doesn't become an infinite loop.

To be honest, although you don't care about supervision, I'd probably have it supervised anyway. In this case a simple_one_to_one supervisor with a restart strategy set to temporary so it doesn't respawn fits. Your servers will be collected in one place then, not just floating in limbo, and you'll get supervisor reports, which can't be a bad thing. Sadly you won't get run away restart protection because there are no restarts here so you still need to worry about that (unless you change temporary to transient). Your effective point of arbitration would then be supervisor:start_child/2 and you would pass the desired name of the process as a parameter.

Answer 2

As Michael suggested, I would also go for supervision.

You can use the {via,Module,ViaName} in your gen_server:start_link to use other names than atoms. See here for details: http://erlang.org/doc/man/gen_server.html#start_link-4

For example, with gproc

gen_server:start_link({via, gproc, {n, l, {?MODULE, Name}}, ?MODULE, [], []).

Don't forget to use the same {via, gproc, ...} structure when calling the gen_server, in stead of just using the Name:

gen_server:call({via, gproc, {n, l, {?MODULE, Name}}, {execute_command, Command}). 

I tend to define the via like this:

-define(SERVER(Name), {via, gproc, {n, l, {?MODULE, Name}}}).

And then use it like:

gen_server:start_link(?SERVER("Testing"), ?MODULE, [], []).
gen_server:call(?SERVER("Testing"), {execute_command, Command}).

You can then start it in a supervisor with a simple_one_for_one strategy and a temporary child specifitation, like this:

Supervisor

-module(my_cool_sup).

-behaviour(supervisor).

%% API
-export([start_link/1, start_child/1]).    
%% Supervisor callbacks
-export([init/1]).

-define(SERVER, ?MODULE).

%% Helper macro for declaring children of supervisor
-define(CHILD(ChildName, Type, Args), {ChildName, {ChildName, start_link, Args}, temporary, 5000, Type, [ChildName]}).

%%====================================================================
%% API functions
%%====================================================================

start_link() ->
    supervisor:start_link({local, ?SERVER}, ?MODULE, []).

start_child(Name) ->
    supervisor:start_child(?SERVER, [Name]).

%%====================================================================
%% Supervisor callbacks
%%====================================================================

init([]) ->
    RestartStrategy = {simple_one_for_one, 1, 5},

    Children = [?CHILD(my_cool_server, worker, [])],

    {ok, { RestartStrategy, Children} }.

gen_server

-module(my_cool_server).

-behavior(gen_server).

%% API
-export([start_link/3, execute_command/3]).
%% gen_server callbacks
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
     terminate/2, code_change/3]).

-define(SERVER(Name), {via, gproc, {n, l, {?MODULE, Name}}}).

%%%===================================================================
%%% API
%%%===================================================================

start_link(Name) ->
    gen_server:start_link(?SERVER(Name), ?MODULE, [], []).

execute_command(Name, Command) ->
    gen_server:call(?SERVER(Name), {execute_command, Command}). 

%%%===================================================================
%%% gen_server callbacks
%%%===================================================================

%% Your normal gen_server callbacks here...

Now you can use my_cool_sup:start_child("My cool name"). to start you child processes. They will be supervised and if they are already started it will return already_started but not throw an error.

Check start_child for more details: http://erlang.org/doc/man/supervisor.html#start_child-2