How can the "packet" option of socket in Erlang accelerate the tcp transmission so much?

Question

It takes only 8 seconds to transfer 1G data through two different ports on localhost using {packet,4}, while the same task can't be finished within 30 seconds using {packet,raw}. I know if use the latter method, the data will arrive in tens of thousands small pieces (on archlinux the size is 1460 bytes). I've learned some aspects of TCP/IP protocol and have been thinking about this question for days but still can't figure out what is the EXACT difference. Sincerely look forward to some bottom-up explanation.

-module(test).

-export([main/1]).

-define(SOCKOPT, [binary,{active,true},{packet,4}]).

main(_) ->
    {ok, LSock} = gen_tcp:listen(6677, ?SOCKOPT),
    spawn(fun() -> send() end),
    recv(LSock).

recv(LSock) ->
    {ok, Sock} = gen_tcp:accept(LSock),
    inet:setopts(Sock, ?SOCKOPT),
    loop(Sock).

loop(Sock) ->
    receive
        {tcp, Sock, Data} ->
            io:fwrite("~p~n",[bit_size(Data)]),
            loop(Sock);
        {tcp_closed, Sock} -> ok
    end.

send() ->
    timer:sleep(500),
    {ok, Sock}=gen_tcp:connect("localhost", 6677, ?SOCKOPT),
    gen_tcp:send(Sock, binary:copy(<<"1">>, 1073741824)),
    gen_tcp:close(Sock).

$ time escript test.erl
8589934592

real    0m8.919s
user    0m6.643s
sys     0m2.257s

Answer 1

When you use {packet,4} erlang first reads 4 bytes to get length of your data, allocates a buffer to hold it and reads data into buffer after getting each tcp packet. Then it sends the buffer as one packet to your process. This all happens inside builtin read code, which is rather fast.

When you use {packet,raw} erlang sends a message to your process after receiving each tcp packet of data, so for each tcp packet it does many more things.

Answer 2

When the data is received in small pieces, the kernel buffer at receiver end fills up fast. It will reduce the congestion window size at sender side forcing the sender to push data at lower rate.