Why is Heroku's heroku-18 stack only compatible with Puma versions 3.7.x?

Question

By trial-and-error, I determined that upgrading from Heroku's heroku-16 stack to heroku-18 stack would require that I update Puma in my Rails app.

Below is the failure to build native extensions for the puma gem. Can you explain why this failure occurs with puma gem versions before v3.7?

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/tmp/build_893b3955637fcd96d49136f0c3564c51/vendor/bundle/ruby/2.4.0/gems/puma-3.6.2/ext/puma_http11
/tmp/build_893b3955637fcd96d49136f0c3564c51/vendor/ruby-2.4.5/bin/ruby -r
./siteconf20190110-284-13d7ke9.rb extconf.rb
checking for BIO_read() in -lcrypto... yes
checking for SSL_CTX_new() in -lssl... yes
checking for openssl/bio.h... yes
creating Makefile

current directory:
/tmp/build_893b3955637fcd96d49136f0c3564c51/vendor/bundle/ruby/2.4.0/gems/puma-3.6.2/ext/puma_http11
make "DESTDIR=" clean

current directory:
/tmp/build_893b3955637fcd96d49136f0c3564c51/vendor/bundle/ruby/2.4.0/gems/puma-3.6.2/ext/puma_http11
make "DESTDIR="
compiling http11_parser.c
In file included from ext/puma_http11/http11_parser.rl:7:0:
ext/puma_http11/http11_parser.rl: In function ‘puma_parser_execute’:
ext/puma_http11/http11_parser.rl:111:17: warning: comparison between signed and
unsigned integer expressions [-Wsign-compare]
ext/puma_http11/http11_parser.rl: At top level:
cc1: warning: unrecognized command line option ‘-Wno-self-assign’
cc1: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1: warning: unrecognized command line option ‘-Wno-parentheses-equality’
compiling io_buffer.c
compiling mini_ssl.c
mini_ssl.c: In function ‘get_dh1024’:
mini_ssl.c:90:5: error: dereferencing pointer to incomplete type ‘DH {aka struct
dh_st}’
   dh->p = BN_bin2bn(dh1024_p, sizeof(dh1024_p), NULL);
     ^~
mini_ssl.c: In function ‘engine_init_server’:
mini_ssl.c:139:3: warning: ISO C90 forbids mixed declarations and code
[-Wdeclaration-after-statement]
   ID sym_cert = rb_intern("cert");
   ^~
mini_ssl.c:144:3: warning: ISO C90 forbids mixed declarations and code
[-Wdeclaration-after-statement]
   ID sym_ca = rb_intern("ca");
   ^~
mini_ssl.c:166:3: warning: ISO C90 forbids mixed declarations and code
[-Wdeclaration-after-statement]
   DH *dh = get_dh1024();
   ^~
mini_ssl.c:170:3: warning: ISO C90 forbids mixed declarations and code
[-Wdeclaration-after-statement]
   EC_KEY *ecdh = EC_KEY_new_by_curve_name(NID_secp521r1);
   ^~~~~~
mini_ssl.c: In function ‘engine_init_client’:
mini_ssl.c:197:3: warning: ‘DTLSv1_method’ is deprecated
[-Wdeprecated-declarations]
   conn->ctx = SSL_CTX_new(DTLSv1_method());
   ^~~~
In file included from /usr/include/openssl/ct.h:13:0,
                 from /usr/include/openssl/ssl.h:61,
                 from mini_ssl.c:15:
/usr/include/openssl/ssl.h:1642:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *DTLSv1_method(void)) /* DTLSv1.0 */
 ^
mini_ssl.c: In function ‘engine_shutdown’:
mini_ssl.c:333:8: warning: unused variable ‘buf’ [-Wunused-variable]
   char buf[512];
        ^~~
mini_ssl.c:332:11: warning: unused variable ‘err’ [-Wunused-variable]
   int ok, err;
           ^~~
mini_ssl.c: In function ‘engine_init’:
mini_ssl.c:350:8: warning: unused variable ‘buf’ [-Wunused-variable]
   char buf[512];
        ^~~
mini_ssl.c:349:11: warning: unused variable ‘err’ [-Wunused-variable]
   int ok, err;
           ^~~
mini_ssl.c:349:7: warning: unused variable ‘ok’ [-Wunused-variable]
   int ok, err;
       ^~
mini_ssl.c: At top level:
cc1: warning: unrecognized command line option ‘-Wno-self-assign’
cc1: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1: warning: unrecognized command line option ‘-Wno-parentheses-equality’
Makefile:241: recipe for target 'mini_ssl.o' failed
make: *** [mini_ssl.o] Error 1

make failed, exit code 2

Gem files will remain installed in
/tmp/build_893b3955637fcd96d49136f0c3564c51/vendor/bundle/ruby/2.4.0/gems/puma-3.6.2
for inspection.
Results logged to
/tmp/build_893b3955637fcd96d49136f0c3564c51/vendor/bundle/ruby/2.4.0/extensions/x86_64-linux/2.4.0-static/puma-3.6.2/gem_make.out

An error occurred while installing puma (3.6.2), and Bundler cannot continue.
Make sure that `gem install puma -v '3.6.2'` succeeds before bundling.

Answer 1

The Heroku-16 stack is based on Ubuntu 16.04, which ships with OpenSSL 1.0.2 while the Heroku-18 stack is based on Ubuntu 18.04 which ships with OpenSSL 1.1.0.

Between these OpenSSL versions, there were quite some API and behavior changes which often requires some changes to software using OpenSSL APIs directly (like Puma does).

In Puma 3.7.0, there was a change to make it compatible with OpenSSL 1.1.0 which thus makes it the first version to be compatible with that version.