Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Google Cloud TCP external load balancer and TLS not self signed

Is it possible to directly expose a server behind a L4 load balancer, with a public certificate?

This server is inside a Kubernetes pod. There is a TCP loadbalancer service in front of it which creates the external L4 LB.

My problem is that the TLS traffic does not reach the container inside the pod. So if you succeeded with a similar configuration, I would be interested into knowing.


Update

I did not mention that the traffic is GRPC.

Here is what I did: I have a domain and a corresponding official certificate. I want to secure the grpc connection.

I tried two approches:

  • with google ESP container, I put the cert as an nginx secret, pass it to the container, set an ssl-port. Behind the ESP in the same pod, I have my grpc server

In this case I get a message like this on the client side:

D0610 14:38:46.246248584 32401 security_handshaker.cc:176] Security handshake failed: {"created":"@1591792726.246234613","description":"Handshake failed","file":"../deps/grpc/src/core/lib/security/transport/security_handshaker.cc","file_line":291,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}

I see some TLS exchanges with wireshark but no log in esp.

  • without ESP, I put the cert in my GRPC server. There the GRPC server fails with something like this:

error:1408F10B:SSL routines:ssl3_get_record:wrong version number

In the google ESP documentation I see that I have to prove the domain belongs to me and upload the cert (but where)?


Update 2

As of today, I see no evidence that it is feasible.

IMO, the main issue is that the L4 has the IP corresponding to the domain name of the certificate. Hence the pods don't have the correct IP to prove that they can use the cert so their request towards roots are denied (I have no proof of that as I can't get debug info from nginx in the ESP. I have seen a request with the pure GRPC server solution though).

like image 764
unludo Avatar asked Jun 09 '20 18:06

unludo


Video Answer


1 Answers

The issue was in TLS exchange.

By installing the cert in the ESP, it works fine with a web browser and the certificate is indicated valid, whereas with a GRPC client, the TLS handshake fails. Adding additional trace info helped.

By checking my certificate (not self signed but attached to my domain), I found that there is an intermediate certificate provided with it. I installed it along with the domain certificate (in the same crt file) and then it worked.

I don't know exactly why it is behaving like this but maybe it's due to the root_cert file in grpc client lib being too old.

By the way for a domain cert, there is no specific requirement regarding CN and subjectAltName for the certificate. It works without it. So it must only apply to self signed certs as I have seen elsewhere.

I had another issue that disturbed my task: be careful not to name the service port of the L4 load balancer with 'http2' inside. I had some side effect, making another deployment fail due to this. In fact when you do https, don't put http2 in the name.

Anyway it is now working and answers the request for the bounty. Thanks to all who tried to help :)

like image 183
unludo Avatar answered Oct 11 '22 17:10

unludo