In gRPC & AWS we talked about why ALBs and ELBs won’t work for gRPC. The good news was that the new Network Load Balancers (NLBs) do work for gRPC, but unfortunately that means that you can no longer rely on AWS Certificate Management for SSL. If you're like me you've probably become pretty accustomed to AWS Certificate Manager. It is fabulous and trivially easy to use, but like I said it won't work for NLBs.
So... how do we do it the hard way?
Let's (Not) Encrypt
Doing SSL on your own you’ve got a couple options. Our initial thought was to use LetsEncrypt, because, well because it’s free. There were two big gotchas there that stopped us.
First, LetsEncrypt has a very short duration of 90 days. The intent is that this will force you to make sure you have a good automated solution to updating your keys. That’s a nice idea, but it’s also a sword of Damacles every 90 days. More unfortunately I found the documentation around the certbot refresh utilities to be sparse and/or pretty confusing.
The second, and for us the showstopper, was that the LetsEncrypt CA is not included in the Java default CA (technically some versions of java 8 do). That means we’d have to ensure that all environments trying to run the client had a special CA added... this is basically a non-starter for a client we intend our customers to run.
So that left us with purchasing a cert and getting our private key to be available for all our container instances running in ECS.
Certs in ECS
The easiest path forward here was to create an EFS file system that is available to all of ECS. Then we can put the key in there, and through the magic of Docker host mounts, all apps will be able to see it. The docs to make this happen are quite good. The first step is to make an EFS file system and have all of your ECS nodes mount it on startup.
This is what you'll add to your
user_data.sh That's the file that gets run everytime you start a new instances in your ECS cluster.
The next piece is to make sure your containers have access to this mounted filesystem. This comes in two pieces. First making it available to the task.
That makes the instance's mounted directory available to the task. I didn't want to have to setup another EFS if I want to do this again, but I don't necessarily want my containers to have access to all of my EFS. My solution here picks out a subdirectory
/efs/ssl of the EFS mount and makes that available as
The next thing to do is to make our container aware that this volume is available and assign it a path where our app will look.
This tells our container that the
efs volume can be found at
Loading the PrivateKey into gRPC
Finally we need to tell gRPC to actually load these two files, thecert chain and key.
This loads the private key and certChainFile that you got from DNSimple into gRPC.
One Last PCKS Gotcha
One last gotcha to round it out? Sure. It turns out that there isn’t exactly agreement on what format these private keys should be in. gRPC will give you a cryptic error on the private key that you download from DNSimple until you run this magic incantation to make it pcks8.
openssl pkcs8 -topk8 -nocrypt -in key_from_dns_simple.key -out key_that_grpc_likes.key.pcks8
I won’t claim to know exactly why that is necessary, but I’m guessing you are more interested in getting the darn thing to work :)
Overall, doing SSL termination in gRPC is very doable, but may require you to get down and dirty with a few things you may be unfamiliar with.