TLS Support

Overview#

Nodes communicate with each other via HTTP on the private port (default: 8374). This port is used to reverse proxy application requests between nodes, to share gossip information between nodes, and to service RPC requests originating from maelctl.

By default this port is not encrypted. In some environments, such as when you’re running your own physical hardware, this is acceptible provided that port 8374 is firewalled from the public internet. However, in most cases it is desirable to encrypt this port.

Maelstrom supports TLS encryption of the private port. To enable TLS in your cluster:

• Use openssl to generate a key pair
• Ensure that the key pair files are copied to each node before starting maelstromd
• Set the appropriate environment variable to point at the directory containing the key pairs
  • If more than one key pair exists in the directory, all key pairs are trusted, but the key with the lexigraphically smallest file name will be used to as the active key on that node.
  • This allows keys to be rotated while the system is running

Generate key pairs#

Step 1: Create tls.conf file#

Run this bash script, editing the [dn] block as desired. The maelstrom.example.com name should be used as-is, and does not need to resolve to a valid IP address. This is used to satisfy SAN requirements.

cat <<EOF > tls.conf
[req]
default_bits = 2048
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn

[dn]
C = ES
ST = MyState
L = MyCity
O = MyOrg
emailAddress = email@example.com
CN = example.com

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = maelstrom.example.com
EOF

Step 2: Create keys using openssl#

# create directory to store keys
mkdir -p /etc/maelstrom/tls
cd /etc/maelstrom/tls

# create three certs
openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3560 \
  -keyout server_a.key -out server_a.crt \
  -config tls.conf

openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3560 \
  -keyout server_b.key -out server_b.crt \
  -config tls.conf

openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3560 \
  -keyout server_c.key -out server_c.crt \
  -config tls.conf

Step 3: Set env var to enable TLS#

Set this environment variable:

export MAEL_PRIVATE_TLS_DIR=/etc/maelstrom/tls

When maelstromd is restarted with this variable set, the given directory will be scanned for .key and .crt files. The lexigraphically smallest file name will be used as the active key. The other keys will be trusted but will not be actively used.

Using TLS with maelctl#

maelctl will respect the MAEL_PRIVATE_TLS_DIR env var. If that variable is set, it will use the lexigraphically smallest file name in that directory to sign requests.

Key rotation#

You may periodically wish to rotate your keys. If you use the 3 key setup described above, this is a fairly simple process:

# Move 'a' to 'c', effectively deprecating it
mv -f server_a.key server_c.key
mv -f server_a.crt server_c.crt

# Move 'b' to 'a', promoting it to the active key
mv -f server_b.key server_a.key
mv -f server_b.crt server_a.crt

# Create a new 'b' key which will be unused now, but will be rotated active next
openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3560 \
  -keyout server_b.key -out server_b.crt \
  -config tls.conf

Sync this directory to each node in the cluster and then restart maelstromd on each node.