TLS Configuration

Use this guide to configuring Transport Layer Security (TLS) for Aerospike on your network. For more information about Aerospike's TLS implementation, see Transport Layer Security (TLS).

Aerospike Database Enterprise Edition (EE) and Aerospike Database Enterprise Edition for United States Federal (FE) support Transport Layer Security (TLS) for encrypting the following network traffic:

• Client-to-cluster - traffic between clients and all Database Nodes
• Cluster-to-cluster - cross-datacenter replication (XDR) traffic between clusters
• Node-to-node - fabric and heartbeat traffic between nodes in the same cluster

Planning TLS encryption and type

If you are planning to use TLS encryption, contact Aerospike Support for a smooth migration.

Refer to the TLS Guide for an overview of the TLS functionality. If you require standard authentication or mutual authentication (PKI authentication with mTLS), select what to encrypt and the certificate type.

When using TLS, you can choose to encrypt:

• Traffic between your clients and cluster nodes
• Fabric and/or heartbeat traffic between cluster nodes
• Cross-datacenter (XDR) traffic

Select the certificate match type:

• Cluster-name match
• Wildcard hostname match
• Match with common server certificate
• Hostname match with individual server certificate

Once the TLS deployment architecture is determined, generate the appropriate certificates and use the following procedure.

caution

Dynamically rotating ECDSA private keys and certificates as well as password-protected private keys is not supported in Aerospike server versions prior to 4.7.0.5, 4.6.0.8, 4.5.3.10, 4.5.2.10, 4.5.1.15, 4.5.0.19. More details in the FAQ on TLS.

Enabling and validating TLS

Procedure

Prerequisites

1. The cluster-name setting:

   To enable TLS, your cluster must use the cluster-name setting, and the name must be the same for all the nodes in your cluster.

   Use asadm to dynamically set the cluster-name parameter for your cluster:

   asinfo -v 'set-config:context=service;cluster-name=CLUSTER-NAME'

   Replace CLUSTER-NAME with the name of your cluster.

   Once the asadm command is executed, all the nodes in the cluster detect the change and update their configuration settings accordingly, but you still need to edit the configuration file on each node to ensure that the configuration change is made permanent.

   In the service context of the configuration file, add the cluster-name parameter:

   service {
       ... 
       cluster-name  CLUSTER-NAME
       ...
   }

   Use the asadm tool to confirm that the cluster-name parameter is set correctly:

   show config like cluster-name

   The info command of the asadm tool provides general information about the state of your cluster. Make sure that no migrations are occurring before proceeding with enabling TLS.

2. Certificate requirements for standard authentication:

   • Server certificate: A server certificate is a digital certificate issued to a server by a trusted certification authority (CA) containing a name, a signature and a public key.

     The cert-file configuration parameter must be set and must point to the server certificate. It is the all-in-one .pem file, which includes the server’s own certificate. If the certificate is chain-signed, it must also include the chained CA Certificate files in certificate chain order.

     Sometimes, a server certificate may not be a single certificate but a chain of certificates. A certificate chain is an ordered list of certificates. The chain begins with the SSL certificate, and each certificate in the chain is signed by the entity identified by the next certificate in the chain. Any certificate that sits between the OpenSSL certificate and the root certificate is called a chain or intermediate certificate. The intermediate certificate is the signer/issuer of the OpenSSL certificate. The root CA Certificate is the signer/issuer of the Intermediate Certificate.

   • Server Key: This is the private key for the server certificate. The key-file configuration parameter must be set. It is the .pem file with the private key associated with the certificate.

   • The client must have the relevant root CA (Certificate Authority) certificate configured. A CA certificate has the same structure as a server certificate and contains a name, a signature, and a public key. It is used to verify that the server certificate has the right signature and is correct for its name.

   • For the example described here (cluster name matching), all Aerospike cluster nodes must use same set of certificates.

   • The certificates must have the cluster-name defined in the Subject/CN field or the X509v3 Subject Alternative Name (SAN) field. For example:

     Subject: C=US, ST=California, L=San Jose, O=Acme, Inc., OU=Aerospike Cluster, CN=aerocluster

3. Certificate requirements for mutual authentication:

   In addition to the files required for the client to authenticate the cluster nodes, the following files are also required for cluster nodes to authenticate clients:

   • Client certificate: The certificate must be a valid certificate chain, and must be signed by the same CA (Certificate Authority) as the server.

   • Client Key: The private key for the client certificate.

   • Root CA certificate: The client’s signing CA certificate. The ca-file/ca-path configuration parameter must point to this file or to the path containing the relevant CA certificate.

Enable TLS on the cluster nodes

1. Copy the certificates to all the nodes in the cluster.

2. Set the required TLS parameters in the configuration file.

   Standard authentication

   • In the network context:

     • Add a sub-context named tls with the following parameters:

       • cert-file: the path to the server certificate.
       • key-file: the path to the key file.

   • Add a sub-context named service with the following parameters:

     • tls-address: the bind address for TLS, which is the IP address at which the server listens for client connections. Defaults to any if not set, which will bind to all available interfaces.

     • tls-port: the TLS-enabled-port at which the server listens for client connections, heartbeat connections, and fabric connections.

     • tls-authenticate-client: set to false.

     • tls-name: specifies which TLS parameters to use for the given context TLS connections. The TLS parameters are configured under the matching tls sub-context.

   Example configuration:

   network {
     tls CLUSTER-NAME {
        cert-file /home/user/cluster_chain.pem
        key-file /home/user/key.pem
     }
     service {
         ...
         address 10.0.0.100
         port 3000
         tls-address 10.0.0.100
         tls-port 4333
         tls-name CLUSTER-NAME
         tls-authenticate-client false
         ...
     }
   }

   Mutual authentication

   Mutual authentication requires the same configuration settings as standard authentication, with the following additions:

   • tls-authenticate-client: set to any.

   • ca-file: the path to the CA certificate.

   Example configuration:

   network {
     tls CLUSTER-NAME {
        cert-file /home/user/cluster_chain.pem
        key-file /home/user/key.pem
        ca-file /etc/aerospike/certs/rootCA.pem
     }
     service {
         ...
         address 10.0.0.100
         port 3000
         tls-address 10.0.0.100
         tls-port 4333
         tls-name CLUSTER-NAME
         tls-authenticate-client any
         ...
     }
   }

Restart the node and repeat for each node in the cluster

1. Restart the aerospike service on each node:

   sudo service aerospike restart

2. Check the stability of the cluster with asadm:

   Use the info command:

   info

Validate the TLS setup on the server side

At this point in the process, applications are still able to communicate with the Aerospike cluster via clear ports. Before proceeding to the next step, check to make sure your TLS port is active and available. One way to test your TLS port is to run asadm with the --port argument. You can also use the asbench benchmark tool to run queries against a particular port on your Aerospike cluster.

Update client applications and remove the non-TLS port

Once you've validated that your TLS port is active:

1. Update all client applications to connect with the TLS port.

2. Remove the non-TLS port from your configuration file.

3. Restart each node of the cluster in turn.

TLS configuration parameters

Refer to the Configuration Reference for details regarding required and optional parameters.

Legend

• (*) required parameters for TLS
• (**) required parameters for standard authentication or mutual authentication
• (+) optional parameters for mutual authentication

network {
  tls tlsname1 {                    # (*) This is the TLS name to be referred to in subsequent network
                                    # sub-stanzas that woudld require TLS. Will be populated in the
                                    # tls-name wire field for incoming connections. Could be set to
                                    # CLUSTER-NAME or HOSTNAME or user-defined.
                                    # Set to `tlsname1` in this example.  
    cert-file path-to-file          # (**) Required for standard authentication or mutual authentication
    key-file path-to-file           # (**) Required for standard authentication or mutual authentication
    ca-file path-to-file            # (+) For mutual authentication or for XDR.
                                    # Only one of ca-file / ca-path required.
    ca-path directory-path          # (+) For mutual authentication or for XDR.
                                    # Only one of ca-file / ca-path required.
    protocols TLSv1.2               # Protocol version to include
                                    # note - In version 4.6 the default protocols configuration parameter
                                    # was changed from “-all,+TLSv1.2” to “TLSv1.2”.
    cipher-suite ALL:!COMPLEMENTOFDEFAULT:!eNULL
                                    # Ciphers to includes.
    cert-blacklist path-to-file     # Blacklist including serial number of certs to revoke
  }

  tls tlsname2 {
    ca-file /tls/ca-cert2.pem
    cert-file /tls/rem-chain2.pem
    key-file /tls/rem-key2.pem
    ...
  }

  tls tlsname3.source.domain {
    ...
  }

  tls CLUSTER-NAME {
    ...
  }

  ...

  service {
    port 3000
    address 192.168.80.100          # Bind address for non TLS traffic.
                                    # Default to `tls-address` if not specified.

    tls-port 4333                   # (*) Enables tls
    tls-address 192.168.90.150      # Bind address for TLS.
                                    # Will default to `address` if not specified.
    tls-authenticate-client false|any|user-defined
                                    # If not specified, default is any, which is mutual authentication.
                                    # Multiple tls-authenticate-client entries can be defined in order
                                    # to accept multiple subject names. At typical example would be to
                                    # use different subject names between local clients and XDR clients
                                    # (when specifying the subject name of course).
    tls-name CLUSTER-NAME|HOSTNAME|user-defined  
                                    # (*) Which TLS config to use (from tls sub-stanza).
                                    # This is also the name that would be populated in the tls-name
                                    # wire field. (For example, tlsname1 to refer to the first
                                    # tls sub-stanza described above).
  }

  heartbeat {
    tls-port 3012
    tls-name tlsname2
    tls-mesh-seed-address-port 192.168.90.151 3012
    ...
  }

  fabric {
    tls-port 3011
    tls-name tlsname2
    ...
  }

  ...
}

...

info

The TLS name (specified as part of the tls sub-stanza and referred to within the service, fabric and heartbeat sub-stanzas as well as XDR stanzas) has the ability to be substituted. For example:

• tls / tls-name CLUSTER-NAME means the server will use what's defined in cluster-name as the tls-name attribute.
  
• tls / tls-name HOSTNAME means the server will use what's returned by the underlying OS's "hostname" system call as the tls-name attribute.
  
  

info

ca-path: it may be required to run a "c_rehash" to generate symlinks based on hash values.

Configuring XDR with TLS

For the details on configuring XDR with TLS, see Securing with TLS.

XDR configuration stanza for pre-5.0 servers

The following xdr stanza applies only to versions of Aerospike Server prior to 5.0.

xdr {
  dc DC1 {
    node-address-port 10.10.1.1 3010 tlsname2
                                    # Remote nodes IP address along with TLS name a cluster node expects
                                    # the remote DC to present on XDR connections.
    ...
    tls-name tlsname3.source.domain # The remote cluster should have a tls-authenticate-client directive
                                    # specifying this TLS name, or `false`, or `any`.
    ...
  }

  dc DC2 {
    ...                           # Can have its own set of tls configs
  }

  ...
}

Blacklisting certificates

Certificates that have been compromised can be blacklisted, preventing malicious applications from connecting to the server. To compile a list of blacklisted certificates, edit the cert-blacklist configuration parameter.

The serial number assumes hex representation, with no non-hex characters.

To add a certificate's information to the blocklist, you must have the certificate available. To extract the properly formatted serial number, use the following command:

cat chain.pem | openssl x509 -noout -serial -issuer | \
awk 'match($0,/[^=]*=(.*)/, a) { printf "%s", a[1] } END {printf "\n"}'

Upgrade intra-node connections to TLS

Rolling upgrades from non-TLS to TLS for intra-node connections is supported. The upgrade happens in two rounds:

1. Enable TLS for heartbeat and fabric across all nodes in a rolling fashion.

• Keep non-TLS ports enabled while doing so.
• This first step results in a full cluster supporting non-TLS as well as TLS.

note

Under normal circumstances, a cluster configured with both non-TLS and TLS fabric/heartbeat will default to TLS. However, downgrade attacks are possible, where an attacker prevents the cluster from establishing TLS connections, forcing the cluster to fall back to non-TLS.

2. Disable non-TLS for heartbeat and fabric across all nodes in a rolling fashion.

• At this point, the cluster only supports TLS connection for node to node communication. This prevents downgrade attacks.

Upgrade XDR connections to TLS

To enable TLS on an existing deployment with XDR:

1. Enable TLS on destination clusters

• For each destination cluster, use the procedure described in Enabling and validating TLS.

2. Upgrade source cluster

• For each source cluster node, upgrade XDR's DC configuration to use the newly configured TLS port.

3. Shut down clear ports (optional)

• An additional rolling restart is required to shut down the destination cluster node's clear port (3000 by default).

Examples

Using cluster-name standard authentication

Server Configuration

service {
   ...
   cluster-name as-cluster-west
}

network {

    tls CLUSTER-NAME {
        cert-file   /home/citrusleaf/x509_certificates/chainless_cluster_chain.pem
        key-file    /home/citrusleaf/x509_certificates/Chainless_cluster/key.pem    
    }

    service {
        tls-port 4333
        tls-name CLUSTER-NAME
        tls-authenticate-client false
        ...
    }

    heartbeat {
        ...
        protocol v3
    }
}

C Client Benchmark Call

target/benchmarks -h "192.168.113.203:as-cluster-west:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem

Java Client Benchmark Call

java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:as-cluster-west:4333" -tlsEnable

Using hostname match with a common server certificate, mutual authentication (without name validation), and client-certificate blacklisting

Server Configuration

network {


    tls HOSTNAME {

        ca-file      /home/citrusleaf/x509_certificates/Platinum/cacert.pem  # Root CA's cert

        cert-file    /home/citrusleaf/x509_certificates/multi_chain.pem
        key-file     /home/citrusleaf/x509_certificates/MultiServer/key.pem

        cert-blacklist /root/certs_subject/blacklist.txt
    }

    service {
        tls-port 4333
        tls-name HOSTNAME
        tls-authenticate-client any
        ...
    }

}

C Client Benchmark Call

target/benchmarks -h "192.168.113.203:t3.t-cluster.aerospike.com:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem --tlsChainFile ~/x509_certificates/client_chain.pem --tlsKeyFile ~/x509_certificates/Client/key.pem

Java Client Benchmark Call

java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> Djavax.net.ssl.keyStore=<KeyStorePath> -Djavax.net.ssl.keyStorePassword=<KeyStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:t-cluster.aerospike.com:4333" -tlsEnable

Using hostname match with individual server certificate, and server-certificate blacklisting

Server Configuration

network {

    tls HOSTNAME {

        cert-file    /home/citrusleaf/x509_certificates/server1_chain.pem  
                     # note - different for each server node.
        key-file     /home/citrusleaf/x509_certificates/Server1/key.pem    
                     # note - different for each server node.
    }

    service {
        tls-port      4333
        tls-name      HOSTNAME
        tls-authenticate-client false

    }
}

C Client Benchmark Call

target/benchmarks -h "192.168.113.203:t3.t-cluster.aerospike.com:4333" --tlsEnable --tlsCaFile ~/x509_certificates/Platinum/cacert.pem --tlsCertBlackList ~/bad_serials.txt

Java Client Benchmark Call

java -Djavax.net.ssl.trustStore=<TrustStorePath> -Djavax.net.ssl.trustStorePassword=<TrustStorePassword> -jar target/aerospike-benchmarks-*-jar-with-dependencies.jar -h "192.168.113.203:t-cluster.aerospike.com:4333" -tlsEnable -tlsRevoke 0xfaa7f3e06e288466c5420e0c7ccd4c45

Generating Self-Signed TLS Certificates

Developers may want to generate self-signed TLS certificates for their own development environment. The following gives an example for doing this.

Prepare the certificate directories

Prepare certificate directories for input, local, output, and follow this directory structure:

mkdir -p rootca 2>/dev/null
rm -rf rootca/* 2>/dev/null
mkdir -p rootca/input
mkdir -p rootca/local
mkdir -p rootca/output
cd rootca

Configure OpenSSL

cat <<'EOF' > openssl.conf
HOME                    = .
oid_section             = new_oids

[ new_oids ]
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7

[ ca ]
default_ca      = CA_default            # The default ca section

[ CA_default ]
dir             = rootca                # Where everything is kept
certs           = $dir/certs            # Where the issued certs are kept
crl_dir         = $dir/crl              # Where the issued crl are kept
database        = $dir/index.txt        # database index file
                                        # several certs with same subject
new_certs_dir   = $dir/newcerts         # default place for new certs

certificate     = $dir/cacert.pem       # The CA certificate
serial          = $dir/serial           # The current serial number
crlnumber       = $dir/crlnumber        # the current crl number
                                        # must be commented out to leave a V1 CRL
crl             = $dir/crl.pem          # The current CRL
private_key     = $dir/private/cakey.pem# The private key
x509_extensions = usr_cert              # The extensions to add to the cert
name_opt        = ca_default            # Subject Name options
cert_opt        = ca_default            # Certificate field options
default_days    = 365                   # how long to certify for
default_crl_days= 30                    # how long before next CRL
default_md      = default               # use public key default MD
preserve        = no                    # keep passed DN ordering
policy          = policy_match

[ policy_match ]
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_anything ]
countryName             = (2 letter code, for example, US)
stateOrProvinceName     = State or Province Name (full name)
localityName            = Locality Name (such as, city)
organizationName        = Organization Name (such as, company)
organizationalUnitName  = Organizational Unit Name (such as, section or department)
commonName              = Common Name (such as, server FQDN or YOUR name)
emailAddress            = joedo@joedocompany.com

[ req ]
default_bits            = 2048
default_keyfile         = privkey.pem
distinguished_name      = req_distinguished_name
attributes              = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert
string_mask = utf8only

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code, for example, US)
countryName_default             = US
countryName_min                 = 2
countryName_max                 = 2
stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Some-State
localityName                    = Locality Name (such as, city)
0.organizationName              = Organization Name (such as, company)
0.organizationName_default      = Your company name
organizationalUnitName          = Organizational Unit Name (such as, section)
commonName                      = Common Name (such as, server FQDN or YOUR name)
commonName_max                  = 64
emailAddress                    = joedo@joedocompany.com
emailAddress_max                = 64

[ req_attributes ]
challengePassword               = A challenge password
challengePassword_min           = 4
challengePassword_max           = 20
unstructuredName                = An optional company name

[ usr_cert ]
basicConstraints=CA:FALSE
nsComment                       = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer


[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true

[ crl_ext ]
authorityKeyIdentifier=keyid:always

[ proxy_cert_ext ]
basicConstraints=CA:FALSE
nsComment                       = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

[ tsa ]
default_tsa = tsa_config1       # the default TSA section

[ tsa_config1 ]
dir             = ./demoCA              # TSA root directory
serial          = $dir/tsaserial        # The current serial number (mandatory)
crypto_device   = builtin               # OpenSSL engine to use for signing
signer_cert     = $dir/tsacert.pem      # The TSA signing certificate
                                        # (optional)
certs           = $dir/cacert.pem       # Certificate chain to include in reply
                                        # (optional)
signer_key      = $dir/private/tsakey.pem # The TSA private key (optional)
signer_digest  = sha256                 # Signing digest to use. (Optional)
default_policy  = tsa_policy1           # Policy if request did not specify it
                                        # (optional)
other_policies  = tsa_policy2, tsa_policy3      # acceptable policies (optional)
digests     = sha1, sha256, sha384, sha512  # Acceptable message digests (mandatory)
accuracy        = secs:1, millisecs:500, microsecs:100  # (optional)
clock_precision_digits  = 0     # number of digits after dot. (optional)
ordering                = yes   # Is ordering defined for timestamps?
                                # (optional, default: no)
tsa_name                = yes   # Must the TSA name be included in the reply?
                                # (optional, default: no)
ess_cert_id_chain       = no    # Must the ESS cert id chain be included?
                                # (optional, default: no)
ess_cert_id_alg         = sha1  # algorithm to compute certificate
                                # identifier (optional, default: sha1)
EOF

Generate the Certificate Authority (CA) root directory

openssl genrsa -out local/rootCA.key 2048
openssl req -x509 -new -nodes -key local/rootCA.key -sha256 -days 3650 -out local/rootCA.pem -subj "/C=UK/ST=London/L=London/O=fed/OU=Support/CN=rootca.fed"

Generate TLS certificate requests

The following example generates a TLS certificate for the server, and a certificate for the admin user. This pair of certs can used for Mutual TLS (mTLS) with the clients, as well as PKI authentication for the users.

The tls-name of the TLS context used in the service stanza must also be set as Common Name (CN) of the server's certificate. For example, CN=server.

The tls-name of the TLS context used in the service stanza may also be set as the certificate's X509v3 Subject Alternative Name (SAN) field.

For PKI auth the CN of the user's certificate must match their username. For example CN=admin.

openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/server.req -keyout output/server.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Server/CN=server"
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/admin.req -keyout output/admin.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Admin/CN=admin"

Example: Generate a user certificate

For each user, create a user certificate

export USERNAME=app
openssl req -new -nodes -config openssl.conf -extensions v3_req -out input/${USERNAME}.req -keyout output/${USERNAME}.key -subj "/C=UK/ST=London/L=London/O=fed/OU=Admin/CN=${USERNAME}"
openssl x509 -req -extfile openssl.conf -in input/${USERNAME}.req -CA local/rootCA.pem   -CAkey local/rootCA.key  -extensions v3_req -days 3649 -outform PEM -out output/${USERNAME}.pem -set_serial 310
openssl verify -verbose -CAfile local/rootCA.pem output/${USERNAME}.pem

Sign the certificates

openssl x509 -req -extfile openssl.conf -in input/server.req -CA local/rootCA.pem   -CAkey local/rootCA.key  -extensions v3_req -days 3649 -outform PEM -out output/server.pem -set_serial 110
openssl x509 -req -extfile openssl.conf -in input/admin.req -CA local/rootCA.pem   -CAkey local/rootCA.key  -extensions v3_req -days 3649 -outform PEM -out output/admin.pem -set_serial 310

Verify the certificates

openssl verify -verbose -CAfile local/rootCA.pem output/server.pem
openssl verify -verbose -CAfile local/rootCA.pem output/admin.pem

TLS-related configuration parameters and references

List of the TLS related configuration parameters:

• To be used in the network's tls sub-stanza:

• ca-file

• ca-path

• cert-blacklist

• cert-file

• cipher-suite

• key-file

• key-file-password

• protocols

• To be used in the network's service, fabric, heartbeat sub-stanzas and xdr stanza:
• tls-name
• tls-port (service, fabric and heartbeat sub-stanzas only)
• tls-authenticate-client (service sub-stanza only)
• tls-address (service, fabric and heartbeat sub-stanzas only)
• tls-mesh-seed-address-port (heartbeat sub-stanza only)