Skip to main content
Loading

XDR static and dynamic configuration

Aerospike's Cross-Datacenter Replication (XDR) service is for inter-cluster replication of namespaces, sets, or bins. XDR is included with Aerospike Enterprise Edition. The XDR service is built into the Aerospike Server Daemon, asd.

tip

Aerospike Database updated XDR network behavior in versions 6.1.0.31, 6.3.0.19, 6.4.0.13, and 7.0.0.6. To get the most out of XDR, we recommend that you upgrade to one of these releases.

For a theoretical and architectural overview, see XDR Architecture.

Other XDR information

Terminology

The act of sending data from one datacenter to another is sometimes called shipping.

  • A cluster from which records are shipped is called a local or source cluster.
  • A cluster to which records are shipped is called a remote or destination or target cluster.

In many examples, to represent the user-defined name of a datacenter, the abbreviation "DC" is often used, like DC1, DC2, and so on.

Static configuration vs dynamic configuration​

You can configure XDR in several ways: statically or dynamically.

  • Static configuration for XDR means that all the XDR-related parameters are included in the Aerospike configuration file, as shown in Basic static XDR stanza: xdr, dc, node-address-port, and namespace.

  • Dynamic configuration for XDR means that the XDR-related parameters are set with the asadm's manage config command or asinfo's set-config command. Prior to Aerospike version 5.7, the XDR subsystem should be configured statically with at least one DC. Starting with Aerospike version 5.7, the XDR subsystem always accepts dynamic configuration commands because it is always initialized even if XDR is not configured statically. See Dynamic configuration with asadm.

Aerospike recommends configuring XDR dynamically so that all nodes start shipping at the same time. Also, changing the configuration file preserves the configuration through subsequent restarts.

Basic static XDR stanza: xdr, dc, node-address-port, and namespace​

The basic xdr stanza of the configuration file:

  • Enables XDR.
  • Defines target datacenters, including IP addresses and ports of cluster nodes.
  • Specifies namespaces to ship.
xdr {
    dc dataCenter1 {
        node-address-port someIpAddress1 somePort1
        namespace nameSpaceName1A { 
        
        }
        namespace nameSpaceName1B {
           // optional - can be all empty
           bin-policy <all/only-changed/changed-and-specified/...>
           enable-compression <true/false>
           forward <true/false>
           ignore-expunges <true/false>
           max-throughput 50000
           transaction-queue-limit 32768
           write-policy <auto/update/replace>
           remote-namespace <someRemoteNameSpace>
           ship-only-specified-sets <true/false>
           // if ship-only-specified-sets set to true
           ship-set <someSetName1>
           ship-set <someSetName2>
           // refer to the configuration reference manual
           // for the full list of optional parameters
           ...
        }
    }
    dc dataCenter2 {
        node-address-port <someIpAdress2> <somePort2>
        namespace nameSpaceName2A {
           ...
        }
        namespace nameSpaceName2B {
           ...
        }
    }
    ...
}

Explanation

ParameterDefault valueDescription
xdrNo value is required.Enables XDR and encloses all other XDR-related parameters.
dcNoneYou yourself declare a variable name of the datacenter that follows the dc parameter.

The xdr stanza can include multiple dc sub-stanzas.

The maximum number of dc definitions is 64.
node-address-portNoneSpecifies the IP address and port of a node in the remote datacenter.

The dc sub-stanza can include multiple node-address-port specifications. Most of these examples show only one node-address-port.

The port specification can be followed by a Transport Layer Security (TLS) variable name you define. See Securing XDR with access control, LDAP, and with TLS.
namespaceNoneSpecifies the name of the namespace data to ship to the remote datacenters. The dc stanza can include multiple namespace declarations. The remote-namespace parameter is required only if the namespace needs to be replicated to a different namespace.

Recommendation: global naming convention for namespaces​

Consider developing a global plan or naming convention for namespaces on all your clusters that participate in XDR.

If you intend to use namespace mapping, especially take into account the restrictions described in Namespace mapping with the remote-namespace parameter.

Such global naming conventions can help clarify the purposes and locations of namespaces in your global deployment, avoid inadvertently violating naming restrictions, and minimize confusion that might occur due to a misunderstanding of or lack of knowledge of existing namespaces or namespace mappings.

Namespace mapping​

Usually, records from a local namespace are shipped to a remote namespace with the same name.

In Aerospike Server version 5.2+, a namespace can be shipped to a remote namespace with a different name. Refer to the remote-namespace configuration parameter for details.

xdr {
  dc DC1 {
    node-address-port 10.0.0.1 3000
    namespace A {
       # Map local namespace A to remote namespace B
       remote-namespace B
    }
  }
}

Dynamic configuration with asadm​

note

Tools package 6.0.x or later is required to use asadm's manage config commands. Otherwise, see Dynamic configuration with asinfo.

With asadm's manage config command, you can configure datacenters dynamically without restarting any database node.

  • First, use manage config to define the dc and its corresponding name-address-port parameters, as shown with the clause below.
asadm -e "enable; manage config xdr create dc DC1"
asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:3000"
  • Then, add the desired namespace to that defined dc, as shown with the clause below:
asadm -e "enable; manage config xdr dc DC1 add namespace someNameSpaceName"
  • It is strongly recommended to also add this same configuration to your configuration file in order to not lose it upon restart.

Rewind or restart shipment​

With the asadm's manage config command, you can rewind a namespace's shipment of records a specific number of seconds or restart shipment completely.

When you are rewinding, the namespace to rewind must not be currently configured for that datacenter. That is, if you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time, but you must remove an existing namespace and add it back to rewind. Be aware that removing the namespace from that datacenter discards the existing XDR queue, so that if there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.

In the example below, for datacenter DC1 and namespace biggy, the rewind 600 clause rewinds shipment to 10 minutes before the current time:

asadm -e "enable; manage config xdr dc DC1 remove namespace biggy"
asadm -e "enable; manage config xdr dc DC1 add namespace biggy rewind 600"

For small datasets, you can restart the shipment entirely from the beginning with rewind all. For large datasets, be careful with starting a complete rewind, which can increase server loads. The example below completely restarts shipment for datacenter teenytiny and namespace itsybitsy:

asadm -e "enable; manage config xdr dc teenytiny remove namespace itsybitsy"
asadm -e "enable; manage config xdr dc teenytiny add namespace itsybitsy rewind all"

Dynamic configuration with asinfo​

With the asinfo set-config command, you can configure datacenters dynamically without restarting any database node.

The example parameters shown below can all be entered on a single asinfo set-config command line but are shown separately for clarity.

  • First, use asinfo set-config to define the dc and its corresponding name-address-port parameters, as shown with the parameters and the action=add clause below.
asinfo -v "set-config:context=xdr;dc=DC1;action=create"
asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:3000;action=add"
  • Then, add the desired namespace to that defined dc, as shown with the parameter and the action=add clause below:
asinfo -v "set-config:context=xdr;dc=DC1;namespace=someNameSpaceName;action=add"
  • For good measure, you should also add this same configuration to your configuration file.

Rewind or restart shipment​

With the asinfo set-config command, you can rewind a namespace's shipment of records a specific number of seconds or restart shipment completely.

When you are rewinding, the namespace to rewind must not be currently configured for that datacenter. That is, if you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time, but you must remove an existing namespace and add it back to rewind. Be aware that removing the namespace from that datacenter will discard the existing XDR queue, so that if there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.

In the example below, for datacenter DC1 and namespace biggy, the action=add;rewind=600 clause rewinds shipment to 10 minutes before the current time:

asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=remove"
asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=add;rewind=600"

For small datasets, you can restart the shipment entirely from the beginning with action=add;rewind=all. For large datasets, be careful with starting a complete rewind, which can increase server loads. The example below completely restarts shipment for datacenter teenytiny and namespace itsybitsy:

asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=remove"
asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=add;rewind=all"

Example configuration parameters for XDR topologies​

This section presents configuration file parameters that correspond to different supported topologies described in XDR Architecture. Only the basic XDR stanzas are shown, with no optional parameters.

Active-passive​

In active-passive topology, data are written to only a single node and then shipped to other datacenters.

Datacenter A ships to B at 10.2.0.2Datacenter B at 10.2.0.2
xdr {
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
      }
   }
}

Mesh (active-active)​

In this example, each datacenter ships to all the others but not to itself.

Datacenter B at 10.2.0.2Datacenter A at 10.0.0.1Datacenter C at 10.3.0.3
xdr {
   dc A {
      node-address-port 10.0.0.1 3000
      namespace someNameSpaceName {
         ...
      }
   }
   dc C {
      node-address-port 10.3.0.3 3000
      namespace someNameSpaceName {
         ...
      }
   }
}

xdr {
   dc A {
      node-address-port 10.0.0.1 3000
      namespace someNameSpaceName {
         ...
      }
   }
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
     }
   }
}

xdr {
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
      }
   }
   dc C {
      node-address-port 10.3.0.3 3000
      namespace someNameSpaceName {
         ...
      }
   }
}

Mesh with bin-convergence (active-active)​

In this example, each datacenter ships to all the others with bin-convergence feature turned on. Important things to note:

  • src-id must be different in all the DCs.
  • conflict-resolve-writes must be set to true in the namespace-level in all the DCs.
  • bin-policy must not be all.
Datacenter B at 10.2.0.2Datacenter A at 10.0.0.1Datacenter C at 10.3.0.3
namespace {
   ...
   conflict-resolve-writes true
}

xdr {
   src-id 2
   dc A {
      node-address-port 10.0.0.1 3000
      namespace someNameSpaceName {
         ...
         bin-policy only-changed
         ship-bin-luts true
      }
   }
   dc C {
      node-address-port 10.3.0.3 3000
      namespace someNameSpaceName {
         ...
         bin-policy only-changed
         ship-bin-luts true
      }
   }
}

namespace {
   ...
   conflict-resolve-writes true
}

xdr {
   src-id 3
   dc A {
      node-address-port 10.0.0.1 3000
      namespace someNameSpaceName {
         ...
         bin-policy only-changed
         ship-bin-luts true
      }
   }
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
         bin-policy only-changed
         ship-bin-luts true
     }
   }
}

namespace {
   ...
   conflict-resolve-writes true
}

xdr {
   src-id 1
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
         bin-policy only-changed
         ship-bin-luts true
      }
   }
   dc C {
      node-address-port 10.3.0.3 3000
      namespace someNameSpaceName {
         ...
         bin-policy only-changed
         ship-bin-luts true
      }
   }
}

Linear chain​

In linear chain topology, data are shipped in a straight line from one datacenter to another and another and so on.

In the example below, in datacenter B, forward true is needed to ship records from Datacenter B to Datacenter C.

Datacenter A at 10.0.0.1 ships to B at 10.2.0.2Datacenter B at 10.2.0.2 ships to C at 10.3.0.3Datacenter C at 10.3.0.3
xdr {
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
      }
   }
}
xdr {
   dc C {
      node-address-port 10.3.0.3 3000
      namespace someNameSpaceName {
         forward true
         ...
      }
   }
}
To avoid a loop, Datacenter C must not point back to A.

Star​

In this example, Datacenter A ships to all other datacenters, with no shipment among those other datacenters.

Datacenter B at 10.2.0.2:3000Datacenter A at 10.0.0.1Datacenter D at 10.4.0.4:3000
xdr {
   dc B {
      node-address-port 10.2.0.2 3000
      namespace someNameSpaceName {
         ...
      }
   }
   dc C {
      node-address-port 10.3.0.3 3000
      namespace someNameSpaceName {
         ...
      }
   }
 dc D {
      node-address-port 10.4.0.4 3000
      namespace someNameSpaceName {
         ...
      }
   }
 dc E {
      node-address-port 10.5.0.5 3000
      namespace someNameSpaceName {
         ...
      }
   }
}
Datacenter C at 10.3.0.3:3000 Datacenter E at 10.5.0.5:3000

Server versions​

The Aerospike servers running in different destination clusters do not need to be the same version. For example, if you have clusters running Aerospike server 6.0, you can add a new cluster with the latest server version and preserve XDR functionality.

Where to Next?​

Last updated on