The routing section of aerospike-pubsub-outbound.yml
The routing
section of the /etc/aerospike-pubsub-outbound/aerospike-pubsub-outbound.yml
controls how Aerospike records are routed to a destination topic in Google Pub/Sub.
A connector can route messages to one or more topics. You can use the static
routing mode to specify in advance the exact names of the topics to route messages to. You can also use three different dynamic routing modes to have the connector determine the names of topics dynamically, based on information in records.
Here is a list of the routing modes:
Static mode
static
Always routes to the specified topic. This is the default mode.skip
Skip dispatch of record to Google Pub/Sub and ack success to XDR.
Dynamic modes
namespace
Uses the namespace of each Aerospike record as the name of the topic to publish messages to.set
Uses the set of each Aerospike record as the name of the topic to publish to. For records without a set, you can specify a fallback route that uses the "static" routing mode.bin
Uses the value of a bin in each Aerospike record as the name of the topic to publish to. Only string, blob, and integer bin types are supported.
When you use one of the dynamic modes, you can transform the names of namespaces, sets, and bins by changing their case, removing whitespace, or using regular expressions to replace characters.
Regional Endpoint
Google Cloud Pub/Sub has global and regional service endpoints. By default, the Pub/Sub connector uses the global gRPC endpoint: pubsub.googleapis.com.
When the connector sends requests to this global endpoint, Pub/Sub automatically routes the request to a nearby region. However, Pub/Sub might route traffic to a region that does not have sufficient quota. If you have additional quota in a particular region, the connector can route requests directly to it by using a regional endpoint, rather than the global endpoint.
For a list of the regions and gRPC service endpoints for Pub/Sub, see the section "Service Endpoints" in "Service APIs Overview" in the Pub/Sub documentation.
Be sure to add the port number 443
.
Static routing
Here are the configuration options:
Option | Required | Expected value | Description |
---|---|---|---|
mode | yes | static | Specifies to write messages to the topic that is named in the destination option. |
destination | yes | The name of the destination topic. | |
regional-endpoint | no | The regional service endpoint to publish messages to. |
Example
routing:
mode: static
destination: name-of-topic
regional-endpoint: us-east1-pubsub.googleapis.com:443
Skip routing
For skip routing the configuration options are
Option | Required | Expected value | Description |
---|---|---|---|
mode | yes | skip | Skip dispatch of record to JMS and ack success to XDR. |
Example
routing:
mode: skip
Routing by namespace names
Here are the configuration options:
Option | Required | Expected value | Description |
---|---|---|---|
mode | yes | namespace | Specifies to write to topics that have names that match names of namespaces in record metadata. |
default | no | The default destination topic to use in case the namespace name is missing in the record or the destination topic is not found. | |
transforms | no | List of transformations to apply to the namespace name. See the Transforming dynamically derived names section for details. | |
regional-endpoint | no | The regional service endpoint to publish messages to. |
Example
routing:
mode: namespace
default: default-topic
transforms:
- trim
- regex:
pattern: '(.*):(.*)'
replacement: '$2:$1'
- regex:
pattern: '$'
replacement: ':please'
- uppercase
regional-endpoint: us-east1-pubsub.googleapis.com:443
Routing by set names
Here are the configuration options:
Option | Required | Expected value | Description |
---|---|---|---|
mode | yes | set | Specifies to write to topics that have names that match names of sets in record metadata. |
default | no | The default destination topic to use in case the set name is missing in the record or the destination topic is not found. | |
transforms | no | List of transformations to apply to the set name. See the Transforming dynamically derived names section for details. | |
regional-endpoint | no | The regional service endpoint to publish messages to. |
Example
routing:
mode: set
transforms:
- trim
- regex:
pattern: '(.*):(.*)'
replacement: '$2:$1'
- regex:
pattern: '$'
replacement: ':please'
- uppercase
regional-endpoint: us-east1-pubsub.googleapis.com:443
Routing by bin values
Here are the configuration options:
Option | Required | Expected value | Description |
---|---|---|---|
mode | yes | bin | Specifies to write to topics that have names that match values in the specified bin. |
bin | yes | The name of the bin to pick value from. | |
default | yes | The default destination topic to use in case the bin is missing in the record or the destination topic is not found. | |
transforms | no | List of transformations to apply to the bin value. See the Transforming dynamically derived names section for details. | |
regional-endpoint | no | The regional service endpoint to publish messages to. |
Example
routing:
mode: bin
bin: category
default: test-topic
transforms:
- trim
- regex:
pattern: '[^A-Za-z0-9]'
replacement: '-'
- lowercase
regional-endpoint: us-east1-pubsub.googleapis.com:443
Transforming dynamically derived names
You can configure a list of transforms that will be applied, in order, to the record's set name, namespace path, or bin value to derive the name of the destination topic.
The following transformations are supported:
lowercase
Converts to lowercase.uppercase
Converts to uppercasetrim
Trims leading and trailing whitespace.regex
Matches against a regex pattern and replaces all occurrences with a replacement. The regex and replacement use Java regex syntax.
Example
The following transform configuration trims the route, replaces all non-alphanumeric characters with '-', and then converts the result to lowercase.
routing:
mode: bin
bin: category
default: test-topic
transforms:
- trim
- regex:
pattern: '[^A-Za-z0-9]'
replacement: '-'
- lowercase
Custom Routing
Record can be also be routed with custom code. See Routing Transform.
Example
routing:
mode: custom
class: com.aerospike.connect.outbound.example.GenerationRouter