Bin policy
XDR version 5.0 introduced the bin projection functionality (via the ship-only-specified-bins
configuration parameter).
Starting with version 5.2, a new functionality was added to ship the changed bins. These two
functionalities create interesting combinations which can cater to different
use-cases. The new policy introduced in version 5.2 is called bin-policy
.
The ship-only-specified-bins
configuration is removed in 5.2 and the new bin-policy
supersedes it with extra functionality. Allowed values for this config are:
Config Value | Explanation |
---|---|
all (default) | This is the default mode. The entire record will be shipped to the destination always. |
no-bins | [Introduced in Server 6.0]. Only allowed for connector type DCs. Bins will not be shipped, only record's metadata will be shipped to the destination. |
only-changed | Only the changed bins in a write will be shipped. The bins which did not change will not get shipped. |
only-specified | [Removed as of version 5.3]. Only the bins specified via the ship-bin config will be shipped no matter which bins got changed as part of the write. |
changed-and-specified | It is like the && operator. In other words, it is the intersection of changed and specified bins. A bin will be shipped if it got changed and also is specified via the ship-bin config. The bins which got changed but not specified via ship-bin config will not be shipped |
changed-or-specified | It is like the || operator. In other words, it is the union of changed and specified bins. Both the changed bins in a write and the list of bins specified via the ship-bin config will be shipped |
Overhead
If the XDR bin-policy
is set to
only-changed
, changed-and-specified
or changed-or-specified
, there will be extra overhead
because we need to maintain extra information to provide the necessary functionality. The overhead is:
- 6 bytes for each bin to maintain last-updated-time (LUT) and some metadata.
- deleted bins will become bin tombstones. A bin tombstone costs the same as a regular bin with no data. These bin tombstones will be cleaned up only on a subsequent write based on the
xdr-bin-tombstone-ttl
configuration.
Refer to the Capacity Planning Guide for further details.
Note that the bin-policy
config is at the dc/namespace sub-context. If at least one dc/namespace
sub-context uses the above policies, there will be extra overhead. At a later point in time,
if none of the dc/namespace level configs uses the above policies (switches to all
), the
overhead will be removed on a subsequent write operation as the extra information is no longer
necessary.
Interactions
This bin-policy
config has interactions with other configs like ignore-bins
, write-policy
and connector
. Some combinations of these do not make sense and are restricted. The general philosophy is as follows:
- If the user chooses a
bin-policy
which will pick specified bins (viaship-bin
config)ignore-bin
config is not honored at the same time. - If the user chooses a
bin-policy
which ships changed bins they cannot setwrite-policy
toreplace
- If the user chooses the
no-bins
bin-policy it will be allowed only ifconnector
config is true.
Config Value | ignore-bin honored? | write-policy=replace allowed? | connector |
---|---|---|---|
all (default) | yes | yes | any |
no-bins | NA | no | true |
only-changed | yes | no | any |
only-specified | no | no | any |
changed-and-specified | no | no | any |
changed-or-specified | no | no | any |
Examples
The following examples demonstrate the rules and the interactions between bin-policy
, ship-bin
, and ignore-bin
. If the state of record and config are as follows.
- Record has 8 bins : b1, b2, b3, b4, b5, b6, b7, b8
- Bins configured via
ship-bin
: b1, b5 - Bins configured via
ignore-bin
: b2, b6
Case-1
Say a write comes for b1, b2, b3. Lets see (pictorially) which bins get shipped under different policies.
b1 | b2 | b3 | b4 | b5 | b6 | b7 | b8 | |
---|---|---|---|---|---|---|---|---|
ship-bin | ship-bin | |||||||
ignore-bin | ignore-bin | |||||||
write | write | write | ||||||
--------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
all | ships | ships | ships | ships | ships | ships | ||
only-changed | ships | ships | ||||||
only-specified | ships | ships | ||||||
changed-and-specified | ships | |||||||
changed-or-specified | ships | ships | ships | ships |
Case-2
Say a write comes for b6, b7, b8. Lets see (pictorially) which bins get shipped under different policies.
b1 | b2 | b3 | b4 | b5 | b6 | b7 | b8 | |
---|---|---|---|---|---|---|---|---|
ship-bin | ship-bin | |||||||
ignore-bin | ignore-bin | |||||||
write | write | write | ||||||
--------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
all | ships | ships | ships | ships | ships | ships | ||
only-changed | ships | ships | ||||||
only-specified | ships | ships | ||||||
changed-and-specified | ||||||||
changed-or-specified | ships | ships | ships | ships | ships |
Use cases
Below are example configs for some common usecases
Shipping specific bins if they changed
In this example, for namespace someNameSpaceName
, only binName1
and binName2
are shipped.
xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy changed-and-specified
ship-bin binName1
ship-bin binName2
}
}
}
Excluding specific bins
In this example, for namespace someNameSpaceName
, all bins are shipped except binName1
and binName2
.
xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
ignore-bin binName1
ignore-bin binName2
}
}
}
Ship changed bins
In this example, for namespace someNameSpaceName
, all bins that are modified by a write will be shipped.
xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy only-changed
}
}
}
Ship changed bins plus some additional bins
In this example, for namespace someNameSpaceName
, all bins that are modified by a write plus binName1
and binName2
will be shipped.
xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy changed-or-specified
ship-bin binName1
ship-bin binName2
}
}
}
Ship changed bins but not some bins
In this example, for namespace someNameSpaceName
, all bins that are modified by a write except binName1
and binName2
will be shipped.
xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy only-changed
ignore-bin binName1
ignore-bin binName2
}
}
}