Clustering
From Halon Security
This chapter will cover the basics on how to cluster multiple SPG/VSP appliances together to build a fast and fault tolerant mail cluster, which appear as one or more appliances. This clustering covers both a shared administration interface, configuration and mail quarantine.
Contents
|
The concept
Not only for large businesses, but also for small and mid-sized companies clustering may be a important concept to protect your network from unexpected events. But even if you don't feel that you need the extra benefits of clustering right now, it's good for you to know that when your business grows, the mail cluster may follow.
Clustering are done for may reasons, including
- Risk mitigation
- Higher performance
- Improved manageability with multiple appliances
Our clustering is based on a multi-master topology, all appliances in the cluster are connected to each other and configuration can be preformed on any appliances in your cluster. This reduces the risk of a single point of failure, and allowing appliances to be added and taken out of the cluster without breaking the overall topology.
Together all appliances share a "cluster" configuration, which includes everything except your local network settings (and a few other "unclusterable" values, like storage disk path), while we also support private values on selected parts of the configuration, like host names et.c. (advanced users may set even more private keys, than presented in the web administration).
Preparations
When first building a cluster, the two appliances need to have somewhat compliant configurations. The most vital requirement is that the two appliances have the same number of network addresses and that they match the topology (in terms of inbound/outbound, internal/external).
If an error occurs while doing the initial synchronization it may because of what we call a "Unresolved Dependency"; that is when a configuration key, for example "netaddr:2" is in the shared configuration, while "Appliance 2" only has "netaddr:1" in it's private configuration. You need to add another network address and put the appliance once more in Join mode and repeat the last step where you try to synchronize the configuration on Appliance 1 to Appliance 2.
Before you start...
- If you already have one or more remote systems configured, it might be easier to remove them all and before you start.
- If you have Quarantine -> Clustering -> Remote Systems activated, you should disable this (nothing is removed, the quarantine will just not be clustered) (because if you don't, it might prevent you from completing the tips above). Once the cluster is configured, you must activate quarantine clustering again if you wish to use this feature.
Build your cluster from scratch
Read below how to cluster two appliances. Additional appliances (when adding number three, four, and so on) can be added in a more simple fashion; see below. The guide below assumes that you have not yet configured remote systems on any of the appliances. If so, begin by removing those.
| Step | Appliance 1 | Appliance 2 | Comment |
|---|---|---|---|
|
1 |
On Administration -> Users, add a new user called for example "cluster". Use a difficult password (since you will probably not change it that often). |
On Administration -> Users, add a new user called for example "cluster". Use the same password as on Appliance 1. |
This is optional. The benefit of creating an additional user is that you may then change the password of normal users (such as "admin") without breaking the cluster. |
|
2 |
Add Appliance 2 as a remote system (type cluster@ip-address in the field "Remote System URL" at the upper left corner of the administration interface, below the logo). |
Add Appliance 1 as a remote system (type cluster@ip-address in the field "Remote System URL" at the upper left corner of the administration interface, below the logo). | |
|
3 |
Go to Administration -> Clustering and activate "Push Configuration" |
Go to Administration -> Clustering and activate "Push Configuration" | |
|
4 |
Go to Administration -> Clustering and push the "Join Now" button. | ||
|
5 |
Go to Administration -> Clustering and press "Push Now". |
This is all it takes to cluster two appliances together. Confirm in the log on Administration -> Clustering on "Appliance 1" that the push was successful.
Adding the third (and following) appliance
| Step | Already clustered appliance | New appliance | Comment |
|---|---|---|---|
|
1 |
On Administration -> Users, add a new user called for example "cluster". Use the same password as the other appliances in the cluster. |
Use the same username/password as with all the other appliances in the cluster. | |
|
2 |
Go to Administration -> Clustering and activate "Push Configuration" | ||
|
3 |
Go to Administration -> Clustering and push the "Join Now" button. | ||
|
4 |
Add the new appliance as a remote system (type cluster@ip-address in the field "Remote System URL" at the upper left corner of the administration interface, below the logo). |
This is all it takes to add more appliances to your cluster. Confirm in the log on Administration -> Clustering on the selected appliance that the push was successful.
Screenshots
 Signing on to a member of the cluster |
 Viewing the cluster configuration on VSP 1 |
 Viewing the cluster status overview |
 Some values are not shared in the cluster |
 while some values can be private if you want |
Cluster demystified
This chapter will turn out clustering magic into logic. It's highly recommended to read this chapter now, since it can be somewhat stressful once the cluster is broken and this knowledge is really needed.
Our cluster consists of a shared configuration, which is "Pushed" to the cluster each time you modify the configuration on one of the appliances. In order to keep the clustering logic sane, this can only be done when all appliances are synchronized.
How do we know when to synchronize the configuration?
Well, each configuration revision is assigned a UUID (Universally Unique Identifier or a unique randomised id if you like). This UUID can be seen in the configuration when exported (eg. uuid="bb30efa7-3c0f-4cb1-b0dd-fb601286d303"), but it cannot be changed manually, and any attempts to import a configuration manually with a custom UUID value will make us generate a new one internally, this is simple because one should not try to out smart our clustering mechanism.
The clusters common goal is the have the latest UUID generated (configuration revision) shared on all appliances in the cluster, this is achieved by Pushing the configuration to all other appliances in the cluster which are synchronized.
All appliances keep track of a backlog (history file) of UUIDs which have either been Pushed or imported from the cluster (Accepted).
We consider a unit to be "in-sync":
- If the latest UUID is somewhere in our backlog (this is a very common scenario when preforming any kind of reconfiguration).
- This will also allow a unit to be shut down while all the other appliances are reconfigured; and when the unit comes back online again, it's latest configuration will still be in our backlog; so the cluster will Push the new configuration onto the unit and the cluster will be in-sync once again.
- If the latest UUID is empty (the unit is in Join mode and is about to be overwritten).
We will consider a unit "out of sync" if:
- The latest UUID is not in our backlog, this can happen if:
- You preform a configuration on two appliances too fast, so the first unit haven't had time to synchronize the cluster in between.
- The communication between two or more appliances are down, while the configuration is changed on these appliances.
We will consider all unit to be synchronized when:
- All appliances have the same UUID (Clustering -> Overview).
How do I know when the cluster is synchronized?
Cluster -> Overview show each appliances latest UUID. when all are the same, the clustering icons will all show green.
How do I resolve a conflict?
The easiest way is to put the broken unit in "Join"-mode as if it were to enter the cluster for the first time and accept the shared configuration.
How do I test a configuration on just one of the appliances?
You must disable Pushing of configuration, this will prevent this unit from synchronize it's changes on to the cluster. This is done by disabling "Push Configuration" (Administration -> Clustering).
How do I keep the changes I made?
1 Activate "Push Configuration" and the configuration should be synchronized from this unit.
How do I discard the changes I made?
1 Push "Re-join" and force a sychronization from one of the other clustered appliances (Push Now) or wait 5 minutes and watch the UUID be synchronized (Clustering -> Overview).
2 Activate "Push Configuration" (it's important that you don't activate it until the cluster is repaired).
Can I cluster different models of SPG/VSP?
Yes, as long as...
- ...the configuration works on the smallest model (eg. if outbound mode is used, all units must have the license).
- ...licensed users must not be the same, but its highly recommended (or else the license could be exceeded on one cluster node).
Licenses are not synchronized or shared in the cluster, just the configuration!
