Here is a step by step to set it up. I made it short and concise based on CFEngine Enterprise 3.5.3.
For a full version, please visit this online documentation from 10Gen.
1. Prepare 3 machines
eg. (hub1, IP = 10.0.0.4), (hub2, IP = 10.0.0.5), (hub3, IP = 10.0.0.6)
2. Install CFEngine hub package on all 3 machines and all of them should have same key pairs and license file for our better life in key management
3. Set up mongodb replica set (do (a)(b) on all 3 machines. (c)(d)(e)(f)(g)(h) only on hub1 which is our primary in this case.)
a. In /var/cfengine/masterfiles/update/mongod.conf, uncomment the "# replSet = <replicaset-name>" and give appropriate name eg. "cfenginers0". Also we need to remove "bind_ip" for networking connections
# bind_ip 127.0.0.1
replSet = cfenginers0
b. Start mongod as follows:
/var/cfengine/bin/mongod --dbpath /var/cfengine/state --config /var/cfengine/masterfiles/update/mongod.conf
c. Connect to one of the mongd instances, lets say 10.0.0.4
d. Initialize replication with default config - this also will add 10.0.0.4 to the replica set
e. Add the other two members ***
f. Allows the current connection to allow read operations to run on secondary members
g. Display the replica set configuration object
h. Confirm successful replica set configuration after a while (less than a min)
4. Bootstrap each hub to itself
/var/cfenigne/bin/cf-agent -B 10.0.0.4
/var/cfengine/bin/cf-agent -B 10.0.0.5
/var/cfengine/bin/cf-agent -B 10.0.0.6
5. Bootstrap hub for clients is hub1 so /var/cfengine/ppkeys should be duplicated to hub2 and hub3 from time to time. Otherwise, clients cannot talk to both of them when CFEngien on hub1 failed. (assuming trustkey => "false"; in copy_from body).
*** Note that before running rs.add() command, mongod processes should be running on secondary nodes