Host missing from cf-key -s, Mission Portal or API

Symptom

A host that you know is bootstrapped to a hub does not show in the Mission Portal, cf-key -s or the API queries. The connections seem to work, as running cf-agent -Kf update.cf on the host and cf-hub -q rebase -H <HOST-IP> on the hub completes without errors.

Steps to diagnose duplicate keys

SQL Query Based Detection

Many times a hostname change in the database is a good indication that a key has been duplicated.The following query will show you host keys that have changed their hostname along with all the hostnames known to have that key.

SELECT VariablesLog.VariableValue AS "Host name", VariablesLog.HostKey AS "CFEngine ID", VariablesLog.ChangeTimeStamp AS "Change time" FROM VariablesLog WHERE VariablesLog.VariableName = 'fqhost' AND VariablesLog.ChangeOperation = 'CHANGE'

Manual Detection

A common case is that there is one or more other hosts that use the same key as that host.

To diagnose this:

1. Run cf-hub -q delta -H <HOST-IP> -v (insert appropriate host IP) and find the line "Server is TRUSTED, received key <HOST-KEY> MATCHES stored one." (note what is in place of <HOST-KEY>).
   
   
2. Log in to the Mission Portal -> Inventory ->Filter CFEngine ID and input <HOST-KEY> (as found above).
   
   
3. Check which IP address is reported as using that host key (same as CFEngine ID).
   
   
4. If there is a different IP/hostname reported, that host has a duplicate key and you can go to the next section to resolve it. If not, wait a few minutes and check back (this is required as we manually collected reports above).
   

Automatic Detection

You can use the detect_duplicate_keys.sh script to help identify duplicate keys. It works by comparing sampling the last seen database and detecting duplicate keys based on IP. The README.md provides additional information on usage.

Steps to resolve duplicate keys

Manual Remediation

You should now know about two hosts that share keys. We will need to regenerate keys for hosts so that they don't share keys.

1. Log in to one of the hosts that is sharing a key.
   
2. /etc/init.d cfengine3 stop
   
3. mv /var/cfengine/ppkeys/localhost.* /root
   
4. cf-key
   
5. /var/cfengine/bin/cf-agent -B <HUB-IP>

The host now has a different key, and should show up in reporting within 15 minutes.

If the problem persists, you might have more hosts that share key, so you can run the steps to diagnose above again.

Automatic Remediation

Once you have collected a list of keys that are known to be duplicates you can use the remediate_duplicate_key.cf policy to automatically remediate duplicate keys. Each host executing the policy will look in a text file that contains a list of keys known to be duplicated. If the host detects it has a duplicated key the host will delete its key, regenerate a new one, and re-bootstrap to the hub. Note that trust must be open on the hub for this to work. The README.md provides additional information on usage.