Just a quick update for anyone who wants to know the resolution. Discovered that when they upgraded from 13.3 to 13.4 they only did the first server (server1) and not the other two servers. Once we updated the other two servers everything started updating like it should. Haven't tested the enrollment part as we still need to get the SAN certificate but I suspect once that's done we'll be set to go. Thanks for all the help Chan. 

Are your devices Android, Android-Enterprise, iOS or Windows devices?

Did Soti implement the HA for you?

Do you have any hardware/software load balancer between your deployment servers and the outside?

Currently a mix of Android-Enterprise and Windows CE. We are moving away from the CE devices and replacing them with Android-Enterprise

I can't answer the second question as I've only been here a month. I'm assuming not. 

Currently there are no load balancers that I'm aware of in front of the systems. If there was it might make things easier with the certificate and I had considered seeing if we could put the system behind the corporate balancer. 

Can you try enrolling a generic Android or Android+ device (i.e. NOT Android Enterprise), and see if you get exactly the same symptoms in all your tests as your Android Enterprise device?

What is your MobiControl server version/build number?   And version of the Microsoft Server hosting your Mobicontrol server modules?

Unfortunately I don't have an android or android+ device available to do so. Keep in mind we are facing the issue with both the android devices and the CE devices. Thanks for your help. \

Edit missed the version: 13.4.0.4266

Server 2012 R2, not sure of the SQL back end but think it's either 2008 or 2012. Need to check with the DB team. 

Further edit: I have checked the firewall rules on all three servers and all three of them are set exactly the same way. The firewall allows any/any on the domain but doesn't specifically have 5494 open. I tried opening it on one of the two problem servers and no change. We have a private cell connection through our providers. 

Generic Android and Android plus platforms have very different security requirements from the latest Android Enterprise.   With no spare device, I will factory reset one of the Android Enterprise device and install generic Android or Android+ agent to do the tests if I were you.   As it's not possible to look in details your network and server configurations, the test results with generic Android/Android+ can help to rule out wrong guesses of the cause of your problem.

Do you have managed Google account or managed Google Play account for your Android Enterprise platform integrated in MobiControl?

No, we only use Mobicontrol to deploy to the devices. We don't have a google account for them. I think I'm wrong about the android-enterprise. These are hardened Panisonics but they run as android+. 

Edit: Android 5.1.1 just for the version number. 

Edit Redoux: Ok the devices are android-enterprise/work capable but we aren't using those capabilities. We've been using the android+ agent on the devices so far. 

So, they are Android 5.1.1 device running Android+ agents.  Then not having a load-balancer should not be a major concern.

Can all the three servers communicate freely among themselves? Or do you have Windows firewall on each MS server instance with very strict exceptions for ports such as 5494, 5495, ... etc? 

Have you tried adding a new add-devices rule and see if you can successfully enrol your devices?

For each of your MobiControl servers, is your Deployment Server Extension (DSE) certificate  a self-signed certificate or a paid certificate from a  CA vendor?  Is its common name an FQDN or an IP address?  

All three servers are completely open to each other. There is an any/any rule for all three that allows all traffic. 

A new add devices rule was on my list of things to do for another thing I wanted to try but I'll give it a whirl and see if that helps.

Originally it had the self signed mobi certificate which isn't supported for scan to config. I got a third party vendor certificate for server1 and am working on getting certificates for the other two servers. I don't remember, does Mobicontrol support a wildcard certificate with SANs (if I can get one)?

As for the connections currently the devices are set to connect like this:

server1.company.com

ip of server1

server2.company.com

ip of server2

server3.company.com

ip of server3

Which I've had issues in other places with. I plan on changing them to be URL only. 

The fact that all three servers are completely open to each other rules out possibilities of firewall issue.

Mobicontrol do support wildcard certificate with SAN.

From what you said so far,  I believe your certificate(s) and/or add-devices rule was not properly generated/installed in the right sequence after your colleague(s) had made changes to the server architecture and connection parameters.  I would just add a new devices rule and try enrollment again first if I were you.

Please be warned that some wrong operation or operation order can render your enrolled device permanently uncontrollable without re-enrollment, and might not be recoverable even if you restore a backup of the database or of your server/VM instances.  As it might be a security concern to reveal your architecture and configurations in an open forum, I suggest you contact Soti support team directly to open an official support ticket to debug the above-mentioned issues.

The Deployment/Management servers in an HA implementation should always be running the same version AND build number.   You are actually very lucky that the database does not get corrupted while there have been multiple servers of different versions operating on it for an extended period of time.