Urgent: Error Message after Agent Upgrade to 15.1.1 nearlie all 8000 TC77 devices are affected

Hi all,

we have updatet all our TC77 to agent 15.1.1 now all 8000 TC77 got an error message that mobicontrol has been stopped... "close or wait" if i choice wait the error message pops up after a couple of minutes. for our employees it's not possible to work with this error message. i have already opend an tickt. but maybe someone of you have an idea and an fast solution . btw. rollback is not possible.

device is zebra TC77 with android 8.1

thx in advance

mark

3 years ago
Android
ANSWERS
MD
Mark Doepelheuer
3 years ago

S
Scott
3 years ago

There was a similar bug in an agent a couple of years ago related to a wifi ssid/profile. Disabling wifi on the device would keep the agent from crashing.   Probably not related but disable wifi and see if that makes any difference

MD
Mark Doepelheuer
3 years ago

Thx Scott, i can remember but switch off wlan did not solve the problem

MD
Matt Dermody Diamond Contributor
3 years ago

We just started getting reports of this today as well, affecting devices on 15.1.1. I really do NOT like how the agents auto-update in the background from Google Play even though we don't approve it in Managed Play. This is such a loose / uncontrolled variable that it's scary that SOTI could break something without having any level of recourse. I'll be opening a case as well. I think we're likely going to need an emergency new build that we can upgrade the devices to since a downgrade isnt an available option. 

MD
Mark Doepelheuer
3 years ago

Thanks for the feedback Matt, it's not nice to hear that we are not the only ones affected but it reassures me a bit because it shows me that it's not up to us and as you mentioned Soti needs to take immediate action here. i have already forwarded the first logs.

MD
Matt Dermody Diamond Contributor
3 years ago

Have you been able to glean anything from the logs? I have been trying to piece together what is leading to the agent crash to see if there is any Profile we can maybe unassign or change to reduce the frequency of the issue while we wait on a more official fix. The release notes for the app don't reveal much other than there being some specific Zebra related updates that were recently added. I am happy to see the return of kill_application scripting for Zebra AE devices but not at the expense of the stability of all the devices!

MD
Mark Doepelheuer
3 years ago

No, unfortunately I could not see anything from the logs. With us it is now also already 11:55 pm and I have no more nerves, now soti must react. We have installed the new release of the settings manager in parallel, but I have also tested whether it is because of that, which is unfortunately not so.

MD
Matt Dermody Diamond Contributor
3 years ago

Have you seen this issue occurring repeatedly on the same devices? What I've noticed is that the issue seems to go away on a device after a reboot. I am almost wondering if this is just an error that came up as a direct result of the new agent installation from Google Play and that maybe it isn't as significant of an issue if a reboot can clear it out. 

M
Michael
3 years ago

Totaly agree on the risk of an uncontrolled update of the mobicontrol agent, from Google play.

We also use Zebra devices and mitigates this by using Zebras app manager to disallow ("DisallowApplicationUpgrade") upgrades of the Mobicontrol agent. This prevents Google play to update the agent.

https://techdocs.zebra.com/mx/appmgr/
Requires a MX version of at least 10.3.

Works pretty well on our TC26 devices that uses Managed google play. They are still on agent version 15.0.1

MD
Matt Dermody Diamond Contributor
3 years ago

Great call. I am going to build a Package for that ASAP to at least stop the bleeding in our environments. Unfortunately it appears that it requires 10.3 AND the devices to be on A10 according to this hover over info link.

This info button conflicts with the OSX 8.13+, indicating that it should work on A8.1 as well, along with MX 10.3 that it shows right to the side. 

MD
Matt Dermody Diamond Contributor
3 years ago

Following up on this. I tested applying this MX setting on an A8.1 device with MX 10.3 and the MX failed to apply so that does seem to verify the info pop up showing minimum of A10 and MX 10.5 versus the other text on that table of supportability. 

RC
Raymond Chan Diamond Contributor
3 years ago

I do not have any TC77 device to do any tests or debugging.  However, I would like to point out the importance of stringent and thorough tests on device-agent/-firmware compatibility and stability BEFORE  performing agent/firmware upgrade to large number of production devices (8000 TC77 in your case) or upgrade of the Mobicontrol server.   In the worst case of an improper/wrong agent/firmware upgrade, device may get out of control or totally unusable (e.g. agent crashing, or some warning/error pop-up messages/notifications on the screen blocking various kiosk items, etc.), making it necessary to recall (if a remote fix cannot be found) and  wipe/re-enroll problematic device.  This can be extremely painful and costly. 

Thus,  I always stress the above in my training program to all my corporate/governmental customers in Hong Kong.  I hope you will do similar tests BEFORE all future upgrades to EACH production device model in your system.

MD
Mark Doepelheuer
3 years ago

Thx for your feedback Raymond, we have tested the agent with an couple of devices before the installation. but it's not the first time that soti releases an bugy agent or mc release, i think there must be more in house tests before they should release any software and they should implement an better quality management. we have an other problem with this agent and our samsung devices, since the new agent is released we are not able to enroll devices over mobile data, so and we have no impact to this agent because it's provided by the playstore. Something can always happen, but normally you should be able to rely on the provided software to work otherwise it should be called "beta".

M
MartinsKl
3 years ago

At the moment I uninstalled zebra plugin on our ps20 devices and that seems to be a quick workaround until I can update devices.

MD
Matt Dermody Diamond Contributor
3 years ago

Good to know! Have you lost any significant functionality other than Remote Control with the plugin uninstalled?

M
MartinsKl
3 years ago

For now not really, but most configuration on these devices is with MX. Certificate is issuing now at least and AE commands are working.

Z
ZPMOD@SOTI
3 years ago

Hello Mark,

Thank you for requesting a response from SOTI Support Staff. 

Soti Support is aware of this issue and actively tracking impacted customers. We have a few internal cases with developers to find a solution.

The current workaround seems to be as noted in the discussion here to uninstall the zebra plugin.
Though caution is needed for implementing this into production as it may impact some management capabilities. 

If anyone else is experiencing this issue I would suggest reaching out to Support@soti.net to raise a case so we may further track and bring awareness of the issue to our internal team. 

I'd like to further note about the issue with our agent updating with the profile set to 'never update' as discussed in this thread,

Per our MobiControl release version 15.4.2 this issue should be resolved (MCMR-22799) - https://docs.soti.net/soti-mobicontrol/release-notes/v154/1542/

Regards,  

Technical Support | SOTI Inc. |1.905.624.9828 | support@soti.net | www.soti.net

MD
Matt Dermody Diamond Contributor
3 years ago

Thank you for confirming what I already suspected to be the case, that the "Never Update" feature of MobiControl didn't previously work. From the timeline it appears the bad version of the agent went public in the Play Store on December 15th, and the fix that provided the ability to disable that auto-update wasn't released until December 22nd. By the time the MobiControl patch was released, 7 days later, it would have already been too late for most end customers, especially given that it would have required a MC upgrade. And here we are almost a full month later and the agent still has yet to be fixed. This is a bad look for SOTI and Android Enterprise management in general. 

Bad agent gets released to the Play Store on December 15th:

Ability to prevent the auto-update to the bad agent was not released until December 22nd, and requires an MC upgrade:

MD
Mark Doepelheuer
3 years ago

Hi,

to uninstall the plugin is not option, because a lot of settings which we have configured are just working with installed plugin. in my eyes it's to critical to uninstall a part like the plugin on 8000 devices which are already in use.

mark

MD
Matt Dermody Diamond Contributor
3 years ago

Agreed Mark. At this point I do not understand why SOTI has not recompiled a prior working version of the APK under a higher build number and provided it directly to those experiencing the issue. We could use such a custom compile as a downgrade via upgrade in order to at least temporarily rollback to a known good version. I can understand why they wouldnt want to upload a version like that into the Play Store, but we could at least downgrade our devices directly via File Sync as needed.

MD
Matt Dermody Diamond Contributor
3 years ago
Z
ZPMOD@SOTI
3 years ago
Z
ZPMOD@SOTI
3 years ago

Hello 

Looking to follow up on this thread.

Has the new agent been a success in testing to resolve the issue?

Regards,  

Technical Support | SOTI Inc. |1.905.624.9828 | support@soti.net | www.soti.net |