Have you changed your update schedule to something more frequent than the default of 2 hours?

No, the update schedule is set for two hours.

Are the devices roaming frequently and if so, are they roaming seamlessly? Do they have to completely re-authenticate on every roam?

Also what agent version is currently installed on the devices?

They roam in between many access points, but it is seamless and the signal strength remains strong.

I've noticed this issue happen even when a scanner is stationary. There is something happening on the scanners that is causing the agent to crash and reload (at least that's what I believe is happening).

The signature in the log where the device switches to User mode in the middle of the disconnect and reconnect seems especially unusual. Are you using The SOTI Lockdown in conjunction with another lockdown like EHS?

I am not using SOTI Lockdown (it is not enabled at all in any profile). I am, however, using EHS as a lockdown app.

To add to my previous reply I believe that is a normal process when the agent first loads (it switches to User mode), which is why I believe the agent on the scanner is actually crashing and reloading by itself.

Here is another scanner that I actually have disabled in SOTI, but it still populates the log with the User mode entry.

And another... 

Have you enabled debug device logging on a device and examined the log?  That might shed some light on what is happening device side.

I have sent a link to this thread to the support staff that I am working with on this problem.  I'm glad I am not the only one with this issue, but sad to know you are struggling just as bad as we are!

I've been dealing with this problem since last year April/May time and opened another new ticket that is going on 4 months old which have the developers stumped.  

I opened a case with Zebra and Soti for this exact problem.  Zebra brushed it off and said it's not the device or firmware.  Soti has been more responsive and have been investigating this with tons of logs and time on the phone for almost 4 months.  They felt there was a memory leak and released Zebra legacy agent 14.2.1 but that is still not resolving my problem.  I have tried all Soti agents from 12.0 to 14.2.1 and they all act the same. 

For us, at one point, TC8000 LG12 was a good firmware on wifi, but it and previous versions had a lot of bugs that I submitted to Zebra and Soti, some of which were resolved.

• Enterprise keyboard to crash when scanning if the device goes from lockdown to admin mode to lockdown (resolved in LG13)
• A Device would randomly reboot when scanning (resolved in LG14)
• Random characters submitted when scanning (random - but not reported for a while)
• Network disconnect issues (constant since LG10, better with LG12, but LG12 had the above 3 bugs)
• WAP connections stay connected too long and do not roam accordingly (supposedly resolved in multiple LG change logs)

Now to make things even more fun, I am getting reports where TC8000 devices are rebooting when network connection is lost for 30 seconds to 2 minutes!  That has never happened before!

In a way I'm also relieved I'm not the only one experiencing this issue. I still haven't found a fix. Like you I have updated to every subsequent Agent release and LifeGuard patch and still the problem persists. I'm convinced that there is an issue with the Agent in combination with the TC8000. There is just something about the scanner that causes the Agent to consistently crash. We also have Zebra VC80x devices and they are rock solid with SOTI. The Agent never crashes on those.

I also tried to send logs to SOTI, but they couldn't find the root cause. I finally had to give up on support because it was taking too much of my time.

I am still using my workaround to force close the Agent after I initially configure the scanners, which is not ideal at all since I can't remotely manage them afterwards, but at least it prevents the scanners from locking up and no one complains.

I hope SOTI is able to find something and provide a solution. I wish I could have given you better news. I'm sure you have felt the same pain I have on this problem.

What are you doing to stop the agent, just sending a kill command remotely? 

My only work around is having everyone reboot at the start of a shift and after they have a problem, which significantly lowers the incident rate.  Due to that (in my eyes) it seems to point directly to a memory leak somewhere.

I actually ran into this exact situation today.

TC8000 AOSP, LG13+HF7

MobiControl v15.1.1.1184

Agent v14.2.1.1083 

Ryan if you haven't solved this in over a year that doesn't give me a high level of confidence that we're going to get a quick resolution any time soon. If we can't fix it ourselves, perhaps we can at least placate it with some adjustments like Nathan came up with in terms of disabling the agent.

I'm about to experiment with changing the connection setting from persistent to scheduled, is that what you did Nathan, or did you set it to Manual, or something else entirely?

I am force closing the agent on the scanners themselves. However, in order to do so there are a couple things you have to do in the WebConsole first or the scanner won't let you force stop it.

In the MobiControl WebConsole:

   1. In your TC8000 profile settings you must go into Feature Control > Security > then turn on Allow Removal of the MobiControl        Android Agent

   2. Next you have to disable the device(s) in the WebConsole by highlighting the device(s) > then click the three dots > then click        Disable

On the TC8000 scanner:

   1. Go into Settings > Security > Device administrators > uncheck MobiControl

   2. Go into Settings > Apps > MobiControl > Force Stop. Then do the same with MobiControl Enterprise Service

Initially, I was also having each user reboot at the start of their shift. However, it was always inevitable that their scanner would completely lock up at some point during the day due to this issue and the only way to get it working again was to do a hard reset, sometimes multiple times a day. That was an unacceptable solution since our distribution needs rely on a stable connection with no interruptions. Once I figured out how to kill the agent on the scanners then there were no more problems with them locking up.

I made some headway on this today but am still struggling with RCA. 

I found that whatever application the end users were using (Zebra Enterprise Browser in this case) would crash at it seemingly vied for foreground control with the lockdown/kiosk.

I couldn't have the LoB app crashing in the middle of use so I opted to remove the lockdown from the picture so that end users could at least work without interruptions. This was fairly successful in that the App crashes stopped, but this also left the devices wide open without a lockdown, not ideal. I may end up putting EHS on the devices as an alternative if I can't figure this out over the next couple of days. 

After the lockdown was gone I noticed that the disconnects and the entering user mode issues were still showing in the server logs. I've been pouring through the various device level logs ever since and have found a strong correlation to the SOTI service crashing on the device at the point that the disconnect shows in the server.

In this example we see two disconnects (1:55:38 and 2:00:52)

I can trace these to the System logs captured by RxLogger, where I can see the corresponding SOTI service crash at 1:55:37.944 and 2:00:52. 

I was able to consistently map this in the logs confirming that every instance of the Disconnect + User Mode + Connect signature in the server logs coincides with this service crash at the device level. 

I'm now compiling multiple instances of the the device logs capturing this issue to see if I can discern any common log signature pattern happening immediately before each one the service crashes. I have not found anything concrete yet but I do see regular instances of this Connection Error

Here is a more complete snippet of what the logs look like during the event:

Interesting stuff. I'm glad someone is looking into this that's way smarter than me.

Just fyi, I can tell you that EHS won't make a difference for you because that's what we use and the agent still crashes. At first we used SOTI's built-in lockdown feature, but it had the same problem. In fact, it was actually worse because each time the agent would reload it would kill whatever application the user was using. It's why we switched to EHS.

EDIT: Looks like I read your post wrong. In this case using EHS should prevent the app crashes that you're speaking about. Sounds like it's similar to what was happening when we were using SOTI's lockdown app.

I do have a case open on this as well but based on how the previous cases have gone I'm hoping we can maybe flush this out as a community since this does appear to be widespread. I am now going back through all of our other TC8000 SOTI deployments to see if I can find any common pattern. I have found some environments that do not show this issue at all so now I'm trying to figure out what the differences might be.  

Thanks for the feedback on the EHS, I know it won't solve the underlying disconnect issue but it will definitely put the devices in a better position than having the mission critical app crashing regularly. 

Hi,

we are using TC8300 and havent encountered anything like this.

But i am curious about RAM-Usage on your Devices where the MobiControl-Services keeps crashing.

TC8000 have 1 GB Ram, as far as i know , we had issues with the same amount of Ram, where Android Dalvik or Something else i dont remember the name correctly, was closing MobiControl to freeup ram for system use.

Just an idea. :) have a nice week.

We have the TC51, TC52, TC70x, TC72, MC3300, TC8300, WT6000, VC80x, VC8300, ET51, L10, and MC9300 all deployed across numerous SOTI environments and I am only seeing this on the TC8000. Based on your call out am actually curious if it affects the TC70/TC75 as that was the same device family as the TC8000. 

I have been trying to establish a pattern between different TC8000+SOTI environments and have been tracking the following variables to try and determine if there are any commonalities. I haven't found anything concrete yet. 

Just wanted to check in. Have you had any luck finding a pattern?

Currently running with EHS while SOTI Devs investigate the issue. The constant disconnects are still present but the symptoms for the end users are at least abated by using a different lockdown utility. The initial indication is that the TC8000 doesn't have enough RAM but I am still hopeful they can come up with a fix. 

I have a mix of 512 MB and 1 GB models and they both do the same thing. I was thinking RAM also and my worry is that SOTI won't be able to patch it and I'll be forced to replace our scanners. That would not be ideal.

Hi Nathan,

if its a ram issue,

we had the same problem for 1GB Models, we now upgraded devices.

There is no fix other then to have "tiny" and native android apps so they dont use up the much needed ram.

Using more or less profile options wasnt doing anything for me except when i left antivirus unticked.

The main problem with lockdown is that, when Android free´s up ram for your "production app" it closes mobicontrol because it isnt in the foreground. Mobicontrol restarts automatically again and when this happens the lockdown closes down all applications currently running because of security reasons i guess.

Sounds about right.

I should make a correction to my previous post. We have 1 GB and 2 GB models and both do the same thing. 512 MB was an error and I don't think those exist.

We have a few hundred of these TC8000 devices at one of our facilities running Velocity(telnet emulation) and the default installed browser running a web app. Because of the small screen size, we installed the Soti Surf browser during one of web app updates, and put it into kiosk mode. Immediately upon full production ramp up the next day, we started running into issues like you were seeing where the device agent disconnects and then reconnects repeatedly. Upon reconnect, the lockdown screen re-enables, kicking them out of the browser. In our case, that meant the web app logs them out and they have to start over. 

After troubleshooting for a while, I managed to get them back to stable by

• Uninstalling Soti Surf and pointing lockdown at the default browser.
• Changing the Lockdown type to Package blocking and disabling Samsung Package Disabling. 
• Rebooting the devices.

In my case, it only seemed to work when I completed all these steps, not just one. Not sure if this will help with your situation, but figured the info couldn't hurt. 

OS Version 5.1.1 (Security Patch Level: 2019-04-05)

MX Version:7.2.10.2

Agent Version: 14.2.1.1083

Plugin: 1.16.10.121

But did this solution stop the MobiControl agent from constantly crashing and reconnecting? I think we've established that there is some sort of memory leak whether you use SOTI lockdown or not. In my case I'm not using SOTI lockdown anymore, but instead using Zebra EHS so when the agent does crash it won't interfere with their work.

I am in the same boat now Nathan. Using EHS instead of the lockdown so that the issue is at least masked to the end users but the constant agent crashes and reconnects are still occurring under the surface. SOTI and Zebra are looking into this again but it is very much looking like some kind of memory issue as the MobiControl agent isn't the only thing that is being force closed. We're also seeing the devices spontaneously unenroll in the same environment and think that it might be related, so be on the look out for that as well!

Yes, I've noticed that as well.

The agent did stop crashing after I made those changes and lockdown is working as intended. Also, we never had issues until after I installed the Soti browser. Before they were just using Velocity and the default browser without any issues with lockdown. 

Interesting. Alright, I'll do some testing and see if it helps.

Hi all, a short addition.

At our company we also had the disconnect/reconnect issue and here it also led to glitches in the WMS app running on the devices because of the Soti lock screen. EHS covered this.

Eventually configuring 802.11R for fast roaming on the device itself, finally seemed to have solved the reconnects.
Did anybody maybe also test this? 

We have been experimenting with a lot of different Wireless settings per Zebra & SOTI's recommendation and are still coming up short. This continues to be a major issue across multiple environments now and we still don't have a proper resolution. We need an engineering fix ASAP.

It's unfortunate. I don't really see any hope for this model. We've invested so much money that we can't upgrade at least for another few years.

Just corious, are you using native apps or any webview applications like apache cordova etc. ?

Have you MobiControl Antivirus applied to your devices?

We started having the exact same disconnect/connect problem in SOTI on all our devices in all our Stores at once - except on devices running a single-app kiosk mode.
After a lot of debugging in SOTI, we found that the root cause was simply a new firmware on all Wi-Fi Access points.

We use Meraki model MR33, and they were updated to software version MR 26.6.1
The disconnect/connect problem in SOTI disappeared, as soon as the Meraki Access points were upgraded to version 26.8.1

We're having similar issues with our TC75X/TC77 devices... The agent will restart multiple times during the day most commonly during more memory-intensive functions in our Cordova app (Scan Bot image scanning, or signing) 
We don't want to move away from lockdown due to its speed control safe mode feature. 

After working with Soti support we are testing Lockdown > LaunchWithRecents
The agent still crashes, but the user no longer needs to fully log back into the app and navigate/reenter information until they are where they left off. LaunchWithRecents seems to cache & reload where they left off, helping save some time and at least mitigating the issue. 

We have a few other strange symptoms, such as the datawedge profile will seemingly stop outputting data. The scanner fires/lights up but nothing enters into the field. 

Been keeping an eye out on this thread and am hoping some progress with the agent happens, we've been experiencing these issues since we updated the server last December.

FYI. Getting this tested ASAP. 

From what I can tell the frequency of the issue seems to be greatly reduced but I am still seeing evidence of it in the logs. I have removed the SOTI lockdown and only have the EHS now so it is tough to tell if it is still a full agent crash that is happening or not yet. I will need to comb through the logs more to find out. 

I can't even download the latest agent to the SOTI server. It says network error.

I managed to get the latest agent update working and now there is a new LG version 17. So far the results are promising. I have left a scanner running for over 24 hours and the logs show that it has not disconnected even once. Will keep testing.

Nope. Even with all the latest updates it still constantly disconnects and reconnects. So very disappointing.

Yup. I had hope based on your last update but we're sadly seeing the same. 

Paging SOTI Support!

Hello Matt,

Thank you for requesting an answer from SOTI Support, we will review this thread and will get back to you at earliest.

Regards,

Hi Matt,

Thank you for requesting a response from SOTI Support.

I'm seeing two common behaviors here, potentially a RAM issue (lack of it / memory leak) and an issue with connectivity. Internally the connectivity issue does point to an environmental issue. Meaning using separate networks such as 4G / unrelated WiFi network results in a change of behavior.

One thing to note is there are a lot of variables and no concrete common pattern right now. Therefore I suggest if you haven't already, then raise a Support Case via support@soti.net / support.eu@soti.net. Then link this post and work with our Technical Support Specialists in order to fully understand the problem and drill it down.

Physical reproduction of the issue is important, along with network logs and full verbose logs on the MobiControl side. Ideally RXLogger / ADB Logcat / Wireshark, on top of that the MobiControl Database is ideal too.

We'll message everyone affected for their case numbers where applicable and then will see what can be done to get this moving efficiently. If you're missed out and you're affected by this or very similar, then please reach out to me through the private message system.

Regards,

This is a widespread problem liable to impact any MobiControl + Zebra TC8000 environment. 

We have at least 3 cases open on this issue and opened a 4th one yesterday I believe, as the reports continue to come in. That combined with one additional environment that we're still confirming and then Uline and Scansource listed in this thread brings the total to at least 7 different end customer environments that are affected by this.

I tend to agree that there is a memory issue of some kind although I think it may be a storage issue and not necessarily RAM. The TC8000 is limited in both categories relative to most other Android devices under management so that could make sense. 

The connectivity issues do not appear to be network related. They are the byproduct of instability of the SOTI agent that eventually crashes and relaunches on the device. This is perceived as a connectivity issue when observing this from the management console since frequent connections are being made. However, after pouring through the logcat output and other rxlogger logs on numerous affected devices it is clear that the agent service is crashing and restarting, often incessantly, leading to disconnects and reconnects at the MobiControl server level.

It's not clear if this is an underlying SOTI problem or a Zebra problem but people buy MobiControl because they have Zebra devices and there is a very strong partnership between the two companies so I would hope you could get together to get this figured out. 

I see that SOTI has been responding to a lot of old threads on this channel which is bumping this thread down. Bumping back to the top for visibility since this is still an ongoing and widespread issue. 

Have you had any feedback on this?

looks similar to an issue i have with TC70 & TC56 as below so keen to see what the outcome is

The TC70 has the same core architecture of the TC8000 so I am not surprised to see the issue popping up on that device model as well. That would be the first time that I've seen it reported on Atlas device like the TC56 however. 

Please open a case with SOTI so that we can increase the visibility. 

I'd also prepare to switch from the SOTI Kiosk/Lockdown to EHS if you aren't already using it because the devices will pretty soon become unbearable to use as the SOTI Lockdown gets reloaded every time the agent is restarted, leading to interruptions for the end user. 

Hi All,

Sharing an update regarding devices that have low RAM and are affected by V14 Agent's increased RAM usage. There are a couple of issues raised for Zebra devices internally, which mainly cover TC8000/TC75 & TC700H. The Product Management team is aware of these issues and are working on how best to address these concerns.

In the meantime you can try the following as a potential workaround (The ultimate purpose is to reduce a device's RAM usage):

1. Use the following script:

writeprivateprofstring DeviceFeature DisableDozeMode 0
apply featurecontrol
sleep 15
reset /s

- Setting DisableDozeMode to 0 will re-enable battery optimization on the MobiControl Agent itself.

2. Apply the packages mentioned below in a blacklist profile.

- Note you need to modify the list [1] as per your needs. Remove any package that you're using from this list and add the others to a blacklist profile (ARC / Application Run Control).

[1] Packages:

com.symbol.osx.quickbrowser
com.symbol.nlt.reader
com.android.dreams.phototable
com.android.videoeditor
com.google.android.street
net.soti.mobicontrol.motorola.installer
com.android.musicvis
com.google.android.music
com.android.exchange
com.rhomobile.appgallery
com.android.noisefield
com.android.email
com.android.magicsmoke
com.android.musicfx
com.google.android.apps.books
com.android.phasebeam
com.google.android.videos
com.google.android.talk
com.android.inputmethod.pinyin
com.android.galaxy4
com.symbol.dataanalytics
com.android.dreams.basic
com.google.android.apps.plus
com.google.android.play.games
com.android.wallpaper
com.symbol.mxmf.csp.analyticsmgr
com.google.android.youtube
com.google.android.apps.magazines
com.google.android.googlequicksearchbox
com.android.voicedialer
com.android.soundrecorder
com.symbol.simulscan.demo
com.symbol.simulscan.res
com.symbol.pttexpress
com.symbol.msrn
com.google.android.gm
com.google.android.apps.docs
com.android.calculator2
com.android.calendar
com.android.providers.calendar
com.google.android.syncadapters.calendar

Make sure to test everything first on an affected device.

Regards,

Thank you for those suggestions. I'll give them a try.

I'll give it a shot as well. Thanks!

Has Zebra given up on this? Doesn't seem like a solution will ever come to fruition. I've already started the process of looking into upgrading our devices to the TC8300 model. This has been a constant headache for two years.

Zebra and SOTI have been back and forth on it for a few months now. I think they have gone through several iterations of the agent trying to get it resolved but crashes are still occurring. Agreed, we are starting to hit the refresh cycle on the TC8000 sooner than this issue is getting resolved. 

Being plagued terribly now trying to keep these things working.  Sounds like a great time for a Zebra BuyBack program to upgrade to TC8300 :D 

This one is official cooked. No fix coming. The best you can do to mitigate the impacts of the issue to the end user is to switch from the SOTI Lockdown/Kiosk profile to Zebra Enterprise Home Screen for lockdown. The agent will still continue to crash in the background but the lockdown won't constantly reload and interrupt the end user from their business app.

One thing we have seen, that I did not see mentioned is This problem does not occur on our TC8000s running Android 4.4.3 but only on the ones running Android 5.1.1. We are going to downgrade a device and test to see if the problem remains. Not an answer or even a very good work around but maybe a clue as to where the core problem may be.

It's worth a shot. Thanks for the suggestion.

Business leadership has asked me to push back on Zebra regarding this problem because we are trying to retire devices still under warranty that can't run more than 2 or 3 days without requiring a factory reset - so a new case has been opened with Zebra - let's see where this goes....

Good luck! 

SOTI has a pretty extensive report that they've published that details why this won't be possible unless the RAM can be upgraded on the TC8000 or the OS can be updated to a newer dessert flavor with better memory management. I don't think either are going to be a reality.

We have otherwise resorted to manually configuring TC8000s via StageNow or running older versions of MobiControl with older, less memory intensive agent versions. 

Zebra gave me a copy of the same list of apps to blacklist as above as their recommended solution. 

Super fun!  I couldn't figure out why some of the devices applied to the policy had the apps removed and some didn't, then as I was watching my device I saw all the apps re-enable.

Funny thing - yeah they remove right away after the profile is installed, but if you refresh details of the device in Soti, or the device checks in, all the blacklisted apps are re-enabled!

I really just want to get rid of these devices.

What you may be seeing is the effects of the lockdown being applied and unapplied. The blacklist is actively applied on the device while the lockdown is enabled. If you are disabling the lockdown manually in order to inspect the device to see if the blacklist is working then you are also actually disabling the blacklist in the process. It's somewhat of a schrodingers cat situation, you can't observe the state of the blacklist while the lockdown is actively applied because the lockdown is blocking your visibility to the underlying apps and if you disable the lockdown to try to observe what apps are disabled then the blacklist also becomes unapplied in the process, thereby re-enabling those apps again. If you really wanted to blacklist the apps so that they're always disabled an not only actively disabled while the lockdown is active then instead use Zebra's AppMgr MX CSP to disable the apps instead of the SOTI blacklist. 

Just created the .xml file and tested - apps disabled in front of my eyes - time to watch the memory usage. I have a few left over devices that unenrolled themselves listed in Soti and it seems they are all under 200mb available ram at that point.

Hi, did anyone solved this issue? Was blacklisting successful (Soti or Mx way) ? Can I use Lockdown afterwards or it's better to stick with EHS?