Please provide more details, e.g.

-  What device model, firmware version and agent version?

-  Any recent change(s) in MobiControl server implementation (patches) or in hosting MS-Windows server?

-  What options (DC entity and definition, schedule, etc) were used in data collection rule?

-  How and what was observed to confirm data collection of custom data is not successful?

etc.

Zebra TC77 - Android O

Agent: 1360_1662

No changes that I know of on the server, was working at least a couple of weeks ago.

No specific Data Collection Rule for Custom Data. Data is not showing in the Device Details tab.

I can see that the New data definition (and the old ones removed) when the agent checks in (looking in the detailed Device log). 

This appears to be something with the console not processing / updating from the device.

Also check your Search Sync Integrity, you may have to manually sync.

Matt, 

Where is this located? The system Admin has me locked out of some things and I have never seen that screen.

Found it. Is set at Day. Integrity is 100% Attempted Manual refresh.

100% Integrity should be good. Keep in mind that version 14 introduced the quick filter device inventory view that also introduced a separate stats db of sorts that is synced off of the primary db so that it can be quickly queried against without affecting the performance of that primary db. It seems that this synchronization tends to drift at times so you may observe different properties reported for a particular device in the inventory view than are actually technically reported from the device at the same time. The manual sync can clear up those discrepancies but it can otherwise be pretty frustrating. 

If there is no data collection rule for custom data in your case,  I believe you are actually referring to some default device information field(s) that does not seem to get updated as frequent as you expected.

In such case, I suggest you to modify the title of your discussion to replace the term "Custom Data" with "Device information", as the term "custom data" has special meaning in MobiControl.

You don't have to define a data collection rule to make use of custom data.  Any defined custom data elements are displayed on the device information screen in a separate panel.  The information displayed SHOULD reflect the values following the most recent device checkin.  I use a couple of different custom data values in concert with file sync rules and packages to control large executions.  eg, File sync rule stages large files and then runs script to change the custom data value.  A package that has the custom data value as a filter then applies and executes the update and as part of that resets the custom data value back to default.

I have though seen instances where the custom data appears to stop updating even though I know the value has changed on the device.  Not sure if the device stops reporting it or the server stops reporting it correctly.  I haven't had a chance to investigate with SOTI though.

You can change a custom data value with a script?

Hi Scott,

Who told you custom data is updated after each check-in?  If you prefer not to use data-collection rule to indirectly affect scheduling and updating of data to the server, be my guest.   Maybe you spend more time to do more testing to figure out what is a sensible way to use custom data.

Poor choice of words on my part.  The "default" custom data value is a file sync rule that is updated only on "file does not exist".

The staging file sync includes the same custom data file with a different value.  The "after sync" script just copies the custom data file over the top of the default one, effectively changing the value.  Next check-in the device will report the changed custom data value which causes the "execute" package to apply.  The execute package also deletes the changed data file and then next sync the default value will come back down and the package will no longer apply.

I do it that way so I can stage large pushes in the background and then I use the "Battery = 100" in profile assignment as a rough proxy for "device is charging" and can be rebooted without affecting the user.  Ugly but basically working for me for now.

What you said does not seem to be related in any way to what Jim asked in the first place in this thread.

As I mentioned in earlier post, Jim's title of the discussion seems to be irrelevant because I don't think he is using "Custom Data".

Raymond, 

Scott's reply is exactly what I am asking. I am using CUSTOM DATA and I am using for the exact reasons that he is stating. He is also correct about it not needing Data Collection Rules and that it shows on its own panel in the Device screen.

Jim

Hello Raymond,

Who told you custom data is updated after each check-in?

No one told me.  Is it your contention that a device check-in does NOT update the custom data value?  If I am incorrect I would appreciate it if you could point me (and the forum) to the correct information so we can all be better informed.  From your response I take it that you believe my previous post to be not "sensible".  If you believe that to be the case, along with your response, can you please elaborate as to where I am misinformed and what your suggested approach would be?

In my testing, here is what I have observed:

A custom data value will be managed essentially the same way as other intrinsic data items like "Battery Status".  I don't have to use a data collection rule to view the last value.  ie, The Battery Status will show the value at the time of the latest checkin.  A custom data value works the same way.  Am I incorrect?  If so, please explain how/where.

A data collection rule does not affect when the data is transmitted to the server.  eg, A rule collecting data every five minutes does not transmit data to the server every five minutes.  It is collected locally on the device and transmitted in bulk when the device checks in.  Am I incorrect?  If so, please explain how/where.

Thanks

Hi Scott,

I don't know which server and agent version you are using, or if there are some recent improvements made that I am not aware of.   

The various tests I did over the past few years showed that custom data are not guaranteed to be updated to the server after each update.   Custom data are captured  by the device agent at scheduled interval (best-effort, but not guaranteed) to the buffer defined in data-collection rule.   However, the data-collection buffer may not be  flushed after each schedule update.   That was the reason I asked different Soti support teams many times in the past about way to flush collected data stored in the data-collection rule buffer, but could still not get a solution so far.

If your "special" use of custom data actually involve synchronization using file-sync rule, then there is a higher chance that the "custom data" field in a sync'ed  file gets immediately read back via the device agent back to the server. But my previous tests showed that flushing still is not guaranteed for every file-sync.  

If Soti has already made the improvements and/or feature I requested related to custom data collection back to server in recent server/agent version,(s)  I would be more than happy to know about and use.

Hello Raymond,

I'm still on 14.1.5 and so don't have any idea if it behaves differently on other versions.

I believe though that the file sync has nothing to do with the custom data collection.  I'm just using that as the means to effect a change in the underlying custom data file.  Nothing changes on the server after a file sync (related to the custom data value).  I have to sync to change the underlying file and then at some later point do a check-in to cause the device to read the value and communicate it back to the server.  Since I only care about the value at the time of the check-in I don't need a data collection rule so there is no data collection rule buffer involved.  It just reports the current data value at the time of check-in (again, in my experience).

You might be correct that it is not guaranteed to happen at every check-in but my experience leads me to believe it happens immediately enough of the time for me to call it "good enough".

Have you tried overwriting the file on the device side (e.g. with simple file copy operation with a file manager app), and then performed a check-in and confirmed that the new custom data is reflected in your web-console?  I did tests like that in the past and confirmed that the new data in the overwritten file were not reported on the server side every time.

Raymond, 

I am the original poster of this problem. Yes I have tried updating the file that is read. The does not seem to be processing the custom data values. I have removed old ones from the the folders and they are not updating on the devices information pages either. I am in the process of getting a call setup with the SE. I believe that there must be a process on the database that is not getting triggered. 

I originally opened this thread to see if others have experienced this issue, not for everyone to get into a flame war. 

Everyone, thanks for the constructive replies. Like everything else, SOTI is not very transparent about how all these processes work. 

Jim

Hi Jim,

If you are really using custom data (defined in INI/XML file), you have to use data collection rule to schedule sampling of the INI/XML file by the device agent, and stored in data collection buffer for eventual flushing to the server.  This has been the way it's designed to be used many years ago.

Though you may be able to see the custom data value in the device information tab,  the timestamp associated with the value is not shown.  The value shown is just the last sample since the last buffer flush, which can be sampled tens of minutes earlier.   So, to be sure about the relevance of the data shown,  you have to go to the Collected Data tab to see both the value and timestamp of each custom data sample collected.

Hey Jim,

I as well am not interested in any personal disputes but rather a collegial approach to sharing, learning, and solving issues.  To that end, I have conducted some testing on my devices.  I have one device with a custom data definition (WebView Update) and no defined data collection rules.  I enabled debug logging on the device.

12:18:05 first checkin (server value already at 0)

Device Log:
2019-07-23 12:18:05.190|pool-1-thread-1|D|AP|[CustomDataStorage][readAll] - customDataConfig: CustomDataConfig{name='WebView Update', scheme='XML', path='/sdcard/DCIM/WebViewUpdateReady.xml', parameters={XP=/state/ready}}|
2019-07-23 12:18:05.206|pool-1-thread-1|D|AP|[CustomData][Snapshot]WebView Update0|
...
2019-07-23 12:18:05.213|pool-1-thread-1|D|AP|Snapshot:| 
...
2019-07-23 12:18:05.214|pool-1-thread-1|D|AP| CustomData=OSUpdate="0";WebView Update="0";|

Copy file with data value =1 to device then checkin again:

Device Log:
2019-07-23 12:18:52.973|pool-1-thread-1|D|AP|[CustomDataStorage][readAll] - customDataConfig: CustomDataConfig{name='WebView Update', scheme='XML', path='/sdcard/DCIM/WebViewUpdateReady.xml', parameters={XP=/state/ready}}|
2019-07-23 12:18:52.987|pool-1-thread-1|D|AP|[CustomData][Snapshot]WebView Update1|
...
2019-07-23 12:18:52.994|pool-1-thread-1|D|AP|Snapshot:| 
...
2019-07-23 12:18:52.995|pool-1-thread-1|D|AP| CustomData=OSUpdate="0";WebView Update="1";|

(Server now shows value=1)

overwrite file on device (back to 0)
request checkin at 12:20:31

Device Log:
2019-07-23 12:20:30.888|pool-1-thread-1|D|AP|[CustomDataStorage][readAll] - customDataConfig: CustomDataConfig{name='WebView Update', scheme='XML', path='/sdcard/DCIM/WebViewUpdateReady.xml', parameters={XP=/state/ready}}|
2019-07-23 12:20:30.903|pool-1-thread-1|D|AP|[CustomData][Snapshot]WebView Update0|
...
2019-07-23 12:20:30.911|pool-1-thread-1|D|AP|Snapshot:| 
...
2019-07-23 12:20:30.912|pool-1-thread-1|D|AP| CustomData=OSUpdate="0";WebView Update="0";| 

(server now shows value=0)

Then I created a data collection rule for same value with a schedule of every 30 minutes
12:34:01 (Server, Checkin, Data collection configured)

Device Log:
2019-07-23 12:34:01.384|pool-1-thread-1|D|AP|[CustomDataStorage][readAll] - customDataConfig: CustomDataConfig{name='WebView Update', scheme='XML', path='/sdcard/DCIM/WebViewUpdateReady.xml', parameters={XP=/state/ready}}| 
2019-07-23 12:34:01.401|pool-1-thread-1|D|AP|[CustomData][Snapshot]WebView Update0| 
...
2019-07-23 12:34:01.413|pool-1-thread-1|D|AP|Snapshot:| 
...
2019-07-23 12:34:01.415|pool-1-thread-1|D|AP| CustomData=OSUpdate="0";WebView Update="0";| 
...
2019-07-23 12:34:02.051|pool-1-thread-1|D|AP|[LegacyScriptExecutor][execute]script command: writeprivateprofstring [DataCollection, I11, PT=5;EX=`XML:///sdcard/DCIM/WebViewUpdateReady.xml?XP=/state/ready`], result: ScriptResult{description='OK', resultType=OK}|

12:35:46 copy file with value=1
12:35:55 device check in

Device Log:
2019-07-23 12:35:55.607|pool-1-thread-1|D|AP|[CustomDataStorage][readAll] - customDataConfig: CustomDataConfig{name='WebView Update', scheme='XML', path='/sdcard/DCIM/WebViewUpdateReady.xml', parameters={XP=/state/ready}}| 
2019-07-23 12:35:55.627|pool-1-thread-1|D|AP|[CustomData][Snapshot]WebView Update1| 
...
2019-07-23 12:35:55.637|pool-1-thread-1|D|AP|Snapshot:| 
...
2019-07-23 12:35:55.639|pool-1-thread-1|D|AP| CustomData=OSUpdate="0";WebView Update="1";| 

(Server value did NOT change to 1)

Waited one minute and tried again:
12:37:03 device check in

Device Log:
2019-07-23 12:37:04.036|pool-1-thread-1|D|AP|[CustomDataStorage][readAll] - customDataConfig: CustomDataConfig{name='WebView Update', scheme='XML', path='/sdcard/DCIM/WebViewUpdateReady.xml', parameters={XP=/state/ready}}| 
2019-07-23 12:37:04.051|pool-1-thread-1|D|AP|[CustomData][Snapshot]WebView Update1| 
...
2019-07-23 12:37:04.058|pool-1-thread-1|D|AP|Snapshot:| 
...
2019-07-23 12:37:04.060|pool-1-thread-1|D|AP| CustomData=OSUpdate="0";WebView Update="1";| 

(Server value changed to 1)

So, it appears to me that the checking and gathering of custom data values can be done independently of the data collection rule process as the logs indicate that the customdata snapshot sampling always finds the correct underlying value on a checkin.  Multiple reasons are possible to explain why the last attempt did not correctly report on the server:

• Creating a data collection rule might possibly interfere with the snapshot reporting. (doubtful based on the lack of evidence on the device log)
• A bug in the agent causes it to not report the value it showed in the device log.  (doubtful that the device log would show the correct value but report something different to the server)
• A bug in the server caused it to not report the most recent data value. (more likely but would need to examine server logs and possibly test with more data values than just 0/1 in order to track changes over time.

However, I think your original issue, assuming your situation mirrors mine is due to some transient server issue.  Have you bounced the server?  Enable debug logging on the server and an affected device and pull the logs after a checkin and see what is reported.

Good luck

OK, since this was bugging me I went ahead and dug a little more.  Switched the server logs to debug and found what appears to be a reasonable answer.  Looks like there are two ways to get the custom data updated:

• Value state snapshot during checking
• Data collection rule

When a device checks in you will see in the deployment server log:

2019-07-23 15:34:47,045 (0x00007458) [DEBUG] MessageProtocolHandler::Process() <290> CommDebug:Receive. DataLink: Comm.Client.55286 [8867948f33f105010017110522502717], Message: DeviceInfoMsg: {"Snapshot":{"SotiKeyString": (...lots of deleted device data... ,"CustomData": "OSUpdate=\"0\";WebView Update=\"0\";","SupportedApis": "1100,812","FreeSpace": 22957924352, ...etc}

When a device uploads data from a collection rule you will see:

2019-07-23 15:34:47,326 (0x00003fb0) [DEBUG] MessageProtocolHandler::Process() <242> CommDebug:Receive. DataLink: Comm.Client.55286 [8867948f33f105010017110522502717], Message: BinaryDataMsg: {"DataType":9,"BinaryData":{"Length":32},"TimeStamp":"2019-07-23T20:34:47.3266686Z","Type":33,"SequentialId":0,"Priority":1,"IsResponse":false,"IsNotify":false,"IsRequest":true,"IsCompressed":false,"IsEncrypted":true,"IsGzipCompressed":false,"ErrorCode":0,"SessionId":0} 

There will be one of these for each set of collected data.  It appears the Snapshot data is always obtained before the collection rule data.  My tests indicate that they are probably applied in that order also so whatever the collection rule supplies (if anything) will be what is displayed on the server.  The server logs show that the Snapshot data is ALWAYS accurate as to the current state at the time of check-in and if there is no data collection rule then the server value will update reliably.  I can break it however, by adding a data collection rule.  I add a rule that collects every 5 minutes, let it report a couple of times so I know exactly when the rule is running on the device.  Then right after I know the rule has checked locally, I update the file and then do a check-in.  The server data will not update (correctly).  The server logs show that the snapshot data is correct but since the collection rule runs independently of check-in, it overwrites the custom data value with it's stale data.  Checking in again immediately will allow the snapshot to update and the server will then show the correct value.  The short answer is, if you don't need historical data and you only want to use the custom data value as a switch at check-in, DON'T create a data collection rule.  Otherwise you can run into collisions.  That is on my server (14.1.5).  That might not be true of earlier or later versions but is how I'm operating now.

Spent an hour on the phone with SOTI and here is what I have learned at least for my problem. No solution yet... but.

1. Custom Data IS updated when the device checks in (per SOTI)

2. In my case Agent is collecting the data and sending it to the server. Data is in Devcustomdata table.

3. Virtual groups WILL find the new Custom Data

4. Data is still not displaying on in the Custom Data Panel of the Device Details page LIKE IT IS SUPPOSED TO

I have a follow up call with SOTI in the morning and will provide more details as I learn them.

Jim

Hi Scott,

Thank you for your update.  I've to check when Soti has started adding the feature to read custom data after check-in.  As long as I have the time,  tracking that wouldn't be too difficult as I have the installation file for each and every MobiControl release.

The feature you said definitely wasn't there in my previous tests/struggles with different Soti support teams, nor were there a single Soti guy who explicitly told me that we could have the collected data snapshot after doing a device check-in (possibly because it just wasn't there at those times).   Otherwise, I wouldn't have to ask them to give me a solution (or file a feature-request) to flush the data-collection buffer, as my primary target at that time was also to get at least the most recent value for use in alert rule, etc.

Perhaps they secretly made some changes afterwards.  Unfortunately, as you said, the change seem to cause inconsistency or other problems when legacy data collection rule is used.  When there is spare time,  I likely need  to do another round of tests on the recent v13.4 and v14.3+ servers, which are the production versions my company have been recommending to our corporate customers. Your v14.1.x is therefore not the version I will be testing.

I have been really busy this summer..... but thought I should post an update on this. 

SOTI did confirm that in the server version that I am running (14.1.2) that having a period (.) in the custom name field name does cause a problem. They also confirmed that the code that was causing the problem was rewritten in a later version (14.2 or 14.3, I don't remember). 

I have not yet upgraded our server so I have not been able to confirm.

FYI, also recently found another issue with Custom Data on my server version. The server will recognize paths and file names that are not in the UNIX format, but the agent on the device DOES require that the Custom Data be defined with the correct UNIX paths and file names (slashes and case). This was very confusing since the Custom Data values showed on the server but would not be recognized on the device (Kiosk). 

I don't think you previously mentioned your custom data name include '.' character.   I often avoid all special character except underscore.   Also, the filename/pathname restrictions may differ on different device models/firmware-versions.

However, it's not that hard.  Back to a few years ago, it took me less than 20 minutes to figure out the right formats that worked.