Low Disk Watermark error on MobiControl Search

As part of MobiControl we have implemented an advanced search service, which uses Elasticsearch as the search engine.  This gives our users the ability to specify complex searches by using simple filter expressions.  Elasticsearch also acts as our backend search service which populates the device list on the new MobiControl web console.

What we have found recently is that there is a point where the search service stops functioning, and the search database goes into read only

As part of MobiControl we have implemented an advanced search service, which uses Elasticsearch as the search engine.  This gives our users the ability to specify complex searches by using simple filter expressions.  Elasticsearch also acts as our backend search service which populates the device list on the new MobiControl web console.

What we have found recently is that there is a point where the search service stops functioning, and the search database goes into read only mode.  This occurs due to a “low disk watermark” that has been set in the elasticsearch.yml file.  The intention of the watermark is to protect the system from running out of space and causing corruption in the database. 

The below is an extract from the search logs which can be found in the following folder: Program Files/MobiControl/Search/logs.

[2019-04-01T10:28:33,981][INFO ][o.e.c.r.a.DiskThresholdMonitor] [WIN-T7I8IK4T8T9] low disk watermark [1gb] exceeded on 
[jA6sAP7DR0yb2khflZKdyw][WIN-T7I8IK4T8T9][C:\Program Files\SOTI\MobiControl\Search\data\nodes\0] free: 991.1mb[0.6%], replicas will 
not be assigned to this node

When this protection is enabled the database is locked and there are administrative actions that will release the lock. 

The following procedure can be followed to release the lock:

1. Ensure that there is enough available space on the server where the search service is hosted. The space required in the elasticsearch.yml file.

         In the above extract the minimum available space required is 512mb, read only mode will be activated if the available disk space          goes below this.

2. Stop all services in the MobiControl Admin Utility.
3. Delete the contents in the following folder: Program Files/MobiControl/search/data.
4. Start all services in the MobiControl Admin Utility. Be sure to start the MobiControl Search service after restarting the MS & DS.
5. Open MobiControl and navigate to Global Settings, edit MobiControl Search Sync Interval. Within this window select “Sync Now”.

      6.   The sync will update the “Last Sync” date and timestamp. It will also show what the search integrity is.

It is important to note; devices are very volatile.  It is quite normal to see search integrity below 100%, this is due to the change in device states and the time it takes to update all the relevant tables with the changes.  It is acceptable to have a search integrity of 95% or higher.  If it goes below this, then a manual sync may be required. 

Search integrity is calculated as (TotalDevices – OutOfSyncDevices)/TotalDevices, where OutOfSyncDevices is the number of devices that were out of sync BEFORE the synchronization started.

For more information on the Elasticsearch Low Disk Watermark, refer to this link - https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html