Syncing Data

Owned by James Maina
Last updated: Oct 25, 2016
3 min read

The data sync functionality is handled by Cloudant Sync. Cloudant Sync is a  replication-protocol-compatible datastore for devices that don't want or need to run a full CouchDB instance.  Cloudant replication functionality is used to synchronise data between the local datastore and a remote database, either a CouchDB instance or a Cloudant database. Many datastores can replicate with the same remote database, meaning that cross-device synchronisation is achieved by setting up replications from each device to the remote database.





Setup and Configuration

Dependencies

OpenSRP uses cloudant-sync-datastore-*.jar files setup in the core module lib folder as dependencies and doesn't load the cloudant dependencies through gradle as is norm. This is because the Cloudant sync library code has been modified to include a timestamp field (updated_at) to store the date and time a document is saved in the cloudant database.

After a user submits a form, the form submission is converted to events or clients and saved in the client cloudant database ready for syncing to the server couchdb. To be able to sync data from the client to the server couchdb, the following configurations are required:

PropertyLocationNotes
datastore_manager_directoryCloudantDataHandler.dataThis is the directory where cloudant will create the client db file. Set to 'data' by default
datastore_nameCloudantDataHandler.opensrp_clients_eventsThe name of the cloudant database in the client app. Set to 'opensrp_clients_events'
dbURL
port- AllConstants.CloudantSync.COUCHDB_PORTThis is a combination of the server IP address (by default using the IP specified in the settings screen when logging in), server couchdb port number (set to 5984 by default) and the server couch database name
databaseName
AllConstants.CloudantSync.COUCH_DATABASE_NAME
The server side couch database name to be synced with the client app
COUCH_DATABASE_USER
AllConstants.CloudantSync
Server couchdb  username
COUCH_DATABASE_PASS
AllConstants.CloudantSync
Server Couchdb password
syncFiltersassets/sync_filters.json
{
  "_id": "_design/cloudantFilter",
  "filters": {
    "locationId": "function(doc, req){ if ( doc.locationId != req.query.locationId ){ return false; } return true; }",
    "providerId": "function(doc, req){ if(doc.type != \"Event\"){ return true; } if ( doc.providerId == req.query.providerId ){ return true; } return false; }",
    "team": "function(doc, req){ if(doc.type != \"Event\"){ return true; } if(req.query.team){ if(req.query.team.split(',').indexOf(doc.providerId) >= 0){ return true; } } return false; }"
  }
}

Sync filters are special couchdb design document functions that allow data querying from the database returning only specific documents that pass filter rules. Filter rules are composed of fields in the documents being selected whereby the values for those fields are compared against the remotely passed filter values from cloudant and available in the 'req' object. The above sample shows what would be required in the server couchdb for cloudant to sync data by location, provider or team. The sync filters are defined in the client app and automatically loaded through CloudantSyncHandler class replicationFilterSettings() method and added to the server couchdb in getReplicationFiler() method. Every time a static instance of CloudantSyncHandler class is initialized, replicationFilterSettings() method is called to compare the contents of the local sync filters JSON file with the server version to see if there are changes and if so update the server with the new sync filters configs. Once the filters are set, Cloudant sync is then setup to do a filtered pull from the server database by instantiating the PullFilter class and passing the server side filter document and the filter params as key/value pairs (PullFilter("filterDoc/filterFunctionName", parameters)). NB As shown in the example above team filter expects a comma separated list of team members to compare against the providerId value in the EC documents. More on cloudant filtered replication here

Every time a user clicks on the save button in the forms, saveFormSubmission() method is called in the activity hosting the form to start the form save process which involves breaking down the formsubmission to EC models and saving to the client app cloudant database as indicated in the flow diagram above. After the records are saved in the cloudant database, an Intent Service is started in FormUtils.generateClientAndEventModelsForFormSubmission() method by calling startReplicationIntentService(). This starts ReplicationIntentService which handles the replication process by calling CloudantSyncHandler.startPullReplication() method to pull documents from the server and CloudantSyncHandler.startPushReplication() to save data to the server couch database. Once the replication process is complete, the EC data needs to be processed further into the case model for displaying in the registers.

To sync EC data on clicking the refresh button in the home screen org.ei.opensrp.service.FormSubmissionSyncService.sync() method is called.