Syncing Data
- James Maina
The data sync functionality is handled by Cloudant Sync. Cloudant Sync is a replication-protocol-compatible datastore for devices that don't want or need to run a full CouchDB instance. Cloudant replication functionality is used to synchronise data between the local datastore and a remote database, either a CouchDB instance or a Cloudant database. Many datastores can replicate with the same remote database, meaning that cross-device synchronisation is achieved by setting up replications from each device to the remote database.
Setup and Configuration
Dependencies
OpenSRP uses cloudant-sync-datastore-*.jar files setup in the core module lib folder as dependencies and doesn't load the cloudant dependencies through gradle as is norm. This is because the Cloudant sync library code has been modified to include a timestamp field (updated_at) to store the date and time a document is saved in the cloudant database.
After a user submits a form, the form submission is converted to events or clients and saved in the client cloudant database ready for syncing to the server couchdb. To be able to sync data from the client to the server couchdb, the following configurations are required:
| Property | Location | Notes |
|---|---|---|
| datastore_manager_directory | CloudantDataHandler.data | This is the directory where cloudant will create the client db file. Set to 'data' by default |
| datastore_name | CloudantDataHandler.opensrp_clients_events | The name of the cloudant database in the client app. Set to 'opensrp_clients_events' |
dbURL | port- AllConstants.CloudantSync.COUCHDB_PORT | This is a combination of the server IP address (by default using the IP specified in the settings screen when logging in), server couchdb port number (set to 5984 by default) and the server couch database name |
| databaseName | AllConstants.CloudantSync.COUCH_DATABASE_NAME | The server side couch database name to be synced with the client app |
COUCH_DATABASE_USER | AllConstants.CloudantSync | Server couchdb username |
COUCH_DATABASE_PASS | AllConstants.CloudantSync | Server Couchdb password |
| syncFilters | assets/sync_filters.json | {
"_id": "_design/cloudantFilter",
"filters": {
"locationId": "function(doc, req){ if ( doc.locationId != req.query.locationId ){ return false; } return true; }",
"providerId": "function(doc, req){ if(doc.type != \"Event\"){ return true; } if ( doc.providerId == req.query.providerId ){ return true; } return false; }",
"team": "function(doc, req){ if(doc.type != \"Event\"){ return true; } if(req.query.team){ if(req.query.team.split(',').indexOf(doc.providerId) >= 0){ return true; } } return false; }"
}
}
Sync filters are special couchdb design document functions that allow data querying from the database returning only specific documents that pass filter rules. Filter rules are composed of fields in the documents being selected whereby the values for those fields are compared against the remotely passed filter values from cloudant and available in the 'req' object. The above sample shows what would be required in the server couchdb for cloudant to sync data by location, provider or team. The sync filters are defined in the client app and automatically loaded through CloudantSyncHandler class replicationFilterSettings() method and added to the server couchdb in getReplicationFiler() method. Every time a static instance of CloudantSyncHandler class is initialized, replicationFilterSettings() method is called to compare the contents of the local sync filters JSON file with the server version to see if there are changes and if so update the server with the new sync filters configs. Once the filters are set, Cloudant sync is then setup to do a filtered pull from the server database by instantiating the PullFilter class and passing the server side filter document and the filter params as key/value pairs (PullFilter("filterDoc/filterFunctionName", parameters)). NB As shown in the example above team filter expects a comma separated list of team members to compare against the providerId value in the EC documents. More on cloudant filtered replication here |
Every time a user clicks on the save button in the forms, saveFormSubmission() method is called in the activity hosting the form to start the form save process which involves breaking down the formsubmission to EC models and saving to the client app cloudant database as indicated in the flow diagram above. After the records are saved in the cloudant database, an Intent Service is started in FormUtils.generateClientAndEventModelsForFormSubmission() method by calling startReplicationIntentService(). This starts ReplicationIntentService which handles the replication process by calling CloudantSyncHandler.startPullReplication() method to pull documents from the server and CloudantSyncHandler.startPushReplication() to save data to the server couch database. Once the replication process is complete, the EC data needs to be processed further into the case model for displaying in the registers.
To sync EC data on clicking the refresh button in the home screen org.ei.opensrp.service.FormSubmissionSyncService.sync() method is called.
This site is no longer maintained. Please visit docs.opensrp.io for current documentation.