Technical Debt

Owned by Pierre Dane
Jan 20, 2020
4 min read

Android Multi-Team Login

  • Allow users from other teams to login to the Android device without having to delete the database

    • Create a database per team

  • Ensure that data sync is complete even for users that are not logged in

    • The app should be able to access the database of other teams in order to sync

    • The database can be deleted if all data is synced to the server

  • Change database encryption so that the encryption key is generated on the device and stored on the server

    • Retain encryption at the team level

BIOPHICS clean-up

  • Validate that all records added or updated in the Android app are visible in the Biophics database

    • Currently, household members and household updates do not reflect

    • Ensure that events, locations, clients, plans and tasks are all streamed to the server

  • Ensure that all fields from each form are present in Biophics in correctly labelled columns

    • Check column names

    • Check completeness

TBD: Need confirmation from Ona that streaming of entities to the warehouse (Biophics or other) is the agreed path forward

Configuration file for Android

  • Add a JSON config file to the settings of the country in order to allow configuration of certain fields. Initially, these settings will be restricted to Circle of Selection attributes

    • On/Off for Circle of Selection restriction

    • Admin password for the circle of selection

    • The radius for circle of selection

  • Other configuration settings to be added in future

    • peer-2-peer on/off

    • in-app indicators on/off

    • in-app offline map download on/off

    • base layer switching on/off

    • API key (DG vs Mapbox vs etc)

This epic is a prerequisite for form configuration


Improve device-server sync

  • Optimise sync DOWN from server to device

    • Package data so that device processing is reduced

    • Improve initial sync (authenticate API call, OpenMRS location dependancy)

    • Optimise timeout settings per user/program

      • instrument sync time

  • Optimise sync UP from device to the server

    • Ensure that ALL data is synced

  • Display sync status on Android app

    • Show sync UP complete/incomplete with green/red flags OR give more detailed sync statistics per entity

    • Show sync down complete status per entity

Improve server to data warehouse sync

  • Optimise data sync from the server to the data warehouse

    • Decide on a strategy - to discuss:

      • event streams using Nifi or Kafka or both (reduce Nifi workload) vs materialized views

      • creation of entity tables in the warehouse using JSON query language

      • packaging of entities e.g. bundle related entities with each event (may add to payload size)

      • pre-processing of data into flattened files before sync

      • use of FHIR

      • use of JSON schemas (can be used for the materialization of event data in the warehouse)

      • Abandon custom syncing for Postgres and use a replicated node (possibly some issue with long-running queries on a read-only replica being cancelled by incoming transactions - reported by Dickson)

      • optimisation of OpenSRP API for ETL/streaming

  • Simply the configuration of the process to remove the need for code release

  • Provide monitoring to show completeness and timeliness of warehouse data (sync time, number of records per entity vs raw OpenSRP)

  • Not covered:

    • Move periodic database actions (initialization, queries, refreshes etc) out of NiFi

    • Add health monitoring component to Postgres deployments

    • Use database migrations instead of using NiFi for database initialization and all client-specific setup

Optimise Reveal stack

  • Reduce the hardware requirements for the Reveal Stack

    • Remove Nifi from OpenSRP server (replace with Java code - large performance impact)

    • Remove Nifi from the data warehouse (replace pipeline with another product e.g. Kafka)

    • Remove OpenMRS user dependency (high LOE)

    • Remove OpenMRS location dependency (medium LOE)

  • Full removal of Nifi and OpenMRS may not be possible but moving functionality out of each will reduce the server load and the requirements for RAM and CPU)

  • Initial server specs:

Remove OpenMRS dependency - locations

  • Remove dependency of OpenSRP on OpenMRS for locations

    • Document interaction between OpenSRP and OpenMRS locations

    • Replace OpenMRS hierarchy generation (could use JSON query language and inner joins)

Remove OpenMRS dependency - users

  • Allow users and roles to be managed within Reveal without the need for OpenMRS

    • Transition user accounts from OpenMRS to OpenSRP

    • Build UI in OpenSRP WebUI for user management

    • Add Role-Based Access Controls to OpenSRP so that we can limit access of the data to only authorized roles (TBD if this is a separate requirement as these do not exist at present?)

Technical Debt - Small Tasks

  • Bucket for small tasks not included in other epics

    • Use flavours for APK builds (rather than variables)

      • Determine use cases for this vs configuration file - is it still necessary with config file? Perhaps so depending on time constraints

      • At present variable are used to set the language and included forms

    • Sticky sessions

      • Remove these to improve load balancing and HA

    • Design reskin

      • Possibly apply Tobys design to the WebUI (Low priority)

    • Align Team Assignment REST POST and GET API keys (e.g. location in POST vs locationId in GET)

Technical Debt - Superset Improvements

  • Improvements to Superset

    • Standardise export functionality via Superset UI to ensure that large amounts of data can be downloaded for analysis (currently not feasible to download all Zambia data)

    • Standardise the Superset slice functionality - either virtual tables or underlying tables

Unit and Integration testing

  • Improve unit and integration testing across the platforms

    • WebUI has 90% coverage - maintain this

    • Android has very little coverage - increase coverage as new functionality is developed or as bugs are fixed

Upgrade Java Spring

Upgrade Java Spring dependency to keep utilize new Spring functionality and increase development speed

  • Version 5.2 and above

  • This work is funded elsewhere and will have a time impact on Reveal, not budgetary impact

WebUI Functionality

  • Fix WebUI functionality that is incomplete or not functioning as expected

    • Search boxes are not working (decision to fix search for FI plans in Plan and Monitor pages and to hide other search functionality)

    • Disable Reactive plan creation for Thailand (TBC)

    • Only show plan types that are relevant to the country (e.g. Hide IRS from Thailand)

    • Ensure that plan start and end dates are either used or hidden)