R6 and Technical Debt Reveal Platform Narrative

Owned by Craig Appl
Last updated: Mar 11, 2020
12 min read

Last Updated 6 Mar 2020

Release 6 (R6) represents a significant shift in Reveal from a fixed software system that supports pre-defined Malaria workflows toward a platform that can support dynamic workflow definition across multiple contexts. We’re shifting the terminology from “system” to “platform” with release 6 because we are laying the foundation to support a more broad range of services. This document presents a narrative of the major changes that will go into Reveal in 2020 to set the tone and establish a common understanding by which we will build the underlying features.

Big Changes in Reveal

R6 represents the third large iteration of the Reveal platform over the past 18 months. The first version of Reveal was delivered in December 2018 and focused on supporting structure based IRS workflows with a map-first user experience. The second major change focused on expanding that experience by implementing plans, tasks, families, and family members. It was delivered in June 2019. This third change focuses on making the relationship between plans, tasks, families, family members, and forms dynamic so that we can build workflows based on the patterns implemented to date. R6 will be accompanied by a major shift in how users, roles and access controls are handled.

Current State

Reveal is built to deliver a specific set of indoor residual spraying, mass drug administration and focus investigation workflows. It has evolved over time to support planning, interventions at multiple levels within the health system, and longitudinal health records for malaria interventions. The system allows implementers to perform full life cycle operations on entities such as locations, households, household members, and points of interest. These items operate within a fixed technical pattern that requires code changes to update. For example, forms are all hard coded for each deployment, so any changes require a release and deployment.

Below is a summary of features that are available in the system:

  • Configuration

    • Manual import of locations using a Python script

    • Manual user and practitioner creation from a CSV

    • Form cascading drop-down values

    • Android Client admin password is able to be turned on or off and set globally or at the team level

    • The distance required to activate the admin password is configurable in the server

    • User accounts can be created through the OpenMRS user interface

    • English and Thai support in the Android client and Web UI

  • Planning

    • IRS map based planning

    • IRS and FI Routine and Recurring plan generation from the Web UI

    • Team Management (Add practitioners, teams and link them to plans and locations)

    • Automatic creation of Focus Investigation plans from Thailand when cases are reported in the BIOPHICS system

    • Update existing plans by changing their state

  • Tasking

    • Automatic bulk task generation in the server when a plan’s status becomes active

    • Offline task generation when a location, household or household member is added

    • Offline task generation when Mass Drug Administration Round 1 is completed

    • Reset tasks to their original state

    • Sync task changes across teams

    • Automatic Index Case Confirmation generation, completion and display

  • Data Collection Forms Available (across all implementations)

    • Structure Based Forms

      • Add Location

      • Register Household

      • Update Household

      • Indoor Residual Spray

      • Potential Area of Transmission (PAOT)

      • Mosquito Collection

      • Larval Dipping

      • IRS Verification

      • CB Spray Area

    • Household and Household Member Based Forms

      • Add Household Member

      • Update Household Member

      • Bednet Distribution

      • Blood Screening

      • Case Confirmation

      • Mass Drug Administration Dispense (Step 1)

      • Mass Drug Administration Adherence (Step 2)

    • Operational Area Based Forms

      • Behaviour Change Communication

    • Other Forms

      • Daily Summary

      • Field Officer

      • SA Decision

      • Mobilization

      • Team Leader

  • Android Client Features

    • Peer-to-Peer sync

    • In App Reporting

      • Operational area charts as the user makes changes

    • Map tile downloader so users can specify the areas they wish to download before they go out to the field and better manage the memory on the device

  • Reporting

    • IRS tabular drill-down reporting with country-specific customisations

    • Focus Investigation plan views with links to other plans in that operational area

    • IRS CSV Exports

    • Data Warehouse access and reporting through Apache Superset

    • Custom reporting solution for the Thailand BIOPHICS system

Reveal Future State

Release 6 and the accompanying technical debt work are comprised of a number of core features that are broken out into individual tasks that need to be developed. This section lists those features and builds a narrative to describe how the pieces fit together.

Feature List

  • Form Configuration by system administrators

  • Dynamic Task generation

  • Entity persistence (households, household members, locations) and updating tasks offline across plans

  • Web UI CSV Import for risk files

  • Web UI importing coverage targets

  • Web UI see historical index cases

  • Remove the OpenMRS dependency

    • Add user accounts, roles and rights to OpenSRP

    • Remove OpenMRS Location dependency

  • Streamlining Data Warehouse sync process to reduce resource consumption

Narrative

Our target is to develop Reveal into a platform that is dynamic, configurable and able to be deployed with minimal coding. We will build off of the features listed above to achieve this. The future allows system administrators to configure the list of activities available in plans, link them to forms that are defined in the server and generate tasks when plans are set to active for any intervention type at the operational area, structure, household, and household member levels. households, household members and locations will persist across plans so that you only have to register a household once. Any changes to these will update tasks in other active plans on the user’s device. The web UI will be improved to support this dynamic task generation in a straightforward way including determining if a particular plan type should follow the tabular drill down pattern, or plan view. Map based planning in the web UI will also be improved so users can import coverage targets different geographic levels and import risk files to better inform intervention planning. System administrators will be able to create users and assign their roles with improved access controls that support granting specific permissions in the Android client and web UI. Finally, the entire platform will be able to be run on less hardware than is currently required because we are removing OpenMRS and streamlining the data warehouse process.

Technical Design

This section establishes a core set of terminologies and outlines the technical design of the Reveal Platform to inform software development. This section builds off of core functions in the OpenSRP system, so we will briefly describe each component to establish a common baseline.

Background

Data Model

Reveal is built on an Event Client (EC) data model that stores each event that’s saved in the system. Once the event is stored, it updates a set of convenience “client” tables that are used to support queries across the system. We primarily sync events across the system and apply to each event to create up to date client tables in each tablet, OpenSRP server and data warehouse. The form is the primary way for inputting or creating events in the system. This dynamic structure allows us to generate different types of form inputs and generate client tables conveniently for the platform. It’s also important to note that this structure is similar to the pattern implemented in CommCare and HAPI FHIR.

We have a number of other components that augment the EC data model. These include specific, structured domain objects for plans, tasks, locations, and roles. These components are first class entities that are able to be linked to each other and entities in the EC data model. Note that SQLite doesn’t support relationships, so we do this in Java code in Android. Plans, tasks, and locations have fields that allow us to link them to each other. These are defined in the Reveal Data Dictionary. For example, The “for” field in a task is dynamic depending on the “code”. So, if “code”:”IRS” the “for” relates to the id of the location. If the “code”:”blood_screening” the “for” relates to the family member. Tasks are linked to the location with the “group” field and are linked to the plan by the “planIdentifier” field. This pattern is modeled after the FHIR R4 standard.

R6 doesn’t require any major changes to the data model. We’re building on top of this structure to make it so that forms, plans, activities and tasks can be configurable for multiple intervention types. Therefore, we need to updated how the Android client, OpenSRP server and Data warehouse present the information that’s entered by the system administrator.

Android Client User Experience Patterns

Reveal implements a number of user experience patterns across the system. The core workflow is map-first with the ability to view a list of tasks as a secondary navigation element. The system also supports interacting with households and household members once a location has been selected for person-centred interventions. This section defines the current state of these patterns. Future sections focus on how we will change these patterns.

  1. Hamburger Menu Plan and Operational Area Selection

    1. This pattern allows the user to select the plan and operational area they are interacting with. Once selected, the map navigates to the operational area, populates colors on structures, loads tasks, and responds to user taps based on the information in the task. The list of available plans is based on the user’s Plan And Location assignment that was done in the web UI. Once selected, the operational area presents a shortlist of locations that are assigned in that plan.

  2. Post Form Save Action

    1. The platform performs actions after each form is saved in the system. We call these post form save actions. We program logic into the application to update different components in the system as soon as the user hits save on a form. For example, the system creates a household, head of household and creates new tasks when a register family form is saved. In this case, the creation of the household and head of household are core to the platform, but task generation is something that will ultimately need to be configurable. This logic is hard coded every time a form is saved and is the logic that allows the app to update colors on the map, change the business status, present different information in the card view and generate new tasks on the fly.

  3. Server Side Task Generation when the Plan becomes Active

    1. The platform has a regular query that identifies if a plan has become active. Once active, logic is run on the server that generates tasks for all activities in the plan. There’s logic in the plan to generate these tasks based on the state of locations, household registrations and family members at the time of generation. It’s possible for tasks to be generated on the server with a different version of information that’s unsynced on the Android client. These tasks are then downloaded to the Android client and activate the ability to tap a location.

  4. Map View: Add a location

    1. Users are able to add locations of different types by tapping a plus sign in the bottom left corner of the screen. A form pops up that allows the user to zoom in on the map and drop a marker. The lower part of the form allows the user to define the location’s type so we can generate a task with a specific code after the form is saved.

  5. Map View: Tap Structure>Evaluate status>Open Form, Card View, or household

    1. This is a core pattern in the system, where a user can tap any structure on the map, the underlying task is evaluated and the appropriate next steps is executed. In some cases a new form is opened, in others, a card is displayed or the household profile is opened. The action after tapping is hard coded logic to perform differently based on the status of the structure and any outstanding tasks assigned. This logic informs the next steps in the user’s workflow.

  6. Task List View: Tap task> Open Form or household

    1. The task list view displays individual tasks if they are assigned to the operational area or structure and displays a grouping of tasks if there is more than one task assigned to a household. The system opens a form when a user taps a task that’s assigned to an operational area or structure. If the task has already been completed, it opens up the form. The grouped task always opens the household’s member task list view.

  7. Household View: Add Household Member>Generate Task

    1. Every time you add a household member, a specific set of tasks are added for that household member. This logic is hard coded in the instance of focus investigations.

High Level Feature Description (SECTION STILL IN PROGRESS)

This section defines the features to be developed at a high level and describes how they will impact the data model and user experience based on the information provided in the previous sections. We will focus in on the first three features that have the greatest impact across the platform.

Form Configuration by system administrators

The target MVP is to develop a centralized way to update existing forms in the system.

From the scoping doc:

  • As a system administrator, I would like to be able to create and update Reveal forms without needing to compile and release a new APK

    • Changes will include new form fields, hiding of forms fields but not updates to existing form fields (e.g. data type or the meaning of the form field)

    • Changes can include modifications to calculated fields, and changes to skip logic

    • Changes will also include the workflow rules for task generation based on form actions (Dynamic Tasking in the Client) 

  • Forms will be authored and updated in the underlying JSON documents 

  • Changes to form fields should be reflected in the data warehouse

  • The forms available to each implementation should be configurable

  • Form changes should be immediately testable in a real-time test application (as opposed to Android Studio or other IDEs where code needs to be compiled)

  • A minimum dataset should be defined for each form which will be used for dashboard reports and should not be changed

Additional stuff:

  • We need to link the form name to the task code so we know which form to open

  • The add location form may need to be updatable to add different location types (not sure this is a requirement)

  • We need to have a unified way for defining business status across the forms

  • We need a unified way for defining color coding for each form/task type

  • We need to guarantee certain fields are persisted in the client table and are not missing from individual forms

  • We need to add versioning of forms so that we can dynamically push updates to the data warehouse and downstream processes

  • We need to think through the link between a plan activity code and a form name. There must be a 1-to-1 on the code in the activity.

  • We need to think through the process of defining post form save actions on a per form basis so that a user has a clear expectation of what is available in (dynamic task generation is involved in this).

  • We need to figure out what to do when you try to edit a form on an old version and you can only download the new version of the form.

  • We need to come up with a process for identifying the difference when a new form version is uploaded and validate the change so that we can enforce rules. i.e. no deletion of fields.

  • We probably need jsonlint on the upload to validate the JSON

  •  

Dynamic Task generation

The goal of this feature is to read from the plan when choosing to generate tasks client side when a post form save action is fired. Currently, we hard code the tasks that are generated, but we should make this part of the plan.

  • Identify the points where a post form save action should create another task

  • Develop logic to read the codes and interventionUnit from the plan and execute the creation of tasks based on business rules

  • Create a place where a system admin can define the business rules for each code/interventionUnit combo in the system including adding combos

  • Figure out how to handle edge cases where multiple tasks are generated on the same entity. For example, what would we do if multiple tasks were generated on the same structure? How would we deal with that from a UX perspective.

  • Figure out how to handle conflicts that are generated because two tablets generated client side tasks for the same location.

  • Figure out roll back procedures if the user chooses to reset tasks (currently a feature in the system)

Entity persistence (households, household members, locations) and updating tasks offline across plans

The goal of this feature is to make it easier for users to access information that was already input to the system across plans. Ultimately, we will depend on the dynamic tasking work to deliver this functionality.

Identify problems currently faced by Thailand

  • Families are created in a tablet in Plan A but aren’t synced, Plan B kicks off in the server which thinks a family hasn’t been registered so it creates a register family task. Plan B has different logic for the existing family members in the system if it knew a family was registered.

  • Locations don’t have a lifecycle. We can’t delete locations in the app.

Scope Ideas

  • We need to add logic that updates tasks across plans if a household is registered anywhere

  • We need to add logic that updates tasks across plans if we remove a person from a household

  • We need to add logic that updates tasks across plans if we update a location

  • We need to figure out the logic to address the client side cases where a register family task is assigned, but there’s already a family registered to that location.

  • We need to expose this logic to the processes defined in the dynamic task generation so that we don’t get into a case where we generate or remove tasks

  • We probably need to block all other processes when we’re evaluating tasks

  • We probably need to address the reset task issue and how we actually remove a registered family on old plans if something changed in the current plan