This document defines the software requirements for the distribution point features for Reveal. A distribution point is a centralized location where individuals meet to receive care. The primary workflows are funded under a grant for Neglected Tropical Diseases (NTDs) which focuses on distributing medication among children who are attending schools in Eswatini. The functional requirements are outlined on NTD School Workflow and the data dictionary. This document applies those functional requirements into the Reveal technical architecture.
...
(Outside of the system) The user updates a template based on the structure that’s defined for the project
User logs into the system
User chooses a plan (draft or active)User chooses a distribution that they want to upload the CSVselects “Admin” in the navigation and finds “student lists”
User clicks a button to find the file on their computer
User clicks the upload button and the CSV is uploaded to the server
Details are passed in with the CSV to show what types of entities to generate and the mapping of the template.
CSV is parsed server side and validated (See CSV Validator section)
An error is returned to the user if the CSV is invalid or it doesn’t pass a security check
An event is created for each row in the OpenSRP server
Each event is tagged to the plan and location
A task is created for each even that’s uploaded if the plan is active. Otherwise, tasks are created when the plan becomes active.
(Existing functionality) A client is created in the Android client for each event when it’s received
The
...
We need to build user can upload more than one csv - list of all csv uploads shown
User can download a csv of students - they can do this at each level of the location hierarchy down to school
Mock created by Roger for csv upload functionality:
...
The OpenSRP server needs to be enhanced in the following ways in order to deliver this workflow:
We need to build a REST API in the OpenSRP server that can receive the CSV file and perform the event generation
We need to build the foundation for a server side task generator in OpenSRP that can generate tasks as a service when the CSV is uploaded. This task generator will be improved upon in the coming months so that we can move the Nifi flows that generate tasks for Focus Investigation, IRS and MDA to Java server code so that the task generation system functions better.
Note that this will involve engaging the queuing software in OpenSRP so that we don’t overwhelm the server with a spike of activity.
We need to build an import template pattern that is configurable for OpenSRP. This would allow us to create a server side configuration that validates the template that was received and create the events. Ultimately, this will be able to be used generically to import events and, optionally, tag them to the appropriate jurisdiction and plan.
We need to be able to define settings that map the event/client metadata tags to the labels in the CSV template, sync them down to the Android client and make them available for display and filter in the Android client. We have three points that need to be dynamically assigned in the app. We need to “tag” each client in the register view to up to 3 of those points so they can be searched against.
See the subsection labeled CSV Upload to Event Generator
We need to be able to generate clients by using a CSV upload. Each row will present an event
...
How do we roll back these items in the event someone messes up?
We need to define the logic behind generating
We need to define the error handling.
(Business question) Users don’t have a view into all of the entities that are registered to a distribution point. Therefore, they have no idea if their list was uploaded correctly. How should we mitigate this?
In other words, how will we handle mistakes when a user uploads information that they don’t want to be there. What’s the life cycle we should create to remove the likelihood of duplicates?
We need to come up with unique id generation so that we can
We are building a bulk event generator in this CSV upload process. It would be valuable to support JSON format as well as CSV.
We need to determine if we should have different variations of the template and optional fields so that we don’t need different templates here.
Consider removing the upload after the plan is generated.
How will we provide a list of children to the user.
OpenSRP Web UI
We need to develop the following features in the OpenSRP Web UI in order to deliver the features above.
...
UI for uploading CSV files to send to the OpenSRP server’s REST API
...
Change the manage plans form so that users can either select jurisdictions or distribution points
...
We are building a bulk event generator in this CSV upload process. It would be valuable to support JSON format as well as CSV.
We need to determine if we should have different variations of the template and optional fields so that we don’t need different templates here.
Consider removing the upload after the plan is generated.
OpenSRP Web UI
We need to develop the following features in the OpenSRP Web UI in order to deliver the features above.
UI for uploading CSV files to send to the OpenSRP server’s REST API
Change the manage plans form so that users can either select jurisdictions or distribution points
Adjust the manage plans and add new plan view to include MDA plan types and NTD plans
Web UI Reporting Needs
Dashboards will follow the current IRS model
Drill down to school level with the same set of stats (Region, Inkhundla, School)
At the school level, cut statistics by age category
6-10
11-15
16-18
The reporting needs are defined in the data dictionary from Akros. This will require changes to the Canopy flow to make sure the data is available to the data warehouse and Superset.
Web UI Reporting Needs
Dashboards will follow the current IRS model
Drill down to school level with the same set of stats (Region, Inkhundla, School)
At the school level, cut statistics by age category
...
6-10
11-15
16-18
The reporting needs are defined in the data dictionary from Akros. This will require changes to the Canopy flow to make sure the data is available to the data warehouse and Superset.
CSV Upload → Event Generator
We need to develop a server side event generator that generates events server side for each row that’s available in the import. We aim to make the CSV event generator generic so that we can import different types of entities into OpenSRP in the future. We need to reference the event template with the CSV so that we can identify which fields should be mapped to which events when they are created. This template should be defined within the OpenSRP server so that it’s CSV to event objects or make it so that system administrators can configure these entities and evolve their generation over time.
[This section needs to be enhanced by showing the business logic that’s executed when the CSV is uploaded. Craig Appl will do this]
CSV Validator
CSVs need to go through a validation process that can generate an event that will sufficiently generate a client when the event is loaded in the Android device. This section defines the details of the CSV Validator.
Perform a security validation to reduce risks on import (see the Security section below)
Validate the structure of each field in the template
(Stretch) identify if there is an existing client in the system with the same information. Provide feedback to the user.
[This section needs to be improved by identifying how we will address duplicates. Craig Appl will update this.]
Plan Definition and Task changes
This section defines the changes to the plan generation and task process for NTDs. The most simple solution is to slightly modify the plan definition template for NTD distribution point workflows. In the long run, we need to accommodate NTDs in the dynamic tasking workflows that are scheduled to be developed in June.
Plan Definition Sample
We will add the following attributes to a plan definition. All core task sections are defined in The All Events Data Dictionary
[Craig needs to complete this section]
Outstanding questions on this section:
Should the plan account for adverse event reporting in some way? How will we do counts of adverse events in the reports by creating a complete task when there’s an adverse event reported?
Outstanding Items on the Plan Definition and Task Changes
The current Task paradigm in Reveal generates all tasks when the plan is changed to active. We have a work item to persist entities in the Android client by the end of June as well as support dynamic tasking client side. Craig has a hunch that there is a core redesign in how and when we generate tasks in Reveal. We went through a lot of pain in the past because we only allowed tasks to be generated once instead of doing it on a rolling basis while events happened in the system. We need to look at the whole picture of what needs to be delivered in June and figure out if we can work on that as part of this work. This section focuses on those changes so we can make a clear decision.
[This section needs more thought and research from Craig based on the long term timelines for Reveal.]
We need to rethink when tasks are generated server side.
Prior workflows generated tasks at one time when the plan was transitioned from draft to active
We have the dynamic tasking feature which may overlap
We need to
Task Data Elements
Security
This section defines security requirements for these features.
Users should not have the capability to download large lists that contain personally identifiable information about the clients.
Access to download rosters will, at maximum, provide identifiers that can be linked to people registered in the system.
All uploads need to be validated to include the following:
Valid CSV architecture with UTF-8 encoding
Individual fields are parsed and validated to remove exposure to SQL injection threats
Individual fields are parsed and validated to remove exposure to cross-site scripting threats
(We probably need to add more here)
Things to be discussed from scoping so far
We need to validate the assumption that we will support importing rosters against active plans.
Do we only import rosters against active plans?
Should we support importing rosters against plans in draft state?
Should we generate tasks when a plan is shifted to active like we currently do?
Out of Scope
...
Canopy Components
A NiFi and Superset instance will need to be deployed (along with all necessary backend data stores) to support the OpenSRP Web UI dashboards and task generation features. The NiFi instance will have two main flows:
Latest version of the Reveal OpenSRP connector, or a generic version if at all possible
Customized NTD task generation logic. Initially this will be single-pass generation of all tasks when a plan becomes active.
If it is impossible to guarantee all clients affected by a plan will be registered in OpenSRP when the plan becomes active, a second iteration which does some form of dynamic task generation in response to client changes will need to be researched and built.
The Superset instance will have the following slices exposed:
Aggregate target statistics per juridiction+level
Non-aggregated target statistics
Due to the the complexity of these statistics, they will probably need to be recomputed on a (smart or dumb) periodic basis vs aggregated for every request. As a first iteration a simple time interval will be used, but if this is not sufficient we can iterate with marked-as-dirty tables and triggers or other solutions.
CSV Upload → Event Generator
We need to develop a server side event generator that generates events server side for each row that’s available in the import. We aim to make the CSV event generator generic so that we can import different types of entities into OpenSRP in the future. We need to reference the event template with the CSV so that we can identify which fields should be mapped to which events when they are created. This template should be defined within the OpenSRP server so that it’s CSV to event objects or make it so that system administrators can configure these entities and evolve their generation over time.
Business logic that’s executed when events are generated:
Ronald Kudoyi , please work on this section
Generate an event
Possibly generate a client in the OpenSRP Server→ See the note in the Nifi section about generating tasks when the events haven’t been synced down to the Android client, which generates clients in the EC data model.
CSV Validator
CSVs need to go through a validation process that can generate an event that will sufficiently generate a client when the event is loaded in the Android device. This section defines the details of the CSV Validator.
Perform a security validation to reduce risks on import (see the Security section below)
Validate the structure of each field in the template
(Stretch) identify if there is an existing client in the system with the same information. Provide feedback to the user.
We do not allow the user to delete children from the system using the csv upload because that could affect active plans and reporting. We do, however, allow them multiple csv uploads in order to add children that have been missed or update information on children already in the system.
In order to prevent duplicates, the user will be asked to download the csv roster from the system, make changes or add students in that sheet, and re-upload it. This downloaded csv will contain a unique ID for all students already in the system. When that csv is uploaded, the system will run a check for these IDs and update those children’s data, but not create new entities.
Plan Definition and Task changes
This section defines the changes to the plan generation and task process for NTDs. The most simple solution is to slightly modify the plan definition template for NTD distribution point workflows. In the long run, we need to accommodate NTDs in the dynamic tasking workflows that are scheduled to be developed in June.
Distribution Point Locations
We need to add metadata to distribution points in the location hierarchy. Ultimately, we need to be able to assign plans at distribution points. So, we need to assign the following metadata to the appropriate distribution points.
We need to update the add point form to include a type “Distribution Point” Data Dictionary Link Distribution points are not jurisdictions. They are a point in the lowest location of the hierarchy, similar to how we handle facilities in OpenMRS and DHIS2.
JSON Location Change
(Ronald Kudoyi , please update this so that we can clearly mark how a location is a distribution point in the location JSON. I don’t know if this is done through tags somehow.)
| Code Block |
|---|
//This code shows the changes made to locations that represent a distribution point |
Questions on this section:
Ronald Kudoyi , do we need to update the locations API to return distribution points so it’s easier for the web UI to get this list?
I think the answer is no, but we need to talk with the Web UI team to make sure they know how to get this list per jurisdiction. (See the next section)
Plan Definition Sample
This section includes the plan definition sample. All core task sections are defined in The All Events Data Dictionary
Notes:
The first activity is to dispense the medication with a goal of greater than or equal to 100% of eligible people
The second activity is to report adverse events with a goal of less than or equal to 2% of people
| Code Block |
|---|
[
{
"identifier": "69b2f14d-f555-4ba6-b880-596f68e1888c",
"version": "1",
"name": "MDA-Point-Plan-1-2020-03-18",
"title": "MDA Point Distribution Plan 1",
"status": "active",
"date": "2020-03-18",
"effectivePeriod": {
"start": "2020-03-18",
"end": "2020-06-01"
},
"useContext": [
{
"code": "interventionType",
"valueCodableConcept": "MDA-Point"
},
{
"code": "taskGenerationStatus",
"valueCodableConcept": "False"
}
],
"jurisdiction": [
{
"code": "c138d117-bf30-48b2-804e-964b4891ea51"
}
],
"goal": [
{
"id": "mdaDispense",
"description": "Dispense medication to each eligible person.",
"priority": "medium-priority",
"target": [
{
"measure": "Percent of eligible people.",
"detail": {
"detailQuantity": {
"value": 100,
"comparator": "u003eu003d",
"unit": "Percent"
}
},
"due": "2020-06-01"
}
]
},
{
"id": "mdaAdverseEvent",
"description": "Report any adverse events from medication.",
"priority": "medium-priority",
"target": [
{
"measure": "Percent of people who reported adverse events.",
"detail": {
"detailQuantity": {
"value": 100,
"comparator": "<=",
"unit": "percent"
}
},
"due": "2020-06-01"
}
]
}
],
"action": [
{
"identifier": "83c5adc1-5d0b-4b1a-9f68-deed475cf103",
"prefix": 1,
"title": "MDA Dispense",
"description": "Dispense medication to each eligible person.",
"code": "MDA Dispense",
"timingPeriod": {
"start": "2020-03-18",
"end": "2020-06-01"
},
"reason": "Routine",
"goalId": "mdaDispense",
"subjectCodableConcept": {
"text": "Person"
},
"taskTemplate": "MDA_Point_Dispense"
},
{
"identifier": "ed9fbdf9-f817-40f0-9c1b-ea4eb1acb134",
"prefix": 2,
"title": "MDA Adverse Event(s)",
"description": "Report any adverse events from medication.",
"code": "MDA Adverse Event",
"timingPeriod": {
"start": "2020-03-18",
"end": "2020-06-01"
},
"reason": "Routine",
"goalId": "mdaAdverseEvent",
"subjectCodableConcept": {
"text": "Person"
},
"taskTemplate": "MDA_Adverse_Event"
}
]
}
] |
Task Structure
We need to develop two new types of task to account for the distribution point, the MDA Dispense task and the MDA Adverse Event Task. The MDA Dispense task is generated for each person who is eligible to receive medication. The MDA Adverse Event task is generated whenever an adverse event form is generated in the Android client and it’s automatically marked as complete. This allows us to use the plan reporting structure to support reporting adverse events. Below are task templates for each task type.
MDA Dispense
| Code Block | ||
|---|---|---|
| ||
{
"identifier": "cd53f605-4398-4713-9bc1-43805a302c06",
"planIdentifier": "69b2f14d-f555-4ba6-b880-596f68e1888c", //The id of the plan
"groupIdentifier": "c138d117-bf30-48b2-804e-964b4891ea51", //The location id of the distribution point
"status": "Ready", //Underlying task status
"businessStatus": "Active", //Business status from the form
"priority": 3,
"code": "MDA Dispense", // The code from the action
"description": "Dispense medication to each eligible person.", //Description of the task
"focus": "83c5adc1-5d0b-4b1a-9f68-deed475cf103", //The id of the plan action
"for": "", //The baseEntityId of the person record
"executionStartDate": "2020-04-22T12:44:53.382+03:00", //Start date of the task execution period
"executionEndDate": "", //End date of the task execution period
"reasonReference":"", //This is blank for MDA info
"authoredOn": "2020-04-22T12:44:53.382+03:00",
"lastModified": "2020-04-22T12:44:53.382+03:00",
"owner": "testUser"
} |
MDA Adverse Event
| Code Block |
|---|
{
"identifier": "cd53f605-4398-4713-9bc1-43805a302c06",
"planIdentifier": "69b2f14d-f555-4ba6-b880-596f68e1888c", //The id of the plan
"groupIdentifier": "c138d117-bf30-48b2-804e-964b4891ea51", //The location id of the distribution point
"status": "Ready", //Underlying task status
"businessStatus": "Not Started", //Business status from the form
"priority": 3,
"code": "MDA Adverse Event", // The code from the action
"description": "Report any adverse events from medication.", //Description of the task
"focus": "ed9fbdf9-f817-40f0-9c1b-ea4eb1acb134", //The id of the plan action
"for": "", //The baseEntityId of the person record
"executionStartDate": "2020-04-22T12:44:53.382+03:00", //Start date of the task execution period
"executionEndDate": "", //End date of the task execution period
"reasonReference":"", //This is blank for MDA info
"authoredOn": "2020-04-22T12:44:53.382+03:00",
"lastModified": "2020-04-22T12:44:53.382+03:00",
"owner": "testUser"
} |
Task Business Logic
This section defines the business logic for each task type that needs to be executed when we generate tasks either in the Android client or OpenSRP server/Nifi.
MDA Dispense
These tasks are generated in the OpenSRP server when the plan is made active.
One task is generated for each eligible child registered at each distribution point
Child eligibility is defined as
Generate task if 6.0 years<=age<=18.0 years
One task is generated in the Android client when a child is registered following the child eligibility above
Business Status and Color coding
Status one [Green] = Child is eligible, has a Mass Medicine Administration form, and mmaDrugAdmin = 'Yes' (aka drugs were in fact dispensed)
Status two [Red] = Child is eligible, has a Mass Medicine Administration form, and mmaDrugAdmin = 'No' (aka drugs were not dispensed)
Status three [Yellow] = Eligible, and does not have a Mass Medicine Administration form - this is the default status
MDA Adverse Event
These tasks are generated in the Android client only when an adverse event form is complete
One task is generated after the form is saved and the status is marked as complete immediately
Business Status (No Color Coding in the system) (Data element from dictionary)
Severe Reaction
Moderate Reaction
Mild Reaction
Changes to the Nifi Flow
We need to make a number of changes to the Nifi process:
We have existing MDA plans with useContext.interventionType = “MDA” these are now “MDA-Community” plans.
The Nifi processes built for MDA up to this point need to account for the change to the use context in 1.
We need to build new workflows in Nifi to account for business logic changes for useContext.interventionType = “MDA-Point” outlined in the Task Generation section above.
When we generate tasks right now, we generate them against clients that have been uploaded to the server. In this case, we won’t have clients in the server. We will have a mix of clients and events because the imported events haven’t hit the Android app to generate clients in the EC data model. Ronald Kudoyi , can you recommend how to address this? We could alternatively generate clients in the OpenSRP server when we import a roster.
Security
This section defines security requirements for these features.
Users should not have the capability to download large lists that contain personally identifiable information about the clients.
Access to download rosters will, at maximum, provide identifiers that can be linked to people registered in the system.
All uploads need to be validated to include the following:
Valid CSV architecture with UTF-8 encoding
Individual fields are parsed and validated to remove exposure to SQL injection threats
Individual fields are parsed and validated to remove exposure to cross-site scripting threats
(We probably need to add more here)
Out of Scope
Linking people who were registered in the distribution point workflows to families or structures
Outstanding Items on the Plan Definition and Task Changes
The current Task paradigm in Reveal generates all tasks when the plan is changed to active. We have a work item to persist entities in the Android client by the end of June as well as support dynamic tasking client side. Craig has a hunch that there is a core redesign in how and when we generate tasks in Reveal. We went through a lot of pain in the past because we only allowed tasks to be generated once instead of doing it on a rolling basis while events happened in the system. We need to look at the whole picture of what needs to be delivered in June and figure out if we can work on that as part of this work. This section focuses on those changes so we can make a clear decision.
[This section needs more thought and research from Craig based on the long term timelines for Reveal.]
We need to rethink when tasks are generated server side.
Prior workflows generated tasks at one time when the plan was transitioned from draft to active
We have the dynamic tasking feature which may overlap
We need to
Android client
We need to create tasks in the Android client as follow