API response time benchmarks

Owned by jmuindi
Last updated: Apr 26, 2021
5 min read

Documentation for API response time benchmarks

Definition of Performance Testing 

Performance testing is a form of testing that focuses on determining how a system performs in terms of responsiveness and stability under a particular workload.

The purpose of Performance Testing is not to find functional defects but to eliminate performance bottlenecks in the software or device.

Load testing is about putting your application under heavy load. So if we’re talking about a server, that probably means firing lots of requests at it. You’re interested in general in what happens in the system. You might not be looking for anything specific but you want to see when unusual things start to happen — slowdowns, bad responses, unusual behaviour.

Benchmarking suggests more of a specific goal. One wants to see how a certain factor (or factors) hold up under a specific amount of a certain stress. It might not be an excessive amount of stress, just a regular amount. For example, if one expects 300 requests per second to hit a server, to run a benchmark test to see how response time of the server performs under these conditions. Or run a benchmark of twice or 10x the normal load. But the aim is for a specific result — to be able to say “with this traffic, 95% of requests are resolved in n milliseconds”.

Attributes: 

  1. Speed

  2. Scalability 

  3. Stability

  4. Reliability 

 

Why is Performance Testing/Load Testing/Benchmarking important?

  1. It’s a high time we incorporated it at Ona

  2. How fast our platform responds under a certain workload 

  3. It uncovers what needs to be improved. (products under development) 

  4. It will help with determining whether the software 

  5. It’s good to understand what currently happens; how have we conducted P.T in the past? What were the outcomes? 

  6. To know the limits of our software systems. Based on real data; not estimates

  7. We can achieve our milestone; scaling Ona Data. If we know how much Ona data can handle; we know we need to scale to a certain amount. 

 

Types of performance testing

  1. Load testing - Measures system performance as the workload increases. Workload could mean concurrent users or transactions. The system is monitored to measure response time and system staying power as workload increases. That workload falls within the parameters of normal working conditions.

  2. Stress testing also known as fatigue testing is meant to measure system performance outside of the parameters of normal working conditions. The software is given more users or transactions that can be handled. The goal of stress testing is to measure the software stability. At what point does software fail, and how does the software recover from failure?

  3. Spike testing - a type of stress testing that evaluates software performance when workloads are substantially increased quickly and repeatedly. The workload is beyond normal expectations for short amounts of time.

  4. Endurance testing - also known as Soak testing - is an evaluation of how software performs with a normal workload over an extended amount of time. The goal of endurance testing is to check for system problems such as memory leaks. 

  5.  Scalability testing - used to determine if software is effectively handling increasing workloads. This can be determined by gradually adding to the user load or data volume while monitoring system performance. Also, the workload may stay at the same level while resources such as CPUs and memory are changed.

  6.  Volume testing - determines how efficiently software performs with a large, projected amounts of data. It is also known as flood testing because the test floods the system with data

Common Problems Observed in Performance Testing

  • Speed issues - slow responses and long load times for example — often are observed and addressed.

  • Bottlenecking - This occurs when data flow is interrupted or halted because there is not enough capacity to handle the workload.

  • Poor scalability - If software cannot handle the desired number of concurrent tasks, results could be delayed, errors could increase, or other unexpected behavior could happen that affects:

    • Disk usage

    • CPU usage

    • Memory leaks

    • Operating system limitations

    • Poor network configuration

  • Software configuration issues - Often settings are not set at a sufficient level to handle the workload.

  • Insufficient hardware resources - Performance testing may reveal physical memory constraints or low-performing CPUs.

  •  

Tools used in Performance Testing

  1. JMeter for the web applications (onadata and OpenSRP)

    1. Tested Authentication and Submissions

  2. Locust

  3. Apache bench 

  4. Tsung

Metrics used in Performance Testing

  1. Response time

    1. Total time to send a request and get a response.

  2. Wait time

    1. A.k.a average latency, this tells the developers how long it takes to receive the first byte after the request is sent.

  3. Average load time

    1. The average amount of time it takes to deliver every request is a major indicator of quality from a user’s perspective.

  4. Peak response time

    1. This is the measurement of the longest amount of time it takes to fulfill a request. A peak response time that is significantly longer than average may indicate an anomaly that will create problems.

  5. Error rate 

    1. This is a percentage of requests resulting in errors compared to all requests. These errors usually occur when the load exceeds capacity.

  6. Turn around time (TATs)

  7. Concurrent users

    1. This the most common measure of load — how many active users at any point. Also known as load size.

    2. Concurrent users - many virtual users are active at any given time

    3. Requests per second - How many requests are handled.

    4. Throughput - often bandwidth consumed, but, in general, the maximum rate at which something can be processes

  8. Transactions passes/failed (Success Rate )

    1. A measurement of the total numbers of successful or unsuccessful requests.

  9. Throughput

    1. Measured by kilobytes per second, throughput shows the amount of bandwidth used during the test.

  10. CPU Utilization 

    1. How much time the CPU needs to process requests.

  11. Memory Utilization 

    1. How much memory is needed to process the request.

First test the entire service; then go to separate sections. 

Simulate different strategies for different APIs Not all endpoints are hit equally.

Tasks: 

  1. Environment setup - 

  2. Scripting ; how to generate data (QA)

  3. Running the scripts

  4. Reporting

  5. Indicators to measure load testing - should be guided by what the engineers think a healthy system should look like.

Performance Testing Steps

 

Important notes to go through: 

Keep track of Grafana, Greylog and such. Need to involve other teams to separate what is in testing and what is not. Can be the preview environment, not staging.