We recently ran load tests for the Linx Host Service. In these tests, the CPU load and memory were monitored under heavy load to provide better insights into how the REST API Request Limits perform for the different virtual machines (Azure B-Series) that run Linx.
About Load Testing: Load testing simulates real-world conditions and examines how the system behaves during normal and high loads. They will determine if your system or software can handle high loads given the high demand of end-users. This test is typically applied when a software development project nears completion.
Method
Load testing was done using JMeter. A web service with four endpoints with empty tasks was tested to determine the behaviour of the system as the load increases from zero-load situation to a situation until the server`s CPU usage reached 100 % without the API returning any errors or timing out.
Four endpoints were tested for Load as described in the following table
| Label | Method | Headers | Parameters | Body Data | Response |
| DummyGET | Get | None | None | None | None |
| DummyPOST | Post | Content-type and Accept | None | None | None |
| GetWithResponse | Get | None | None | None | String |
| PostWithResponseAndContetn | Post | Content-type and Accept | None | Yes | String |
We started by trying to find the maximum throughput that our API can sustain. We define this metric as the number of requests per second that will bring our API server to the maximum CPU utilization. The test was carried out in 2 phases; first, requests were done consecutively on each of the four endpoints, and then requests were done in parallel on four endpoints.
Challenges
- In parallel tests, the actual number of concurrent users and transactions per seconds depends on the server response time.
- In parallel tests, only four endpoints were tested. The result may vary with the number of endpoints.
- Users were added during a fixed period of time. 60 seconds was chosen as the ‘spread over time’ in all tests, which is not a real-life case scenario.
- The example does not accurately reflect a normal user’s usage pattern as the Linx server deployment and load injector were in the same data center.
Result
To find each endpoint limit on each machine, traffic was generated by increasing the number of users slowly until errors were noted in JMeter. In the consecutive test, traffic was simulated on four different endpoints using linear injection of users for a fixed time period of sixty seconds (60 sec). Each user would send 10 requests on each endpoint consecutively.
Consecutive Test Results
| B1ms (1 vcpus, 2 GiB memory) |
B2ms (2 vcpus, 4 GiB memory) |
B2ms (2 vcpus, 8 GiB memory) |
B4ms (4 vcpus, 16 GiB memory) |
B8ms (8 vcpus, 32 GiB memory) |
|
| No. of requests | 80,000 | 120,000 | 120,000 | 320,000 | 320,000 |
| Mean response time (ms) | 15 | 5 | 5 | 6 | 6 |
| Average requests/s | 1250 | 2653 | 2823 | 4064 | 6524 |
To test how increasing the requests on the four endpoints simultaneously would affect performance, traffic was simulated on all four different endpoints at the same time. Requests were sent on all four endpoints at the same time using linear injection of users for fixed time period of sixty seconds (60 sec). Each user would send 10 requests.
Parallel Test Results
| B1ms (1 vcpus, 2 GiB memory) |
B2ms (2 vcpus, 4 GiB memory) |
B2ms (2 vcpus, 8 GiB memory) |
B4ms (4 vcpus, 16 GiB memory) |
B8ms (8 vcpus, 32 GiB memory) |
|
| No. of requests | 48,000 | 140,000 | 120,000 | 240,000 | 240,000 |
| Mean response time (ms) | 9 | 17 | 18 | 351 | 2041 |
| Average requests/s | 811 | 1736 | 1848 | 3212 | 5510 |
Conclusion
REST API requests of all types and to all endpoints are limited by the number of CPU of the virtual machines. As we increased the number of virtual CPU, we could see that Linx RESTHost performance increased to serve more requests and the mean response time was acceptable. As the number of requests increased above approximately 5500 requests per second, we started having errors with 500/Internal Server Error.