Category: Software

Microservices have become the defacto standard for building cloud native applications for the past decade and several enterprises have large internal projects to decompose their monolithic applications into collection of microservices to realize the benefits of this service oriented architecture.

As the move to a microservice world accelerates , I have been noticing an aspect of software engineering that needs to be revisited. Typically for any large project to be successful and sustainable, you have to define a solid testing strategy. This involves building several types of tests which are usually classified as unit, integration and end-to-end (e2e) tests. QE teams follow a system called “Testing Pyramid” to help build these tests which suggests that you invest a lot more in unit tests since they help test the business logic without the noise of external dependencies. Martin Fowler’s article is an excellent read regarding this topic.

But in the world of microservices, most of the business logic isn’t centralized inside a particular microservice and it is usually a complex orchestration across different services. In such a world, having a larger corpus of unit test compared to integration tests doesn’t provide the protection you need to run your cloud application. The focus has to shift to a larger battery of integration tests which provide the coverage of the intricate dependencies between the microservices and increase the quality of your releases. It looks more like a “Testing Diamond”.

In the world of cloud services, Availability is a key metric measured by your customer to ensure they are getting what they are paid for. But Availability (sometimes also referred to as uptime) is an IT service metric which usually measures the time when the service was completely inaccessible to its users. It doesn’t translate well to a cloud service based architecture where the chances of your service being completely down is far fewer than the service not behaving properly. Typically when a team measures the Availability of their service, they use a combination of synthetic tests & monitors which run on a schedule and count the minutes when the service was unavailable. Once you have this number, you can calculate the availability using the formula for the specified time window –

Availability = (total mins - outage mins)/(total minutes)%

The problem with this metric is that it doesn’t accurately capture what your users are experiencing in terms of a usable product. For example, if you provide a web service where search is a key scenario and 20% of a particular user’s search are failing, you have a highly available service but it is still a bad experience for your user.

To combat this gap, we have usually required for all our team’s services to define & measure another metric called Service Reliability. Reliability as a metric is very common in the world of electronics and is measured as Mean Time Between Failure (MTBF) and Mean Time To Repair (MTTR) and usually not used in the world of cloud services. A simple definition of Service Reliability is the ratio of the number of times your service behaves correctly over all the times it was invoked to serve its user. If you provide RESTful API based service, you can use a simple formula such as –

Reliability = (total number of success (HTTP status 2xx))/(total number of invocations)%

You could add further attributes/dimension to this formula which can track things such as scenario (for example, ‘search’ or ‘checkout’), customer (a unique customer id). By using such a metric, you can easily identify the overall quality of service provided by your application and can also easily drill into hotspots with respect of scenarios and customers. Another advantage of measuring & ensuring high Service Reliability, you are also ensuring you are providing a highly available service as defined by the Availability metric.

One of the key requirements for running a cloud service is to have a solid understanding of your service’s scalability. Scalability is usually defined as a service’s ability to manage increased demands and is assessed by running scale tests. Scale test usually involves testing the service or a collection of services with synthetic traffic to mimic production loads. It involves a series of test runs where the synthetic traffic is slowly ramped up (concurrent users, and/or size of the data/request) while observing the service’s behavior under load. 

In my experience of reviewing scale test results, I have consistently noticed teams focus on the behavior of their service with varying traffic patterns where the topology of the service is kept constant. In other words, the number of instances of a service in cluster or the hardware features used by the service (CPU/memory/disk) are unchanged. The failure to observe the behavior of the service when you vary the topology is a missed opportunity for the team to further bolster their system against a surge in usage. 

By answering these 3 key questions as an outcome of a scale test, you are ensuring you have a clearer grasp of knowing when your service is going to break under load and you have a clear plan on addressing the spike before it breaks.

1. Does your service scale linearly with capacity? (Horizontally scalable) – For example, if you add an additional node to your cluster, do you see linear growth in concurrent users or requests/second handled by your service.
2. How long does it take to add new capacity in production? (Elasticity) – For example, depending on your service architecture & deployment, it might take minutes or hours or days to add new nodes. 
3. Can you predict when you need to add new capacity? (Monitoring & Alerting) – Given the answer to the time required to add capacity, you will have to work backward to figure out when you would need to start that work. This will ensure you have the right monitors and alerting setup to address increased usage.