Some time ago we wrote a post about Synthetic Testing and Real User Monitoring, and explained why these are complementary, not competing, approaches. Similarly, Synthetic Monitoring and APM – Application Performance Monitoring – are similar in that they are often regarded as two competing solutions, as if one were an alternative to the other. They are actually perfectly complementary tools to achieve optimal performance for your application. Whether you develop web applications, SaaS products or mobile apps, you‘ll find both approaches to be absolutely necessary in your software operations. Here, I explain why.
Let’s look at APM first. These tools are normally used to collect data from a live environment. APM, or more broadly, passive monitoring solutions help gather important information on the app performance, web transactions, application backend metrics, server behaviors, etc.. Several application monitoring solutions track database and browser performance as well.
There are many kinds of passive monitoring solutions that operate at different levels of the stack, with different timing granularity, with the opportunity to define custom metrics. For example, one can have server monitoring, infrastructure monitoring, user monitoring, high frequency metrics, and others. What is clear with passive monitoring solutions is that one is not enough. You probably need a portfolio of tools to have a clear understanding of what is happening with your live application. Most tools have alerts systems that provide real-time information on key metrics and are designed to notify you in real time of events.
However, passive monitoring is not enough.
It doesn’t matter how comprehensive your monitoring architecture is, it doesn’t provide the full picture. First of all, live traffic is noisy. You have no control over what your users are doing and how many users you have at any given point. If you have a performance issue, that makes it really hard to troubleshoot. Was that expected behavior? Did they just hit a corner case previously unanticipated? Is a combination of the live workloads that crashed the system? So even though the technology is designed to enable real-time troubleshooting, the reality is that since it’s not a controlled environment, you might not be able to rootcause performance issues in a timely manner.
Second and most important, the information produced by APM is delivered after the fact. APM is like calling AAA after an accident. It’s a great service to have, but it’s much better to prevent the accident in the first place.
The third aspect is that APM may not give you the complete end-to-end picture. For example if you’re monitoring the app server, and you’re having issues with your DNS service, the metrics delivered by the app server might show a healthy situation. While in reality your users are affected by the failing DNS.
Synthetic Monitoring and Performance Testing
This explains why you need to add synthetic monitoring (on production environments) and performance testing (in dev & test) to the mix. While APM can inform you about performance after the fact, performance testing can help you prevent bad things from happening and synthetic monitoring can give you a complete picture of the health of your service, on a regular basis.
Synthetic monitoring is usually done on your live/production environment, and utilizes synthetic traffic. Performance testing is usually performed on a pre-production/staging environment. Having a pre-production environment as close as possible to your production environment will help you derive the most meaningful results. You can actually use user scenarios in dev&test and convert them into monitors in production. This achieves another objective, which is the ability of breaking down silos between your dev/test team and the ops team, and enable them to share data and metrics.
With synthetic monitoring, you have total control over the scenarios that are executed by your monitors. That makes it a lot easier to isolate and troubleshoot problems.
In addition, with synthetic monitoring you always have a true picture of the health of your service. With passive monitoring, you rely on your live users to give you data. With synthetic monitoring, you have data even if you don’t have a lot of live users or if your live users are not executing certain workloads.
And last but not least, with synthetic monitoring you measure end-to-end performance – a good indication of user experience — which gives visibility into the entire application delivery chain, enabling greater transparency and targeted troubleshooting.
In performance testing users are simulated but the traffic is absolutely real. You can apply different types of traffic to discover the breaking point of your application with user growth. You can use performance testing to test with traffic that is higher than anything your actual application has seen – yet – so that you can prepare for peak of traffic. Performance testing can also help you identify performance degradations that might have resulted from code changes, infrastructure changes or 3rd party changes. It basically answers the question: “Can you trust this build to deliver the same user experience your users are counting on?”. In performance testing you have total control over the amount of traffic and the workloads your users execute. That makes it a lot easier to troubleshoot.
And, you could also test performance enhancements, under different configurations and platforms easily.
Used together, synthetic monitoring and APM make a great team, so to speak. Use both to make sure you deploy a reliable product.