How-To
Detecting Anomalous Spending on Your AWS Account
One of the more unsettling aspects of running business processes in the cloud is that costs can sometimes be unpredictable. Amazon and other cloud service providers bill their subscribers based on the resources they use.
Although many business processes use about the same amount of resources from one month to the next, there is always the possibility that abnormal usage will result in a sky-high bill at the end of the month. While abnormal use can stem from activity spikes, it can also sometimes be caused by a simple configuration change that had an unanticipated impact on resource usage.
Amazon has long given its AWS subscribers tools to help them track their online spending. The Budgets tool for example, allows subscribers to be notified when their resource usage reaches various budget thresholds. As useful as the Budgets tool may be however, it isn't perfect. Imagine for a moment that an organization makes an accidental configuration change that causes resource usage to dramatically increase. Assuming that the organization has set up budget notifications, it will receive a notification at the predetermined budget thresholds. However, hitting a budget notification threshold does not stop the spending. Resources will continue to be used at an elevated rate until someone notices the alert, tracks down the cause of the increased usage and then takes steps to rein in the resource usage.
But what if it were possible to detect these types of situations (and correct them) before your budget is depleted? Amazon makes this possible through a feature called the Cost Anomaly Detection tool.
As previously mentioned, business processes running in the cloud generally consume a predictable level of resources. The AWS Cost Anomaly Detection tool uses machine learning to learn your organization's normal resource usage patterns. By knowing what is normal for your organization, the tool can recognize abnormal resource usage. It may even be able to alert you of the abnormal usage before your budget thresholds are released, and it can assist with root cause analysis.
To set up cost anomaly detection, log into AWS, open the AWS Cost Management Service and then click on Cost Anomaly Detection. Next, click on the Get Started button. When you do, AWS offers to give you an optional tour. When the tour completes (or if you skip the tour), you will be taken to the Choose Monitor Type screen, shown in Figure 1.
[Click on image for larger view.] Figure 1: You will need to tell AWS how to monitor your spending. (source: AWS).
The first thing you will need to do is to tell AWS how to monitor your spending. Amazon recommends using the AWS Services option for most situations. Choosing this option causes AWS to monitor the various services you use and compare your usage against historical patterns. There are however, some other options.
One such option is to monitor a specific linked account. This can be useful is several people in you want to monitor a specific person's usage. You also have the option of monitoring usage by cost category or by tag.
Once you have chosen the monitor type you want to use, the next thing you will need to do is name the monitor you are creating. Amazon makes you do this is because it is possible to create multiple monitors. If you are monitoring AWS services, you will probably only need a single monitor. However, if you are monitoring based on a linked account, cost category or cost allocation tag, then you will likely need several monitors in order to comprehensively monitor your AWS deployment.
After you have entered a monitor name, click Next and you will be taken to the Alert Subscription screen, which you can see in Figure 2.
[Click on image for larger view.] Figure 2: The Configure Alert Subscription screen lets you tell AWS how you want to be notified when an anomaly occurs. (source: AWS).
As you can see in the figure, alert subscriptions have at least some similarities to budgets in that you will need to specify a threshold dollar amount. The idea behind this however, is that you probably don't want AWS to send you alerts for very minor anomalies.
Besides entering a dollar amount, you will also need to choose the alerting frequency. You can choose to receive a daily or weekly summary of anomalous spending that exceeded your predefined threshold, or you can choose to be notified for each individual incident.
Finally, you will need to choose how you want to be alerted. You can receive alerts through Amazon SNS or through Amazon chatbots. When you are done, click the Create Monitor button to create the subscription.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.