Skip to main content

Amazon CloudWatch

1. Introduction​

Amazon CloudWatch is AWS’s central monitoring and observability service that collects real‑time data from AWS resources, on‑premises systems, and even other clouds. It offers you the ability to:

  • Monitor performance: Track metrics (like CPU Utilization, NetworkIn, etc.) for every AWS service.
  • Collect logs: Centralize application and system logs from a variety of sources.
  • Set alarms: Automatically trigger notifications or actions when specified thresholds are exceeded.
  • Visualize data: Build dashboards that display metrics in different visual formats (line, stacked area, pie charts, numbers) with customizable time ranges and the ability to export data (e.g., as CSV).
  • Automate responses: Integrate with services such as Auto Scaling, Lambda, SNS, and even recover EC2 instances automatically.

CloudWatch is designed to provide deep operational insight—from troubleshooting basic performance issues to enabling sophisticated automation and security monitoring.

2. Foundational Concepts​

Understanding these core concepts is essential before setting up or optimizing CloudWatch.

2.1 Namespaces​

  • What They Are: Containers that group metrics by service (e.g., AWS/EC2, AWS/Lambda) or by custom application.
  • Key Point: Every metric must be associated with a namespace to prevent accidental aggregation. Custom metrics you publish will reside in your chosen namespace.

2.2 Dimensions​

  • Definition: Name/value pairs used to further refine and identify a metric (for example, an EC2 metric might include dimensions like InstanceId or Environment).
  • Limits: You can use up to 30 dimensions per metric.
  • Usage: Dimensions let you filter, group, and drill down into metric data.

2.3 Resolution and Detailed Monitoring​

  • Standard Resolution: Metrics are typically recorded with 1‑minute granularity.
  • High Resolution: For more granular insights, metrics can be recorded down to 1 second (or at 5, 10, or 30‑second intervals for custom metrics).
  • EC2 Detailed Monitoring:
    • Default Behavior: EC2 instances emit metrics every 5 minutes.
    • Enhanced Monitoring: When detailed monitoring is enabled (an additional cost), metrics are pushed every 1 minute—up to 10 detailed monitoring metrics per instance—allowing for faster detection and auto‑scaling responsiveness.
  • Note: Certain metrics (for example, memory usage) are not collected by default on EC2 and must be sent as custom metrics.

2.4. Aggregation and Periods​

  • Aggregation Functions: CloudWatch supports functions like Sum, Average, Minimum, Maximum, and percentiles.
  • Periods: The time interval over which data points are aggregated (e.g., 60, 300, or 3600 seconds).

2.5 Timestamps and Visualization​

  • Timestamps: Each metric data point is tagged with a timestamp.
  • Dashboards: CloudWatch dashboards allow you to filter by time (custom ranges, such as one month), view data points (as numbers, line charts, or stacked areas), and even download the data as CSV or share visualizations with others.

3. Basic Components and Setup​

This section details the core building blocks you will work with, including metrics, logs, alarms, agents, and encryption.

3.1 CloudWatch Metrics​

Metrics provide time‑ordered data about resource and application performance.

Standard vs. Custom Metrics​

  • Standard Metrics: Automatically provided by AWS services (e.g., CPU Utilization for EC2).
    • Example: EC2 metrics are pushed every 5 minutes by default unless detailed monitoring is enabled.
  • Custom Metrics:
    • Definition: User‑defined metrics that you publish using the PutMetricData API call.
    • Usage Details:
      • What You Can Track: Beyond default metrics, you can push application-specific data such as memory usage, disk activity, or even the number of active logins.
      • Custom Dimensions: You can add any number of attributes (like instance.id or environment.name), up to 30 dimensions per metric.
      • Resolution Options: Specify standard (60‑second intervals) or high resolution (1, 5, 10, or 30 seconds).
      • Timestamps Flexibility: You can push data points with timestamps up to two weeks in the past or two hours into the future—be sure your instance’s system time is correctly synchronized.
    • Visualizing Metrics: In the CloudWatch console, you can filter metrics by region, namespace, and dimensions. Choose different visual representations (line chart, stacked area, pie chart, or numerical display) and export data as CSV files.

3.2 CloudWatch Logs and Metric Filters​

CloudWatch Logs is used to store, analyze, and query log data.

Log Groups and Streams​

  • Log Groups: Logical groupings that typically represent an application or service. You also set log retention policies here (from 1 day to 10 years, or indefinitely).
  • Log Streams: Individual sequences of log events within a log group (for example, a specific log file or container instance).

Data Ingestion​

  • Sources:
    • Logs can be sent via the SDK, the older CloudWatch Logs Agent, or the newer CloudWatch Unified Agent.
    • Many AWS services automatically send logs (Elastic Beanstalk, ECS, Lambda, VPC Flow Logs, API Gateway, CloudTrail, and Route 53).

Querying and Exporting Logs​

  • CloudWatch Logs Insights:
    • CloudWatch Logs Insights is an interactive, fully managed log analytics service within AWS CloudWatch that allows you to search, query, and analyze log data quickly and efficiently. Using its purpose-built query language, you can extract, aggregate, and visualize information from your logs to troubleshoot issues, monitor performance, or gain operational insights in near real-time. This tool is particularly useful for examining large volumes of log data across multiple log groups, enabling you to pinpoint problems, identify trends, and even create dashboards based on your query results.
    • Note: Logs Insights is designed for querying historical (not real‑time) data.
  • Export Options:
    • Batch Export: Use the CreateExportTask API call to export logs to Amazon S3 (this batch export may take up to 12 hours).
    • Real‑Time Streaming: Use subscription filters to stream logs in near real‑time to destinations like Kinesis Data Streams, Kinesis Data Firehose, Lambda, or Amazon OpenSearch.

Metric Filters​

Metric Filters in AWS CloudWatch are essentially rules you create to scan your log data for specific terms or patterns as it’s ingested into CloudWatch Logs. When a log event matches the filter pattern you’ve defined, CloudWatch converts that match into a metric data point. This lets you monitor your logs in a more actionable way—for example, you can count occurrences of the word “ERROR” and then use that count to trigger an alarm or display a graph.

Key points:

  • Pattern Matching: You define filter patterns (which can range from simple keywords to more complex expressions) to search through log events.
  • Metric Creation: Each time the pattern matches a log event, a specified metric is either created or incremented by a defined value.
  • Monitoring & Alerts: The resulting metrics can be visualized on CloudWatch dashboards or used to set alarms, helping you quickly react to issues like increased error rates or performance problems.
  • Value Extraction: Beyond simple counts, you can extract numerical values from logs (for instance, latency numbers) and publish those as metrics.

This process transforms raw log data into actionable insights, enabling proactive monitoring of application health and performance.

3.3 CloudWatch Alarms​

CloudWatch Alarms monitor your metrics and take actions when thresholds are breached.

Core Features​

  • States:
    • OK: Normal conditions.
    • INSUFFICIENT_DATA: Not enough data to determine state.
    • ALARM: Threshold breached.
  • Evaluation Periods:
    • Can be set as short as 10 or 30 seconds for high‑resolution metrics, or longer as needed.
  • Actions:
    • EC2 Actions: Stop, terminate, reboot, or recover instances.
    • Auto Scaling: Trigger scale‑out or scale‑in events.
    • SNS Notifications: Send alerts that can also trigger Lambda functions for custom automation.
  • Composite Alarms:
    • Combine multiple alarms using AND/OR logic to reduce noise.
    • Example: Monitor both high CPU and abnormal IOPS together so that an alert is only sent when both conditions are met.
  • EC2 Instance Recovery:
    • Uses status checks (instance, system, and attached EBS volumes) to automatically recover an instance on a different host while preserving its IP addresses, metadata, and placement groups.
  • Testing:
    • The CLI command set-alarm-states can be used to test alarms and verify that the correct actions are triggered.

3.4 CloudWatch Agents and Log Collection​

To collect metrics and logs from your own infrastructure (EC2 or on‑premises), you deploy agents.

Agent Options​

  • CloudWatch Logs Agent (Legacy):
    • Only sends log files to CloudWatch Logs.
  • CloudWatch Unified Agent (Recommended):
    • Collects both system‑level metrics and logs.
    • Provides more granular data such as:
      • CPU Metrics: Detailed breakdown (active, guest, idle, system, user, steal).
      • Disk Metrics: Free, used, total space, disk IO (reads/writes, bytes, IOPS).
      • Memory Metrics: Free, inactive, used, total, cached.
      • Network Metrics: Number of TCP/UDP connections, packets, and bytes.
      • Process Information: Counts of running, sleeping, blocked, or idle processes.
      • Swap Usage: Free, used, and percentage usage.
    • Centralized Configuration: Easily managed via SSM Parameter Store.
  • Troubleshooting Tips:
    • Check the agent’s JSON configuration file (including the specified region, debug mode, and run-as user).
    • Verify that the default namespace (typically CWAgent) is being used.
    • Ensure connectivity to AWS endpoints by reviewing security group rules, network ACLs, and IAM permissions.
    • Confirm that the system time is accurate and synchronized.

3.5 Data Security: CloudWatch Logs Encryption​

To protect sensitive log data, CloudWatch Logs supports encryption at the log group level using AWS KMS.

  • How It Works:
    • Encryption: By default, logs are encrypted. You can also specify your own Customer Managed Key (CMK).
    • Association Methods:
      • Existing Log Groups: Use the CLI/API with the associate-kms-key command.
      • New Log Groups: Create a log group with the create-log-group command that directly associates a CMK.
    • Important Consideration:
      • This association cannot be done via the CloudWatch console.
      • You must update the KMS key policy to grant CloudWatch Logs (for example, logs.eu-west-2.amazonaws.com) the necessary permissions (encrypt, decrypt, re‑encrypt, generate data key, describe).

4. Advanced Monitoring and Analysis​

With the basics in place, CloudWatch provides advanced features to enhance observability, user experience, and security.

4.1 CloudWatch Evidently​

CloudWatch Evidently is an AWS service that helps you experiment with and manage the rollout of new features in your applications. In essence, it provides a controlled way to perform A/B tests or feature flag evaluations by directing a fraction of your users to different feature variants and then measuring the impact on key metrics like error rates, performance, or conversion rates.

Key aspects include:

  • Experimentation: You can define experiments to test new features against your current baseline. Evidently collects data on user behavior and performance metrics so you can determine which variant works best.
  • Feature Management: Using feature flags, you can gradually roll out new functionality to a subset of users, monitor its performance, and then decide to roll it out to a larger audience or roll it back if needed.
  • Data-Driven Decisions: By integrating with CloudWatch’s monitoring and analytics, Evidently helps you make informed decisions based on real-time metrics and user feedback.
  • Integration: It works seamlessly with other CloudWatch services, allowing you to visualize experiment results on dashboards and set alarms based on the observed metrics.

Overall, CloudWatch Evidently empowers you to safely test and launch new features while minimizing risk and ensuring a smooth user experience.

4.2 CloudWatch Synthetics Canary​

CloudWatch Synthetics Canary is a feature within AWS CloudWatch Synthetics that enables you to create and run automated, script-based tests—called canaries—that simulate real user interactions with your APIs, websites, or applications. These canaries continuously monitor your endpoints from AWS-managed locations to verify availability, performance, and correct functionality. They can be scheduled to run at regular intervals, helping you detect issues before your customers experience them.

Functionality:

  • Canary Creation: You write canary scripts (in Node.js or Python) that reproduce typical user flows—such as browsing a product page, adding items to a cart, and completing checkout—to mimic real-world usage.

Key Features:

  • Scripted Monitoring: Leverage the flexibility of writing custom scripts in Node.js or Python to simulate complex customer interactions.
  • Headless Browser Access: Use a headless version of Google Chrome for visual testing, enabling the capture of screenshots and visual comparisons to baseline images.
  • Blueprints and Tools:
    • Heartbeat Monitor: Continuously verify endpoint availability and capture screenshots to monitor uptime.
    • API Canary: Test the read and write functions of REST APIs.
    • Broken Link Checker: Identify non‑functional links in your web applications.
    • Visual Monitoring: Automatically compare current screenshots against baseline images to detect unintended visual changes.
    • Canary Recorder & GUI Workflow Builder: Record user interactions and generate scripts automatically, simplifying the process of creating canaries.

Integration and Alerting:

  • Detailed Reporting: When a canary detects issues—such as failed transactions, slow responses, or errors—it logs detailed metrics and diagnostic information in CloudWatch.
  • Alarm Triggering: These failures can trigger CloudWatch alarms. For example, an alarm may invoke a Lambda function to perform automated remediation tasks, such as updating DNS records via Route 53 to reroute traffic to a healthy region.

Overall, CloudWatch Synthetics Canary combines scripted, automated testing with robust visual and functional monitoring capabilities. It enables proactive management of application health by simulating real user interactions, providing detailed insights, and integrating with AWS alarm systems for rapid response.

4.3 CloudWatch Contributor Insights​

CloudWatch Contributor Insights is a feature in AWS CloudWatch that analyzes log data to help you understand which factors—such as specific IP addresses, error codes, or request paths—are contributing most to the activity in your logs. It works by applying user-defined rules to aggregate and summarize log events in near real time. The insights generated allow you to quickly pinpoint trends, identify outliers, and troubleshoot issues by highlighting the “top contributors” behind spikes or anomalies in your system’s behavior.

Key aspects include:

  • Aggregation of Log Data: Contributor Insights automatically groups log events based on fields you specify (like status codes, API endpoints, or instance IDs), and then calculates metrics (e.g., counts or percentages) to show which contributors have the greatest impact.
  • Real-Time Analysis: The service processes your log data continuously, so you can monitor trends and shifts in your environment as they occur.
  • Visualization & Dashboards: The aggregated results are displayed as tables and graphs in CloudWatch dashboards, making it easier to spot performance bottlenecks or recurring issues.
  • Integration with CloudWatch Alarms: You can set up alarms based on the insights provided to trigger automated responses or notify your team when unexpected changes in the contributor patterns are detected.

By turning raw log data into actionable insights, CloudWatch Contributor Insights helps you better understand the root causes of operational issues and improves your ability to optimize and secure your applications.

4.4 CloudWatch Anomaly Detection​

CloudWatch Anomaly Detection is a feature within AWS CloudWatch that leverages machine learning to automatically analyze your metric data and detect unusual behavior. It works by building a dynamic model of the historical behavior of your metrics—taking into account seasonality, trends, and variations—to establish an expected range of values. When new data points fall outside of this expected range, the system flags these as anomalies, which can then trigger alarms or prompt further investigation.

Key features include:

  • Machine Learning-Based Analysis: Automatically models the normal behavior of your metrics without needing to manually set static thresholds.
  • Customizable Sensitivity: You can adjust the sensitivity of the anomaly detection model to fine-tune the balance between catching true anomalies and reducing false positives.
  • Integration with CloudWatch Alarms: Anomaly Detection can be embedded within CloudWatch Alarms, so that when a metric deviates from its modeled behavior, you’re alerted immediately.
  • Visual Insights: In the CloudWatch console, anomaly detection bands are overlaid on your metric graphs. These bands visually indicate the expected range, making it easier to spot when actual values stray into abnormal territory.
  • Proactive Monitoring: By continuously analyzing metric trends in near real-time, it enables you to detect performance degradation, unexpected traffic patterns, or operational issues before they impact your end users.

Overall, CloudWatch Anomaly Detection helps you maintain system health by automatically highlighting when something unusual is happening, enabling proactive troubleshooting and timely response.

4.5 Integrating CloudTrail with CloudWatch​

  • Stream CloudTrail logs (recording all API calls) into CloudWatch Logs.
  • Create metric filters to detect abnormal patterns (for example, if the StartInstances API call occurs more than 30 times within a defined period).
  • Trigger composite alarms to alert security teams when thresholds are exceeded, enhancing overall security monitoring.

5. Conclusion​

Bringing together all components and advanced features allows you to build a robust observability and automation strategy.

  • Unified Monitoring:
    • Use standard and custom metrics alongside log data to get a full picture of system performance.
    • Visualize data with customizable dashboards that support various chart types and export options.
  • Automation and Incident Response:
    • Combine CloudWatch Alarms (and composite alarms) with EC2 instance actions, auto‑scaling, and SNS/Lambda integrations to automate responses.
    • Test alarm setups with CLI commands to ensure that actions are triggered correctly.
  • Data Security and Compliance:
    • Secure your log data using KMS encryption, ensuring that proper key policies and permissions are in place.
  • Deep Analysis:
    • Leverage CloudWatch Logs Insights, Contributor Insights, and Anomaly Detection to drill down into performance issues and unusual behaviors.
  • Extended Integration:
    • Enhance CloudTrail’s monitoring capabilities by streaming API call data into CloudWatch and setting up targeted alarms.
  • Best Practices:
    • Regularly review and update agent configurations, IAM roles, and network settings.
    • Use centralized configuration via SSM for CloudWatch Unified Agents.
    • Verify that system time is synchronized and that your agents are reporting under the correct namespace (default is typically CWAgent).

By combining these elements—from fundamental metric collection and visualization to advanced automation and security integrations—you can tailor CloudWatch to meet the unique needs of your environment. For further details, consult the official AWS documentation and continuously refine your setup to optimize performance, reduce downtime, and enhance security.