CloudWatch Logs and Metric Filters

Introduction¶

CloudWatch Logs collects log data into log groups (one per app/service), each containing log streams (one per instance/task/function). The value is in what you do with it: retention, metric filters, alarms, and queries.

Getting logs in¶

• Lambda — automatic, to /aws/lambda/<function>.
• ECS/Fargate — the awslogs log driver in the task definition, or awsfirelens (Fluent Bit) for routing/filtering.
• EC2 / on-prem — the CloudWatch agent (amazon-cloudwatch-agent), configured to tail files and also ship metrics.
• EKS — Fluent Bit DaemonSet (Container Insights) or the CloudWatch agent.
• Application SDK — PutLogEvents directly (rarely needed).

Emit structured JSON log lines — metric filters and Logs Insights parse them far better than free text.

Retention and cost¶

New log groups keep logs forever by default and you pay for storage every month. Set retention on every group:

aws logs put-retention-policy --log-group-name /aws/ecs/api --retention-in-days 30

Cost levers:

• Ingestion is the big one (~$0.50/GB). Don't log DEBUG in production, don't log full request/response bodies, sample high-volume events.
• Storage — set retention (7–30 days for app logs, longer only where required).
• Archive cold logs to S3 via a subscription filter + Firehose if you need long retention cheaply.
• Logs Insights queries are billed per GB scanned — narrow the time range and log group.

Metric filters — logs → metrics → alarms¶

A metric filter scans incoming log events for a pattern and increments a CloudWatch metric.

aws logs put-metric-filter \
  --log-group-name /aws/ecs/api \
  --filter-name 5xx-count \
  --filter-pattern '{ $.status >= 500 }' \
  --metric-transformations \
     metricName=Http5xx,metricNamespace=MyApp,metricValue=1,defaultValue=0

Filter pattern syntax:

• JSON: { $.level = "ERROR" }, { $.status >= 500 && $.path = "/api/*" }
• Space-delimited: [ip, id, user, timestamp, request, status_code=5*, size]
• Plain text: "ERROR" or ?"Exception" ?"panic"

Then alarm on the metric:

aws cloudwatch put-metric-alarm --alarm-name api-5xx \
  --namespace MyApp --metric-name Http5xx --statistic Sum --period 60 \
  --threshold 10 --comparison-operator GreaterThanThreshold \
  --evaluation-periods 5 --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:...:alerts

defaultValue=0 matters — without it, the metric has no data points when there are no errors, and the alarm can't evaluate.

Logs Insights — ad-hoc queries¶

fields @timestamp, @message, status, path, duration_ms
| filter status >= 500
| stats count() as errors by bin(5m)
| sort @timestamp desc
| limit 100

# Top slow endpoints
fields path, duration_ms
| filter ispresent(duration_ms)
| stats avg(duration_ms) as avg, pct(duration_ms, 99) as p99, count() as n by path
| sort p99 desc

Save frequent queries. Insights auto-discovers fields from JSON logs.

Subscription filters¶

Stream matching log events in near-real-time to:

• Lambda — custom processing / alerting.
• Kinesis Data Streams / Firehose — to S3, OpenSearch, or a third-party SIEM.
• Another account's log group (central logging).

Verification and troubleshooting¶

aws logs tail /aws/ecs/api --follow --format short
aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName'  # groups with no retention!
aws logs get-log-events --log-group-name ... --log-stream-name ...

• No logs appearing — the task/instance role lacks logs:CreateLogStream + logs:PutLogEvents (execution role for ECS), the log group doesn't exist and logs:CreateLogGroup isn't granted, or the agent isn't running / misconfigured path.
• Metric filter shows nothing — pattern doesn't match (test with the console "Test pattern"), logs aren't JSON so $.field fails, or the filter was created after the events (filters are not retroactive).
• Alarm stuck INSUFFICIENT_DATA — no data points because errors are rare and there's no defaultValue=0, or wrong namespace/dimensions.
• Surprise bill — a chatty log group with no retention, or DEBUG logging in prod, or a tight-loop error logging thousands of lines/sec. Find it: aws logs describe-log-groups --query 'reverse(sort_by(logGroups,&storedBytes))[:10]'.
• Logs Insights slow / expensive — scanning a huge group over a wide window. Narrow both; consider a metric filter for anything you query repeatedly.
• Cross-account logging not working — the destination needs a resource policy allowing the source account's logs service.

Related tools and reading¶

• On-site: JSON formatter, Timestamp converter.
• Related posts: Monitoring what actually matters, Alert fatigue and how to fix it.