AWS Secrets Manager Rotation

Introduction¶

Secrets Manager can rotate a secret automatically — generate a new value, update the target system, verify it works, and promote it — without your app noticing. The mechanism is a rotation Lambda and a set of version staging labels.

Staging labels¶

A secret has multiple versions, each tagged with labels:

• AWSCURRENT — the version apps get by default (GetSecretValue with no version).
• AWSPENDING — the new version being rotated in, not yet live.
• AWSPREVIOUS — the prior AWSCURRENT, kept for rollback / in-flight clients.

Rotation moves AWSPENDING → AWSCURRENT atomically at the end.

The four steps¶

The rotation Lambda is invoked once per step with {SecretId, ClientRequestToken, Step}:

1. createSecret — generate a new secret value (e.g. GetRandomPassword), store it as a new version labelled AWSPENDING. Idempotent — if AWSPENDING already exists, do nothing.
2. setSecret — apply the pending value to the target system (change the DB user's password, update the API key at the provider). This is the only step that touches the external system.
3. testSecret — verify the pending value works (connect to the DB with it, call the API). Fail here and rotation stops with the old secret still AWSCURRENT — safe.
4. finishSecret — move the AWSCURRENT label to the pending version. Rotation complete.

AWS provides ready-made rotation Lambdas for RDS/Aurora, Redshift, and DocumentDB; for anything else you write one (templates in the Serverless Application Repository).

Single-user vs multi-user (databases)¶

• Single-user — the rotation Lambda changes the same user's password. Simple, but there's a brief window where connections using the old password fail until the app re-reads the secret. Use AWSPREVIOUS grace and pooled reconnection to smooth it.
• Multi-user (alternating) — you have two DB users (app_a, app_b). Rotation alternates: while app_a is live, it resets app_b's password and switches AWSCURRENT to app_b. Zero failed connections — the currently- live user is never touched. Preferred for production databases. Requires a "master" secret with privileges to alter both users.

Configuring rotation¶

aws secretsmanager rotate-secret \
  --secret-id prod/db/app \
  --rotation-lambda-arn arn:aws:lambda:eu-west-1:123:function:SecretsManagerRDSPostgreSQLRotationMultiUser \
  --rotation-rules '{"ScheduleExpression": "rate(30 days)", "Duration": "2h"}'

The Lambda needs: secretsmanager:GetSecretValue/PutSecretValue/ UpdateSecretVersionStage/DescribeSecret on the secret, network access to the target (VPC config + security group to the DB), and permission to read the master secret.

The app side¶

Apps should fetch the secret at startup and cache it with a TTL (or use the Secrets Manager Lambda extension / caching client), and re-fetch on an auth error before failing. Don't fetch on every request (cost + latency); don't cache forever (you'll miss the rotation).

Verification and troubleshooting¶

aws secretsmanager describe-secret --secret-id prod/db/app \
  --query '{Rotation:RotationEnabled, Last:LastRotatedDate, Versions:VersionIdsToStages}'
aws secretsmanager get-secret-value --secret-id prod/db/app --version-stage AWSPENDING
aws logs tail /aws/lambda/<rotation-fn> --follow

• Rotation stuck, AWSPENDING exists but never promotes — testSecret failed (Lambda can't reach the DB — VPC/SG/subnet, or the new password wasn't actually applied), or setSecret errored. Lambda logs have the step and error. Fix, then rotate-secret again (it resumes from AWSPENDING).
• Secrets Manager can't find the specified secret in the Lambda — the Lambda's resource policy / IAM doesn't grant it access to this secret, or a KMS Decrypt permission on the secret's key is missing.
• Apps get auth failures during/after rotation — single-user rotation window; move to multi-user, or make the app re-read on auth error, or use the caching client with a short TTL.
• InvalidRequestException: You can't create this secret because a rotation is in progress — a previous rotation didn't finish. describe-secret shows VersionIdsToStages; if AWSPENDING is orphaned, cancel and re-trigger.
• Multi-user rotation: "current credentials are the same as pending" — the alternating-users logic can't find the second user, or the master secret lacks ALTER privileges on it.
• Rotation Lambda times out — default 30s may be short for a slow DB connection through a VPC; raise the timeout, and check it's not waiting on a NAT/endpoint that isn't there.
• KMS AccessDeniedException on GetSecretValue from the app — the app's role needs kms:Decrypt on the secret's CMK (if not the AWS-managed key), and the key policy must allow it.

Related tools and reading¶

• On-site: JSON formatter, Password generator.
• Related posts: AWS IAM policy evaluation logic, Handling secrets in CI without leaking them.