Skills operations

Skills Operations

Deployment, secret-readiness, collector invocation, smoke-check, and rollback runbook.

Skills Operations

Last updated: 2026-08-21

Use this runbook to deploy Skills, confirm data collection readiness, and avoid leaking secret material. Development is the normal validation environment; production serves explicitly approved releases.

Safety Rules

Never print PATs, passwords, Cognito tokens, .env values, or raw deployment logs that could contain credentials.
Treat AWS Secrets Manager values as secret-bearing even when checking whether a secret is ready.
Enable collector schedules only when the target environment has an approved Azure DevOps PAT secret.
Require explicit approval before changing production. Confirm the current stack parameters and preserve or deliberately change each schedule state and expression.

Safe Secret Readiness Check

This pattern checks whether a secret still has the scaffold replacement marker without printing the secret:

aws secretsmanager get-secret-value \
  --secret-id "$SKILLS_PAT_SECRET_ID" \
  --query SecretString \
  --output text |
node -e 'let input="";process.stdin.on("data",(c)=>input+=c);process.stdin.on("end",()=>{process.stdout.write(input.includes("REPLACE_WITH_APPROVED_AZURE_DEVOPS_PAT")?"placeholder\n":"non-placeholder\n")})'

Use AWS_PROFILE and AWS_REGION for the intended account before running the check.

Deploy

Development deploy with collector schedules enabled requires the existing approved dev secret ARN:

AWS_PROFILE=iDPCC-DEV-New \
AWS_REGION=us-east-1 \
STACK_NAME=skillz-idpcc-dev \
ENVIRONMENT_NAME=skills-idpcc \
COGNITO_DOMAIN_PREFIX=skills-idpcc-dev-852507783007 \
AZURE_DEVOPS_PAT_SECRET_ARN="$DEV_SKILLS_PAT_SECRET_ARN" \
DASHBOARD_DOMAIN_NAME=skills.idpcc.ceirr-network-dev.org \
DASHBOARD_HOSTED_ZONE_ID=Z05416731M9AYVKD2JHC \
scripts/deploy-skills-aws.sh

The deploy script defaults to the existing development CloudFormation stack.

For an approved production release, use the existing production stack and secret reference and pass the live schedule settings explicitly:

AWS_PROFILE=iDPCC-PROD-PowerUser \
AWS_REGION=us-east-1 \
STACK_NAME=skillz-idpcc-prod \
ENVIRONMENT_NAME=skills-idpcc-prod \
COGNITO_DOMAIN_PREFIX=skills-idpcc-prod-643366142385 \
AZURE_DEVOPS_PAT_SECRET_ARN="$PROD_SKILLS_PAT_SECRET_ARN" \
DASHBOARD_DOMAIN_NAME=skills.idpcc.ceirr-network.org \
DASHBOARD_HOSTED_ZONE_ID=Z03823772NH0I14KS76J9 \
COLLECTOR_SCHEDULE_EXPRESSION='rate(15 minutes)' \
COLLECTOR_SCHEDULE_STATE=ENABLED \
RECOMMENDATION_SCHEDULE_EXPRESSION='rate(1 day)' \
RECOMMENDATION_SCHEDULE_STATE=ENABLED \
scripts/deploy-skills-aws.sh

Use the full deploy script when the current site depends on newer API routes or Lambda behavior. A static-only sync is safe only when the deployed backend contract already matches the site.

Manual Collector Run

After a development deploy with an approved secret, invoke the collector once before trusting the schedule:

aws lambda invoke \
  --function-name "$SKILLS_COLLECTOR_FUNCTION_NAME" \
  --payload '{}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/skills-collector-result.json

Then run the recommendation function:

aws lambda invoke \
  --function-name "$SKILLS_RECOMMENDATION_FUNCTION_NAME" \
  --payload '{}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/skills-recommendation-result.json

The Lambda responses should contain counts and status fields only. If an invocation fails because of source access or secret readiness, disable schedules before finishing the ship.

Smoke Checks

Run these checks after each deploy:

curl -fsSI https://skills.idpcc.ceirr-network-dev.org/
curl -fsS https://skills.idpcc.ceirr-network-dev.org/data/skills.json
curl -fsSI https://skills.idpcc.ceirr-network.org/
curl -fsS https://skills.idpcc.ceirr-network.org/data/skills.json

When the CloudFront default distribution domain is known, confirm it redirects to the meaningful hostname:

curl -sI "https://$SKILLS_CLOUDFRONT_DOMAIN/index.html?host-check=1" | sed -n '1,8p'

Expected result: 301 with a Location header on the meaningful hostname for the target environment.

Rollback

Static-site rollback: redeploy from the previous merge commit and invalidate CloudFront.
Collector safety rollback: keep the existing AZURE_DEVOPS_PAT_SECRET_ARN and redeploy with COLLECTOR_SCHEDULE_STATE=DISABLED and RECOMMENDATION_SCHEDULE_STATE=DISABLED.
CloudFront hostname issue: remove the custom-domain parameters only for an emergency test stack; active environments should normally stay on their meaningful URLs.

Record every rollback or schedule-state change in Skills Status.

Note: the CloudFormation stack names still carry the original pre-rename spelling so existing Cognito users, DynamoDB data, and CloudFront resources are updated in place instead of recreated. Keep those names in operator commands, not product-facing copy.