Verification & Testing

Confirm your deployment and run first commands

Deployment Checklist

Before testing commands, verify each component:

AWS Infrastructure

  • CloudFormation stack shows CREATE_COMPLETE
  • ECS service has 1 running task
  • API Gateway endpoint responds to health check
  • CloudWatch logs are being created
  • Secrets Manager contains credentials

Microsoft Teams (if using Teams)

  • Azure Bot shows Teams channel as Running
  • Webhook URL is configured correctly
  • Teams app is installed
  • Bot appears in Teams chat/channel

Google Chat (if using Google Chat)

  • Google Chat API enabled and configured
  • Service account credentials stored in CloudFormation
  • Chat app appears in Google Chat search
  • Bot is added to a space or direct message

Bedrock

  • Claude Sonnet 4.5 shows Access granted
  • Model is available in deployment region

Your First Commands

The commands below work in both Microsoft Teams and Google Chat. In Teams, use @Ohlala SmartOps to mention the bot. In Google Chat direct messages, you can type commands directly without mentioning.

1. Test Connection

Teams: @Ohlala SmartOps hello

Google Chat (direct message): hello

Expected Response: Friendly greeting confirming the bot is working

2. Get Help

Teams: @Ohlala SmartOps help

Google Chat: /help

Expected Response: Interactive card with available commands and examples

3. Check Instance Status

show me my EC2 instances

Expected Response: List of your EC2 instances with status information

4. Health Report

/health

Expected Response: Detailed health metrics for your instances

5. Natural Language Query

which instances are running in us-east-1?

Expected Response: Filtered list based on your query

Advanced Testing

Test SSM Integration

@Ohlala SmartOps check disk space on i-1234567890abcdef0

  • Verifies SSM command execution
  • Returns disk usage information

Test Cost Analysis

@Ohlala SmartOps analyze my EC2 costs

  • Checks CloudWatch metrics access
  • Provides cost optimization suggestions

Test Multi-Instance Commands

@Ohlala SmartOps show me all stopped instances

  • Tests filtering and analysis capabilities
  • Demonstrates natural language understanding

Monitoring Your Deployment

CloudWatch Metrics

Monitor key metrics in CloudWatch:

  1. ECS Service

    • CPU utilization (should be <50%)
    • Memory utilization (should be <70%)
    • Task count (should be 1)

  2. API Gateway

    • Request count
    • 4XX/5XX errors (should be minimal)
    • Latency (should be <3 seconds)

  3. Bedrock Usage

    • Token consumption
    • API throttling events
    • Model invocation errors

Common Issues & Solutions

Issue: Bot Not Responding

Quick Diagnosis:

# Check health endpoint
curl https://your-api.execute-api.region.amazonaws.com/prod-stackname/health

Solutions:

  1. Check ECS task is running
  2. For Teams: Verify webhook URL in Azure Bot configuration
  3. For Google Chat: Verify webhook URL in Chat API Configuration
  4. Ensure the app is installed correctly
  5. Review CloudWatch logs

Issue: “Model Access Required” Error

Symptom: Bot responds but shows Bedrock error

Solution:

  1. Go to Bedrock Console then Model access
  2. Enable Claude Sonnet 4.5
  3. Wait for “Access granted”
  4. Retry command (no restart needed)

Issue: No Instances Found

Symptom: Bot works but doesn’t see EC2 instances

Checks:

  1. Instances are in same region as deployment
  2. Instances have SSM agent installed
  3. IAM permissions are correct
  4. Try: list all instances in all regions

Issue: Commands Timeout

Symptom: Bot shows “thinking” but never responds

Solutions:

  1. Check ECS task memory/CPU
  2. Look for Bedrock throttling
  3. Verify network connectivity
  4. Scale ECS service if needed

Issue: Teams Authentication Failures

Symptom: 401/403 errors in logs for Teams requests

Solutions:

  1. Regenerate Azure Bot credentials
  2. Update Secrets Manager
  3. Restart ECS service
  4. Check tenant ID is correct

Issue: Google Chat Authentication Failures

Symptom: 401/403 errors in logs for Google Chat requests

Solutions:

  1. Verify service account JSON is correctly formatted (single line)
  2. Check the service account has the correct permissions
  3. Verify the Project ID matches the service account
  4. Check the Authentication Audience setting in Chat API Configuration

Best Practices

  1. Start simple: Use basic commands first
  2. Be specific: Include instance IDs for targeted actions
  3. Use natural language: The bot understands context
  4. Review suggestions: Always verify before applying changes

Success Indicators

Your deployment is successful when:

  • Bot responds within 2-3 seconds
  • All test commands work
  • No errors in CloudWatch logs
  • Costs align with expectations
  • Team members can use the bot

Next Steps

Now that your bot is working:

  1. Explore Features

    • Try advanced commands
    • Experiment with natural language queries
    • Review health and cost reports

  2. Train Your Team

    • Share the bot with team members
    • Create usage guidelines
    • Document common workflows

Getting Help

If you encounter issues:

  1. Check Documentation

  2. Contact Support

  3. Community Resources

Congratulations

You’ve successfully deployed Ohlala SmartOps. Your AI-powered infrastructure assistant is ready to help manage your AWS environment through natural language conversations in Microsoft Teams, Google Chat, or both.

Happy automating!