AWS Core Services for DevOps: EC2, ECS, EKS, S3, Lambda

Introduction

AWS services for compute, storage, networking, and serverless workloads often work together in the same application architecture. Knowing what each service owns helps teams choose an appropriate deployment model and troubleshoot the pieces beneath higher-level platforms.

This guide covers multi-account foundations, EC2, ECS, EKS, S3, and Lambda, with operational trade-offs and common production failures. It also highlights identity, observability, and security considerations that shape day-to-day DevOps work.

AWS Multi-Account Architecture

AWS resources are deployed to specific geographic regions, and regions are independent of each other. Each region has multiple availability zones (AZs)—physically separate data centers with independent power, networking, and cooling. Deploying across multiple AZs protects against single-datacenter failures.

# List available regions
aws ec2 describe-regions --output table

# Get current region
aws configure get region

Account structure shapes your AWS environment. Organizations use consolidated billing to manage multiple accounts under a single payer. Common patterns include separate accounts per environment (dev, staging, production), per team, or per application domain.

Organization
├── Management Account (billing, SCPs)
├── Security Account (GuardDuty, Security Hub)
├── Dev Account
├── Staging Account
└── Production Account

Service Control Policies (SCPs) at the organization level restrict what can be done in member accounts. This enforces guardrails without managing IAM in every account.

flowchart TD
    A[AWS Organization] --> B[Management Account]
    A --> C[Security Account]
    A --> D[Dev Account]
    A --> E[Staging Account]
    A --> F[Production Account]
    C --> G[GuardDuty]
    C --> H[Security Hub]
    D --> I[Dev VPC]
    E --> J[Staging VPC]
    F --> K[Production VPC]
    K --> L[ALB]
    L --> M[EKS Cluster]
    M --> N[ECS Tasks]
    N --> O[S3 Artifacts]

EC2 Instance Types and ASGs

EC2 provides virtual machines in the cloud. Instance types determine the CPU, memory, storage, and networking capacity. The naming pattern is family, generation, and size—for example, t3.micro is a burstable general purpose instance, third generation, micro size.

# Launch an EC2 instance
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type t3.micro \
  --key-name my-key-pair \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0

# Describe instance status
aws ec2 describe-instance-status --instance-ids i-0abcdef1234567890

Auto Scaling Groups (ASGs) automatically adjust capacity based on demand. You define minimum, maximum, and desired capacity, along with scaling policies that trigger adjustments based on metrics like CPU utilization or request count.

# ASG CloudFormation snippet
AutoScalingGroup:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    MinSize: 2
    MaxSize: 10
    DesiredCapacity: 2
    VPCZoneIdentifier:
      - !Ref PrivateSubnet1
      - !Ref PrivateSubnet2
    LaunchTemplate:
      LaunchTemplateId: !Ref LaunchTemplate
      Version: !GetAtt LaunchTemplate.LatestVersionNumber
    TargetGroupARNs:
      - !Ref TargetGroup
    HealthCheckType: ELB
    HealthCheckGracePeriod: 300

ASGs work with Elastic Load Balancers to distribute traffic across healthy instances. The load balancer performs health checks and removes unhealthy instances from the rotation automatically.

ECS Task Definitions and Services

Amazon Elastic Container Service (ECS) manages Docker containers on a cluster of EC2 instances or using AWS Fargate serverless compute. Task definitions describe what containers to run and how much resources they need.

{
  "family": "webapp",
  "containerDefinitions": [
    {
      "name": "webapp",
      "image": "123456789.dkr.ecr.us-east-1.amazonaws.com/webapp:latest",
      "memory": 512,
      "cpu": 256,
      "essential": true,
      "portMappings": [
        {
          "containerPort": 8080,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/webapp",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

An ECS service maintains a desired count of task instances and automatically replaces failed tasks. It integrates with Application Load Balancers for traffic distribution and Auto Scaling for dynamic capacity adjustment.

# Register a new task definition revision
aws ecs register-task-definition --cli-input-json file://task-definition.json

# Update service to use new revision
aws ecs update-service \
  --cluster production \
  --service webapp \
  --task-definition webapp:2

Fargate removes the need to manage EC2 instances for container workloads. You specify CPU and memory requirements, and AWS handles the underlying infrastructure. This simplifies operations at the cost of less granular control over the compute environment.

EKS Cluster Management Basics

Amazon Elastic Kubernetes Service (EKS) provides a managed Kubernetes control plane. AWS handles the master nodes; you manage the worker nodes and workloads.

# Create an EKS cluster
aws eks create-cluster \
  --name production \
  --role-arn arn:aws:iam::123456789:role/eks-cluster-role \
  --resources-vpc-config subnetIds=subnet-0123456789abcdef0,subnet-0123456789abcdef1,securityGroupIds=sg-0123456789abcdef0

# Update kubeconfig
aws eks update-kubeconfig --name production

# Verify cluster access
kubectl get svc

EKS manages the Kubernetes control plane across multiple AZs for high availability. Worker nodes join the cluster via a node group, which can be managed by AWS (EKS Managed Node Groups) or self-managed.

# Node group configuration
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: production
  region: us-east-1

managedNodeGroups:
  - name: compute
    instanceType: t3.medium
    desiredCapacity: 3
    minSize: 2
    maxSize: 10
    volumeSize: 50
    ssh:
      allow: true

Kubernetes deployments, services, and ingresses work the same on EKS as on any Kubernetes cluster. For workload access to AWS services, configure IAM Roles for Service Accounts (IRSA) or EKS Pod Identity. IRSA uses the cluster’s OIDC provider and a narrowly scoped IAM role associated with a Kubernetes service account.

S3 for Artifact Storage

Amazon S3 stores objects in buckets. For DevOps, S3 typically holds build artifacts, deployment packages, and infrastructure state. S3 integrates with everything on AWS through IAM policies and resource-based bucket policies.

# Create a bucket for artifacts
aws s3 mb s3://my-app-artifacts --region us-east-1

# Upload a build artifact
aws s3 cp ./dist/app.tar.gz s3://my-app-artifacts/prod/

# List bucket contents
aws s3 ls s3://my-app-artifacts/prod/

# Enable versioning for artifact history
aws s3api put-bucket-versioning \
  --bucket my-app-artifacts \
  --versioning-configuration Status=Enabled

Lifecycle policies automate archival and deletion. Move old artifacts to cheaper storage classes automatically, or delete artifacts older than a retention period.

{
  "Rules": [
    {
      "ID": "ArchiveOldArtifacts",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "prod/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}

Lambda for Serverless Workloads

AWS Lambda runs code in response to events without provisioning servers. You pay only for the compute time consumed—billed in milliseconds. Lambda is ideal for event-driven tasks, API backends, and background processing.

// Lambda handler for processing S3 uploads
exports.handler = async (event) => {
  const s3Event = event.Records[0].s3;
  const bucket = s3Event.bucket.name;
  const key = decodeURIComponent(s3Event.object.key.replace(/\+/g, " "));

  console.log(`Processing file: ${bucket}/${key}`);

  // Process the file...
  const result = await processUpload(bucket, key);

  return {
    statusCode: 200,
    body: JSON.stringify({ result }),
  };
};

Lambda runs in a Lambda-managed VPC by default and can reach public AWS services and the internet. To connect to resources in your own VPC, such as an RDS database, configure the function with subnets and security groups from that VPC. A function attached to private subnets needs a NAT gateway for general internet access; use VPC endpoints when it only needs supported AWS services.

# Create a Lambda function
aws lambda create-function \
  --function-name my-processor \
  --runtime nodejs20.x \
  --role arn:aws:iam::123456789:role/lambda-execution-role \
  --handler index.handler \
  --zip-file fileb://function.zip \
  --vpc-config SubnetIds=subnet-0123456789abcdef0,SecurityGroupIds=sg-0123456789abcdef0

For more on managing AWS costs, see our post on Cost Optimization which covers EC2, Lambda, and S3 cost optimization strategies.

For more on securing AWS workloads, see Cloud Security for IAM best practices, VPC design, and encryption patterns, and Network Security for security groups, NACLs, and VPC endpoint configuration.

Trade-off Analysis

Production Failure Scenarios

AWS Observability Hooks

EC2 and ASG monitoring:

# Get EC2 instance metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abcdef1234567890 \
  --start-time 2026-03-24T00:00:00 \
  --end-time 2026-03-25T00:00:00 \
  --period 3600 \
  --statistics Average

# Check ASG health status
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names my-asg \
  --query 'AutoScalingGroups[0].Instances[*].[InstanceId,HealthStatus,LifeCycleState]'

ECS monitoring:

# Check service health and running task count
aws ecs describe-services \
  --cluster production \
  --services webapp \
  --query 'services[0].{runningCount:runningCount,desiredCount:desiredCount,pendingCount:pendingCount}'

EKS monitoring:

# Check node health and pod distribution
kubectl get nodes -o wide
kubectl get pods -o wide --all-namespaces | grep -v Running

# Get cluster control plane health
aws eks describe-cluster \
  --name production \
  --query 'cluster.{status:status,version:version,endpoint:endpoint}'

Key CloudWatch metrics to alert on:

Common Pitfalls / Anti-Patterns

Using the default VPC. The default VPC has permissive security groups and is shared across all accounts in a region. Production workloads should use dedicated VPCs with explicit networking controls.

Attaching IAM policies directly to users instead of roles. Direct user policies create credential management nightmares and are harder to audit. Always use IAM roles for EC2, Lambda, and other compute services.

Not configuring ASG health checks properly. If your health check is too lenient, unhealthy instances stay in service. If it is too strict, instances get replaced during legitimate load spikes. Match health check type to your application needs.

Storing secrets in object metadata or instance user data. Metadata and user data can be exposed to principals that can inspect the object or instance configuration, and may be copied into logs or support output. Put secrets in Secrets Manager or Systems Manager Parameter Store and grant workloads narrowly scoped access.

Attaching Lambda to a VPC without planning its routes. A VPC-attached function can reach only what its subnet routes, security groups, and network ACLs allow. Use private subnets for private resources, add a NAT gateway if the function needs general internet access, or use VPC endpoints for supported AWS services. Lambda creates and reuses managed Hyperplane ENIs for the subnet and security-group combination; first-time network setup can delay activation, but it is not a fixed per-invocation cold-start penalty.

Security and Compliance Notes

• Give each workload a dedicated least-privilege role: an instance profile for EC2, a task role for ECS (separate from its execution role), a workload identity such as IRSA for EKS pods, and an execution role for Lambda. Keep deployment permissions separate and restrict which roles a deployer can pass with iam:PassRole.
• Use organization-level guardrails for account boundaries, then verify resource settings in every account and region. Keep CloudTrail management events in a central security account with restricted deletion and retention; enable S3 data events for buckets that need object-level audit. Use AWS Config to record resource configuration history, and aggregate GuardDuty or Security Hub findings for review.
• For data that needs an audit trail or retention control, scope S3 bucket policies to approved roles and prefixes, enforce TLS and default encryption, and use a customer-managed KMS key when access to key use must be independently controlled or audited. Set lifecycle, versioning, and retention rules to match the data policy.
• Store credentials in Secrets Manager or Systems Manager Parameter Store and let the workload role retrieve only what it needs. Do not place secrets in user data, task definitions, images, or routine logs; redact sensitive fields before sending logs to CloudWatch or a central account.

Capacity Estimation and Benchmark Data

Use these numbers for initial capacity planning. Actual performance varies by workload characteristics.

EC2 Instance Type Families

Lambda Performance Parameters

S3 Performance Targets

Service Limits for Planning

AWS service quotas vary by account, region, and resource. Check the Service Quotas console or API for current values and request increases where supported.

Quick Recap Checklist

• EC2 gives full control but maximum operational burden; use for legacy workloads and specific hardware needs.
• ECS with Fargate removes EC2 management; best for teams wanting containers without Kubernetes complexity.
• EKS runs Kubernetes on AWS; choose it when the team needs Kubernetes APIs and accepts the added cluster operations.
• Lambda is ideal for event-driven, short-running workloads; not suitable for long processes or stateful operations.
• Multi-account AWS organizations enforce guardrails via SCPs and simplify billing tracking.

Interview Questions

Further Reading

• Cost Optimization - EC2, Lambda, and S3 cost optimization strategies
• Cloud Security - IAM best practices, VPC design, and encryption patterns
• Network Security - Security groups, NACLs, and VPC endpoint configuration
• AWS Whitepapers - Official architecture guides
• AWS Well-Architected Framework - Guidance for reviewing workload architecture
• Lambda VPC configuration - Networking behavior and Hyperplane ENIs
• Amazon EKS IRSA guide - Associate IAM roles with service accounts
• Amazon S3 performance guidance - Request scaling and workload patterns

Conclusion

AWS Onboarding Checklist

# 1. Create organization and enable SCPs
aws organizations create-organization
aws organizations enable-service-control-policy --service-principal ALL

# 2. Set up VPC for production
aws ec2 create-vpc --cidr-block 10.0.0.0/16
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24

# 3. Create ECS cluster with Fargate
aws ecs create-cluster --cluster-name production --capacity-providers FARGATE

# 4. Set up CloudWatch alarms for critical metrics
aws cloudwatch put-metric-alarm \
  --alarm-name EC2-High-CPU \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --threshold 80 \
  --period 300 \
  --evaluation-periods 1

# 5. Enable S3 versioning on artifact bucket
aws s3api put-bucket-versioning \
  --bucket my-artifacts \
  --versioning-configuration Status=Enabled