paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 13, 2019

Getting Started with AWS EC2 and S3

Understanding compute and storage fundamentals on AWS, from instance provisioning to bucket design patterns

Introduction

Amazon Web Services popularized the idea that compute and storage could be rented by the hour, provisioned through an API, and torn down just as quickly as they were created. Two services sit at the center of that shift: Elastic Compute Cloud (EC2) and Simple Storage Service (S3). Nearly every AWS architecture, no matter how sophisticated it eventually becomes with managed databases, container orchestration, or serverless functions, traces its lineage back to these two primitives. Understanding them well is not a beginner's exercise to be rushed through - it's foundational knowledge that informs decisions you'll make for years.

This guide is written for engineers who already understand general backend development and distributed systems concepts but want a grounded, practical introduction to EC2 and S3 specifically. Rather than walking through console screenshots, we'll focus on the underlying architecture, the trade-offs embedded in common configuration choices, and code examples that reflect how these services are actually used in production systems - through infrastructure-as-code and SDKs rather than manual clicking.

Context: Why EC2 and S3 Exist and What Problems They Solve

Before cloud computing, provisioning a server meant procurement cycles measured in weeks: ordering hardware, racking it, installing an OS, and configuring networking. Capacity planning was a guessing game with expensive consequences in both directions - overprovision and you waste capital, underprovision and you can't handle a traffic spike. EC2 collapsed that timeline to minutes by virtualizing compute and exposing it through an API. You request an instance with a given CPU, memory, and network profile, and it exists moments later, billed by the second or hour depending on the pricing model.

S3 solved a parallel but distinct problem: durable, available storage for unstructured data at a scale that no single filesystem or disk array could reasonably provide. Traditional storage systems required you to think about disk capacity, RAID configurations, and backup schedules. S3 abstracts all of that away behind a simple HTTP API: you put objects in, you get objects out, and Amazon handles replication, durability, and scaling transparently. The service is designed for eleven nines of durability (99.999999999%) for objects stored across multiple facilities, a figure AWS has published consistently in its S3 documentation.

Together, EC2 and S3 represent the two foundational categories of cloud infrastructure - compute and storage - decoupled from each other and from physical hardware. This decoupling is itself an architectural principle worth internalizing: stateless compute paired with durable external storage is a pattern you'll see repeated throughout distributed systems design, well beyond AWS.

Deep Technical Explanation

EC2 Architecture Fundamentals

An EC2 instance is a virtual machine running on AWS's underlying hypervisor infrastructure, historically Xen-based and increasingly migrated to AWS's own Nitro hypervisor. Instances are grouped into families optimized for different workload shapes: general purpose (the t and m series), compute-optimized (c series), memory-optimized (r and x series), and storage-optimized (i and d series). Choosing a family is less about raw benchmarks and more about matching the ratio of vCPU to memory to your actual workload - a cache-heavy service wants more memory per core, while a batch processing job might want raw CPU throughput.

Instances are launched from Amazon Machine Images (AMIs), which are snapshots of a root volume plus metadata describing the OS, architecture, and any pre-installed software. AWS publishes official AMIs for common Linux distributions and Windows Server, but production teams typically build custom AMIs baked with their application dependencies using tools like Packer, reducing cold-start configuration time. Storage attaches via Elastic Block Store (EBS), a network-attached block storage service that persists independently of the instance lifecycle - an important distinction from instance store volumes, which are physically attached and ephemeral, disappearing if the instance stops.

Networking is governed by Virtual Private Cloud (VPC) constructs: subnets, route tables, security groups, and network ACLs. Security groups act as stateful virtual firewalls at the instance level, while NACLs apply stateless rules at the subnet boundary. This layered model gives you fine-grained control over traffic flow, which matters significantly once you're running anything beyond a single public-facing server.

S3 Architecture Fundamentals

S3 organizes data into buckets, which are globally namespaced containers, and objects, which are the actual data blobs stored within them along with metadata and a unique key. Unlike a traditional filesystem, S3 has no real concept of directories - the "folder" structure you see in the console is a visual convenience built from delimiters in object key names, typically forward slashes. This matters for performance: S3 request rates scale well because of how keys are partitioned internally, but predictable prefix patterns (like sequential timestamps) can create hot partitions if you're not careful at very high throughput.

S3 offers multiple storage classes trading off cost against retrieval latency and frequency: S3 Standard for frequently accessed data, S3 Standard-IA and One Zone-IA for infrequent access, S3 Glacier and Glacier Deep Archive for long-term archival, and S3 Intelligent-Tiering, which automatically moves objects between tiers based on access patterns. Lifecycle policies let you codify transitions between these classes automatically, which is often the single highest-leverage cost optimization available to teams storing large volumes of data.

Implementation and Practical Examples

Provisioning EC2 and S3 manually through the console is fine for experimentation, but production systems should define infrastructure as code. Here's a Terraform example provisioning an EC2 instance with an IAM role granting scoped access to a specific S3 bucket - a pattern you'll use constantly:

resource "aws_s3_bucket" "app_data" {
  bucket = "my-company-app-data-prod"
}

resource "aws_iam_role" "ec2_s3_access" {
  name = "ec2-s3-access-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy" "s3_read_write" {
  name = "s3-scoped-access"
  role = aws_iam_role.ec2_s3_access.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:GetObject", "s3:PutObject", "s3:ListBucket"]
      Resource = [
        aws_s3_bucket.app_data.arn,
        "${aws_s3_bucket.app_data.arn}/*"
      ]
    }]
  })
}

resource "aws_iam_instance_profile" "ec2_profile" {
  name = "ec2-s3-instance-profile"
  role = aws_iam_role.ec2_s3_access.name
}

resource "aws_instance" "app_server" {
  ami                  = "ami-0c55b159cbfafe1f0"
  instance_type        = "t3.medium"
  iam_instance_profile = aws_iam_instance_profile.ec2_profile.name
  subnet_id            = aws_subnet.private.id

  tags = {
    Name = "app-server"
  }
}

Notice that the IAM policy scopes permissions to a single bucket ARN rather than granting broad s3:* access - this is the principle of least privilege applied concretely, and it's the single most common security gap in real-world AWS deployments. Attaching the role via an instance profile means the application running on the EC2 instance can call S3 without ever handling long-lived access keys, since the AWS SDK automatically retrieves temporary credentials from the instance metadata service.

On the application side, interacting with S3 from Python using boto3 looks like this in a realistic data-processing context:

import boto3
from botocore.exceptions import ClientError
import json
from datetime import datetime

s3_client = boto3.client("s3")
BUCKET_NAME = "my-company-app-data-prod"

def upload_processed_record(record: dict, partition_date: datetime) -> str:
    """Upload a processed record with a partitioned key for efficient querying."""
    key = (
        f"processed/year={partition_date.year}/"
        f"month={partition_date.month:02d}/"
        f"day={partition_date.day:02d}/"
        f"{record['id']}.json"
    )
    try:
        s3_client.put_object(
            Bucket=BUCKET_NAME,
            Key=key,
            Body=json.dumps(record),
            ContentType="application/json",
            ServerSideEncryption="AES256",
        )
        return key
    except ClientError as e:
        error_code = e.response["Error"]["Code"]
        raise RuntimeError(f"Failed to upload {key}: {error_code}") from e

def generate_presigned_download_url(key: str, expiration_seconds: int = 3600) -> str:
    """Generate a time-limited URL for a client to download an object directly."""
    return s3_client.generate_presigned_url(
        "get_object",
        Params={"Bucket": BUCKET_NAME, "Key": key},
        ExpiresIn=expiration_seconds,
    )

The partitioned key structure (year=/month=/day=) mirrors the Hive-style partitioning convention used by Athena and Glue, which means data landing in S3 this way becomes immediately queryable by analytics tools without a separate transformation step. The presigned URL pattern is equally important: it lets you grant temporary, scoped access to a private object without making the bucket public or routing large file transfers through your own application servers.

Trade-offs and Common Pitfalls

EC2's flexibility comes with an operational burden that's easy to underestimate early on. Unlike fully managed compute options, EC2 instances require you to think about patching cycles, OS-level security updates, and capacity planning. Autoscaling groups mitigate the capacity problem but introduce their own complexity: health checks need to accurately reflect application readiness, and scaling policies that react too aggressively to short-lived spikes can cause thrashing, where instances are repeatedly launched and terminated. Teams that skip this operational investment often end up running oversized fleets around the clock simply because right-sizing felt too risky to attempt.

S3's pitfalls tend to be less about operations and more about data governance and cost. Because buckets have no inherent size limit and uploads are so frictionless, it's common for organizations to accumulate years of unreferenced data with no lifecycle policy attached, quietly inflating storage bills. Public bucket misconfiguration has also been the root cause of numerous widely reported data exposure incidents - not because S3's default permissions are insecure (buckets are private by default), but because overly permissive bucket policies or ACLs get applied during debugging and never get rolled back. AWS's S3 Block Public Access setting, enabled by default on new buckets since 2023, exists specifically to prevent this class of mistake, but it can be explicitly disabled, which teams sometimes do without fully understanding the implications.

Cost attribution is a subtler shared pitfall across both services. EC2 costs are relatively easy to reason about per-instance, but data transfer charges - particularly cross-region or internet egress traffic from S3 - can surprise teams that haven't modeled them. A well-architected system minimizes unnecessary data movement, for instance by processing data in the same region it's stored, rather than treating network transfer as a free resource.

Best Practices

Start with the principle of least privilege for every IAM role and policy you write, scoping permissions to specific resources and actions rather than relying on managed policies with broad wildcard access. This single habit prevents the majority of security incidents that stem from compromised credentials, because a limited blast radius turns a potential catastrophe into a contained incident. Pair this with enabling AWS CloudTrail and S3 server access logging from day one; retrofitting audit logging after an incident has already occurred is far less useful than having it continuously running.

For EC2, treat instances as disposable and immutable where possible. Rather than SSHing into a running instance to apply a fix, bake the fix into a new AMI or container image and replace the instance through your deployment pipeline. This discipline, often summarized as "cattle, not pets," makes autoscaling and disaster recovery dramatically simpler because you never depend on the specific state of any one running machine.

For S3, apply lifecycle policies as a matter of course rather than an afterthought, and enable versioning on buckets containing anything you can't easily regenerate. Versioning combined with a lifecycle rule that expires old versions after a reasonable window gives you protection against accidental deletion or overwrite without unbounded storage growth. Finally, enable default encryption at the bucket level (SSE-S3 or SSE-KMS) so you don't rely on every application developer remembering to set encryption headers on every request.

Key Takeaways

Conclusion

EC2 and S3 are deceptively simple on the surface - launch a server, store a file - but the depth of configuration options underneath reflects decades of accumulated operational wisdom about running infrastructure at scale. Engineers who take the time to understand the architectural reasoning behind IAM scoping, storage class design, and network boundaries end up building systems that are cheaper, more secure, and easier to operate than those who treat these services as black boxes to be clicked through once and forgotten.

The patterns covered here - scoped IAM roles, infrastructure as code, partitioned storage keys, lifecycle automation - aren't AWS-specific tricks so much as general distributed systems principles expressed through a particular vendor's API. That's ultimately the most useful way to internalize this material: not as a list of AWS features to memorize, but as a concrete implementation of ideas about compute, storage, and security that apply wherever you build systems next.

References