Spacelift

This page documents how the AWS/IaC practice lab actually got built — two working environments, run through Spacelift, with real gotchas along the way. See also: How to Use Spacelift for the step-by-step version, and MAC, the agent that did most of the hands-on work.

Watch it happen

Brett sketches the architecture, MAC turns it into OpenTofu config, Spacelift plans it and waits for a human to confirm before anything touches AWS. That's the whole loop:

Why it exists

Practice, ahead of an interview with Spacelift itself. The goal wasn't a tutorial-grade demo — it was a real AWS account, real infrastructure, real failure modes, so the mistakes happened here instead of live.

The two labs

Orbit Labs (two-stack)

Two independent Spacelift stacks, one Git repo each, linked by a Spacelift-level stack dependency rather than a Terraform data source:

• Networking stack — VPC (10.0.0.0/16), one public subnet, an internet gateway, and a public route table. Outputs subnet_id.
• App stack — consumes subnet_id as var.subnet_id. Builds a key pair, two security groups (web: 80/443/22 from an admin-only CIDR; db: 3306 from the web SG only), a pinned-AMI web fleet across prod/dev/test environments, and one dedicated database instance.

The app stack cannot plan until the networking stack has applied and produced that output — enforced on Spacelift's Dependencies tab, not just documented convention.

Pulsar (single-instance)

A simpler, single self-contained stack built later specifically to avoid the stack-dependency wiring above for quick experiments: one VPC, one t3.micro/t3.small instance, no cross-stack linkage. 443 open to the world by design (for testing real internet-facing TLS); 22 restricted to an admin IP. An Elastic IP keeps the address stable across stop/start and resize cycles.

Build order

1. Push the networking/base stack first; wait for FINISHED.
2. Push the app stack; it fails outright with a missing-variable error if pushed before the dependency exists.
3. Poll with spacectl, not the browser — the web UI lags behind real state.
4. Pull outputs (spacectl stack outputs --id <stack-id>) for live IPs once finished.

What actually broke

• AMI drift wiped a fleet. A data "aws_ami" { most_recent = true } lookup meant a routine, unrelated push replaced the entire web fleet the moment AWS published a new image — new IPs, and any state that only lived on local disk was gone. Fix: pin the AMI as a fixed-default variable, bump it deliberately.
• The AWS console reads a literal Name tag. Resources tagged with lowercase name are valid but invisible in the console's Name column. Every resource needs Name = "...", capital N.
• Deleting a stack doesn't clean up its dependency link. Emptying the networking stack's config to tear it down left the app stack's dependency reference dangling — the next run on either stack failed with a vague missing-input error until the link was removed and, later, explicitly re-added before rebuilding.
• Free-tier restrictions fail mid-operation. Requesting a t3.medium resize got rejected by a FreeTierRestrictionError — but only after AWS had already stopped the instance to attempt the resize, leaving it stopped (and, without an Elastic IP already attached, on a fresh IP).
• Uploaded-file PHP execution was silently broken. Because the vhost proxies every *.php request straight to PHP-FPM at the server-config level, MediaWiki's and WordPress's default "disable script execution in the uploads directory" .htaccess protection didn't actually apply — confirmed by dropping a harmless .php test file in wp-content/uploads/ and watching it execute. Fixed with an explicit SetHandler none + Require all denied override in both upload directories.

The pattern that's worth keeping

Instance power state as a declarative resource, not a manual CLI step:

resource "aws_ec2_instance_state" "web" {
  instance_id = aws_instance.web.id
  state       = "stopped"  # flip to "running", push, confirm
}

Flip the string, push, Spacelift applies it. No compute billing while stopped; the Elastic IP means the address survives the whole cycle.

Why Spacelift specifically

• Plan and apply are structurally separate, with a mandatory human confirm gate before anything touches AWS.
• Stack dependencies are a first-class, enforced object — not a README instruction to run things in order.
• Full run history via spacectl stack run list — every change, when, and why, queryable later.
• The automation API key is scoped read-only by design; it hard-fails on anything that tries to trigger a deploy. Read access for tooling and write access for a human are two separate credentials.

That last point is the one that actually matters most: it's the reason handing real, live infrastructure work to MAC was something to trust rather than something to worry about.