Chapter 4. Trivy Image Workflow

1) Build → Scan → Decide (quarantine first)

Tag every fresh build as quarantine until it passes policy.

Makefile additions

QUAR := $(IMAGE_REPO):quarantine-$(SHA)
GOOD := $(IMAGE_REPO):good-$(TAG)

build:
	docker build -t $(IMAGE) app
	docker tag $(IMAGE) $(QUAR)   # mark as quarantine by default

sca:
	@echo "== Trivy (image scan) =="
	docker run --rm -v $(PWD):/workspace \
	  -v $(PWD)/.trivy-db:/root/.cache/trivy \
	  aquasec/trivy:latest image --ignore-unfixed \
	  --format json -o /workspace/artifacts/trivy-image.json $(QUAR) || true

We scan $(QUAR) specifically so it never accidentally ships before review.

2) Policy: when to promote

Day-1 default (tune later):

  • PASS if no CRITICAL vulnerabilities and no HIGH with a fix available in direct dependencies.
  • Else: stay in quarantine, open a ticket, or accept temporary risk with expiry.

Automate with a tiny shell gate:

sca-gate:
	@./scripts/sca_gate.sh artifacts/trivy-image.json

scripts/sca_gate.sh (example)

#!/usr/bin/env bash
set -euo pipefail
f="${1:-artifacts/trivy-image.json}"
[ -s "$f" ] || { echo "no trivy report"; exit 1; }
crit=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity=="CRITICAL")] | length' "$f")
high_fix=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity=="HIGH" and (.FixedVersion != null and .FixedVersion != ""))] | length' "$f")
if [ "$crit" -gt 0 ] || [ "$high_fix" -gt 0 ]; then
  echo "❌ Gate fail: CRITICAL=$crit HIGH(w/fix)=$high_fix"
  exit 2
else
  echo "✅ Gate pass"
fi

Make it executable:

chmod +x scripts/sca_gate.sh

3) Promote (or don’t)

If sca-gate passes, promote the image tag we scan:

promote:
	@echo "Promoting $(QUAR) -> $(GOOD)"
	@docker tag $(QUAR) $(GOOD)
	@docker tag $(GOOD) $(IMAGE_LATEST)

If it fails, do not promote. Patch deps, rebuild, and rescan.

4) Deploy to Minikube (from promoted image)

Even without a registry, Minikube can pull whatever you load into the node.

cd: mk-up build sca sca-gate promote mk-image-load k8s-ns k8s-apply k8s-verify
	@echo "✅ Deployed $(GOOD) to Minikube ($(PROFILE))"

Note: k8s-apply should reference $(IMAGE_REPO) and a tag variable; or use kubectl set image … $(GOOD) post-apply.

5) Evidence pack

Every run leaves:

  • artifacts/trivy-image.json (scan result)
  • artifacts/sbom.json (inventory)
  • artifacts/cosign-verify.txt (if signing enabled)
  • Optional: make evidence zips everything for sharing.

6) Optional: sign & verify before promotion

sign:
	@if [ -f cosign.key ]; then cosign sign --key cosign.key $(QUAR); else echo "cosign.key missing; skipping"; fi

verify:
	@if [ -f cosign.pub ]; then cosign verify --key cosign.pub $(QUAR) > artifacts/cosign-verify.txt || true; else echo "cosign.pub missing; skipping"; fi

# Promote only after verify (optional stronger gate)
promote: verify

End-to-end demo (copy/paste)

make build                  # builds + tags :quarantine-<sha>
make sca                    # writes artifacts/trivy-image.json
make sca-gate               # enforces CRITICAL/HIGH(w/fix) policy
make promote                # only if gate passes (tags :good-<tag> + :latest)
make cd                     # loads/promotes to Minikube & deploys
make k8s-port               # get the URL; then run make zap if desired
make evidence               # bundle artifacts into a dated zip

Checklist

  • Quarantine tag exists and is scanned by default
  • Gate script enforces a simple, explainable policy
  • Promote target only runs on gate pass
  • Deployed image equals the promoted tag
  • Evidence saved under ./artifacts/ (scan + SBOM + verify)

Maturity ladder

  • Bronze: scanning + quarantine pattern in place
  • Silver: promotion gated; deploys only promoted images
  • Gold: signing enabled; verify required before promotion
  • Platinum: monthly thresholds tighten (e.g., block HIGH regardless of fix), metrics tracked

Should Fail

Short answer: yes—this is expected with the policy we put in. Your zap-k8s depends on cd, and cd depends on sca-gate. Trivy found 1 HIGH with a fix available, so the gate failed by design and blocked deploy—therefore ZAP didn’t run.

What the log means

  • Built image → tagged :quarantine-dev
  • Trivy scanned :quarantine-dev and wrote artifacts/trivy-image.json
  • Gate rule: fail if any CRITICAL, or any HIGH that has a fix → ❌ Gate fail
  • Because zap-k8s calls cd first, deploy was blocked and ZAP never started

See which vuln tripped the gate

jq -r '
  .Results[]
  | .Vulnerabilities[]?
  | select(.Severity=="HIGH" and (.FixedVersion != null and .FixedVersion != ""))
  | "\(.VulnerabilityID) | \(.PkgName) \(.InstalledVersion) -> fix: \(.FixedVersion) | \(.Title)"
' artifacts/trivy-image.json

Your choices (in class)

A) Fix it (preferred)

Most likely it’s an Alpine base package. Try bumping the final stage base image (and rebuild):

  • If your final image is alpine:3.20@sha256:…, bump to the latest patch (or 3.21):
# In app/Dockerfile final stage:
FROM alpine:3.21

Then:

make build sca
make sca-gate     # should pass if the fix is in the newer base
make cd           # deploys; now zap-k8s will work too

Even better long term: use a minimal base (e.g., distroless nonroot) to shrink CVE surface.

B) Temporarily relax for the demo (skip gate just this run)

If you just need ZAP to run right now, you can bypass the gate once:

# Manually do what cd would do, but without the gate:
make build sca
make promote                       # promote quarantine tag anyway
make mk-image-load k8s-ns k8s-apply k8s-verify
# Now run ZAP without re-running cd (since zap-k8s calls cd)
# Option 1: port-forward + docker ZAP manually:
kubectl -n ship port-forward deploy/app 8080:3000 &
docker run --rm -t --add-host=host.docker.internal:host-gateway \
  -v "$PWD/tools/zap":/zap/wrk -w /zap/wrk \
  zaproxy/zap-stable:latest \
  zap-baseline.py -t http://host.docker.internal:8080 -m 1 -T 3 -s \
  -r report.html -J report.json

C) Add a controlled bypass switch (nice for teaching)

If you want a clean toggle:

Patch your Makefile:

SKIP_GATE ?= 0

sca-gate:
	@if [ "$(SKIP_GATE)" = "1" ]; then \
	  echo "⚠️  Skipping SCA gate (SKIP_GATE=1)"; \
	else \
	  ./scripts/sca_gate.sh artifacts/trivy-image.json; \
	fi

cd: mk-up build sca sca-gate promote mk-image-load k8s-ns k8s-apply k8s-verify

Use it once:

make cd SKIP_GATE=1
make zap-k8s SKIP_GATE=1

(Still produces the Trivy report so you can discuss it, but doesn’t block.)

Quick recap for students

  • The block is intentional: “quarantine → scan → gate → promote.”
  • Today it failed on HIGH (fix available). Either patch (best) or use SKIP_GATE=1 for the exercise, then come back and remediate.

If you paste the jq output for the HIGH finding (ID + package), I’ll suggest the exact Dockerfile or go.mod change to clear it.

Yep—this is exactly the kind of thing the gate is supposed to catch.

You’ve got a HIGH in the Go stdlib bundled into your binary:

CVE-2025-47907|stdlibv1.22.12fix:1.23.12,1.24.6|database/sql:PostgresScanRaceCondition

What it means (plain English)

  • Trivy inspects your gobinary and sees it was built with Go 1.22.12.
  • That Go toolchain version carries a HIGH vuln affecting database/sql (Postgres scanning race).
  • The fixed toolchains are Go 1.23.12 and Go 1.24.6 (no fixed 1.22 per your report).
  • Our gate fails on “HIGH with a fix available” → so it blocked deploy. Correct behavior.

Edit your builder image in app/Dockerfile to a patched Go:

- FROM golang:1.22-alpine AS build
+ FROM golang:1.24.6-alpine AS build

(If you prefer to be conservative: golang:1.23.12-alpine is also listed as fixed.)

Then rebuild/scan:

make build sca
make sca-gate      # should pass now
make cd            # deploys (then you can run make zap-k8s)

Optional but tidy: update your go.mod to align with the toolchain:

go mod tidy -go=1.24   # or -go=1.23

If you can’t upgrade right now (teaching/demo fallback)

Add the toggle we discussed so you can continue the flow while acknowledging risk:

# in Makefile
SKIP_GATE ?= 0

sca-gate:
	@if [ "$(SKIP_GATE)" = "1" ]; then \
	  echo "⚠️  Skipping SCA gate (SKIP_GATE=1)"; \
	else \
	  ./scripts/sca_gate.sh artifacts/trivy-image.json; \
	fi

Use it once:

make cd SKIP_GATE=1
make zap-k8s SKIP_GATE=1

(You’ll still have artifacts/trivy-image.json to show the class why you skipped.)

Want to be extra precise? (Reachability signal)

This CVE is in database/sql for Postgres. If your demo app doesn’t use a DB at all, the vuln is likely not reachable. For a more nuanced gate:

  • Add a govulncheck step on source (reachable vulns) and gate primarily on that, keeping Trivy as a secondary “FYI” signal.
  • Keep today’s simple gate for class, then upgrade policy after students see it in action.

TL;DR for the class

  • The gate blocked deploy because your Go toolchain had a HIGH with a fix.
  • Upgrade the builder image to Go 1.24.6 (or 1.23.12), rebuild, rescan → pass.
  • Or temporarily bypass with SKIP_GATE=1 to continue the exercise, then circle back and remediate.