TestForge Blog
← All Posts
#kubernetes #devops #kustomize #argocd #gitops #multi-env

Kubernetes Dev & Ops in Practice 6 — Multi-Environment Deployment Strategy (Kustomize + ArgoCD)

Separate dev/staging/prod configuration with Kustomize overlays and manage the entire cluster as GitOps using ArgoCD's App of Apps pattern. The final installment in the Kubernetes Dev & Ops series.

TestForge Team ·

The Core Problem with Multi-Environment Management

You operate the same application across dev, staging, and prod. Each environment differs in:

  • Image tag (dev: latest, prod: 1.2.3)
  • Replicas (dev: 1, prod: 3)
  • Domain (dev.example.com vs example.com)
  • Resource requests/limits
  • External connection strings (DB URL, Redis URL)

Managing this by copying files leads to drift — a fix in one environment gets missed in another. Kustomize solves this with a shared base + per-environment overlay structure.

1. Kustomize Directory Structure

k8s/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml
    │   └── patch-deployment.yaml
    ├── staging/
    │   ├── kustomization.yaml
    │   └── patch-deployment.yaml
    └── prod/
        ├── kustomization.yaml
        ├── patch-deployment.yaml
        └── patch-hpa.yaml

2. Base Configuration

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

commonLabels:
  app: my-app
  managed-by: kustomize

base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-registry/my-app:latest
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          envFrom:
            - configMapRef:
                name: my-app-config

3. Overlay Configuration

overlays/dev/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: development

bases:
  - ../../base

patches:
  - path: patch-deployment.yaml

images:
  - name: my-registry/my-app
    newTag: dev-latest

configMapGenerator:
  - name: my-app-config
    behavior: merge
    literals:
      - LOG_LEVEL=debug
      - DATABASE_URL=postgres://dev-db:5432/myapp

overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

bases:
  - ../../base

patches:
  - path: patch-deployment.yaml
  - path: patch-hpa.yaml

images:
  - name: my-registry/my-app
    newTag: "1.2.3"

configMapGenerator:
  - name: my-app-config
    behavior: merge
    literals:
      - LOG_LEVEL=info
      - DATABASE_URL=postgres://prod-db:5432/myapp

overlays/prod/patch-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 2Gi
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10

Preview and Deploy

# Preview rendered output (no deployment)
kubectl kustomize overlays/prod

# Deploy
kubectl apply -k overlays/prod

# Deploy dev
kubectl apply -k overlays/dev

4. ArgoCD App of Apps Pattern

Manage all apps in the entire cluster through a single root ArgoCD Application.

Git Repository Structure

gitops-repo/
├── apps/
│   ├── kustomization.yaml
│   ├── my-app.yaml
│   ├── api-gateway.yaml
│   └── monitoring.yaml
└── manifests/
    ├── my-app/overlays/prod/
    ├── api-gateway/overlays/prod/
    └── monitoring/overlays/prod/

Root Application (App of Apps)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-repo
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Individual Application Definition

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: production
  source:
    repoURL: https://github.com/myorg/gitops-repo
    targetRevision: main
    path: manifests/my-app/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true     # Delete resources removed from Git
      selfHeal: true  # Revert manual changes automatically
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore replicas managed by HPA

5. CI/CD Pipeline Integration

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build and push image
        run: |
          IMAGE_TAG=${GITHUB_SHA::8}
          docker build -t $REGISTRY/my-app:$IMAGE_TAG .
          docker push $REGISTRY/my-app:$IMAGE_TAG
          echo "IMAGE_TAG=$IMAGE_TAG" >> $GITHUB_ENV

      - name: Update image tag in GitOps repo
        run: |
          git clone https://github.com/myorg/gitops-repo
          cd gitops-repo/manifests/my-app/overlays/prod

          kustomize edit set image my-registry/my-app:$IMAGE_TAG

          git config user.email "ci@github.com"
          git config user.name "GitHub Actions"
          git add .
          git commit -m "chore: update my-app image to $IMAGE_TAG"
          git push

ArgoCD detects the Git change and automatically reconciles the cluster.

6. Environment Promotion Strategy

[Developer PR]
    ↓ merge to main
[CI: build image + update dev tag]
    ↓ ArgoCD auto sync
[dev deployment]
    ↓ tests pass → manual promotion
[staging tag update PR]
    ↓ merge
[staging deployment]
    ↓ QA approval
[prod tag update PR + review]
    ↓ merge
[prod deployment]
# Promotion script: staging → prod
CURRENT_TAG=$(kustomize cfg grep --path manifests/my-app/overlays/staging | grep newTag | awk '{print $2}')

cd manifests/my-app/overlays/prod
kustomize edit set image my-registry/my-app:$CURRENT_TAG

git add . && git commit -m "chore: promote my-app $CURRENT_TAG to prod"
git push origin -u promote/my-app-$CURRENT_TAG

7. Drift Detection

# Check sync status via ArgoCD CLI
argocd app list
argocd app diff my-app

# Compare cluster vs Git without applying
argocd app sync my-app --dry-run

# Force reconcile
argocd app sync my-app --force

Series Wrap-Up — The Full Architecture

Here’s the complete Kubernetes dev and ops system built across this series.

[Developer Local]
  Kind cluster + Skaffold/Tilt → fast dev loop

[Code Structure]
  Namespace + RBAC → team and environment isolation
  Workload type selection → right unit for the service

[Networking]
  ClusterIP + Ingress + NetworkPolicy → explicit traffic control

[Deployment Management]
  Helm Chart → templating + version tracking
  Kustomize overlay → per-environment config separation

[GitOps]
  ArgoCD App of Apps → Git as Single Source of Truth
  CI/CD image tag updates → automated deployment pipeline

Each installment builds toward a single coherent flow: code written locally travels through Git and lands in production in a consistent, auditable way.