Skip to content

GitOps with Flux CD#

This GitOps repository implements a complete infrastructure and application management system using Flux CD and Kustomize. It follows a structured, multi-environment approach to manage Kubernetes resources, ensuring automated reconciliation, security, and consistency.

Overview#

The GitOps system is designed to provide:

  • Declarative Infrastructure: All resources (Deployments, Services, Ingress, Alerts) are defined as YAML.
  • Continuous Delivery: Flux CD synchronizes the cluster state with this repository.
  • Multi-Environment Support: Dedicated overlays for production and staging.
  • Secret Ops: Secure secret management using SOPS and age/GPG.
  • Automated Validation: CI pipelines for linting and schema validation.

Architecture#

The repository allows Flux CD to manage itself and the cluster workloads:

graph TD
    Git[Git Repository] -->|Syncs| Flux[Flux CD Controllers]
    Flux -->|Applies| K8s[Kubernetes Cluster]

    subgraph "Repository Structure"
        Base[Apps Base] -->|Inherits| Overlay[Apps Overlay]
        Infra[Infrastructure] -->|Configures| Cluster[Cluster Configs]
    end

    subgraph "Cluster State"
        Flux --> Apps[Applications]
        Flux --> Controllers[Infra Controllers]
        Flux --> Configs[Infra Configurations]
    end

Repository Structure#

The repository is organized to separate concerns and environments:

gitops/
├── apps/                        # Application definitions
│   ├── base/                    # Shared resource definitions
│   │   ├── cnpg/                # CloudNativePG Operator & Clusters
│   │   ├── grafana-stack/       # LGTM Stack (Loki, Grafana, Mimir)
│   │   ├── homepage/            # Dashboard
│   │   ├── media/               # Immich & Media tools
│   │   └── prometheus-blackbox/ # Uptime monitoring
│   ├── production/              # Production environment specific configs
│   │   ├── fluxcd/              # FluxCD specific monitoring/dashboards
│   │   ├── o11y/                # Observability (Uptime Kuma, SLOs)
│   │   ├── searxng/             # Search Engine
│   │   └── ... (overlays for base apps)
│   └── staging/                 # Staging environment overlays
├── clusters/                    # Flux CD entry points
│   ├── prod/                    # Production cluster definition
│   └── staging/                 # Staging cluster definition
├── infrastructure/              # Cluster-wide infrastructure
│   ├── configs/                 # Configs (StorageClasses, Alerts, Issuers)
│   └── controllers/             # Controllers (Cert-Manager, Traefik)
└── scripts/                     # Operational scripts
    ├── create-secret.sh         # Helper to generate encrypted secrets
    └── validate.sh              # CI validation script

Deployment Workflow#

Prerequisites#

  • Kubernetes Cluster (v1.24+)
  • kubectl configured for admin access
  • Flux CLI (v2.0+)
  • SOPS & age (for secret decryption)
  • Gitea (or Git host) SSH access

1. Bootstrap Flux CD#

To initialize the cluster and install Flux CD components:

Production: 

flux bootstrap gitea \
  --token-auth=false \
  --owner=igresc \
  --repository=gitops \
  --branch=main \
  --path=clusters/prod \
  --hostname=git.local.isabelsoler.es \
  --ssh-hostname=git.local.isabelsoler.es \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key

Staging: 

flux bootstrap gitea \
  --token-auth=false \
  --owner=igresc \
  --repository=gitops \
  --branch=main \
  --path=clusters/staging \
  --hostname=git.local.isabelsoler.es \
  --ssh-hostname=git.local.isabelsoler.es \
  --read-write-key

2. Secret Management#

Secrets are encrypted using SOPS. You must have the correct private key configured in the cluster.

Creating a new secret: Use the provided helper script to safely generate and encrypt secrets.

# Usage: ./scripts/create-secret.sh <secret-name> <path-to-values.yaml> <namespace>
./scripts/create-secret.sh my-app-values ./local-values.yaml my-namespace

Manual Encryption: 

# Encrypt a file in place
sops --encrypt --in-place secrets/my-secret.yaml

# Edit an encrypted file
sops secrets/my-secret.yaml

3. Validation & Testing#

Before pushing changes, run the validation suite to ensure manifests are valid and follow best practices.

# Run full validation (Kustomize build + Kubeconform)
bash scripts/validate.sh

# Lint YAML syntax
yamllint .

# Lint Markdown
markdownlint "**/*.md"

4. Deployment#

  1. Commit Changes: Push your changes to the main branch.
  2. Reconciliation: Flux will detect the commit (usually within 1 minute).
  3. Verification: 
    flux get kustomizations
flux get helmreleases --all-namespaces

Key Components#

Infrastructure#

  • Cert-Manager: Automates certificate management (currently disabled in infrastructure/controllers).
  • Longhorn: Distributed block storage.
  • Traefik: Ingress controller.

Observability (O11y)#

  • Grafana Stack: Complete monitoring with Grafana, Loki (logs), Mimir (metrics), and Alloy (collector).
  • Uptime Kuma: Status pages and uptime monitoring.
  • Prometheus Blackbox: External endpoint monitoring.
  • Flux Dashboards: Custom Grafana dashboards for Flux internal metrics.

Applications#

  • Homepage: Central dashboard for all services.
  • Immich: Self-hosted photo and video management.
  • SearXNG: Privacy-respecting metasearch engine.
  • CloudNativePG: PostgreSQL operator for high-availability databases.

Troubleshooting#

Debugging Flux#

If a Kustomization fails to reconcile:

  1. Check Status: 
    flux get kustomizations
  2. Describe Failure: 
    flux describe kustomization <name>
  3. Force Reconciliation: 
    flux reconcile kustomization <name> --with-source

Common Issues#

  • Secret Decryption: Ensure the cluster has the sops-age secret in the flux-system namespace.
  • Missing Resources: Check kustomization.yaml in the directory to ensure your new file is listed in the resources array.
  • Validation Errors: Run scripts/validate.sh locally to catch schema errors before pushing.

Migration & Maintenance#

  • Adding Apps: Create a base in apps/base/, add an overlay in apps/production/, and register it in apps/production/kustomization.yaml.
  • Updates: Flux Image Automation (if enabled) will update image tags in Git. Manual updates require changing the tag or version in the YAML files.