Skip to content

Grafana Stack#

The core observability platform is defined in grafana-stack.

Overview#

  • Namespace: monitoring
  • Components: Loki (Logs), Mimir (Metrics), Alloy (Collector), k8s-monitoring (Meta-chart).
  • Source: Grafana Helm Charts

Components & Architecture#

1. Alloy (Collector)#

Alloy is deployed as a DaemonSet (and singleton Deployment via k8s-monitoring) to collect telemetry.

  • Configuration: Defined in apps/production/grafana-stack/config.alloy and injected via a ConfigMapGenerator named alloy-config.
  • Pipelines:
    • Logs: Discovers Pods/Nodes, relabels Kubernetes metadata (namespace, pod, container), and pushes to loki-distributor.
    • Metrics: Scrapes Pods, Nodes (cAdvisor), OS metrics, and standard Kubernetes ServiceMonitors/PodMonitors.
    • Remote Write: Pushes metrics to http://mimir-nginx/api/v1/push.
    • Static Targets: Explicitly scrapes Docker (192.168.1.32) and Graphite Exporter.

2. K8s Monitoring#

A meta-chart (k8s-monitoring) orchestrates the monitoring configuration.

  • Version: 3.5.3
  • Cluster Name: k3s-prod
  • Integrations:
    • Cert-Manager: Auto-discovered.
    • Node Logs: Scrapes kubelet.service and containerd.service from systemd journal.
    • PrusaLink: Static scrape configuration for 3D printer metrics (192.168.1.32:10009).
  • Features Enabled: Annotation Autodiscovery (prometheus.io/scrape), Cluster Events, Cluster Metrics.

3. Loki (Logs)#

  • Deployment: Distributed microservices mode (implied by chart loki).
  • Version: 6.x.x

4. Mimir (Metrics)#

  • Deployment: Distributed mode (mimir-distributed).
  • Version: 6.x.x

GitOps Strategy#

  • Base: apps/base/grafana-stack defines the HelmRepository and base HelmReleases.
  • Production: apps/production/grafana-stack applies overlays:
    • Loki/Mimir: Updates chart versions.
    • Alloy: Enables clustering and binds the custom config.alloy.
    • K8s-Monitoring: Defines the cluster-specific monitoring rules and destinations.