foundation/AGENTS.md

Foundation Services GitOps Repository

This repository contains the GitOps configuration for foundational services deployed to a K3s home server cluster using Argo CD.

Repository Structure

.
├── apps/                    # Argo CD Application manifests
│   ├── foundation-project.yaml    # Argo CD project definition
│   ├── root-application.yaml      # Root app that syncs all other apps
│   ├── storage-class.yaml         # Storage class configuration
│   ├── cert-manager.yaml          # Cert Manager application
│   ├── istio.yaml                 # Istio service mesh application
│   ├── cloudnative-pg.yaml        # CloudNativePG operator application
│   ├── nats.yaml                  # NATS messaging application
│   ├── kyverno.yaml               # Kyverno policy engine application
│   ├── trivy.yaml                 # Trivy security scanner application
│   └── kustomization.yaml         # Kustomize configuration
├── storage/                  # Storage class configuration
│   ├── storage-class.yaml    # Local path storage with symlink support
│   └── kustomization.yaml   # Kustomize configuration
├── scripts/                  # Utility scripts
│   └── validate.sh          # Validation script for CI/CD
├── Makefile                  # Build and deployment commands
├── .yamllint                 # YAML linting configuration
└── .pre-commit-config.yaml   # Pre-commit hooks configuration

Services Deployed

Core Infrastructure

• Argo CD: GitOps controller (installed via Makefile)
• Cert Manager: Certificate management
• Istio: Service mesh
• CloudNativePG: PostgreSQL operator
• NATS: Messaging system
• Kyverno: Policy engine
• Trivy: Security scanning

Storage

• Local Path Storage: Default storage class with symlinks to /data/pvc/{namespace}/{name} for easier backup and management

Deployment

Initial Setup

1. Create the cluster and install Argo CD:

   make create
   

2. Wait for Argo CD to be ready:

   kubectl wait --for=condition=available --timeout=300s deployment/argocd-server -n argocd
   

3. Get Argo CD admin password:

   kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
   

4. Port-forward to access Argo CD UI:

   kubectl port-forward svc/argocd-server -n argocd 8080:443
   

   Access at: https://localhost:8080 (username: admin)

Deploy Foundation Services

Deploy all foundation services using the Makefile:

make deploy

This will:

1. Create the foundation Argo CD project
2. Deploy the root application that manages all other applications
3. Argo CD will automatically sync all foundational services

Manual Deployment

Alternatively, you can deploy manually:

kubectl apply -k apps/

Argo CD Project

All applications are managed under the foundation Argo CD project, which allows:

• Deployment to any namespace
• Access to any source repository
• Cluster and namespace resource management

Storage Configuration

The default storage class (local-path) is configured to:

• Store volumes in /var/lib/rancher/k3s/storage (K3s default)
• Create symlinks in /data/pvc/{namespace}/{name} for easier backup and management
• Use WaitForFirstConsumer binding mode for optimal pod scheduling

Application Details

Cert Manager

• Namespace: cert-manager
• Source: Helm chart from cert-manager GitHub
• Version: v1.13.3

Istio

• Namespace: istio-system
• Source: Helm chart from Istio releases
• Version: 1.20.0

CloudNativePG

• Namespace: cnpg-system
• Source: CRDs and operator from CloudNativePG GitHub
• Version: v1.22.0

NATS

• Namespace: nats
• Source: Helm chart from NATS Helm repository
• Version: 0.20.0
• Features: JetStream enabled with 10Gi storage

Kyverno

• Namespace: kyverno
• Source: Helm chart from Kyverno Helm repository
• Version: 3.1.0

Trivy

• Namespace: trivy-system
• Source: Helm chart from Aqua Security Helm repository
• Version: 0.20.0

Repository URL

Update the repository URL in apps/root-application.yaml and apps/storage-class.yaml if you're using a different Git repository:

source:
  repoURL: https://gitea.olsen.cloud/homelab/foundation.git
  targetRevision: main

Quality Assurance and Linting

This repository includes several QA tools to ensure manifest quality and correctness.

Available Commands

# Run all checks (lint + validate)
make check

# Validate Kubernetes manifests
make validate

# Lint YAML files
make lint

# Format YAML files
make format

# Install required tools
make install-tools

Tools Used

• kubeconform: Validates Kubernetes manifests against the Kubernetes API schema
• yamllint: Lints YAML files for syntax and style issues
• kustomize: Validates Kustomize configurations and builds

Pre-commit Hooks

The repository includes a .pre-commit-config.yaml for automated checks before commits. Install pre-commit hooks:

pip install pre-commit
pre-commit install

CI/CD Integration

The scripts/validate.sh script can be used in CI/CD pipelines:

./scripts/validate.sh

Validation Details

• Kubernetes Validation: Uses kubeconform to validate all manifests against Kubernetes API schemas
• Kustomize Validation: Ensures all kustomization.yaml files are valid
• YAML Linting: Checks YAML syntax, indentation, and style according to .yamllint configuration

Future Enhancements

When adding new services:

1. Create an Argo CD Application manifest in the apps/ directory
2. Add it to apps/kustomization.yaml
3. Ensure it uses the foundation project
4. Run make check to validate before committing
5. Commit and push - Argo CD will automatically sync

Troubleshooting

Argo CD Applications Not Syncing

• Check Argo CD server logs: kubectl logs -n argocd deployment/argocd-server
• Verify repository access: Argo CD needs read access to the Git repository
• Check application status in Argo CD UI or CLI: argocd app list

Storage Issues

• Verify storage class is default: kubectl get storageclass
• Check local-path-provisioner logs: kubectl logs -n kube-system -l app=local-path-provisioner
• Verify symlinks are created: ls -la /data/pvc/

Service Not Starting

• Check pod status: kubectl get pods -n <namespace>
• Check events: kubectl get events -n <namespace> --sort-by='.lastTimestamp'
• Review Argo CD application sync status