|
|
|
|
@@ -1,901 +0,0 @@
|
|
|
|
|
# Writing Custom Resources
|
|
|
|
|
|
|
|
|
|
This guide explains how to create and implement custom resources in the
|
|
|
|
|
homelab-operator.
|
|
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
|
|
Custom resources in this operator follow a structured pattern that includes:
|
|
|
|
|
|
|
|
|
|
- **Specification schemas** using Zod for runtime validation
|
|
|
|
|
- **Resource implementations** that extend the base `CustomResource` class
|
|
|
|
|
- **Manifest creation** helpers for generating Kubernetes resources
|
|
|
|
|
- **Reconciliation logic** to manage the desired state
|
|
|
|
|
|
|
|
|
|
## Project Structure
|
|
|
|
|
|
|
|
|
|
Each custom resource should be organized in its own directory under
|
|
|
|
|
`src/custom-resouces/` with the following structure:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
src/custom-resouces/{resource-name}/
|
|
|
|
|
├── {resource-name}.ts # Main definition file
|
|
|
|
|
├── {resource-name}.schemas.ts # Zod validation schemas
|
|
|
|
|
├── {resource-name}.resource.ts # Resource implementation
|
|
|
|
|
└── {resource-name}.create-manifests.ts # Manifest generation helpers
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Quick Start
|
|
|
|
|
|
|
|
|
|
This section walks through creating a complete custom resource from scratch.
|
|
|
|
|
We'll build a `MyResource` that manages a web application with a deployment and
|
|
|
|
|
service.
|
|
|
|
|
|
|
|
|
|
### 1. Define Your Resource
|
|
|
|
|
|
|
|
|
|
The main definition file registers your custom resource with the operator
|
|
|
|
|
framework. This file serves as the entry point that ties together your schemas,
|
|
|
|
|
implementation, and Kubernetes CRD definition.
|
|
|
|
|
|
|
|
|
|
Create the main definition file (`{resource-name}.ts`):
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import { createCustomResourceDefinition } from "../../services/custom-resources/custom-resources.ts";
|
|
|
|
|
import { GROUP } from "../../utils/consts.ts";
|
|
|
|
|
|
|
|
|
|
import { MyResourceResource } from "./my-resource.resource.ts";
|
|
|
|
|
import { myResourceSpecSchema } from "./my-resource.schemas.ts";
|
|
|
|
|
|
|
|
|
|
const myResourceDefinition = createCustomResourceDefinition({
|
|
|
|
|
group: GROUP, // Uses your operator's API group (homelab.mortenolsen.pro)
|
|
|
|
|
version: "v1", // API version for this resource
|
|
|
|
|
kind: "MyResource", // The Kubernetes kind name (PascalCase)
|
|
|
|
|
names: {
|
|
|
|
|
plural: "myresources", // Plural name for kubectl (lowercase)
|
|
|
|
|
singular: "myresource", // Singular name for kubectl (lowercase)
|
|
|
|
|
},
|
|
|
|
|
spec: myResourceSpecSchema, // Zod schema for validation
|
|
|
|
|
create: (options) => new MyResourceResource(options), // Factory function
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
export { myResourceDefinition };
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Key Points:**
|
|
|
|
|
|
|
|
|
|
- The `group` should always use the `GROUP` constant to maintain consistency
|
|
|
|
|
- `kind` should be descriptive and follow Kubernetes naming conventions
|
|
|
|
|
(PascalCase)
|
|
|
|
|
- `names.plural` is used in kubectl commands (`kubectl get myresources`)
|
|
|
|
|
- The `create` function instantiates your resource implementation when a CR is
|
|
|
|
|
detected
|
|
|
|
|
|
|
|
|
|
### 2. Create Validation Schemas
|
|
|
|
|
|
|
|
|
|
Schemas define the structure and validation rules for your custom resource's
|
|
|
|
|
specification. Using Zod provides runtime type safety and automatic validation
|
|
|
|
|
of user input.
|
|
|
|
|
|
|
|
|
|
Define your spec schema (`{resource-name}.schemas.ts`):
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import { z } from "zod";
|
|
|
|
|
|
|
|
|
|
const myResourceSpecSchema = z.object({
|
|
|
|
|
// Required fields - these must be provided by users
|
|
|
|
|
hostname: z.string(), // Base hostname for the application
|
|
|
|
|
port: z.number().min(1).max(65535), // Container port (validated range)
|
|
|
|
|
|
|
|
|
|
// Optional fields with defaults - provide sensible fallbacks
|
|
|
|
|
replicas: z.number().min(1).default(1), // Number of pod replicas
|
|
|
|
|
|
|
|
|
|
// Enums - restrict to specific values with defaults
|
|
|
|
|
protocol: z.enum(["http", "https"]).default("https"),
|
|
|
|
|
|
|
|
|
|
// Nested objects - for complex configuration
|
|
|
|
|
database: z.object({
|
|
|
|
|
host: z.string(), // Database hostname
|
|
|
|
|
port: z.number(), // Database port
|
|
|
|
|
name: z.string(), // Database name
|
|
|
|
|
}).optional(), // Entire database config is optional
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
// Additional schemas for secrets, status, etc.
|
|
|
|
|
// Separate schemas help organize different data types
|
|
|
|
|
const myResourceSecretSchema = z.object({
|
|
|
|
|
apiKey: z.string(), // API key for external services
|
|
|
|
|
password: z.string(), // Database or service password
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
export { myResourceSecretSchema, myResourceSpecSchema };
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Schema Design Best Practices:**
|
|
|
|
|
|
|
|
|
|
- **Required vs Optional**: Make fields required only when absolutely necessary
|
|
|
|
|
- **Defaults**: Provide sensible defaults to reduce user configuration burden
|
|
|
|
|
- **Validation**: Use Zod's built-in validators (`.min()`, `.max()`, `.email()`,
|
|
|
|
|
etc.)
|
|
|
|
|
- **Enums**: Restrict values to prevent invalid configurations
|
|
|
|
|
- **Nested Objects**: Group related configuration together
|
|
|
|
|
- **Separate Schemas**: Create different schemas for different purposes (spec,
|
|
|
|
|
secrets, status)
|
|
|
|
|
|
|
|
|
|
### 3. Implement the Resource
|
|
|
|
|
|
|
|
|
|
The resource implementation is the core of your custom resource. It contains the
|
|
|
|
|
business logic for managing Kubernetes resources and maintains the desired
|
|
|
|
|
state. This class extends `CustomResource` and implements the reconciliation
|
|
|
|
|
logic.
|
|
|
|
|
|
|
|
|
|
Create the resource implementation (`{resource-name}.resource.ts`):
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import type { KubernetesObject } from "@kubernetes/client-node";
|
|
|
|
|
import deepEqual from "deep-equal";
|
|
|
|
|
|
|
|
|
|
import {
|
|
|
|
|
CustomResource,
|
|
|
|
|
type CustomResourceOptions,
|
|
|
|
|
type SubresourceResult,
|
|
|
|
|
} from "../../services/custom-resources/custom-resources.custom-resource.ts";
|
|
|
|
|
import {
|
|
|
|
|
ResourceReference,
|
|
|
|
|
ResourceService,
|
|
|
|
|
} from "../../services/resources/resources.ts";
|
|
|
|
|
|
|
|
|
|
import type { myResourceSpecSchema } from "./my-resource.schemas.ts";
|
|
|
|
|
import {
|
|
|
|
|
createDeploymentManifest,
|
|
|
|
|
createServiceManifest,
|
|
|
|
|
} from "./my-resource.create-manifests.ts";
|
|
|
|
|
|
|
|
|
|
class MyResourceResource extends CustomResource<typeof myResourceSpecSchema> {
|
|
|
|
|
#deploymentResource = new ResourceReference();
|
|
|
|
|
#serviceResource = new ResourceReference();
|
|
|
|
|
|
|
|
|
|
constructor(options: CustomResourceOptions<typeof myResourceSpecSchema>) {
|
|
|
|
|
super(options);
|
|
|
|
|
const resourceService = this.services.get(ResourceService);
|
|
|
|
|
|
|
|
|
|
// Initialize resource references
|
|
|
|
|
this.#deploymentResource.current = resourceService.get({
|
|
|
|
|
apiVersion: "apps/v1",
|
|
|
|
|
kind: "Deployment",
|
|
|
|
|
name: this.name,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
this.#serviceResource.current = resourceService.get({
|
|
|
|
|
apiVersion: "v1",
|
|
|
|
|
kind: "Service",
|
|
|
|
|
name: this.name,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
// Set up event handlers for reconciliation
|
|
|
|
|
this.#deploymentResource.on("changed", this.queueReconcile);
|
|
|
|
|
this.#serviceResource.on("changed", this.queueReconcile);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#reconcileDeployment = async (): Promise<SubresourceResult> => {
|
|
|
|
|
const manifest = createDeploymentManifest({
|
|
|
|
|
name: this.name,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
ref: this.ref,
|
|
|
|
|
spec: this.spec,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
if (!this.#deploymentResource.current?.exists) {
|
|
|
|
|
await this.#deploymentResource.current?.patch(manifest);
|
|
|
|
|
return {
|
|
|
|
|
ready: false,
|
|
|
|
|
syncing: true,
|
|
|
|
|
reason: "Creating",
|
|
|
|
|
message: "Creating deployment",
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!deepEqual(this.#deploymentResource.current.spec, manifest.spec)) {
|
|
|
|
|
await this.#deploymentResource.current.patch(manifest);
|
|
|
|
|
return {
|
|
|
|
|
ready: false,
|
|
|
|
|
syncing: true,
|
|
|
|
|
reason: "Updating",
|
|
|
|
|
message: "Deployment needs updates",
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Check if deployment is ready
|
|
|
|
|
const deployment = this.#deploymentResource.current;
|
|
|
|
|
const isReady =
|
|
|
|
|
deployment.status?.readyReplicas === deployment.status?.replicas;
|
|
|
|
|
|
|
|
|
|
return {
|
|
|
|
|
ready: isReady,
|
|
|
|
|
reason: isReady ? "Ready" : "Pending",
|
|
|
|
|
message: isReady ? "Deployment is ready" : "Waiting for pods to be ready",
|
|
|
|
|
};
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
#reconcileService = async (): Promise<SubresourceResult> => {
|
|
|
|
|
const manifest = createServiceManifest({
|
|
|
|
|
name: this.name,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
ref: this.ref,
|
|
|
|
|
spec: this.spec,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
if (!deepEqual(this.#serviceResource.current?.spec, manifest.spec)) {
|
|
|
|
|
await this.#serviceResource.current?.patch(manifest);
|
|
|
|
|
return {
|
|
|
|
|
ready: false,
|
|
|
|
|
syncing: true,
|
|
|
|
|
reason: "Updating",
|
|
|
|
|
message: "Service needs updates",
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return { ready: true };
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
public reconcile = async () => {
|
|
|
|
|
if (!this.exists || this.metadata.deletionTimestamp) {
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Reconcile subresources
|
|
|
|
|
await this.reconcileSubresource("Deployment", this.#reconcileDeployment);
|
|
|
|
|
await this.reconcileSubresource("Service", this.#reconcileService);
|
|
|
|
|
|
|
|
|
|
// Update overall ready condition
|
|
|
|
|
const deploymentReady =
|
|
|
|
|
this.conditions.get("Deployment")?.status === "True";
|
|
|
|
|
const serviceReady = this.conditions.get("Service")?.status === "True";
|
|
|
|
|
|
|
|
|
|
await this.conditions.set("Ready", {
|
|
|
|
|
status: deploymentReady && serviceReady ? "True" : "False",
|
|
|
|
|
reason: deploymentReady && serviceReady ? "Ready" : "Pending",
|
|
|
|
|
message: deploymentReady && serviceReady
|
|
|
|
|
? "All resources are ready"
|
|
|
|
|
: "Waiting for resources to be ready",
|
|
|
|
|
});
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
export { MyResourceResource };
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Resource Implementation Breakdown:**
|
|
|
|
|
|
|
|
|
|
**Constructor Setup:**
|
|
|
|
|
|
|
|
|
|
- **Resource References**: Create `ResourceReference` objects to track managed
|
|
|
|
|
Kubernetes resources
|
|
|
|
|
- **Service Access**: Use dependency injection to access operator services
|
|
|
|
|
(`ResourceService`)
|
|
|
|
|
- **Event Handlers**: Listen for changes in managed resources to trigger
|
|
|
|
|
reconciliation
|
|
|
|
|
- **Resource Registration**: Register references for Deployment and Service that
|
|
|
|
|
will be managed
|
|
|
|
|
|
|
|
|
|
**Reconciliation Methods:**
|
|
|
|
|
|
|
|
|
|
- **`#reconcileDeployment`**: Manages the application's Deployment resource
|
|
|
|
|
- Creates manifests using helper functions
|
|
|
|
|
- Checks if resource exists and creates/updates as needed
|
|
|
|
|
- Uses `deepEqual` to avoid unnecessary updates
|
|
|
|
|
- Returns status indicating readiness state
|
|
|
|
|
- **`#reconcileService`**: Manages the Service resource for network access
|
|
|
|
|
- Similar pattern to deployment but typically simpler
|
|
|
|
|
- Services are usually ready immediately after creation
|
|
|
|
|
|
|
|
|
|
**Main Reconcile Loop:**
|
|
|
|
|
|
|
|
|
|
- **Deletion Check**: Early return if resource is being deleted
|
|
|
|
|
- **Subresource Management**: Calls individual reconciliation methods
|
|
|
|
|
- **Condition Updates**: Aggregates status from all subresources
|
|
|
|
|
- **Status Reporting**: Updates the overall "Ready" condition
|
|
|
|
|
|
|
|
|
|
**Key Design Patterns:**
|
|
|
|
|
|
|
|
|
|
- **Private Methods**: Use `#` for private reconciliation methods
|
|
|
|
|
- **Async/Await**: All reconciliation is asynchronous
|
|
|
|
|
- **Resource References**: Track external resources with type safety
|
|
|
|
|
- **Condition Management**: Provide clear status through Kubernetes conditions
|
|
|
|
|
- **Event-Driven**: React to changes in managed resources automatically
|
|
|
|
|
|
|
|
|
|
### 4. Create Manifest Helpers
|
|
|
|
|
|
|
|
|
|
Manifest helpers are pure functions that generate Kubernetes resource
|
|
|
|
|
definitions. They transform your custom resource's specification into standard
|
|
|
|
|
Kubernetes objects. This separation keeps your reconciliation logic clean and
|
|
|
|
|
makes manifests easy to test and modify.
|
|
|
|
|
|
|
|
|
|
Define manifest creation functions (`{resource-name}.create-manifests.ts`):
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
type CreateDeploymentManifestOptions = {
|
|
|
|
|
name: string;
|
|
|
|
|
namespace: string;
|
|
|
|
|
ref: any; // Owner reference
|
|
|
|
|
spec: {
|
|
|
|
|
hostname: string;
|
|
|
|
|
port: number;
|
|
|
|
|
replicas: number;
|
|
|
|
|
};
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
const createDeploymentManifest = (
|
|
|
|
|
options: CreateDeploymentManifestOptions,
|
|
|
|
|
) => ({
|
|
|
|
|
apiVersion: "apps/v1",
|
|
|
|
|
kind: "Deployment",
|
|
|
|
|
metadata: {
|
|
|
|
|
name: options.name,
|
|
|
|
|
namespace: options.namespace,
|
|
|
|
|
ownerReferences: [options.ref],
|
|
|
|
|
},
|
|
|
|
|
spec: {
|
|
|
|
|
replicas: options.spec.replicas,
|
|
|
|
|
selector: {
|
|
|
|
|
matchLabels: {
|
|
|
|
|
app: options.name,
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
template: {
|
|
|
|
|
metadata: {
|
|
|
|
|
labels: {
|
|
|
|
|
app: options.name,
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
spec: {
|
|
|
|
|
containers: [
|
|
|
|
|
{
|
|
|
|
|
name: options.name,
|
|
|
|
|
image: "nginx:latest",
|
|
|
|
|
ports: [
|
|
|
|
|
{
|
|
|
|
|
containerPort: options.spec.port,
|
|
|
|
|
},
|
|
|
|
|
],
|
|
|
|
|
env: [
|
|
|
|
|
{
|
|
|
|
|
name: "HOSTNAME",
|
|
|
|
|
value: options.spec.hostname,
|
|
|
|
|
},
|
|
|
|
|
],
|
|
|
|
|
},
|
|
|
|
|
],
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
type CreateServiceManifestOptions = {
|
|
|
|
|
name: string;
|
|
|
|
|
namespace: string;
|
|
|
|
|
ref: any;
|
|
|
|
|
spec: {
|
|
|
|
|
port: number;
|
|
|
|
|
};
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
const createServiceManifest = (options: CreateServiceManifestOptions) => ({
|
|
|
|
|
apiVersion: "v1",
|
|
|
|
|
kind: "Service",
|
|
|
|
|
metadata: {
|
|
|
|
|
name: options.name,
|
|
|
|
|
namespace: options.namespace,
|
|
|
|
|
ownerReferences: [options.ref],
|
|
|
|
|
},
|
|
|
|
|
spec: {
|
|
|
|
|
selector: {
|
|
|
|
|
app: options.name,
|
|
|
|
|
},
|
|
|
|
|
ports: [
|
|
|
|
|
{
|
|
|
|
|
port: 80,
|
|
|
|
|
targetPort: options.spec.port,
|
|
|
|
|
},
|
|
|
|
|
],
|
|
|
|
|
},
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
export { createDeploymentManifest, createServiceManifest };
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Manifest Helper Patterns:**
|
|
|
|
|
|
|
|
|
|
**Type Definitions:**
|
|
|
|
|
|
|
|
|
|
- **Options Types**: Define clear interfaces for function parameters
|
|
|
|
|
- **Structured Input**: Group related parameters in nested objects
|
|
|
|
|
- **Type Safety**: Leverage TypeScript to catch configuration errors at compile
|
|
|
|
|
time
|
|
|
|
|
|
|
|
|
|
**Deployment Manifest:**
|
|
|
|
|
|
|
|
|
|
- **Owner References**: Ensures garbage collection when parent resource is
|
|
|
|
|
deleted
|
|
|
|
|
- **Labels & Selectors**: Consistent labeling for pod selection and organization
|
|
|
|
|
- **Container Configuration**: Maps custom resource spec to container settings
|
|
|
|
|
- **Environment Variables**: Passes configuration from spec to running
|
|
|
|
|
containers
|
|
|
|
|
- **Port Configuration**: Exposes application ports based on spec
|
|
|
|
|
|
|
|
|
|
**Service Manifest:**
|
|
|
|
|
|
|
|
|
|
- **Service Discovery**: Creates stable network endpoint for the deployment
|
|
|
|
|
- **Port Mapping**: Routes external traffic to container ports
|
|
|
|
|
- **Selector Matching**: Uses same labels as deployment for proper routing
|
|
|
|
|
- **Owner References**: Links service lifecycle to custom resource
|
|
|
|
|
|
|
|
|
|
**Best Practices for Manifest Helpers:**
|
|
|
|
|
|
|
|
|
|
- **Pure Functions**: No side effects, same input always produces same output
|
|
|
|
|
- **Immutable Objects**: Return new objects rather than modifying inputs
|
|
|
|
|
- **Validation**: Let TypeScript catch type mismatches
|
|
|
|
|
- **Consistent Naming**: Use predictable patterns for resource names
|
|
|
|
|
- **Owner References**: Always set for proper cleanup
|
|
|
|
|
- **Documentation**: Comment non-obvious configuration choices
|
|
|
|
|
|
|
|
|
|
### 5. Register Your Resource
|
|
|
|
|
|
|
|
|
|
Add your resource to `src/custom-resouces/custom-resources.ts`:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import { myResourceDefinition } from "./my-resource/my-resource.ts";
|
|
|
|
|
|
|
|
|
|
const customResources = [
|
|
|
|
|
// ... existing resources
|
|
|
|
|
myResourceDefinition,
|
|
|
|
|
];
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Core Concepts
|
|
|
|
|
|
|
|
|
|
These fundamental patterns are used throughout the operator framework.
|
|
|
|
|
Understanding them is essential for building robust custom resources.
|
|
|
|
|
|
|
|
|
|
### Resource References
|
|
|
|
|
|
|
|
|
|
`ResourceReference` objects provide a strongly-typed way to track and manage
|
|
|
|
|
Kubernetes resources that your custom resource creates or depends on. They
|
|
|
|
|
automatically handle resource watching, caching, and change notifications.
|
|
|
|
|
|
|
|
|
|
Use `ResourceReference` to manage related Kubernetes resources:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import {
|
|
|
|
|
ResourceReference,
|
|
|
|
|
ResourceService,
|
|
|
|
|
} from "../../services/resources/resources.ts";
|
|
|
|
|
|
|
|
|
|
class MyResource extends CustomResource<typeof myResourceSpecSchema> {
|
|
|
|
|
#deploymentResource = new ResourceReference();
|
|
|
|
|
|
|
|
|
|
constructor(options: CustomResourceOptions<typeof myResourceSpecSchema>) {
|
|
|
|
|
super(options);
|
|
|
|
|
const resourceService = this.services.get(ResourceService);
|
|
|
|
|
|
|
|
|
|
this.#deploymentResource.current = resourceService.get({
|
|
|
|
|
apiVersion: "apps/v1",
|
|
|
|
|
kind: "Deployment",
|
|
|
|
|
name: this.name,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
// Listen for changes
|
|
|
|
|
this.#deploymentResource.on("changed", this.queueReconcile);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Why Resource References Matter:**
|
|
|
|
|
|
|
|
|
|
- **Automatic Watching**: Changes to referenced resources trigger reconciliation
|
|
|
|
|
- **Type Safety**: Get compile-time checking for resource properties
|
|
|
|
|
- **Lifecycle Management**: Easily check if resources exist and their current
|
|
|
|
|
state
|
|
|
|
|
- **Event Handling**: React to external changes without polling
|
|
|
|
|
- **Caching**: Avoid repeated API calls for the same resource data
|
|
|
|
|
|
|
|
|
|
### Conditions
|
|
|
|
|
|
|
|
|
|
Kubernetes conditions provide a standardized way to communicate resource status.
|
|
|
|
|
They follow the Kubernetes convention of expressing current state, reasons for
|
|
|
|
|
that state, and human-readable messages. Conditions are crucial for operators
|
|
|
|
|
and users to understand what's happening with resources.
|
|
|
|
|
|
|
|
|
|
Use conditions to track the status of your resource:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
// Set a condition
|
|
|
|
|
await this.conditions.set("Ready", {
|
|
|
|
|
status: "True",
|
|
|
|
|
reason: "AllResourcesReady",
|
|
|
|
|
message: "All subresources are ready",
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
// Get a condition
|
|
|
|
|
const isReady = this.conditions.get("Ready")?.status === "True";
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Condition Best Practices:**
|
|
|
|
|
|
|
|
|
|
- **Standard Names**: Use common condition types like "Ready", "Available",
|
|
|
|
|
"Progressing"
|
|
|
|
|
- **Clear Status**: Use "True", "False", or "Unknown" following Kubernetes
|
|
|
|
|
conventions
|
|
|
|
|
- **Descriptive Reasons**: Provide specific reason codes for troubleshooting
|
|
|
|
|
- **Helpful Messages**: Include actionable information for users
|
|
|
|
|
- **Consistent Updates**: Always update conditions during reconciliation
|
|
|
|
|
|
|
|
|
|
### Subresource Reconciliation
|
|
|
|
|
|
|
|
|
|
The `reconcileSubresource` method provides a standardized way to manage
|
|
|
|
|
individual components of your custom resource. It automatically handles
|
|
|
|
|
condition updates, error management, and status aggregation. This pattern keeps
|
|
|
|
|
your main reconciliation loop clean and ensures consistent error handling.
|
|
|
|
|
|
|
|
|
|
Use `reconcileSubresource` to manage individual components:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
public reconcile = async () => {
|
|
|
|
|
// This automatically manages conditions and error handling
|
|
|
|
|
await this.reconcileSubresource("Deployment", this.#reconcileDeployment);
|
|
|
|
|
await this.reconcileSubresource("Service", this.#reconcileService);
|
|
|
|
|
};
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Subresource Reconciliation Benefits:**
|
|
|
|
|
|
|
|
|
|
- **Automatic Condition Management**: Sets conditions based on reconciliation
|
|
|
|
|
results
|
|
|
|
|
- **Error Isolation**: Failures in one subresource don't stop others
|
|
|
|
|
- **Status Aggregation**: Combines individual component status into overall
|
|
|
|
|
status
|
|
|
|
|
- **Consistent Patterns**: Same error handling and retry logic across all
|
|
|
|
|
components
|
|
|
|
|
- **Observability**: Clear visibility into which components are having issues
|
|
|
|
|
|
|
|
|
|
### Deep Equality Checks
|
|
|
|
|
|
|
|
|
|
Deep equality checks prevent unnecessary API calls and resource churn.
|
|
|
|
|
Kubernetes resources should only be updated when their desired state actually
|
|
|
|
|
differs from their current state. This improves performance and reduces cluster
|
|
|
|
|
load.
|
|
|
|
|
|
|
|
|
|
Use `deepEqual` to avoid unnecessary updates:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import deepEqual from "deep-equal";
|
|
|
|
|
|
|
|
|
|
if (!deepEqual(currentResource.spec, desiredManifest.spec)) {
|
|
|
|
|
await currentResource.patch(desiredManifest);
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Deep Equality Benefits:**
|
|
|
|
|
|
|
|
|
|
- **Performance**: Avoids unnecessary API calls to Kubernetes
|
|
|
|
|
- **Reduced Churn**: Prevents resource version conflicts and unnecessary events
|
|
|
|
|
- **Stability**: Reduces reconciliation loops and system noise
|
|
|
|
|
- **Efficiency**: Lets you focus compute on actual changes
|
|
|
|
|
- **Observability**: Cleaner audit logs with only meaningful changes
|
|
|
|
|
|
|
|
|
|
**When to Use Deep Equality:**
|
|
|
|
|
|
|
|
|
|
- **Spec Comparisons**: Before updating any Kubernetes resource
|
|
|
|
|
- **Status Updates**: Only update status when values actually change
|
|
|
|
|
- **Metadata Updates**: Check labels and annotations before patching
|
|
|
|
|
- **Complex Objects**: Especially useful for nested configuration objects
|
|
|
|
|
|
|
|
|
|
## Advanced Patterns
|
|
|
|
|
|
|
|
|
|
These patterns handle more complex scenarios like secret management, resource
|
|
|
|
|
dependencies, and sophisticated error handling. Use these when building
|
|
|
|
|
production-ready operators that need to handle real-world complexity.
|
|
|
|
|
|
|
|
|
|
### Working with Secrets
|
|
|
|
|
|
|
|
|
|
Many resources need to manage secrets. Here's a pattern for secret management:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
import { SecretService } from "../../services/secrets/secrets.ts";
|
|
|
|
|
|
|
|
|
|
class MyResource extends CustomResource<typeof myResourceSpecSchema> {
|
|
|
|
|
constructor(options: CustomResourceOptions<typeof myResourceSpecSchema>) {
|
|
|
|
|
super(options);
|
|
|
|
|
const secretService = this.services.get(SecretService);
|
|
|
|
|
|
|
|
|
|
// Get or create a secret
|
|
|
|
|
this.secretRef = secretService.get({
|
|
|
|
|
name: `${this.name}-secret`,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ensureSecret = async () => {
|
|
|
|
|
const secretData = {
|
|
|
|
|
apiKey: generateApiKey(),
|
|
|
|
|
password: generatePassword(),
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
if (!this.secretRef.current?.exists) {
|
|
|
|
|
await this.secretRef.current?.patch({
|
|
|
|
|
apiVersion: "v1",
|
|
|
|
|
kind: "Secret",
|
|
|
|
|
metadata: {
|
|
|
|
|
name: this.secretRef.current.name,
|
|
|
|
|
namespace: this.secretRef.current.namespace,
|
|
|
|
|
ownerReferences: [this.ref],
|
|
|
|
|
},
|
|
|
|
|
data: secretData,
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Cross-Resource Dependencies
|
|
|
|
|
|
|
|
|
|
When your resource depends on other custom resources:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
class MyResource extends CustomResource<typeof myResourceSpecSchema> {
|
|
|
|
|
#dependentResource = new ResourceReference();
|
|
|
|
|
|
|
|
|
|
constructor(options: CustomResourceOptions<typeof myResourceSpecSchema>) {
|
|
|
|
|
super(options);
|
|
|
|
|
const resourceService = this.services.get(ResourceService);
|
|
|
|
|
|
|
|
|
|
// Reference another custom resource
|
|
|
|
|
this.#dependentResource.current = resourceService.get({
|
|
|
|
|
apiVersion: "homelab.mortenolsen.pro/v1",
|
|
|
|
|
kind: "PostgresDatabase",
|
|
|
|
|
name: this.spec.database,
|
|
|
|
|
namespace: this.namespace,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
this.#dependentResource.on("changed", this.queueReconcile);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#reconcileApp = async (): Promise<SubresourceResult> => {
|
|
|
|
|
// Check if dependency is ready
|
|
|
|
|
const dependency = this.#dependentResource.current;
|
|
|
|
|
if (!dependency?.exists) {
|
|
|
|
|
return {
|
|
|
|
|
ready: false,
|
|
|
|
|
failed: true,
|
|
|
|
|
reason: "MissingDependency",
|
|
|
|
|
message: `PostgresDatabase ${this.spec.database} not found`,
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
const dependencyReady = dependency.status?.conditions?.find(
|
|
|
|
|
(c) => c.type === "Ready" && c.status === "True",
|
|
|
|
|
);
|
|
|
|
|
|
|
|
|
|
if (!dependencyReady) {
|
|
|
|
|
return {
|
|
|
|
|
ready: false,
|
|
|
|
|
reason: "WaitingForDependency",
|
|
|
|
|
message:
|
|
|
|
|
`Waiting for PostgresDatabase ${this.spec.database} to be ready`,
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Continue with reconciliation...
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Error Handling
|
|
|
|
|
|
|
|
|
|
Proper error handling in reconciliation:
|
|
|
|
|
|
|
|
|
|
```typescript
|
|
|
|
|
#reconcileDeployment = async (): Promise<SubresourceResult> => {
|
|
|
|
|
try {
|
|
|
|
|
// Reconciliation logic...
|
|
|
|
|
return { ready: true };
|
|
|
|
|
} catch (error) {
|
|
|
|
|
return {
|
|
|
|
|
ready: false,
|
|
|
|
|
failed: true,
|
|
|
|
|
reason: 'ReconciliationError',
|
|
|
|
|
message: `Failed to reconcile deployment: ${error.message}`,
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Example Usage
|
|
|
|
|
|
|
|
|
|
Once your custom resource is implemented and registered, users can create
|
|
|
|
|
instances using standard Kubernetes manifests. The operator will automatically
|
|
|
|
|
detect new resources and begin reconciliation based on your implementation
|
|
|
|
|
logic.
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
apiVersion: homelab.mortenolsen.pro/v1
|
|
|
|
|
kind: MyResource
|
|
|
|
|
metadata:
|
|
|
|
|
name: my-app
|
|
|
|
|
namespace: default
|
|
|
|
|
spec:
|
|
|
|
|
hostname: my-app.example.com
|
|
|
|
|
port: 8080
|
|
|
|
|
replicas: 3
|
|
|
|
|
protocol: https
|
|
|
|
|
database:
|
|
|
|
|
host: postgres.default.svc.cluster.local
|
|
|
|
|
port: 5432
|
|
|
|
|
name: myapp
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**What happens when this resource is created:**
|
|
|
|
|
|
|
|
|
|
1. **Validation**: The operator validates the spec against your Zod schema
|
|
|
|
|
2. **Resource Creation**: Your `MyResourceResource` class is instantiated
|
|
|
|
|
3. **Reconciliation**: The operator creates a Deployment with 3 replicas and a
|
|
|
|
|
Service
|
|
|
|
|
4. **Status Updates**: Conditions are set to track deployment and service
|
|
|
|
|
readiness
|
|
|
|
|
5. **Event Handling**: The operator watches for changes and re-reconciles as
|
|
|
|
|
needed
|
|
|
|
|
|
|
|
|
|
Users can then monitor the resource status with:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl get myresources my-app -o yaml
|
|
|
|
|
kubectl describe myresource my-app
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Real Examples
|
|
|
|
|
|
|
|
|
|
These examples show how the patterns described above are used in practice within
|
|
|
|
|
the homelab-operator.
|
|
|
|
|
|
|
|
|
|
### Simple Resource: Domain
|
|
|
|
|
|
|
|
|
|
The `Domain` resource demonstrates a straightforward custom resource that
|
|
|
|
|
manages external dependencies. It creates and manages TLS certificates through
|
|
|
|
|
cert-manager and configures Istio gateways for HTTPS traffic routing.
|
|
|
|
|
|
|
|
|
|
**What it does:**
|
|
|
|
|
|
|
|
|
|
- Creates a cert-manager Certificate for TLS termination
|
|
|
|
|
- Configures an Istio Gateway for traffic routing
|
|
|
|
|
- Manages the lifecycle of both resources through owner references
|
|
|
|
|
- Provides wildcard certificate support for subdomains
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
apiVersion: homelab.mortenolsen.pro/v1
|
|
|
|
|
kind: Domain
|
|
|
|
|
metadata:
|
|
|
|
|
name: homelab
|
|
|
|
|
namespace: homelab
|
|
|
|
|
spec:
|
|
|
|
|
hostname: local.olsen.cloud # Domain for certificate and gateway
|
|
|
|
|
issuer: letsencrypt-prod # cert-manager ClusterIssuer to use
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Key Implementation Features:**
|
|
|
|
|
|
|
|
|
|
- **CRD Dependency Checking**: Validates that cert-manager and Istio CRDs exist
|
|
|
|
|
- **Cross-Namespace Resources**: Certificate is created in the istio-ingress
|
|
|
|
|
namespace
|
|
|
|
|
- **Status Aggregation**: Combines certificate and gateway readiness into
|
|
|
|
|
overall status
|
|
|
|
|
- **Wildcard Support**: Automatically configures `*.hostname` for subdomains
|
|
|
|
|
|
|
|
|
|
### Complex Resource: AuthentikServer
|
|
|
|
|
|
|
|
|
|
The `AuthentikServer` resource showcases a complex custom resource with multiple
|
|
|
|
|
dependencies and sophisticated reconciliation logic. It deploys a complete
|
|
|
|
|
identity provider solution with database and Redis dependencies.
|
|
|
|
|
|
|
|
|
|
**What it does:**
|
|
|
|
|
|
|
|
|
|
- Deploys Authentik identity provider with proper configuration
|
|
|
|
|
- Manages database schema and user creation
|
|
|
|
|
- Configures Redis connection for session storage
|
|
|
|
|
- Sets up domain integration for SSO endpoints
|
|
|
|
|
- Handles secret generation and rotation
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
apiVersion: homelab.mortenolsen.pro/v1
|
|
|
|
|
kind: AuthentikServer
|
|
|
|
|
metadata:
|
|
|
|
|
name: homelab
|
|
|
|
|
namespace: homelab
|
|
|
|
|
spec:
|
|
|
|
|
domain: homelab # References a Domain resource
|
|
|
|
|
database: test2 # References a PostgresDatabase resource
|
|
|
|
|
redis: redis # References a Redis connection
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Key Implementation Features:**
|
|
|
|
|
|
|
|
|
|
- **Resource Dependencies**: Waits for Domain, PostgresDatabase, and Redis
|
|
|
|
|
resources
|
|
|
|
|
- **Secret Management**: Generates and manages API keys, passwords, and tokens
|
|
|
|
|
- **Service Configuration**: Creates comprehensive Kubernetes manifests
|
|
|
|
|
(Deployment, Service, Ingress)
|
|
|
|
|
- **Health Checking**: Monitors application readiness and database connectivity
|
|
|
|
|
- **Cross-Resource Communication**: Uses other custom resources' status and
|
|
|
|
|
outputs
|
|
|
|
|
|
|
|
|
|
### Database Resource: PostgresDatabase
|
|
|
|
|
|
|
|
|
|
The `PostgresDatabase` resource illustrates how to manage stateful resources and
|
|
|
|
|
external system integration. It creates databases within an existing PostgreSQL
|
|
|
|
|
instance and manages user permissions.
|
|
|
|
|
|
|
|
|
|
**What it does:**
|
|
|
|
|
|
|
|
|
|
- Creates a new database in an existing PostgreSQL server
|
|
|
|
|
- Generates dedicated database user with appropriate permissions
|
|
|
|
|
- Manages connection secrets for applications
|
|
|
|
|
- Handles database cleanup and user removal
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
apiVersion: homelab.mortenolsen.pro/v1
|
|
|
|
|
kind: PostgresDatabase
|
|
|
|
|
metadata:
|
|
|
|
|
name: test2
|
|
|
|
|
namespace: homelab
|
|
|
|
|
spec:
|
|
|
|
|
connection: homelab/db # References PostgreSQL connection (namespace/name)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Key Implementation Features:**
|
|
|
|
|
|
|
|
|
|
- **External System Integration**: Connects to existing PostgreSQL instances
|
|
|
|
|
- **User Management**: Creates database-specific users with minimal required
|
|
|
|
|
permissions
|
|
|
|
|
- **Secret Generation**: Provides connection details to consuming applications
|
|
|
|
|
- **Cleanup Handling**: Safely removes databases and users when resource is
|
|
|
|
|
deleted
|
|
|
|
|
- **Connection Validation**: Verifies connectivity before marking as ready
|
|
|
|
|
|
|
|
|
|
**Common Patterns Across Examples:**
|
|
|
|
|
|
|
|
|
|
- **Owner References**: All managed resources have proper ownership for garbage
|
|
|
|
|
collection
|
|
|
|
|
- **Condition Management**: Consistent status reporting through Kubernetes
|
|
|
|
|
conditions
|
|
|
|
|
- **Resource Dependencies**: Graceful handling of missing or unready
|
|
|
|
|
dependencies
|
|
|
|
|
- **Secret Management**: Secure generation and storage of credentials
|
|
|
|
|
- **Cross-Resource Integration**: Resources reference and depend on each other
|
|
|
|
|
appropriately
|
|
|
|
|
|
|
|
|
|
## Best Practices
|
|
|
|
|
|
|
|
|
|
1. **Validation**: Always use Zod schemas for comprehensive spec validation
|
|
|
|
|
2. **Idempotency**: Use `deepEqual` checks to avoid unnecessary updates
|
|
|
|
|
3. **Conditions**: Provide clear status information through conditions
|
|
|
|
|
4. **Owner References**: Always set owner references for created resources
|
|
|
|
|
5. **Error Handling**: Provide meaningful error messages and failure reasons
|
|
|
|
|
6. **Dependencies**: Handle missing dependencies gracefully
|
|
|
|
|
7. **Cleanup**: Leverage Kubernetes garbage collection through owner references
|
|
|
|
|
8. **Testing**: Create test manifests in `test-manifests/` for your resources
|
|
|
|
|
|
|
|
|
|
## Troubleshooting
|
|
|
|
|
|
|
|
|
|
- **Resource not reconciling**: Check if the resource is properly registered in
|
|
|
|
|
`custom-resources.ts`
|
|
|
|
|
- **Validation errors**: Ensure your Zod schema matches the expected spec
|
|
|
|
|
structure
|
|
|
|
|
- **Missing dependencies**: Verify that referenced resources exist and are ready
|
|
|
|
|
- **Owner reference issues**: Make sure `ownerReferences` are set correctly for
|
|
|
|
|
garbage collection
|
|
|
|
|
- **Condition not updating**: Ensure you're calling `this.conditions.set()` with
|
|
|
|
|
proper status values
|
|
|
|
|
|
|
|
|
|
For more examples, refer to the existing custom resources in
|
|
|
|
|
`src/custom-resouces/`.
|