I have 395 notes in Obsidian and ~2,800 memories from past AI sessions. My AI agent knows none of it unless I paste it in manually. And I can only paste what I remember to paste — which defeats the point of having a knowledge store.

So I built a hook that fires before every prompt, searches a unified vector store, and injects the relevant context before the agent thinks. R2R as the RAG backend, pgvector for storage, Ollama for embeddings, all on a homelab Kubernetes cluster. Total latency: under 100ms. Zero tokens consumed in the retrieval path.

The value isn't the search itself. It's what surfaces when you stop choosing what's relevant. Ask about a Kubernetes pattern and a six-month-old Obsidian note appears. Debug a tmux issue and a memory from a previous session shows up with the exact fix. The agent answers from your accumulated knowledge, not just its training data.

Architecture​

Three backend services and one client-side hook.

Each prompt follows this path:

1. You type a prompt in Claude Code
2. The UserPromptSubmit hook fires before the agent sees it
3. The hook searches R2R's unified index (~90ms) containing both Obsidian notes and agent memories
4. Relevant chunks get injected as system-reminder context
5. The agent sees your prompt plus the matched knowledge
6. If a note title looks relevant, the agent drills deeper via the Obsidian MCP server

Two data sources feed a single vector store. A local cron syncs the Obsidian vault into R2R every 30 minutes. A Kubernetes CronJob syncs agent-memory entries every hour. Both land in pgvector with the same embedding model, so a single search covers your curated notes and your session-derived insights.

How Vector Search Works​

The search pipeline runs without an LLM. The key component is an embedding model: nomic-embed-text (274MB), running on Ollama. It converts text into a 768-dimensional vector — a list of 768 floating-point numbers that represent the text's meaning as coordinates in high-dimensional space.

The model was trained on millions of text pairs so that texts with similar meaning land near each other. "Longhorn PVC backup" and "persistent volume restore" end up as nearby points, even though they share zero words.

Ingestion​

When the 395 Obsidian notes were loaded into R2R, each note got chunked into segments. Each chunk went to Ollama, which ran it through nomic-embed-text and returned a 768-number vector. pgvector stored both the vector and the original text:

pgvector row:
  id:        uuid
  text:      "StatefulSet gets a 10Gi Longhorn PVC..."
  embedding: [0.023, -0.187, 0.442, ..., 0.091]  (768 floats)
  metadata:  {title: "r2r-rag-pipeline", source: "obsidian"}

pgvector builds an HNSW index over these vectors so it doesn't compare against every row at query time.

Search​

When you type "how do I backup Longhorn volumes?", the hook sends your prompt to Ollama, gets back 768 numbers, and sends those to pgvector. pgvector computes cosine similarity — the angle between your prompt vector and every stored vector — and returns the closest matches. Identical direction = 1.0, orthogonal = 0.0. The threshold is 0.45; anything below gets dropped as irrelevant.

This is pure linear algebra. Dot products and normalization. That's why it runs in ~100ms with zero token costs.

One store, two sources​

Both Obsidian notes and agent memories get embedded with nomic-embed-text and stored in pgvector. A single HNSW index covers everything — a prompt about "Longhorn backup" finds relevant hits regardless of whether the knowledge came from a note you wrote or a pattern the agent learned.

The metadata tags let you filter by source if needed, but the default search spans both.

The Database Journey​

I started with the CloudNativePG operator. It's the standard for running Postgres on Kubernetes — WAL archiving, automated failover, point-in-time recovery. Production-grade.

It didn't work. Two problems.

First, the pgvector image. CNPG validates container images through a webhook, and pgvector/pgvector:pg17 didn't match the expected image patterns. The webhook rejected the pod.

Second, ImageVolume extensions. CNPG has a mechanism for loading Postgres extensions via ephemeral volumes. The pgvector extension needs to be loaded as a shared library, and the ImageVolume approach hit path resolution issues on my cluster's containerd version.

I spent a day debugging webhook configurations and extension loading. Then I stopped and asked: what am I actually building?

A homelab. Single node. No HA requirement. No point-in-time recovery needed. The data is my Obsidian vault — I have the source of truth on disk. If the database dies, I re-ingest.

A plain StatefulSet with the pgvector/pgvector:pg17 image works:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: r2r-db
  namespace: ai-tools
spec:
  serviceName: r2r-db
  replicas: 1
  template:
    spec:
      containers:
        - name: postgres
          image: pgvector/pgvector:pg17
          env:
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: init
              mountPath: /docker-entrypoint-initdb.d
      volumes:
        - name: init
          configMap:
            name: r2r-db-init

An init ConfigMap creates the vector extension:

CREATE EXTENSION IF NOT EXISTS "vector";

The StatefulSet gets a 10Gi Longhorn PVC, a readiness probe on pg_isready, and a Service. Postgres starts, loads pgvector, R2R connects.

Simplicity wins on a homelab. Save the operator for production.

Ingesting the Obsidian Vault​

R2R accepts documents through a multipart API. The ingestion script walks the vault directory and uploads each markdown file as raw_text. 395 notes, about three minutes.

Two gotchas during ingestion.

Filenames as DocumentType. R2R parses the uploaded filename to determine document type. Obsidian's zettelkasten IDs (1711476153-YXRZ.md) worked fine. But filenames with special characters got parsed as unknown types. The fix: sanitize filenames before upload and always set the content type to text/plain.

Document summary generation. R2R's default pipeline generates a summary for each ingested document by calling the configured LLM. For 395 documents, that meant 395 Haiku calls during ingestion. Slow, expensive, and unnecessary since I only use vector search, not summaries.

One line in the R2R config fixes it:

[ingestion]
provider = "r2r"
skip_document_summary = true

R2R Configuration​

R2R uses LiteLLM under the hood, so you can mix providers. Embeddings run through Ollama (free, local). Haiku is configured as the completion LLM, but the hook only calls R2R's /v3/retrieval/search endpoint — pure vector similarity, no LLM in the loop. Haiku would only fire if you used R2R's RAG endpoint for synthesized answers or re-enabled document summaries.

[completion]
provider = "litellm"
concurrent_request_limit = 16

[app]
quality_llm = "anthropic/claude-3-5-haiku-latest"
fast_llm = "anthropic/claude-3-5-haiku-latest"

[embedding]
provider = "ollama"
base_model = "nomic-embed-text"
base_dimension = 768

The R2R deployment points OLLAMA_API_BASE at the in-cluster Ollama service. Ollama runs on a Pop-OS desktop at 192.168.178.125. A headless Service with manual Endpoints bridges it into the cluster:

apiVersion: v1
kind: Service
metadata:
  name: ollama-pc
  namespace: ai-tools
  annotations:
    description: "Pop-OS desktop - always on"
spec:
  clusterIP: None
  ports:
  - port: 11434
    targetPort: 11434
    name: http
---
apiVersion: v1
kind: Endpoints
metadata:
  name: ollama-pc
  namespace: ai-tools
subsets:
- addresses:
  - ip: 192.168.178.125
  ports:
  - port: 11434
    name: http

Any pod in the cluster reaches Ollama at ollama-pc.ai-tools.svc:11434. No port-forwarding, no NodePorts.

One gotcha: Ollama evicts models from VRAM after five minutes idle. The embedding model kept getting cold-loaded on every RAG request after an idle gap. The fix: keep_alive: -1 in the embed request pins the model permanently.

The RAG Hook​

The core of the system is a Python script that Claude Code runs as a UserPromptSubmit hook. It fires on every non-trivial prompt automatically, with an explicit :rag suffix for verbose output. The hook searches R2R's unified index and prints the results to stdout, which Claude Code injects as context.

Hook registration in ~/.claude/settings.json:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "type": "command",
        "command": "python3 ~/.claude/scripts/__rag_context_hook.py"
      }
    ]
  }
}

The script is 233 lines of stdlib Python. No dependencies beyond what ships with Python 3. Three design decisions matter.

Automatic with escape hatches. The first version was manual-only — you had to type :rag to trigger a search. That meant you only got context when you remembered to ask for it, which defeats the purpose of having a knowledge store. The current version fires on every prompt longer than 20 characters that isn't trivial banter ("ok", "thanks", "ship it"). A regex filter catches these short responses and skips the search. Append :norag to suppress auto-search on a specific prompt. Append :rag for verbose output with scores and timing.

how do I backup longhorn volumes    → auto search, compact output
metallb config issue:rag            → verbose search with scores
just do it:norag                    → no search

Single source, single request. Both Obsidian notes and agent memories live in the same pgvector index. One HTTP call to R2R's /v3/retrieval/search endpoint covers everything. No thread pools, no parallel coordination, no partial failure handling. The search takes ~90ms.

payload = json.dumps({
    "query": query,
    "search_settings": {"limit": limit},
}).encode()
req = urllib.request.Request(
    R2R_URL, data=payload,
    headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=timeout, context=_ssl_ctx) as resp:
    body = resp.read()

Tuned relevance threshold. The score threshold sits at 0.45 for both auto and manual modes — high enough to filter noise, low enough to catch useful tangential hits. Auto mode returns 2 results with a 5-second timeout. Manual mode returns 4 results with a 12-second timeout and retries.

The hook prints results in a format Claude Code injects as a system-reminder:

RAG context (Obsidian vault):
  [kubernetes-networking] (score: 0.782)
    Service mesh configuration requires...

RAG context (agent memory):
  [kubernetes, networking, debugging]
    When troubleshooting DNS in pods, check...

The agent sees this alongside the prompt. If a note title looks relevant, it reads the full note via the Obsidian MCP server, follows wikilinks, cross-references. The hook provides the signal; the agent decides how deep to go.

Results​

The hook adds about 90ms to each prompt. Imperceptible — the agent's thinking time dwarfs it.

The original architecture searched R2R and agent-memory in parallel — two stores, two HTTP calls, a ThreadPoolExecutor to coordinate them. Agent-memory went through the MCP JSON-RPC layer, adding ~1.7s of overhead. Bypassing MCP and querying Redis FT.SEARCH directly helped, but maintaining two search backends meant two failure modes, two timeout configs, and two relevance thresholds to tune.

The current architecture is simpler. Both data sources sync into R2R on a schedule. One search call covers everything. The hook went from a parallel coordinator to a single HTTP request.

The value shows up in unexpected moments. Ask about a Kubernetes pattern and the hook surfaces an Obsidian note you wrote six months ago. Start debugging a tmux issue and it finds a memory from a previous session where you solved something similar. The agent doesn't just answer from its training data — it answers from your accumulated knowledge.

The entire search pipeline is open source and runs locally. Ollama generates embeddings with nomic-embed-text. pgvector stores and searches vectors. The hook is 233 lines of stdlib Python. Zero API calls, zero tokens consumed, zero billing in the retrieval path. The only cloud dependency is Claude itself interpreting the results.

The two sources complement each other even though they share an index. Obsidian holds curated, structured notes — architecture decisions, tool configurations, blog drafts. Agent-memory holds organic, session-derived insights — "this user prefers bun over npm", "the homelab uses Longhorn for storage", "tmux layouts need xdotool without --sync flags." Together they give the agent both your deliberate knowledge and your implicit patterns.

Links​

• R2R -- RAG framework by SciPhi
• pgvector -- vector similarity search for Postgres
• Redis Agent Memory Server -- MCP-native memory service
• Ollama -- local LLM inference
• Claude Code hooks -- pre/post tool execution hooks
• Homelab repo -- full working manifests in gitops/apps/r2r/

Platform engineering matured around one thesis: reduce cognitive load for developers. The CNCF Platforms White Paper codified the pattern. Internal developer portals, golden paths, self-service infrastructure — all designed so human teams ship faster without drowning in complexity.

AI agents now consume the same infrastructure. They clone repos, call CI/CD APIs, create pull requests, query issue trackers. But they don't read your Confluence wiki or file support tickets. They parse tool schemas and call structured APIs.

The bottlenecks shift. Humans hit cognitive overload: too many tools, tribal knowledge, context switching. Agents hit context window limits: too many tokens, irrelevant information, unstructured access. Same problem class, different constraint.

The CNCF platform engineering model transfers directly. Here's the mapping, dimension by dimension.

Two Consumers, One Platform​

The CNCF Platforms White Paper defines platform engineering across dimensions like self-service, golden paths, documentation, and guardrails. Each dimension has a direct analog for AI agents.

The mapping isn't forced. These dimensions emerged because platform engineering solves a universal problem: making infrastructure accessible to its consumers. The consumer type changes the interface, not the architecture.

Where It Gets Concrete​

Three dimensions show the mapping most clearly.

Self-service: from portals to tool registries​

For humans, self-service means a portal where you click "Create Kubernetes Cluster" and get one in minutes. No tickets, no waiting.

For agents, self-service means discoverable tools with structured schemas. An agent calls ToolSearch("kubernetes cluster") and gets back a create_cluster function with typed parameters. No parsing documentation, no guessing at CLI flags.

The same backend serves both. The portal and the tool schema are two interfaces to the same provisioning API.

Golden paths: from templates to skills​

Human golden paths are opinionated templates. create-react-app scaffolds a project. A Backstage template wires up CI/CD, monitoring, and deployment in one click.

Agent golden paths are skills — packaged workflows with clear inputs and outputs. A deploy-service skill encapsulates the same decisions the template made: which registry, which pipeline, which environment. The agent composes skills instead of following a wizard.

Documentation: from READMEs to structured context​

Humans read READMEs, runbooks, and how-to guides. The documentation is narrative, written for comprehension.

Agents consume structured context: CLAUDE.md files with project conventions, tool schemas with parameter descriptions, few-shot examples embedded in tool descriptions. Narrative documentation wastes context window. Structured context is information-dense.

A platform that serves both invests in both formats — or better, generates one from the other.

Three Dimensions the CNCF Model Missed​

The original model doesn't cover three concerns that matter only (or primarily) for AI agents.

Context management. Humans manage context through experience. An SRE who's been on-call for a year knows which alerts matter. That context is implicit and essentially free.

Agents have explicit context budgets. Every file read, every search result, every system prompt consumes tokens from a finite window. A platform at the "Provisional" level dumps entire files into the prompt. At "Optimizing," agents request exactly what they need and the platform delivers it pre-filtered.

Context management is the single highest-leverage investment for an AI agent platform. Bad context means wasted tokens, slower responses, and worse decisions.

Memory. Humans accumulate knowledge over time. Institutional memory lives in people's heads, in Slack history, in wikis.

Agents start fresh every session unless the platform provides memory infrastructure. At the "Provisional" level, there's no persistence — every session rediscovers the same project structure. At "Operational," sessions get annotated and handed off manually. At "Optimizing," agents extract insights automatically and share them across a federated knowledge store.

Authority. Human trust is earned through track record, code review, and organizational culture. A senior engineer merges without approval. A junior gets two reviewers.

Agent authority requires explicit boundaries. Permission systems define which tools an agent can call, which resources it can modify, which actions require human approval. At maturity, these boundaries become dynamic — an agent with a strong track record gets expanded permissions automatically.

The Combined Maturity Model​

The CNCF Platform Engineering Maturity Model defines four levels: Provisional, Operational, Scalable, and Optimizing. Each level applies to both human and AI platform consumers. The table below shows the human level (top row) and AI agent equivalent (bottom row) for each dimension.

The key insight: most organizations are at different levels for human and AI consumers. A platform might be "Scalable" for developers — self-service portal, product mindset, dedicated team — but "Provisional" for agents — ad-hoc API keys, unstructured prompts, no memory.

Knowing where you stand on both axes tells you where to invest.

What Platform Teams Should Do​

Audit your AI agent maturity. Walk through each dimension. Where are you Provisional? Most teams discover they have no structured context strategy and no memory infrastructure.

Invest in structured interfaces first. MCP tool schemas, typed APIs, and discoverable registries are the highest-ROI investment. They make your existing platform accessible to agents without rebuilding anything.

Treat context as a budget. Every token an agent spends reading irrelevant information is waste. Create CLAUDE.md files for your repositories. Write tool descriptions that include parameters and examples. Trim your system prompts.

Build memory infrastructure. Agents that start fresh every session repeat the same discovery work. Even simple persistence — session notes, project context files — eliminates massive redundancy.

Don't build a separate AI platform. The same APIs, the same CI/CD, the same infrastructure serves both audiences. What changes is the interface layer. Add tool schemas alongside your portal. Add structured context alongside your documentation.

The CNCF got the model right. The platform serves its consumers. Now it has two kinds.

AI agents learn things about you as you work. Your coding style, your infrastructure quirks, which tools you prefer, how you name things. That context accumulates across dozens of sessions. But it evaporates when the session ends or when you switch tools. Use Claude Code in the terminal, Cursor in the IDE, and a web-based MCP client on your phone. Three tools, three isolated memory silos. You can share static knowledge with files. But the context that accumulates from your interactions has nowhere to go.

What if the memory layer was separate from the tools entirely? Redis Agent Memory Server does exactly that. It's an open-source memory service that speaks MCP over HTTP. CLI tools (Claude Code, Codex, Gemini CLI), IDE tools (Cursor, Windsurf, Copilot), and any web-based MCP client all connect to the same store. Because the server runs over HTTP, you can expose it beyond the homelab with Cloudflare Tunnels or similar, so tools outside your network connect too. The memory lives on your infrastructure, not theirs.

The server provides two tiers: working memory (session-scoped context that auto-promotes to long-term) and long-term memory (persistent, semantically searchable). It exposes both a REST API and an MCP interface.

I deployed it on my homelab Kubernetes cluster. Four pods, one PVC, zero API costs. Here's how.

Architecture​

Four components, each a separate Kubernetes deployment. Here's what happens under the hood.

Storing. When you tell it to remember something, it converts your text into a list of numbers (an embedding). Redis stores those numbers alongside the raw text. The memory is searchable immediately.

Retrieving. When a tool queries memory, your query becomes numbers too. Redis finds stored memories with the closest numbers, by meaning, not by keyword. An AI tool asking "what do I know about deployment patterns?" finds a memory you stored as "we always use Recreate strategy for PVC-backed workloads" even though the words don't overlap. Unlike wikis or Obsidian vaults where you organize content yourself and search by keyword, here you store unstructured text and the system handles the rest.

Enriching. In the background, a worker asks a small local LLM "what topics are in this text?" and tags each memory with structured metadata. Combined with topic and entity tags, you get precise recall without writing structured queries. Think of it as auto-filing.

The worker handles the async enrichment. Here's the full lifecycle:

Memory Lifecycle​

Storing a memory happens in two phases:

Phase 1 is synchronous. nomic-embed-text converts your text into a 768-dimension vector. Redis stores both the vector and the raw text immediately, so you can search for it right away.

Phase 2 runs in the background. The worker pulls async tasks from Redis (using docket), sends text to llama3.2:3b for topic extraction and NER, and writes structured metadata back. This makes future searches more precise; you can filter by topic or entity, not just vector similarity.

Why Not Just Use OpenAI-backed Memory?​

Most AI memory solutions (Mem0, Zep) require OpenAI or another cloud LLM for embeddings and extraction. Every memory stored costs API tokens. Every semantic search costs more tokens.

Redis Agent Memory Server supports LiteLLM, which means it works with 100+ providers. Including Ollama. Running locally. For free.

With Ollama on a spare Mac and Redis on Kubernetes, the entire memory pipeline runs locally. Memories never leave the network.

Deploying to Kubernetes​

Everything below is the actual deployment. If you just wanted the concept, you can stop here. If you want to run this on your own cluster, keep going.

Prerequisites​

• A Kubernetes cluster. Any distro works: k3s on a single node, kubeadm, Talos, managed. If you don't have one yet, k3s gets you there in one command
• A machine running Ollama with nomic-embed-text and llama3.2:3b pulled. Can be the same node, a spare laptop, a Mac. Needs ~4GB RAM for both models
• ~2.3GB cluster RAM at pod limits for the 4 deployments (actual usage is lower)
• A storage class that supports PVCs (Longhorn, local-path, OpenEBS, anything)
• Optional: ArgoCD for GitOps sync, or just kubectl apply

Everything lives in the ai-tools namespace. ArgoCD syncs from Git.

Redis Stack​

Redis is doing triple duty here: vector index (RediSearch for semantic search), task queue (docket polls it for async worker jobs), and persistent store (AOF + PVC for the actual memories). One pod instead of running Postgres + Pinecone + RabbitMQ separately. Same ops patterns as any other stateful workload: PVC, AOF persistence, health checks. Redis Stack bundles RediSearch (vector similarity) and RedisJSON. The appendonly flag enables AOF persistence, so data survives pod restarts. allkeys-lru eviction keeps memory bounded at 512MB. Oldest entries get evicted when full.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-memory-redis
  namespace: ai-tools
spec:
  replicas: 1
  strategy:
    type: Recreate  # avoid two pods fighting over PVC
  selector:
    matchLabels:
      app.kubernetes.io/name: agent-memory-redis
  template:
    metadata:
      labels:
        app.kubernetes.io/name: agent-memory-redis
        app.kubernetes.io/component: redis
    spec:
      containers:
        - name: redis
          image: redis/redis-stack-server:7.4.0-v8
          env:
            - name: REDIS_ARGS
              value: "--appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru"
          ports:
            - containerPort: 6379
              name: redis
          volumeMounts:
            - name: data
              mountPath: /data
          resources:
            limits:
              cpu: 500m
              memory: 768Mi
            requests:
              cpu: 100m
              memory: 256Mi
          livenessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 15
            periodSeconds: 30
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: agent-memory-redis-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-memory-redis-data
  namespace: ai-tools
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi

Recreate strategy is important because two Redis instances can't share the same PVC.

The Application Pods​

All three application components share environment variables. The key ones:

env:
  - name: REDIS_URL
    value: redis://agent-memory-redis.ai-tools.svc:6379
  - name: LONG_TERM_MEMORY
    value: "true"
  - name: ENABLE_TOPIC_EXTRACTION
    value: "true"
  - name: ENABLE_NER
    value: "true"
  - name: GENERATION_MODEL
    value: ollama/llama3.2:3b    # explicit tag matters!
  - name: EMBEDDING_MODEL
    value: ollama/nomic-embed-text
  - name: OLLAMA_API_BASE
    value: http://ollama.ai-tools.svc:11434  # point at your Ollama service
  - name: REDISVL_VECTOR_DIMENSIONS
    value: "768"                  # nomic-embed-text output size
  - name: DISABLE_AUTH
    value: "true"                 # homelab, internal network

Each component gets a different command:

# API server (default entrypoint)
# No command override needed -- runs uvicorn on port 8000

# MCP server
command: ["agent-memory", "mcp", "--mode", "streamable-http"]

# Task worker
command: ["agent-memory", "task-worker"]

The API server handles REST requests. The MCP server speaks Model Context Protocol over streamable-http. The worker runs docket-based async tasks.

Exposing Ollama to the Cluster​

Ollama can run on any machine on your network: the cluster node itself, a spare laptop, a desktop. If it runs outside the cluster, you need a headless Service with manual Endpoints pointing at whatever IP runs Ollama:

apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ai-tools
spec:
  clusterIP: None
  ports:
  - port: 11434
    targetPort: 11434
    name: http
---
apiVersion: v1
kind: Endpoints
metadata:
  name: ollama
  namespace: ai-tools
subsets:
- addresses:
  - ip: <OLLAMA_HOST_IP>
  ports:
  - port: 11434
    name: http

Now any pod in the cluster can reach Ollama at ollama.ai-tools.svc:11434. If Ollama runs on a cluster node, you can skip the Endpoints and use a regular Service or just point OLLAMA_API_BASE at the node IP directly.

Adding Streamable HTTP Support​

Upstream agent-memory-server supports stdio and sse transport modes for MCP. Neither works well for a Kubernetes deployment where Claude Code connects over HTTP.

stdio requires a local process. sse uses server-sent events but doesn't support the streamable-http transport that Claude Code's HTTP MCP client expects.

A small patch fixes this. Three changes:

1. Add streamable-http as a valid --mode choice in the CLI
2. Set stateless_http=True so Claude Code subagents (which don't complete the MCP initialization handshake) still work
3. Return Response() instead of None to prevent a TypeError when SSE clients disconnect

# cli.py - add streamable-http mode
-    type=click.Choice(["stdio", "sse"]),
+    type=click.Choice(["stdio", "sse", "streamable-http"]),

# cli.py - handle the new mode
+        elif mode == "streamable-http":
+            await mcp_app.run_streamable_http_async()

# mcp.py - stateless mode for subagents
+        kwargs.setdefault("stateless_http", True)

# mcp.py - fix SSE disconnect crash
+            return Response()

The build script clones upstream, applies the patch, and builds:

git clone --depth 1 https://github.com/redis/agent-memory-server.git "$BUILD_DIR"
cd "$BUILD_DIR"
git apply mcp-patches.patch
docker build -f Dockerfile.patched --target standard -t "$IMAGE" .

The resulting image (piotrzan/agent-memory-server:0.13.2-custom-v4) is used by all three application pods. I've submitted a PR upstream to add streamable-http support natively, but until that lands you'll need a custom image.

Connecting MCP Clients​

The MCP server pod exposes port 9000. How you make it reachable depends on your setup: a NodePort, an Ingress, Gateway API, a LoadBalancer, or even kubectl port-forward for testing. The only requirement is that your MCP client can reach the /mcp endpoint over HTTP.

Any MCP client that supports HTTP transport can connect. The configuration is the same pattern across tools:

{
  "mcpServers": {
    "agent-memory": {
      "type": "http",
      "url": "http://<your-mcp-host>/mcp"
    }
  }
}

Replace <your-mcp-host> with whatever hostname or IP reaches your MCP service. That's it. Any MCP-capable tool (Claude Code, Cursor, Windsurf, Codex) now has persistent memory across all sessions on any machine that can reach the endpoint.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: agent-memory
spec:
  parentRefs:
  - name: homelab-gateway
    namespace: envoy-gateway-system
    sectionName: http
  hostnames:
  - "agent-memory.homelab.local"
  rules:
  - backendRefs:
    - name: agent-memory-mcp
      namespace: ai-tools
      port: 9000

What You Get​

Once connected, any MCP client gains these tools:

• create_long_term_memories -- store facts, preferences, patterns
• search_long_term_memory -- semantic search across all stored memories
• set_working_memory -- session-scoped context that auto-promotes to long-term
• get_working_memory -- recall current session state
• memory_prompt -- get a context-enriched prompt with relevant memories injected

Say "remember that I always use bun instead of npm" and it persists. Next session, on a different machine, in a different tool, the memory is there. The semantic search means it finds relevant memories even with different wording.

Why This Matters​

This setup solves a real problem: AI tools are stateless by default. Every session is a blank slate. Memory servers fix that, but most require cloud APIs that cost money and store your data on their infrastructure.

Think about what AI memories contain: details about your codebase, your infrastructure topology, your workflow habits, your credentials naming patterns. With cloud memory services, that data lives in someone else's database, governed by their retention policies. With this setup, the canonical store is yours. Redis on your PVC, embeddings computed by Ollama on your hardware, search indexes you control. When an MCP client retrieves a memory and sends it to Claude or Codex for inference, that's your choice for that interaction. The difference is who holds the source of truth. You decide what gets stored, what gets shared, and what gets deleted.

Running agent-memory-server on Kubernetes with Ollama gives you:

• You own the data -- memories live in your database, governed by your retention policies
• Zero inference costs -- Ollama on a spare Mac handles all embeddings and generation
• Kubernetes ops tooling -- health checks, rolling updates, PVC persistence, ArgoCD GitOps
• Multi-machine memory -- any device pointing at the MCP endpoint shares the same memory store

The stack is ~2.3GB RAM at limits (Redis 768Mi + three app pods at 512Mi each). Actual usage sits well below that; requests total ~640Mi (Redis 256Mi + 3x128Mi).

Links​

• Redis Agent Memory Server -- upstream project
• Model Context Protocol -- the MCP specification
• Ollama -- local LLM inference
• RediSearch -- vector similarity search in Redis
• Homelab repo -- full working manifests in gitops/apps/agent-memory-server/

Local Kubernetes development has rough edges. kind and minikube make cluster creation trivial, but common workflows need extra setup.

Exposing a LoadBalancer service means configuring MetalLB or tunneling through ngrok. Testing a new image means running kind load docker-image every iteration.

vind encapsulates these workflows. Kubernetes nodes run as Docker containers sharing the host daemon, so images pull straight from the local cache. A built-in LoadBalancer controller assigns real IPs without additional configuration.

vind​

vind (vCluster in Docker) runs the Kubernetes control plane and simulated nodes as Docker containers. They share your host's Docker daemon. Currently experimental but functional.

Prerequisites:

• Docker with containerd image store enabled (see below)
• vCluster CLI v0.31.0+ (install guide)
• Linux: bridge kernel modules loaded

Docker setup (required for registry proxy):

# Enable containerd image store
echo '{"features":{"containerd-snapshotter":true}}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

# Verify
docker info | grep "driver-type"
# Should show: driver-type: io.containerd.snapshotter.v1

Linux kernel modules (if node join fails):

sudo modprobe bridge br_netfilter

vind workflow:

# vcluster.yaml - need at least one node for pods to schedule
experimental:
  docker:
    nodes:
    - name: "worker-1"

# 1. Set Docker driver (once)
vcluster use driver docker

# 2. Create cluster with node config
vcluster create my-cluster --values vcluster.yaml

# 3. Deploy (images from your local Docker cache)
kubectl apply -f deployment.yaml
# LoadBalancer gets an IP

Your local Docker images are already available. LoadBalancer services get real IPs reachable from your machine.

vCluster Platform Free Tier​

vind works standalone. Connecting it to vCluster Platform's free tier adds management features.

Enterprise features now free:

• Sleep mode (pause/resume clusters)
• Templates and self-service
• User management and RBAC
• CRD sync and sync patches
• Embedded etcd

Limits:

• 64 vCPU cores
• 32 GPUs
• Unlimited users
• Unlimited host clusters
• Unlimited virtual clusters

Covers homelabs, dev environments, CI pipelines, and conference demos.

Activate free tier:

1. Deploy vCluster Platform following the installation guide
2. Enter your email when prompted in the Platform UI
3. Complete activation via the email link

Once connected, vind clusters appear in the Platform dashboard with options for multi-cluster management, team access sharing, and external node joining.

Registry Proxy​

The registry proxy is why vind iteration is faster than kind. Comparison:

kind:

docker build -t myapp:v2 .           # Build locally
kind load docker-image myapp:v2      # Copy entire image into cluster
kubectl set image deploy/app app=myapp:v2  # Deploy
# Wait for image copy... then deploy

vind:

docker build -t myapp:v2 .           # Build locally
kubectl set image deploy/app app=myapp:v2  # Deploy immediately
# Pod starts, registry proxy serves image from local Docker

The registry proxy runs inside vind and intercepts image pulls. When containerd inside the worker node requests myapp:v2, the proxy checks the host Docker daemon first. Local images are served directly.

This also works for public images. Pull nginx:latest once on your host, every vind cluster reuses it.

# On host
docker pull nginx:latest

# In vind cluster - instant, no network pull
kubectl run nginx --image=nginx:latest

LoadBalancer Services That Work​

With kind, LoadBalancer services stay <pending> without additional setup. NodePort or port-forward become the standard workarounds.

vind includes a LoadBalancer controller that assigns real IPs:

kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --type=LoadBalancer

kubectl get svc web
# NAME   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
# web    LoadBalancer   10.96.45.123   172.19.0.5    80:31234/TCP

That IP is routable from your host:

curl 172.19.0.5
# <!DOCTYPE html>
# <title>Welcome to nginx!</title>
# ...

Multiple LoadBalancer services get different IPs.

Multi-Node Clusters​

Test node affinity, pod anti-affinity, or DaemonSets with multiple workers:

# vcluster.yaml
experimental:
  docker:
    nodes:
    - name: "worker-1"
    - name: "worker-2"
    - name: "worker-3"

vcluster create multi-node --values vcluster.yaml
kubectl get nodes
# NAME       STATUS   ROLES    AGE
# worker-1   Ready    <none>   30s
# worker-2   Ready    <none>   30s
# worker-3   Ready    <none>   30s

Each node is a Docker container. Add more by adding lines to the config.

External Nodes via Platform​

With vCluster Platform Pro, you can join real machines to your vind cluster using vCluster VPN. This enables hybrid scenarios: your laptop runs the control plane, a Raspberry Pi or cloud VM joins as a worker.

# vcluster.yaml
experimental:
  docker:
    nodes:
    - name: local-node
privateNodes:
  vpn:
    enabled: true
    nodeToNode:
      enabled: true

vcluster create hybrid --values vcluster.yaml

The external machine connects through the Platform and appears as a node in kubectl get nodes. Test ARM workloads, edge scenarios, or distribute workloads across machines.

vind vs kind​

Choose vind when​

LoadBalancer services need real IPs without MetalLB. Image iteration is frequent (demos, rapid development). Clusters need pausing to free resources. A management UI would help.

Stick with kind when​

The workflow already works. CI/CD pipelines need maximum compatibility. Minimal dependencies matter more than features.

Resources​

• vind | vCluster in Docker | Across The Tenancy Spectrum - Video walkthrough
• vind GitHub
• vind Documentation
• Launching vCluster Free - Free tier announcement
• Platform Installation Guide

You've typed a 10-word kubectl command. You realize the namespace is wrong. Now you're holding Ctrl+Left six times to get there. Or worse, reaching for the mouse.

Vim solved this for text editing decades ago. Why are we still navigating command lines like it's 1985?

zsh-jumper brings fuzzy search to command line navigation. Press a key, pick a word, cursor jumps there. Or press ; and a letter for instant EasyMotion-style jumps.

The Problem​

Terminal commands grow long:

kubectl get pods -n kube-system --output wide --selector app=nginx

You need to edit kube-system. Options:

1. Arrow keys - Hold until you get there. Slow.
2. Ctrl+Left/Right - Jump word by word. Better, but still multiple presses.
3. Home then Ctrl+Right - Start from beginning. Indirect.
4. Mouse - Works, but your hands leave the keyboard.

None of these scale. A 15-word command means 15 potential targets.

The Solution​

$ kubectl get pods -n kube-system --output wide
                    ▲
              [Ctrl+X /]
                    │
→ [a]kubectl [s]get [d]pods [f]-n [g]kube-system [h]--output [j]wide   ← overlay
┌─────────────────────────────────────┬────────────────────────────────────────┐
│ jump>                               │ kubectl controls the Kubernetes...     │
│ ^S:wrap | ^E:var | ^R:replace | ^M:m│                                        │
│─────────────────────────────────────│                                        │
│> 1: kubectl                         │ Basic Commands (Beginner):             │
│  2: get                             │   create    Create a resource          │
│  3: pods                            │   expose    Expose a resource          │
│  4: -n                              │   run       Run a particular image     │
│  5: kube-system                     │   set       Set specific features      │
│  6: --output                        │                                        │
│  7: wide                            │ Basic Commands (Intermediate):         │
└─────────────────────────────────────┴────────────────────────────────────────┘

The overlay shows letter hints ([a]kubectl [s]get [d]pods) for instant jump. The picker has numbered items for fuzzy search, with --help preview on the right.

Type kube, hit enter. Cursor lands at kube-system. Edit, continue.

Or press ; then g to jump directly to kube-system. No fuzzy searching needed.

The mental model matches how you think: "I want to change the namespace" not "I need to move left 6 words."

Installation​

Requires fzf (or sk, peco, percol). Most of you have fzf already.

# zinit
zinit light Piotr1215/zsh-jumper

# antigen
antigen bundle Piotr1215/zsh-jumper

# oh-my-zsh
git clone https://github.com/Piotr1215/zsh-jumper \
    ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-jumper
# then add zsh-jumper to plugins in .zshrc

Default binding: Ctrl+X /

FZF Actions​

Beyond jumping, the picker offers token manipulation:

Variable extraction converts my-gpu to:

MY_GPU="my-gpu"
kubectl run test --image "$MY_GPU"

Instant mode: Press ; then a hint letter to jump directly. The overlay on your command line shows the mapping.

Replace mode is particularly useful: it deletes the token and leaves cursor in place, so you get full zsh tab completion for the replacement text.

For complex multiline editing, Ctrl+X Ctrl+E (edit in $EDITOR) still shines. zsh-jumper is for quick navigation and token manipulation.

Preview Panel​

The right panel shows contextual help:

• Commands: --help output (with bat highlighting), fallback to tldr, then man
• Files: Content preview via bat (or head)
• Directories: ls -la listing

This makes the picker double as a quick reference. Forgot what flags kubectl accepts? Just invoke the picker.

Why Word Numbering?​

The picker shows numbered words:

1: kubectl
2: get
3: -n
4: --user
5: admin

This prevents ambiguity. If you search for -u, you want -n at position 3, not a substring match inside --user. Numbers ensure exact cursor placement.

Configuration​

All options via zstyle, set before loading the plugin:

# Force specific picker
zstyle ':zsh-jumper:' picker fzf

# Custom picker options
zstyle ':zsh-jumper:' picker-opts '--height=50% --reverse --border'

# Where cursor lands: start (default), middle, end
zstyle ':zsh-jumper:' cursor end

# Custom keybinding
zstyle ':zsh-jumper:' binding '^J'

For tmux users: the plugin auto-detects tmux and uses fzf-tmux for a floating popup instead of inline fzf.

Vi Mode​

If you use vi mode in zsh, bind to both insert and command modes:

zstyle ':zsh-jumper:' disable-bindings yes
bindkey -M viins '^X/' zsh-jumper-widget
bindkey -M vicmd '^X/' zsh-jumper-widget

Workflow Tips​

The built-in actions handle most edits directly:

• Wrong value? Ctrl+R to replace with full tab completion
• Need quotes? Ctrl+S to wrap in "...", '...', or $(...)
• Reuse a value? Ctrl+E to extract to a variable
• Wrong order? Ctrl+M to swap positions

For anything else, jump to the word then use standard zsh:

• Alt+D - Delete word forward
• Ctrl+W - Delete word backward
• Ctrl+K - Kill to end of line

Multiline Commands​

Works with backslash continuations:

docker run -d \
    --name my-container \
    --network host \
    nginx:latest

Line continuation characters are filtered out. You see only actual command tokens in the picker.

Extensibility​

For power users, custom actions and previewers via TOML config:

zstyle ':zsh-jumper:' config ~/.config/zsh-jumper/config.toml

[[actions]]
binding = 'ctrl-u'
description = 'uppercase'
script = '~/.config/zsh-jumper/scripts/uppercase.sh'

[[previewers]]
pattern = '^https?://'
description = 'URL preview'
script = '~/.config/zsh-jumper/scripts/url-preview.sh'

Run zsh-jumper-list to see all registered actions and previewers.

Performance​

CI measures and enforces thresholds on every commit:

Live badges in the README show actual values from the latest CI run. The plugin loads in under 1ms and cleans up properly on unload.

Resources:

• zsh-jumper on GitHub
• fzf - Required fuzzy finder

You install a plugin for multi-cursor editing. Another for shell integration. A third for incremental search preview. Your plugin count grows, startup time increases, and you debug compatibility issues between packages.

Most of these features exist in vanilla Neovim. Have existed since Vim 7. Here are 10 built-in features that replace common plugins.

1. Shell Filter: ! and !!​

Pipe any text through external commands. Every Unix tool becomes a text processor.

The motion after ! defines the range. ip is "inner paragraph", ap is "around paragraph", % is whole file.

" Sort lines 10-20
:10,20!sort

" Reverse selection (visual mode, then !)
:'<,'>!tac

" Format current function as JSON
!i{jq .

This is the Unix philosophy applied to editing. sort, uniq, jq, column, sed - all available without plugins.

2. Visual Block Increment: g Ctrl-a​

Select a column of numbers, press g Ctrl-a. They become a sequence.

item 0          item 1
item 0    →     item 2
item 0          item 3
item 0          item 4

Standard Ctrl-a increments by the same amount. g Ctrl-a creates a sequence where each line increases by one more than the previous.

Works for generating IDs, numbered lists, array indices. No macro needed.

3. Global Command: :g/pattern/cmd​

Run any Ex command on every line matching a pattern.

The inverse is :v/pattern/cmd - runs on lines NOT matching the pattern.

" Delete all comments
:g/^#/d

" Move all imports to top
:g/^import/m0

" Number all TODO lines
:let i=1 | g/TODO/s/TODO/\=i.'. TODO'/ | let i+=1

This is bulk editing without regex gymnastics in the replacement string.

4. Command-line Registers: Ctrl-r​

In : or / prompts, Ctrl-r inserts register contents. Stop typing long paths and patterns manually.

Practical example: cursor on myFunction, want to search for it:

/Ctrl-r Ctrl-w<Enter>

Inserts myFunction into the search prompt. Works in substitute commands too:

:%s/Ctrl-r Ctrl-w/newName/g

The Ctrl-r = variant evaluates expressions:

:echo Ctrl-r =system('date')<Enter>

5. Normal on Selection: :'<,'>norm​

Run normal mode commands on every selected line. This is multi-cursor without plugins.

Select lines visually, then:

The norm command executes keystrokes as if typed in normal mode. Combined with a range, it applies to multiple lines.

" Convert list to array elements
" Before:      After:
" apple        "apple",
" banana  →    "banana",
" cherry       "cherry",

:'<,'>norm I"
:'<,'>norm A",

6. The g Commands​

Navigation jumps you probably don't use:

gi is the one I use most. Edit something, scroll away to reference code, gi puts you back exactly where you were typing.

g; and g, navigate the change list - every position where you modified text. Different from jump list (Ctrl-o/Ctrl-i) which tracks navigation.

7. Auto-Marks​

Vim sets marks automatically. You don't need to remember to set them.

The backtick goes to exact position. Apostrophe goes to line start.

(double backtick) is the most useful - it toggles between current position and previous. Edit something, jump to a definition, ``` ``` returns you instantly.

`[ and `] mark the boundaries of your last operation. Useful for reselecting: gv reselects visual, but `[v`] selects the range you just yanked or changed.

8. Command History Window: q:​

Press q: and your command history opens in a buffer. Full editing. Search with /. Modify any line. Press Enter to execute.

The third one is key. Start typing a complex command, realize you want to edit it properly, hit Ctrl-f. The command line becomes a buffer.

Compose multi-part commands, edit previous attempts, combine parts from different history entries. Then Enter executes the current line.

9. Live Substitution Preview: inccommand​

See substitution results before executing. Add to your config:

vim.opt.inccommand = "split"

Now when you type :%s/old/new/, matches highlight in real-time and a preview window shows what will change.

split shows preview in a split window. nosplit highlights inline only.

This catches regex mistakes before they happen. See exactly what :s/foo\(bar\)/\1/ will do before pressing Enter.

10. Copy/Move Lines: :t and :m​

Duplicate or relocate lines without touching registers. Your yank stays intact.

:t is copy (think "transfer"), :m is move. Both take a destination line number.

" Copy line 5 to after line 10
:5t10

" Move lines 1-3 to end of file
:1,3m$

" Duplicate current line 5 times
:t.|t.|t.|t.|t.

No register pollution. p still pastes what you yanked earlier.

Bonus: Mark-Based Ranges​

Set marks at function boundaries, substitute only within:

ma          " mark start with 'a'
... navigate to end ...
mb          " mark end with 'b'

:'a,'b s/old/new/g

Scoped refactoring without selecting anything. The substitution only affects lines between marks a and b.

Resources:

• Neovim documentation
• Video covering these features
• My dotfiles

You add a new tmux binding. A week later you're hunting through .tmux.conf because you can't remember what you mapped to prefix + g. The shortcuts.yaml you maintain? Three months out of date.

The problem isn't discipline. It's architecture. Any system requiring manual sync between source of truth (config files) and documentation (shortcuts file) will drift.

Parse your config files directly. For workflows that don't map to a single keybinding, use tealdeer custom pages. One popup, two modes, zero maintenance.

The Architecture​

Two types of "help" information exist:

Config-derived - patterns living in actual config files:

• tmux bindings in .tmux.conf
• zsh bindkeys in .zshrc
• aliases in .zsh_aliases
• abbreviations in .zsh_abbreviations

Workflow-derived - multi-step processes and reference material:

• "How do I copy in tmux?" (5 steps, not one keybinding)
• "What's the taskwarrior syntax?" (cheatsheet)

For config-derived, parse the files. For workflow-derived, use tealdeer custom pages.

confhelp​

I built confhelp to solve this. Define regex patterns in TOML, point at your config files:

pip install confhelp
confhelp --init  # creates ~/.config/confhelp/config.toml

# ~/.config/confhelp/config.toml
[tmux]
paths = [".tmux.conf"]
match_line = "^bind"
regex = 'bind(?:-key)?\s+(?:-n\s+)?(\S+)(.*)'
key_group = 1
desc_group = 2
type = "tmux"
truncate = 100

[alias]
paths = [".zsh_aliases", ".zsh_claude"]
regex = "alias\\s+(?:-[gs]\\s+)?([^=]+)=(.*)"
key_group = 1
desc_group = 2
type = "alias"
strip_quotes = true

[abbrev]
paths = [".zsh_abbreviations"]
mode = "abbrev_block"
type = "abbrev"

Run it:

confhelp -b ~/dotfiles            # list all entries
confhelp -b ~/dotfiles --select   # fzf selection, outputs file:line
confhelp -b ~/dotfiles --edit     # fzf selection, opens $EDITOR at line
confhelp -b ~/dotfiles --conflicts # find keys bound multiple times
confhelp -b ~/dotfiles --check    # find patterns your regex missed

Output is pipe-delimited: [type]|key|description|file:line

Each entry includes the exact file and line number. The --edit flag gives you jump-to-definition out of the box.

Set base_dirs in your config.toml and you can just run confhelp without the -b flag.

fzf Integration​

For advanced workflows (like combining with tealdeer), a wrapper script spawns a centered popup with mode switching:

selection=$(confhelp -b "$DOTFILES" | column -t -s'|' | fzf \
    --header='Enter=jump | Ctrl+G=tealdeer' \
    --bind='ctrl-g:become(echo SWITCH_TLDR)' \
    --height=100% \
    --layout=reverse)

if [[ -n "$selection" ]]; then
    file_line=$(echo "$selection" | awk '{print $NF}')
    file=$(echo "$file_line" | cut -d: -f1)
    line=$(echo "$file_line" | cut -d: -f2)
    nvim "+$line" "${DOTFILES}/${file}"
fi

The become() action replaces fzf with a new command without closing the terminal. Critical for smooth mode switching between bindings and tealdeer.

Tealdeer Custom Pages​

Tealdeer reads custom pages from a configured directory:

# ~/.config/tealdeer/config.toml
[directories]
custom_pages_dir = "/home/decoder/.local/share/tealdeer/pages"

Custom pages need .page.md extension:

# tmux-copy-mode

> Tmux copy mode with vi keybindings

- Enter copy mode:

`prefix + [`

- Start selection:

`v` or `V` for line

- Copy selection:

`y`

Ctrl+G in the bindings popup switches to tealdeer browser. Custom pages appear with [custom] marker.

Result​

Add a binding to .tmux.conf, it appears in the popup immediately. No yaml to update.

Gotchas​

1. Tealdeer file extension - Custom pages must end in .page.md, not .md.

2. fzf become() vs execute() - execute() runs a command and returns to fzf. become() replaces fzf entirely. For mode switching, you need become().

3. Column alignment - The parser outputs pipe-separated values. column -t -s'|' aligns them.

Resources:

• confhelp on PyPI
• confhelp on GitHub
• Example wrapper script
• tealdeer docs
• Inspired by Extracto

Photo by Paul Hanaoka on Unsplash

The Kubernetes scheduler is lazy. It places pods when they're created, picks the best node at that moment, and never thinks about it again. Weeks later, one node is at 75% memory while another sits at 40%. The scheduler doesn't care - its job was done the moment the pod started.

Descheduler fixes this. It runs every few minutes, finds imbalanced nodes, and evicts pods so the scheduler gets another chance to place them better. Set it up once, cluster self-balances automatically.

Here's how to configure it for a homelab. Takes about 20 minutes to get right.

Why This Matters​

Kubernetes scheduling is a point-in-time decision. When a pod is created, the scheduler picks the best available node based on current conditions. But clusters are dynamic - nodes get added, pods come and go, resource requests change with updates.

After a few weeks, you end up with hot nodes and cold nodes. Here's what my homelab looked like:

NAME           CPU(cores)   MEMORY%
kube-worker1   438m         75%       <-- carrying the load
kube-worker4   266m         41%       <-- sitting idle
kube-worker6   171m         42%       <-- sitting idle

Worker1 was handling most of the work while worker4 and worker6 did nothing.

Requests vs Actual Usage​

Before configuring descheduler, you need to understand what it actually measures. This is where most people get it wrong.

Descheduler uses REQUESTED resources, not actual usage.

A node might show 5% actual CPU usage but 70% requested. From a scheduling perspective, that node is "full" - the scheduler reserved that capacity even if the pods aren't using it.

Check what descheduler sees:

kubectl describe node kube-worker1 | grep -A5 "Allocated resources"

Allocated resources:
  Resource           Requests         Limits
  --------           --------         ------
  cpu                2368m (69%)      5490m (161%)
  memory             4769334Ki (65%)  9343689984 (126%)

Those percentages (69% CPU, 65% memory) are what descheduler uses. Not kubectl top nodes.

Install Descheduler​

Descheduler runs as a CronJob. The Helm chart is the easiest way:

# values.yaml
kind: CronJob
schedule: "*/5 * * * *"

deschedulerPolicy:
  profiles:
    - name: default
      pluginConfig:
        - name: DefaultEvictor
          args:
            ignorePvcPods: true
            evictLocalStoragePods: false
        - name: LowNodeUtilization
          args:
            thresholds:
              cpu: 55
              memory: 30
              pods: 30
            targetThresholds:
              cpu: 70
              memory: 70
              pods: 50
      plugins:
        balance:
          enabled:
            - LowNodeUtilization

Deploy:

helm install descheduler descheduler \
  --repo https://kubernetes-sigs.github.io/descheduler/ \
  --namespace descheduler \
  --create-namespace \
  -f values.yaml

Understanding Thresholds​

This is where I wasted an hour. LowNodeUtilization has two threshold sets that work differently:

thresholds - defines UNDERUTILIZED nodes

• A node is underutilized when ALL metrics are BELOW these values
• These nodes receive evicted pods

targetThresholds - defines OVERUTILIZED nodes

• A node is overutilized when ANY metric is ABOVE these values
• Pods get evicted FROM these nodes

                    thresholds        targetThresholds
                         |                   |
  UNDERUTILIZED         |    BALANCED       |    OVERUTILIZED
  (receives pods)       |                   |    (evicts pods)
  <---------------------|-------------------|-------------------->
         0%            55%                 70%               100%

Descheduler evicts pods from overutilized nodes. The scheduler then places them on underutilized nodes.

Critical: If no nodes qualify as underutilized, descheduler does nothing. If no nodes qualify as overutilized, descheduler does nothing. Both conditions must be true.

Tuning for Your Cluster​

The default thresholds (cpu: 20, memory: 20) assume your nodes have low resource requests. Most real clusters have higher utilization - mine certainly did.

Check your actual request percentages first:

for node in $(kubectl get nodes -o name | cut -d/ -f2); do
  echo "=== $node ==="
  kubectl describe node $node | grep -A3 "Allocated resources" | grep -E "cpu|memory"
done

Then set thresholds based on what you see:

• If your coldest node is at 45% CPU requests, set thresholds.cpu: 55 (above it)
• If your hottest node is at 75% memory requests, set targetThresholds.memory: 70 (below it)

I started with the defaults and descheduler did nothing. My coldest node had 45% CPU requests - above the 20% threshold. No node qualified as "underutilized" so there was nowhere to put evicted pods.

After checking my actual cluster state:

thresholds:
  cpu: 55      # nodes below 55% CPU are underutilized
  memory: 30   # nodes below 30% memory are underutilized
  pods: 30
targetThresholds:
  cpu: 70      # nodes above 70% CPU are overutilized
  memory: 70   # nodes above 70% memory are overutilized
  pods: 50

Protecting PVC Pods​

Not all pods should be evicted. This is important:

- name: DefaultEvictor
  args:
    ignorePvcPods: true           # Don't evict pods with PVCs
    evictLocalStoragePods: false  # Don't evict pods with emptyDir

Why protect PVC pods? Longhorn and similar storage use ReadWriteOnce volumes. Evicting a pod means:

1. Old pod terminates
2. Volume detaches from old node
3. New pod schedules on different node
4. Volume attaches to new node
5. Pod starts

If step 2 doesn't complete before step 4, you get Multi-Attach errors. The new pod hangs waiting for the volume while the old node still has it attached.

I had ignorePvcPods: false initially. Got several Multi-Attach errors before I figured this out.

Verify It's Working​

After deploying, watch for LowNodeUtilization events:

kubectl get events -A | grep -i "LowNodeUtilization"

prometheus   Normal  LowNodeUtilization  pod/kube-state-metrics-7c8b8bf58c-88qqq
  pod eviction from kube-worker1 node by sigs.k8s.io/descheduler

vpa          Normal  LowNodeUtilization  pod/vpa-updater-f59cccc88-fp2t6
  pod eviction from kube-worker1 node by sigs.k8s.io/descheduler

Check node balance:

kubectl top nodes

Results​

Worker1's request percentage dropped from 75% to 65%. The cluster is balanced now, and descheduler runs every 5 minutes to keep it that way.

Gotchas​

1. Thresholds are percentages of REQUESTS - Don't use kubectl top nodes to set thresholds. Use kubectl describe node to see request percentages.

2. ALL metrics must be below thresholds for underutilized - If you set cpu: 20 but your coldest node has 45% CPU requests, no node qualifies as underutilized. Descheduler does nothing.

3. The comparison is greater-than, not greater-than-or-equal - A node at exactly 70% CPU with targetThresholds.cpu: 70 is NOT overutilized. 70 is not greater than 70.

4. Jobs get auto-deleted - CronJob default history limits are low. If debugging, set successfulJobsHistoryLimit: 3 to keep completed jobs around for log inspection.

5. DaemonSet pods can't be evicted - Don't count them when calculating node utilization. A node with 8 DaemonSet pods still has room for workloads.

Resources:

• Descheduler GitHub
• LowNodeUtilization docs
• My homelab descheduler config

Photo by todd kent on Unsplash

MinIO Community Edition is dead. On December 3, 2025, MinIO Inc. announced maintenance mode: no new features, no PR reviews, no Docker images, no RPM/DEB packages. Critical security fixes only "on a case-by-case basis."

This didn't come out of nowhere. Back in May 2025, they gutted the console - the GUI that made MinIO actually usable. What's left is a glorified file browser. User management, policies, replication config? Moved to the paid AIStor product. The whole thing is open source cosplay now - the repo exists, but it's just a funnel to their commercial offering.

The r/selfhosted and Hacker News threads are worth reading. Thousands of Helm charts and CI/CD pipelines depending on minio/minio images are now broken. Bitnami stopped their MinIO builds too.

Time to migrate. Honestly? For a homelab, Garage is the better choice anyway. 50MB footprint vs 500MB+. Written in Rust. Built-in static web hosting. Actively maintained. MinIO's collapse just forced me to make the switch I should have made earlier.

Here's how to set up Garage on Kubernetes. Takes about 15 minutes.

Why Garage?​

Garage supports multi-node clusters with built-in replication. For a single-node setup where your storage layer (like Longhorn) already handles redundancy, single-node Garage works perfectly.

Deploy Garage​

This section covers a basic Kubernetes deployment. You can adapt it to your setup - Docker, bare metal, whatever. The official docs cover other deployment methods.

Create Namespace and Secrets​

kubectl create namespace garage

# Generate secrets
RPC_SECRET=$(openssl rand -hex 32)
ADMIN_TOKEN=$(openssl rand -hex 32)

# Store them (save these somewhere safe!)
echo "RPC Secret: $RPC_SECRET"
echo "Admin Token: $ADMIN_TOKEN"

kubectl create secret generic garage-secrets \
  --from-literal=rpc-secret=$RPC_SECRET \
  --from-literal=admin-token=$ADMIN_TOKEN \
  -n garage

ConfigMap​

Garage uses a TOML config file. Create a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: garage-config
  namespace: garage
data:
  garage.toml: |
    metadata_dir = "/data/meta"
    data_dir = "/data/blocks"
    db_engine = "lmdb"
    replication_factor = 1

    rpc_bind_addr = "[::]:3901"
    rpc_public_addr = "garage.garage.svc.cluster.local:3901"
    rpc_secret_file = "/secrets/rpc_secret"

    [s3_api]
    s3_region = "garage"
    api_bind_addr = "[::]:3900"

    [s3_web]
    bind_addr = "[::]:3902"

    [admin]
    api_bind_addr = "[::]:3903"
    admin_token_file = "/secrets/admin_token"

Secret File Permissions​

Garage reads secrets from files, not environment variables. When Kubernetes mounts a Secret as a volume, the files default to mode 0644 (world-readable). Garage refuses to start with world-readable secret files - it's a security check built into the binary.

The fix is defaultMode: 0600 on the volume mount:

volumes:
  - name: secrets
    secret:
      secretName: garage-secrets
      defaultMode: 0600
      items:
        - key: rpc-secret
          path: rpc_secret
        - key: admin-token
          path: admin_token

Skip this and Garage exits with "secret file has insecure permissions".

Deployment​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: garage
  namespace: garage
spec:
  replicas: 1
  strategy:
    type: Recreate  # Important for RWO PVCs
  selector:
    matchLabels:
      app: garage
  template:
    metadata:
      labels:
        app: garage
    spec:
      containers:
      - name: garage
        image: dxflrs/garage:v2.1.0
        ports:
        - containerPort: 3900  # S3 API
        - containerPort: 3901  # RPC
        - containerPort: 3902  # Web
        - containerPort: 3903  # Admin
        volumeMounts:
        - name: data
          mountPath: /data
        - name: config
          mountPath: /etc/garage.toml
          subPath: garage.toml
        - name: secrets
          mountPath: /secrets
          readOnly: true
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: garage-data
      - name: config
        configMap:
          name: garage-config
      - name: secrets
        secret:
          secretName: garage-secrets
          defaultMode: 0600
          items:
          - key: rpc-secret
            path: rpc_secret
          - key: admin-token
            path: admin_token

Service​

apiVersion: v1
kind: Service
metadata:
  name: garage
  namespace: garage
spec:
  selector:
    app: garage
  ports:
  - name: s3
    port: 3900
  - name: web
    port: 3902
  - name: admin
    port: 3903

PVC​

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: garage-data
  namespace: garage
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Apply everything, wait for the pod to start.

Initialize Garage​

Garage needs a "layout" before it accepts data. This tells it how much storage to use and which zone the node belongs to.

# Get the node ID
kubectl exec -n garage deploy/garage -- /garage status

# You'll see something like:
# ==== HEALTHY NODES ====
# ID                Hostname  Address         Tags  Zone  Capacity
# 563e...          garage    10.42.0.1:3901              NO ROLE

# Assign storage (use your node ID)
kubectl exec -n garage deploy/garage -- \
  /garage layout assign -z dc1 -c 50GB 563e

# Apply the layout
kubectl exec -n garage deploy/garage -- \
  /garage layout apply --version 1

Now create an access key:

kubectl exec -n garage deploy/garage -- \
  /garage key create garage-admin

# Output:
# Key name: garage-admin
# Key ID: GK0ff60c017ac3f70efb9772f4
# Secret key: 598d28be9a91fc0b0e854454419f091cd6a704b2c121e8a99eab8f9e964e1bf0

Save these. You'll need them for any S3 client.

Test It​

Create a bucket and upload something:

kubectl exec -n garage deploy/garage -- \
  /garage bucket create test-bucket

kubectl exec -n garage deploy/garage -- \
  /garage bucket allow --read --write test-bucket --key garage-admin

From outside the cluster, use AWS CLI:

aws configure --profile garage
# Access Key: GK0ff60c017ac3f70efb9772f4
# Secret Key: (your secret)
# Region: garage
# Output format: json

aws --profile garage --endpoint-url http://garage.garage.svc:3900 \
  s3 ls

That's it. You have a working S3-compatible storage system.

My Homelab Integration​

The above is all you need for basic Garage. The rest of this post covers how I integrated it into my specific homelab setup: GitOps secrets management, ingress routing, automation, and document sync.

My homelab runs on a 7-node Kubernetes cluster (1 control plane, 6 workers) across 3 Proxmox hosts. Storage is Longhorn, ingress is Envoy Gateway, everything deploys via ArgoCD from a GitOps repo.

Secrets with External Secrets Operator​

The basic setup above uses kubectl create secret. That works, but for GitOps you need secrets in your repo - and committing plaintext secrets is a security risk.

I use External Secrets Operator (ESO) with Bitwarden Secrets Manager:

1. Generate secrets locally with openssl rand -hex 32
2. Store them in Bitwarden Secrets Manager (gives you a UUID)
3. Create an ExternalSecret that references the UUID
4. ESO syncs the secret from Bitwarden into Kubernetes

The ExternalSecret references UUIDs, not values - safe to commit:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: garage-secrets
  namespace: garage
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: bitwarden-secretsmanager
    kind: ClusterSecretStore
  target:
    name: garage-secrets
  data:
  - secretKey: rpc-secret
    remoteRef:
      key: 7b5d53a8-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  - secretKey: admin-token
    remoteRef:
      key: 8c6e64b9-xxxx-xxxx-xxxx-xxxxxxxxxxxx

HTTPRoutes with Envoy Gateway​

Instead of NodePort or LoadBalancer, I use Envoy Gateway with HTTPRoutes. For simple services, my httproute-controller generates HTTPRoutes from Service annotations automatically. Here's the Garage config:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: garage-s3
  namespace: envoy-gateway-system
spec:
  parentRefs:
  - name: homelab-gateway
    sectionName: https
  hostnames:
  - "s3.garage.homelab.local"
  - "*.s3.garage.homelab.local"
  rules:
  - backendRefs:
    - name: garage
      namespace: garage
      port: 3900

The wildcard *.s3.garage.homelab.local enables virtual-hosted bucket access (mybucket.s3.garage.homelab.local).

For cross-namespace routing, you need a ReferenceGrant in the garage namespace allowing the HTTPRoute to reference the Service.

WebUI​

Garage has no built-in UI. garage-webui fills that gap - runs as a sidecar, connects to Garage's admin API on port 3903.

See my deployment manifest for the full setup with WebUI.

Justfile Recipes​

I use a justfile for cluster operations. Two recipes for Garage:

• garage-bucket - creates buckets with read/write permissions
• garage-upload - uploads files/folders, extracts credentials from Garage automatically

rclone Bisync​

For syncing local folders to Garage, I use rclone with bisync. My ~/.config/rclone/rclone.conf:

[garage]
type = s3
provider = Other
access_key_id = GK0ff60c017ac3f70efb9772f4
secret_access_key = (your secret)
endpoint = https://s3.garage.homelab.local
region = garage
no_check_bucket = true

Two gotchas:

• region = garage is required. Without it, rclone defaults to us-east-1 and Garage rejects with "AuthorizationHeaderMalformed"
• Self-signed certs need --no-check-certificate on every command

I run bisync daily via cron with local folder as source of truth.

Static Web Hosting​

Garage has built-in static web hosting. Enable it per bucket in the WebUI (or via CLI), access at https://<bucket>.web.garage.homelab.local. No nginx, no separate web server.

Gotchas​

1. Layout must be applied - Garage won't accept data until you assign capacity and apply the layout. The pod starts fine, but S3 operations fail.

2. PVC access mode - Use ReadWriteOnce with Recreate strategy. Rolling updates hang if the old pod holds the PVC.

3. WebUI needs Garage v2 - The webui uses v2 admin API. Don't use Garage v1.x.

4. HTTPRoute namespace - If using Gateway API, routes in default namespace won't match ReferenceGrants for your gateway namespace.

Results​

450MB less RAM. Built-in static web hosting.

Resources:

• Garage Documentation
• garage-webui
• My homelab repo
  • Garage deployment
  • HTTPRoutes
• rclone S3 docs

Mozilla just named a new CEO. The headline promise? Firefox will "evolve into a modern AI browser." I read that and immediately started looking for alternatives.

This is a quick migration guide, not comprehensive documentation. Check LibreWolf's official docs for your specific system.

I was worried about breaking things—my setup has shell aliases for profile switching, wrapper scripts, keyboard shortcuts that launch Firefox, and window management automation. Everything migrated in about 30 minutes. If it worked for me, it'll work for you.

Why Leave Firefox?​

Mozilla's December 2025 announcement outlined three priorities:

1. AI must be "clear and understandable"
2. Revenue diversification beyond search
3. Firefox growing into "a broader ecosystem"

None of these say "leave users alone."

The Alternatives​

My pick: LibreWolf. It's Firefox without Mozilla's telemetry. Same Gecko engine, same extensions, same workflow. uBlock Origin comes pre-installed. Updates track Firefox releases within days (currently at Firefox 146). Firefox Sync is disabled by default but you can enable it.

Installation​

Multiple options. This guide uses Flatpak:

flatpak install flathub io.gitlab.librewolf-community

I used Flatpak because it's available on Pop!_OS. Trade-off: slower startup, profiles live in ~/.var/app/io.gitlab.librewolf-community/.librewolf/.

For webcam access (video calls), grant device permissions:

flatpak override --user io.gitlab.librewolf-community --device=all

If you choose native install, adjust paths from ~/.var/app/io.gitlab.librewolf-community/.librewolf/ to ~/.librewolf/, and binary from flatpak run io.gitlab.librewolf-community to librewolf.

Migrating Data​

LibreWolf can import everything from Firefox in one step:

Settings → General → Import Browser Data → Import Data

This imports bookmarks, passwords, history, and autofill data.

Fix Broken Sites​

LibreWolf's strict tracking protection can break some sites. Enable this:

Settings → Privacy & Security → Enhanced Tracking Protection → Fix major site issues (recommended)

Keep Logins Persistent​

By default, LibreWolf deletes cookies when closed—you'll have to re-login everywhere. Either disable it or add exceptions for sites you use:

Settings → Privacy & Security → Delete cookies and site data when LibreWolf is closed

Setting Up Profiles​

LibreWolf supports Firefox-style profiles:

# Open profile manager
flatpak run io.gitlab.librewolf-community -P

# Or launch specific profile
flatpak run io.gitlab.librewolf-community -P "Work"

If you use multiple Firefox profiles, create matching ones in LibreWolf's profile manager, then copy the data:

# Copy bookmarks and passwords (adjust profile names)
cp ~/.mozilla/firefox/PROFILE_NAME/places.sqlite \
   ~/.var/app/io.gitlab.librewolf-community/.librewolf/PROFILE_NAME/

cp ~/.mozilla/firefox/PROFILE_NAME/logins.json \
   ~/.var/app/io.gitlab.librewolf-community/.librewolf/PROFILE_NAME/

cp ~/.mozilla/firefox/PROFILE_NAME/key4.db \
   ~/.var/app/io.gitlab.librewolf-community/.librewolf/PROFILE_NAME/

Extensions​

Same Firefox Add-ons store. uBlock Origin comes pre-installed.

Extension settings don't transfer automatically, but most extensions support exporting settings to JSON. Vimium-C, uBlock Origin, and others have Settings → Export/Import. Quick way to move your keybindings and configurations.

Updating Dotfiles​

Shell Aliases​

Update any aliases that reference firefox:

alias firefox='flatpak run io.gitlab.librewolf-community'

System-Wide Wrapper​

Aliases don't work in scripts or desktop files. Create a wrapper so all firefox calls use LibreWolf:

sudo tee /usr/local/bin/firefox << 'EOF'
#!/bin/bash
exec flatpak run io.gitlab.librewolf-community "$@"
EOF
sudo chmod +x /usr/local/bin/firefox

This is the key trick—no changes needed to existing scripts. Everything that calls firefox just works.

Default Browser​

xdg-settings set default-web-browser io.gitlab.librewolf-community.desktop

Verify:

xdg-settings get default-web-browser
# io.gitlab.librewolf-community.desktop

What Changes​

The Result​

Everything works. Firefox stays installed as backup for now.

sudo apt remove firefox

Last week I returned from KCD UK where I led a workshop introducing people to vCluster (try it yourself here). At the workshop and at our booth, we fielded dozens of questions from people with wildly different backgrounds.

Some attendees had deep Kubernetes expertise but had never heard of virtual clusters. Others worked with containers daily and were exploring the orchestration layer. People came from different knowledge bases and experience levels.

The most common question? "What is vCluster, and why would I need it?"

The short answer: vCluster is an open-source solution that enables teams to run virtual Kubernetes clusters inside existing infrastructure. These virtual clusters are Certified Kubernetes Distributions that provide strong workload isolation while running as nested environments on top of another Kubernetes cluster.

But to understand why that matters—and whether you need it—you need to see how we got here.

By understanding this evolution, you'll see why virtual Kubernetes clusters solve real multi-tenancy problems that traditional approaches can't.

The Computing Revolution​

In the beginning, we had computers. The ENIAC filled entire rooms in 1945. The Apple II in 1977 and IBM PC in 1981 brought computation to offices and homes. You write code, it runs on that hardware. Simple.

This worked pretty well until you needed to deploy fleets of them. Then the problems became obvious:

The Problems:

• Single point of failure: If the machine fails, your computation stops. No redundancy. No failover. Just downtime.
• Slow scaling: Buying another physical machine means racking it, cabling it, installing the OS, and configuring it. Weeks of lead time for capacity you might only need during peak hours.
• Resource waste: Expensive, inflexible hardware sitting idle.

Better Together: Distributed Computing​

The obvious first step was networking. ARPANET connected four university computers in 1969. The client-server model emerged in the 1980s. By the 1990s, Beowulf clusters let researchers build supercomputers from commodity hardware.

Now you could distribute work. One machine runs your database. Another runs your web server. A third handles batch processing. If one fails, the others keep running. You have redundancy and you can scale by adding more machines.

This was better. But it created new problems:

The Problems:

• Stranded resources: Your database server runs at 80% CPU while your web server idles at 15%. You can't dynamically shift capacity between machines.
• Operational complexity: Tracking which workload runs where, handling networking and security manually, constantly racking hardware and swapping failed drives.
• Cost inefficiency: Wasting money on idle hardware that can't be repurposed without physically moving cables and reinstalling operating systems.

Virtual Machines Let Us Share Physical Hardware​

Then someone had a brilliant idea: what if we could run multiple "virtual" computers on one physical machine?

VMware brought this to x86 in 1998. Xen followed in 2003. KVM merged into Linux in 2007. A hypervisor sits between hardware and operating systems, creating virtual machines that behave like physical hardware.

This was significantly better:

The Benefits:

• Efficient resource sharing: Run multiple VMs on one physical server. Put that 80% CPU database and 15% CPU web server on the same host.
• Dynamic provisioning: Create VM templates, spin them up and down based on demand, migrate between hosts without downtime.
• Dramatic improvement: Resource utilization improved, provisioning went from weeks to minutes.

But then we realized VMs were bloated and heavyweight.

The Problems:

• Massive overhead: Each VM runs its own kernel, system libraries, daemons, and network stack—gigabytes of redundant code.
• Slow operations: Booting takes minutes. Downloading and distributing VM images is slow. Migration means moving entire OS images.
• Management burden: Every VM needs patching, monitoring, and management.

Virtual Machines Can Be Too Slow​

Then someone smart figured out that you could build an abstraction that looks like a regular OS from the perspective of the software running inside, but when that software makes a system call, it goes to the host machine instead.

LXC (Linux Containers) emerged in 2008 using Linux namespaces and cgroups. But Docker in 2013 made containers actually usable for everyone.

The key insight: all that extra OS overhead gets shared. A container image packages your application and its dependencies—nothing more. No full OS. No kernel. Just your code and the libraries it needs.

The Benefits:

• Lightning fast: Containers start in milliseconds instead of minutes.
• Resource efficient: Megabytes instead of gigabytes.
• High density: Run hundreds of containers on a single host.
• Developer productivity: Replicate production environments locally, deploy hundreds of times per day.

Docker became very popular and soon people started building all sorts of different containers. A typical deployed system has at least three components: the application, a database, and maybe a proxy like nginx or a cache like redis.

These components need to work together:

• Database comes online first
• Then redis, then the app, then the proxy
• Each needs to check the health of the previous one
• Your app can't start until the database is ready
• The proxy can't route traffic until the app is healthy

Containers Rarely Work Alone​

You need to orchestrate your containers. Docker Compose did this simply for a single host.

But what about across dozens or hundreds of hosts?

• Which host runs which container?
• How do containers find each other across machines?
• What happens when a host fails?
• How do you route traffic?
• How do you handle dynamic autoscaling based on load?
• How do you distribute containers across hosts for redundancy?
• How do you handle rolling updates without downtime?

Containers at Scale Need Orchestration​

Google had been running Borg internally since 2003, managing billions of containers. They knew the answers. In 2014 they open-sourced Kubernetes and brought that experience to everyone.

The key insight: doing this declaratively is very powerful because it's both standardized and reproducible. You don't say "start this container on node-7." You say "I want 3 replicas of this app, each needs 512MB memory" and Kubernetes figures out where to run them.

Kubernetes treats a cluster of machines as a single compute resource. Scheduling, service discovery, load balancing, self-healing, autoscaling—it handles all of it. The control plane continuously converges reality toward your declared desired state.

This let us build entire compute platforms around this concept. Deploying stuff is really complicated—tons of little knobs and dials needing to be turned and tweaked. In the old days everyone had bespoke frameworks and it was super inefficient.

If we capture those abstractions in a standardized API and make it flexible enough to satisfy lots of use cases, one engineer can work on and scale up many different deployments. The system itself can self-heal if there's a problem.

Platform teams need to serve many internal customers—Team A, Team B, Team C, maybe dozens of teams. You want to give them self-service Kubernetes access. But Kubernetes namespaces aren't strong enough for true isolation.

Consider what actually happens in production:

The Problems:

• CRD conflicts: Team A deploys a service mesh with cluster-wide CRDs. Team B already has a different version of those CRDs. The conflict causes Team B's applications to fail during deployment. No way to isolate cluster-scoped resources.

• Version lock-in: You need to upgrade to Kubernetes 1.30 for a security patch. Team C has a critical application that breaks on 1.30. You can't upgrade safely, but you can't stay on a vulnerable version either.

• Scheduler conflicts: Team D wants a custom scheduler for GPU workloads. Team E needs the default scheduler. There's only one scheduler per cluster.

• Blast radius: A platform team member accidentally installs a broken admission webhook. It impacts every namespace in the cluster. All deployments across all teams start failing. Debugging requires coordinating across 15 teams.

The financial impact is significant:

According to a 2024 ITIC study, 98% of organizations report that a single hour of downtime costs over $100,000, with the average reaching $9,000 per minute across industries. When a shared control plane issue affects multiple teams simultaneously, these costs multiply rapidly.

Kubernetes namespaces provide resource isolation, but they don't provide control plane isolation. Teams share the same API server, scheduler, controllers, and admission webhooks. This creates operational constraints that don't scale.

The alternative? Spin up separate clusters per team. But now you're managing dozens of physical clusters—each with its own control plane, nodes, and operational overhead. Expensive and operationally complex.

Platform teams needed better isolation without multiplying management overhead.

Virtual Clusters: Lightweight Kubernetes Clusters​

This is where we circle back to the beginning. Remember how VMs solved the problem of stranded resources on physical hardware? The same insight applies here.

vCluster runs Kubernetes control plane components (API server, controller manager, and optionally a scheduler) inside a single pod on a host cluster. A syncer component bridges the virtual and host clusters, translating resources between them. The virtual cluster stores its state in SQLite or etcd, completely isolated from the host.

From the tenant's perspective, they have a full Kubernetes cluster:

How It Solves The Problems:

• Independent upgrades: Each team controls their Kubernetes version. Team A tests 1.30 while Team B stays on 1.28. No coordination required.

• Flexible service sharing: Teams choose what to share:

  • Fully shared: Use the host cluster's ingress controller and cert-manager
  • Partially shared: Share some services, run others independently within the vCluster
  • Fully isolated: Run all services independently within the vCluster

• Flexible compute sharing: Teams choose how to share infrastructure:

  • Shared nodes: Multiple vClusters share the same host cluster nodes for cost efficiency
  • Private nodes: Teams can join dedicated nodes directly to their vCluster for workloads requiring complete isolation (GPU training, compliance requirements)
  • Hybrid: Mix shared and private nodes based on workload needs

• True control plane isolation: Each vCluster has its own API server, scheduler, and controllers. A broken admission webhook in vCluster A doesn't affect vCluster B.

The cost efficiency is dramatic. You run dozens of vClusters on a single host cluster, sharing node capacity efficiently. One platform engineer can serve hundreds of teams without managing hundreds of physical clusters.

Recent developments have expanded the flexibility:

• Private Nodes (v0.27) let teams join dedicated nodes directly to their vCluster for workloads requiring complete isolation—like GPU training jobs or compliance-sensitive applications.
• Auto Nodes (v0.28) bring Karpenter-based elastic scaling to any infrastructure, giving teams dynamic compute without manual node management.
• vCluster Standalone (v0.29) removes the dependency on a host cluster entirely, allowing installation directly on bare metal or VMs.

You can tailor isolation to your needs: shared nodes for cost efficiency, private nodes for GPU workloads or security requirements, or standalone for complete infrastructure control.

Each abstraction solved a real problem. Each created a new one.

vClusters solve multi-tenancy but add another abstraction layer. Debugging requires understanding both vCluster and host cluster layers. Networking configuration becomes more complex. Permissions need careful mapping between vCluster and host resources.

But the trade-off is favorable if you need multi-tenant Kubernetes at scale. The cost savings and operational simplification outweigh the added complexity.

The Pattern​

You see the progression:

• Physical computers → VMs (resources were stranded)
• VMs → Containers (OS overhead was too heavy)
• Containers → Kubernetes (orchestration was chaos)
• Kubernetes → vClusters (multi-tenancy was broken)

You trade one kind of complexity for another. You move problems up the stack. Physical constraints become logical constraints. Hardware limitations become software limitations.

The question is never "should we abstract?" The question is "what complexity are we willing to trade?"

Try It Yourself​

If you want hands-on experience with vClusters, try the Killercoda interactive tutorial I used at the workshop. You'll see exactly how vClusters provide isolation, how they integrate with host clusters, and what the operational model looks like.

Understanding the history and first principles makes the technology click. You'll see vClusters not as "yet another tool" but as the logical next step in a progression that started with physical computers six decades ago.

Resources​

Learn More:

• vCluster Documentation
• Killercoda Interactive Tutorial - The hands-on workshop from KCD UK
• Kubernetes Multi-Tenancy Working Group

Watch My Talks:

• Flexible Multi-Tenancy with vCluster - SREDay London 2025 (event page)
• The Future of Kubernetes Tenancy - Webinar with vCluster CEO

Events:

• KCD UK Edinburgh 2025 - Where this blog idea was born

Every abstraction in computing history emerged to solve a real problem. Virtual clusters are no different. If you're running multi-tenant Kubernetes at scale, dealing with version conflicts, or managing dozens of teams with different requirements, vClusters offer the control plane isolation that namespaces can't provide.

Start with the interactive tutorial, explore the documentation, and see how virtual clusters can transform your Kubernetes multi-tenancy strategy.

Building a Deterministic vCluster Validation MCP Server to Ground AI in Real Schemas

You ask an AI to generate a Kubernetes manifest, Helm chart values, or Ansible playbook. It responds instantly with clean, well-formatted YAML. You apply it. Nothing works.

This isn't a bug—it's AI hallucination. The AI knows YAML syntax but hallucinates config options that don't exist, mixes incompatible versions, or confidently suggests deprecated fields. It generates what looks right based on patterns, not what is right according to actual schemas.

The Cost of Hallucinated Configs​

AI hallucinations aren't just inconvenient—they're expensive. From legal briefs with fake citations to financial analyses based on invented metrics, hallucinated AI content causes real damage.

For infrastructure teams, the pattern is consistent: generate config, apply it, watch it fail, spend 30-60 minutes debugging what the AI made up.

The result? Teams stop trusting AI for infrastructure work. Productivity gains vanish. You're back to grep-ing through values.yaml files manually.

Making AI Reliable for vCluster Configs​

vCluster is an open-source solution that enables teams to run virtual Kubernetes clusters inside existing infrastructure. These virtual clusters are Certified Kubernetes Distributions that provide strong workload isolation while running as nested environments on top of another Kubernetes cluster.

Why vCluster configs are hard for AI:

• Multiple major versions with different schemas (v0.19.0 vs v0.24.0)
• Complex nested YAML structure (Helm chart values)
• Validation rules hidden in comments, not schemas
• Frequent deprecations and field migrations
• Version-specific enum values

What AI hallucinates:

• Config options that don't exist in your version
• Deprecated fields that were renamed
• Invalid enum values
• Incompatible option combinations

Example: controlPlane.backingStore.etcd.deploy moved to controlPlane.backingStore.etcd.embedded in v0.20.0. AI trained on mixed docs will confidently suggest the wrong path for your version.

The Solution: Deterministic Validation​

Stop hallucinations by grounding AI in actual schemas with deterministic validation.

I built an MCP server that connects any AI assistant—Claude, ChatGPT, or any MCP-compatible tool—directly to vCluster's GitHub repository. Every query, every validation, every config generation is grounded in the real Helm values for the exact version you specify. The pattern applies to any complex infrastructure configuration—vCluster is just the implementation.

How it works:

1. Fetches configs directly from github.com/loft-sh/vcluster (source of truth)
2. Every tool accepts explicit version params (v0.24.0, main, etc.)
3. Validates against actual schemas—no pattern matching
4. 15-minute smart cache (respects GitHub API limits)
5. Dual interface: MCP (for AI) + CLI (for humans)

Why Model Context Protocol?​

Model Context Protocol (MCP) is an open standard created by Anthropic for connecting AI assistants to external data sources and tools. Instead of building custom integrations for each AI, MCP provides a universal interface.

What makes MCP powerful:

The MCP specification defines a client-server architecture where:

• MCP Clients (AI applications) can connect to any MCP server
• MCP Servers expose tools, resources, and prompts to AI assistants
• Transport layer supports stdio, HTTP, and Server-Sent Events (SSE)

This means one MCP server works everywhere:

Build once, use everywhere. No custom integrations. No vendor lock-in.

The vCluster YAML server is both an MCP server (for AI) and a standalone CLI tool (for humans and automation). Full architecture details in the GitHub repo.

Three Features That Matter​

No More Version Drift​

# Query specific versions
vcluster-yaml query "sync.fromHost" --version v0.19.0
vcluster-yaml query "sync.fromHost" --version v0.24.0

# AI can query multiple versions in parallel
# No state conflicts, no version switching

Your AI assistant now knows exactly which options exist in which version. No more mixing v0.19 examples with v0.24 configs.

No More Silent Failures​

When your AI generates a config, it validates before showing you:

You ask:

"Create a vCluster config with HA etcd and node sync for v0.24"

Claude does:

1. Queries v0.24 schema
2. Generates YAML
3. Validates it automatically
4. Shows you only validated output

What it looks like in practice:

First, Claude catches the error:

Then provides the corrected, validated config:

No copy-paste-debug loop. Every config works on first apply.

No More Context Switching​

# storage: Valid options: "ephemeral", "persistent"

The server extracts rules like enums, dependencies, and defaults—knowledge AI wouldn't have from schema alone.

Try It Now (2 Minutes)​

Works with Claude Desktop, Claude Web/Code, ChatGPT Desktop, or any MCP-compatible AI.

Option 1: Local MCP

Add to your AI's MCP configuration (example for Claude Desktop at ~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "vcluster-yaml": {
      "command": "npx",
      "args": ["-y", "vcluster-yaml-mcp-server"]
    }
  }
}