sirius/gcloud-lab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gcloud-lab/plans/AI_ARCHITECTURE.md
sirius0xdev 9f5d1fb723 clean up and add routes to paas site
2026-04-28 03:31:52 +00:00

1.9 KiB
Raw Blame History

Dual-Tier AI Architecture: L4 Dispatcher & A100 Deep Thinker

This document outlines the cost-optimized, dual-tier LLM architecture deployed in the cluster using OpenClaw, vLLM, and KEDA.

Concept

Instead of running an expensive A100 GPU 24/7 for all requests, we split the cognitive load into two tiers: a lightweight "Dispatcher" and a heavyweight "Deep Thinker." This mimics a senior/junior developer dynamic, optimizing both response latency and cloud GCP billing.

Tier 1: The Dispatcher (L4 GPU)

  • Hardware: 1x NVIDIA L4 (24GB VRAM)
  • Model: Qwen2.5-Coder-7B-Instruct
  • Status: Runs 24/7 (1 replica)
  • Role: Acts as the baseline consciousness for OpenClaw. Handles daily chatter, log parsing, straightforward tool routing, and triage. Lightning-fast token generation at a fraction of the cost.

Tier 2: The Deep Thinker (A100 GPU)

  • Hardware: 1x NVIDIA A100 (80GB VRAM)
  • Model: Qwen3.6-27B-heretic
  • Status: Scaled to zero by default.
  • Role: Activated only for massive context tasks, deep research, and complex multi-file architectural reasoning.

Scaling & Routing Mechanics (KEDA + OpenClaw)

  1. Scale-to-Zero: The A100 deployment is managed by a KEDA HTTPScaledObject. It scales down to 0 replicas after 15 minutes of inactivity.
  2. Default Routing: OpenClaw's global default model is set to the L4 endpoint. All standard messages hit the L4 immediately.
  3. Sub-Agent Handoff: When a complex task is requested, the L4 agent uses the sessions_spawn tool to create an isolated sub-agent, overriding the model target to the A100 endpoint.
  4. Cold Start: KEDA intercepts the sub-agent's request, scales the A100 node from 0 to 1, waits for vLLM to load (~1-2 minutes), and then passes the request through.
  5. Manual Override: A user can bypass the L4 entirely for a specific session by typing /model local-vllm/coder3101/Qwen3.5-27B-heretic in the OpenClaw chat.