sirius/gcloud-lab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gcloud-lab/README.md

370 lines
14 KiB
Markdown
Raw Blame History

# GCloud-Lab DevOps Infrastructure
A cloud-native DevOps laboratory project showcasing modern infrastructure-as-code, GitOps practices, and Kubernetes orchestration on Google Cloud Platform. This project runs a news intelligence system with LLM-powered analysis and a workflow automation platform.
## Table of Contents
- [Project Overview](#project-overview)
- [Architecture](#architecture)
- [DevOps Tools & Technologies](#devops-tools--technologies)
- [Project Structure](#project-structure)
- [Infrastructure Components](#infrastructure-components)
- [Applications](#applications)
- [Getting Started](#getting-started)
- [Security](#security)
---
## Project Overview
This repository contains infrastructure and application configurations for:
1. **News Intelligence Pipeline**: Automated web scraping, LLM-powered summarization, and Telegram distribution
2. **Workflow Automation**: N8N platform for custom integrations
3. **DevOps Reference Architecture**: Demonstrates GitOps, IaC, and cloud-native best practices
---
## Architecture
```text
┌─────────────────────────────────────────────────────────────────────────┐
│ Google Cloud Platform │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ GKE Cluster (devops-lab-cluster) │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Standard │ │ L4 GPU Pool │ │ A100 GPU │ │ │
│ │ │ Node Pool │ │ (SPOT L4) │ │ (SPOT A100) │ │ │
│ │ │ e2-std-2 │ │ 1 node (24/7│ │ 0-1 nodes │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ Cilium CNI + Hubble │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ customer1 namespace │ │ │
│ │ │ - OpenClaw PAaaS (Dual-Tier vLLM: L4 Dispatch / A100 Think)│ │ │
│ │ │ - News Bot Pipeline (DeepSeek-R1 Quant Analyst) │ │ │
│ │ │ - PAaaS Landing Page (GHCR Docker Pulls) │ │ │
│ │ │ - CloudNativePG Isolated Databases │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## DevOps Tools & Technologies
### Infrastructure as Code (IaC)
| Tool | Version | Purpose |
|------|---------|---------|
| **Terraform** | 1.7+ | Infrastructure provisioning for GCP resources |
| **Google Provider** | 7.14.1 | Terraform provider for GCP |
| **Helm Provider** | Latest | Terraform provider for Helm charts |
| **Flux Provider** | 1.7.6 | Terraform provider for Flux bootstrap |
### Container Orchestration & Networking
| Tool | Version | Purpose |
|------|---------|---------|
| **Google Kubernetes Engine (GKE)** | Latest | Managed Kubernetes cluster |
| **Cilium** | 1.18.5 | CNI plugin with eBPF-based networking |
| **Hubble** | 1.18.5 | Network observability and monitoring |
| **Kubernetes Gateway API** | v1 | Ingress routing and traffic management |
### GitOps & Configuration Management
| Tool | Version | Purpose |
|------|---------|---------|
| **Flux CD** | 1.7.6 | GitOps continuous delivery |
| **Kustomize** | v1beta1 | Kubernetes manifest customization |
| **Helm** | 3+ | Kubernetes package manager |
| **SOPS** | Latest | Secrets encryption in Git |
| **Age** | Latest | Modern encryption for SOPS |
### Database
| Tool | Version | Purpose |
|------|---------|---------|
| **CloudNative PG** | 0.26.1 | PostgreSQL Kubernetes operator |
| **PostgreSQL** | 15.2 | Relational database (3-node HA cluster) |
### AI/ML Infrastructure
| Tool | Version | Purpose |
|------|---------|---------|
| **Ollama** | Latest | Local LLM inference server |
| **Gemma2** | Latest | Open-source LLM for text summarization |
| **NVIDIA L4 GPU** | - | GPU acceleration for LLM workloads |
### Development Environment
| Tool | Version | Purpose |
|------|---------|---------|
| **Mise** | Latest | Development tool version manager |
| **Dev Containers** | Latest | Consistent development environment |
| **k9s** | Latest | Kubernetes CLI dashboard |
---
## Project Structure
```
gcloud-lab/
├── modules/ # Terraform IaC modules
│ ├── providers.tf # Provider configurations
│ ├── gke.tf # GKE cluster definition
│ ├── vpc.tf # VPC and subnet configuration
│ ├── nodepool.tf # Standard node pool
│ ├── nodepool-gpu.tf # GPU node pool (SPOT instances)
│ ├── flux.tf # Flux GitOps bootstrap
│ ├── helm.tf # Helm chart deployments (Cilium)
│ └── variables.tf # Input variables
│
├── clusters/ # Cluster configurations
│ └── devops-lab/
│ ├── flux-system/ # Flux CD components
│ │ ├── gotk-components.yaml # Flux controllers
│ │ ├── gotk-sync.yaml # Git repository sync
│ │ └── kustomization.yaml # Flux kustomization
│ ├── customer1.yaml # Customer1 Kustomization
│ ├── infra-controllers.yaml # Infrastructure controllers
│ └── infra-configs.yaml # Infrastructure configs
│
├── infrastructure/ # Infrastructure components
│ ├── controllers/
│ │ ├── base/
│ │ │ └── cnpg/ # CloudNative PG operator
│ │ │ ├── repository.yaml # Helm repository
│ │ │ └── release.yaml # Helm release
│ │ └── staging/
│ │ └── kustomization.yaml
│ └── configs/
│ └── staging/
│ └── kustomization.yaml
│
├── apps/ # Application deployments
│ ├── base/
│ │ └── customer1/
│ │ ├── namespace.yaml # Namespace definition
│ │ ├── deployment.yaml # N8N deployment
│ │ ├── service.yaml # ClusterIP service
│ │ ├── storage.yaml # PersistentVolumeClaim
│ │ ├── configmap.yaml # N8N configuration
│ │ ├── pg-cluster-customer1.yaml # PostgreSQL cluster
│ │ ├── apigateway.yaml # GCP Gateway
│ │ ├── http-route.yaml # HTTP routing
│ │ ├── healthcheck.yaml # Health check policy
│ │ └── news_bot/ # News bot microservices
│ │ ├── scraper-cronjob.yaml
│ │ ├── analyst-cronjob.yaml
│ │ ├── telebot-cronjob.yaml
│ │ ├── scrapy-configmap.yaml
│ │ └── scrapy-urls-configmap.yaml
│ └── staging/
│ └── customer1/
│ └── kustomization.yaml # Staging overlay
│
├── scripts/
│ └── setup # Development setup script
│
├── .devcontainer.json # Dev container configuration
├── mise.toml # Tool version management
├── age.agekey # SOPS encryption key
└── README.md # This file
```
---
## Infrastructure Components
### GKE Cluster
- **Name**: `devops-lab-cluster`
- **Region**: `us-central1-a`
- **Network**: Custom VPC with dual-stack IPv4/IPv6
### Node Pools
| Pool | Machine Type | Scaling | Purpose |
|------|-------------|---------|---------|
| Standard | e2-standard-2 | 1-16 nodes | General workloads |
| GPU (SPOT) | g2-standard-8 + L4 | 0-5 nodes | LLM inference |
### Networking
- **VPC**: `devops-lab-network`
- **Primary CIDR**: `10.0.0.0/16`
- **Pod CIDR**: `192.168.32.0/20`
- **Service CIDR**: `192.168.16.0/24`
- **CNI**: Cilium with advanced datapath
- **Ingress**: GCP L7 Global Load Balancer
### GitOps Flow
```
GitHub Repository
│
▼
Flux Source Controller (watches git, 1min interval)
│
▼
Flux Kustomize Controller (applies manifests)
│
├── infrastructure/controllers → CNPG Operator
├── infrastructure/configs → Cluster configs
└── apps/staging/customer1 → Applications
```
---
## Applications
### 1. Private Assistant as a Service (PAaaS)
A premium, uncensored, privacy-first AI assistant platform with dual-tier cognitive architecture:
- **Tier 1 (Dispatcher):** L4 GPU Spot instance running 24/7. Hosts `Qwen2.5-Coder-7B-Instruct-heretic` via vLLM `v0.9.1` for lightning-fast, cheap triage and tool calling (using the `pythonic` tool parser).
- **Tier 2 (Deep Thinker):** A100 80GB Spot instance scaling from 0-1 via KEDA. Hosts `Qwen3.5-27B-heretic` with `--enable-chunked-prefill` and `--kv-cache-dtype=fp8` for massive multi-file context and reasoning without OOMing or stalling concurrent users.
- **Frontend:** Isolated `openclaw` deployments per tenant, connected to Telegram/Discord via outbound polling (no public ingress required).
- **Landing Page:** Dockerized marketing site built via CI/CD from `openclaw-projects` and deployed to the `staging` kustomization overlay.
### 2. Autonomous News Quant Pipeline (`news_bot`)
An institutional-grade pipeline scraping 371 global feeds to generate actionable futures trading signals:
- **Scraper:** CronJob at `:50` pulling multi-lingual global financial data.
- **Map/Reduce Analyst:** Utilizes DeepSeek-R1 (with a strict 10-step `<think>` protocol) and local open-weights to extract "Market-Moving DNA". Translates events into explicit futures targets (/ES, /CL, /NQ) with R:R, Take Profit, and Stop Loss levels anchored in provided volume/price data.
- **Privacy:** All proprietary technical data stays strictly within the VPC, executing against local models rather than public APIs like OpenAI to protect the trading edge and avoid throttling during market panics.
### 3. N8N Workflow Automation
- **Database**: PostgreSQL (dedicated `n8n` database)
- Custom integrations and webhook catchers.
*(Note: PineScript trading strategies have been migrated out of this IaC repository and live in `openclaw-projects/trading-bots`.)*
## Getting Started
### Prerequisites
- Google Cloud account with billing enabled
- GitHub account with repository access
- `gcloud` CLI authenticated
- Terraform 1.7+
### Local Development Setup
```bash
# Install tools via mise
./scripts/setup
# Or manually
mise trust && mise install
```
### Infrastructure Deployment
```bash
cd modules
# Initialize Terraform
terraform init
# Set required variables
export TF_VAR_github_token="your-token"
export TF_VAR_github_org="your-org"
export TF_VAR_github_repository="gcloud-lab"
# Plan and apply
terraform plan
terraform apply
```
### Accessing the Cluster
```bash
# Configure kubectl
gcloud container clusters get-credentials devops-lab-cluster \
--zone us-central1-a \
--project devops-lab-cluster
# Verify connection
kubectl get nodes
# Use k9s for interactive management
k9s
```
---
## Security
### Secrets Management
- **Encryption**: SOPS with Age encryption
- **Key Storage**: `age.agekey` (do not commit unencrypted)
- **Flux Integration**: Automatic decryption during deployment
### Pod Security
- Non-root containers (UID 1000)
- Filesystem group enforcement
- Privilege escalation disabled
- Resource limits enforced
### Network Security
- Cilium network policies for pod-to-pod isolation
- TLS termination at load balancer
- Private cluster networking with NAT
### Database Security
- Managed roles with secret-based passwords
- Separate users per application (`customer1`, `news_app`)
- HA cluster with automatic failover
---
## Tool Reference
### Terraform Providers
```hcl
google = "~> 7.14" # GCP resources
helm = "~> 2.0" # Helm chart management
flux = "~> 1.7" # GitOps bootstrap
```
### Helm Charts
```yaml
cilium: 1.18.5 # CNI and service mesh
cloudnative-pg: 0.26.1 # PostgreSQL operator
```
### Container Images
```
docker.n8n.io/n8nio/n8n:2.1.4
ghcr.io/cloudnative-pg/postgresql:15.2
ollama/ollama:latest
siriussec/newsscraper:latest
siriussec/summarizer:latest
siriussec/news-messenger:latest
```
---
## Cost Optimization
- **SPOT GPU Instances**: 60-90% savings on LLM workloads
- **Autoscaling**: GPU nodes scale to 0 when idle
- **Resource Limits**: Prevents runaway costs
- **Scheduled Workloads**: CronJobs only run when needed
---
## License
Private repository - All rights reserved.
Reference in a new issue View git blame Copy permalink