sirius/gcloud-lab
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gcloud-lab/README.md
sirius0xdev 4f80f1eede deploy news bot
2026-01-16 20:33:37 +00:00

397 lines
16 KiB
Markdown
Raw Blame History

# GCloud-Lab DevOps Infrastructure
A cloud-native DevOps laboratory project showcasing modern infrastructure-as-code, GitOps practices, and Kubernetes orchestration on Google Cloud Platform. This project runs a news intelligence system with LLM-powered analysis and a workflow automation platform.
## Table of Contents
- [Project Overview](#project-overview)
- [Architecture](#architecture)
- [DevOps Tools & Technologies](#devops-tools--technologies)
- [Project Structure](#project-structure)
- [Infrastructure Components](#infrastructure-components)
- [Applications](#applications)
- [Getting Started](#getting-started)
- [Security](#security)
---
## Project Overview
This repository contains infrastructure and application configurations for:
1. **News Intelligence Pipeline**: Automated web scraping, LLM-powered summarization, and Telegram distribution
2. **Workflow Automation**: N8N platform for custom integrations
3. **DevOps Reference Architecture**: Demonstrates GitOps, IaC, and cloud-native best practices
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────┐
│ Google Cloud Platform │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ GKE Cluster (devops-lab-cluster) │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │
│ │ │ Standard │ │ GPU Pool │ │ Flux CD (GitOps) │ │ │
│ │ │ Node Pool │ │ (SPOT L4) │ │ - Source Controller │ │ │
│ │ │ e2-std-2 │ │ g2-std-8 │ │ - Kustomize Controller│ │ │
│ │ │ 1-16 nodes │ │ 0-5 nodes │ │ - Helm Controller │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ Cilium CNI + Hubble │ │ │
│ │ │ (Network Policies + Observability) │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌───────────────────────┐ ┌─────────────────────────────────┐ │ │
│ │ │ customer1 namespace │ │ cnpg-system namespace │ │ │
│ │ │ ┌─────────────────┐ │ │ ┌───────────────────────────┐ │ │ │
│ │ │ │ N8N │ │ │ │ CloudNative PG Operator │ │ │ │
│ │ │ │ (Workflows) │ │ │ └───────────────────────────┘ │ │ │
│ │ │ └─────────────────┘ │ └─────────────────────────────────┘ │ │
│ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ News Scraper │ │ ┌─────────────────────────────────┐ │ │
│ │ │ │ (CronJob :00) │ │ │ PostgreSQL HA Cluster │ │ │
│ │ │ └─────────────────┘ │ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │ │
│ │ │ ┌─────────────────┐ │ │ │ DB1 │ │ DB2 │ │ DB3 │ │ │ │
│ │ │ │ News Analyst │◄─┼──┼──│(RW) │ │(RO) │ │(RO) │ │ │ │
│ │ │ │ (CronJob :15) │ │ │ └─────┘ └─────┘ └─────┘ │ │ │
│ │ │ │ + Ollama/Gemma2 │ │ └─────────────────────────────────┘ │ │
│ │ │ └─────────────────┘ │ │ │
│ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ Telegram Bot │ │ │ │
│ │ │ │ (CronJob :20) │ │ │ │
│ │ │ └─────────────────┘ │ │ │
│ │ └───────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ GCP L7 Global Load Balancer (HTTPS) │ │
│ │ n8n.sirius-sec.com │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## DevOps Tools & Technologies
### Infrastructure as Code (IaC)
| Tool | Version | Purpose |
|------|---------|---------|
| **Terraform** | 1.7+ | Infrastructure provisioning for GCP resources |
| **Google Provider** | 7.14.1 | Terraform provider for GCP |
| **Helm Provider** | Latest | Terraform provider for Helm charts |
| **Flux Provider** | 1.7.6 | Terraform provider for Flux bootstrap |
### Container Orchestration & Networking
| Tool | Version | Purpose |
|------|---------|---------|
| **Google Kubernetes Engine (GKE)** | Latest | Managed Kubernetes cluster |
| **Cilium** | 1.18.5 | CNI plugin with eBPF-based networking |
| **Hubble** | 1.18.5 | Network observability and monitoring |
| **Kubernetes Gateway API** | v1 | Ingress routing and traffic management |
### GitOps & Configuration Management
| Tool | Version | Purpose |
|------|---------|---------|
| **Flux CD** | 1.7.6 | GitOps continuous delivery |
| **Kustomize** | v1beta1 | Kubernetes manifest customization |
| **Helm** | 3+ | Kubernetes package manager |
| **SOPS** | Latest | Secrets encryption in Git |
| **Age** | Latest | Modern encryption for SOPS |
### Database
| Tool | Version | Purpose |
|------|---------|---------|
| **CloudNative PG** | 0.26.1 | PostgreSQL Kubernetes operator |
| **PostgreSQL** | 15.2 | Relational database (3-node HA cluster) |
### AI/ML Infrastructure
| Tool | Version | Purpose |
|------|---------|---------|
| **Ollama** | Latest | Local LLM inference server |
| **Gemma2** | Latest | Open-source LLM for text summarization |
| **NVIDIA L4 GPU** | - | GPU acceleration for LLM workloads |
### Development Environment
| Tool | Version | Purpose |
|------|---------|---------|
| **Mise** | Latest | Development tool version manager |
| **Dev Containers** | Latest | Consistent development environment |
| **k9s** | Latest | Kubernetes CLI dashboard |
---
## Project Structure
```
gcloud-lab/
├── modules/ # Terraform IaC modules
│ ├── providers.tf # Provider configurations
│ ├── gke.tf # GKE cluster definition
│ ├── vpc.tf # VPC and subnet configuration
│ ├── nodepool.tf # Standard node pool
│ ├── nodepool-gpu.tf # GPU node pool (SPOT instances)
│ ├── flux.tf # Flux GitOps bootstrap
│ ├── helm.tf # Helm chart deployments (Cilium)
│ └── variables.tf # Input variables
│
├── clusters/ # Cluster configurations
│ └── devops-lab/
│ ├── flux-system/ # Flux CD components
│ │ ├── gotk-components.yaml # Flux controllers
│ │ ├── gotk-sync.yaml # Git repository sync
│ │ └── kustomization.yaml # Flux kustomization
│ ├── customer1.yaml # Customer1 Kustomization
│ ├── infra-controllers.yaml # Infrastructure controllers
│ └── infra-configs.yaml # Infrastructure configs
│
├── infrastructure/ # Infrastructure components
│ ├── controllers/
│ │ ├── base/
│ │ │ └── cnpg/ # CloudNative PG operator
│ │ │ ├── repository.yaml # Helm repository
│ │ │ └── release.yaml # Helm release
│ │ └── staging/
│ │ └── kustomization.yaml
│ └── configs/
│ └── staging/
│ └── kustomization.yaml
│
├── apps/ # Application deployments
│ ├── base/
│ │ └── customer1/
│ │ ├── namespace.yaml # Namespace definition
│ │ ├── deployment.yaml # N8N deployment
│ │ ├── service.yaml # ClusterIP service
│ │ ├── storage.yaml # PersistentVolumeClaim
│ │ ├── configmap.yaml # N8N configuration
│ │ ├── pg-cluster-customer1.yaml # PostgreSQL cluster
│ │ ├── apigateway.yaml # GCP Gateway
│ │ ├── http-route.yaml # HTTP routing
│ │ ├── healthcheck.yaml # Health check policy
│ │ └── news_bot/ # News bot microservices
│ │ ├── scraper-cronjob.yaml
│ │ ├── analyst-cronjob.yaml
│ │ ├── telebot-cronjob.yaml
│ │ ├── scrapy-configmap.yaml
│ │ └── scrapy-urls-configmap.yaml
│ └── staging/
│ └── customer1/
│ └── kustomization.yaml # Staging overlay
│
├── scripts/
│ └── setup # Development setup script
│
├── .devcontainer.json # Dev container configuration
├── mise.toml # Tool version management
├── age.agekey # SOPS encryption key
└── README.md # This file
```
---
## Infrastructure Components
### GKE Cluster
- **Name**: `devops-lab-cluster`
- **Region**: `us-central1-a`
- **Network**: Custom VPC with dual-stack IPv4/IPv6
### Node Pools
| Pool | Machine Type | Scaling | Purpose |
|------|-------------|---------|---------|
| Standard | e2-standard-2 | 1-16 nodes | General workloads |
| GPU (SPOT) | g2-standard-8 + L4 | 0-5 nodes | LLM inference |
### Networking
- **VPC**: `devops-lab-network`
- **Primary CIDR**: `10.0.0.0/16`
- **Pod CIDR**: `192.168.32.0/20`
- **Service CIDR**: `192.168.16.0/24`
- **CNI**: Cilium with advanced datapath
- **Ingress**: GCP L7 Global Load Balancer
### GitOps Flow
```
GitHub Repository
│
▼
Flux Source Controller (watches git, 1min interval)
│
▼
Flux Kustomize Controller (applies manifests)
│
├── infrastructure/controllers → CNPG Operator
├── infrastructure/configs → Cluster configs
└── apps/staging/customer1 → Applications
```
---
## Applications
### N8N Workflow Automation
- **URL**: `https://n8n.sirius-sec.com`
- **Image**: `docker.n8n.io/n8nio/n8n:2.1.4`
- **Database**: PostgreSQL (dedicated `n8n` database)
- **Storage**: 1GB persistent volume
### News Intelligence Pipeline
A three-stage data pipeline running as Kubernetes CronJobs:
| Stage | Schedule | Container | Purpose |
|-------|----------|-----------|---------|
| Scraper | `:00` hourly | `siriussec/newsscraper` | Scrapes 100+ global news sources |
| Analyst | `:15` hourly | `siriussec/summarizer` + `ollama/ollama` | LLM-powered summarization |
| Telegram | `:20` hourly | `siriussec/news-messenger` | Distributes summaries to Telegram |
**News Sources Coverage**:
- North America: NPR, AP News, CBC, etc.
- Europe: BBC, Reuters, The Guardian, etc.
- Asia: SCMP, Al Jazeera, Times of India, etc.
- Africa: BBC Africa, News24, etc.
- South America: Buenos Aires Herald, etc.
---
## Getting Started
### Prerequisites
- Google Cloud account with billing enabled
- GitHub account with repository access
- `gcloud` CLI authenticated
- Terraform 1.7+
### Local Development Setup
```bash
# Install tools via mise
./scripts/setup
# Or manually
mise trust && mise install
```
### Infrastructure Deployment
```bash
cd modules
# Initialize Terraform
terraform init
# Set required variables
export TF_VAR_github_token="your-token"
export TF_VAR_github_org="your-org"
export TF_VAR_github_repository="gcloud-lab"
# Plan and apply
terraform plan
terraform apply
```
### Accessing the Cluster
```bash
# Configure kubectl
gcloud container clusters get-credentials devops-lab-cluster \
--zone us-central1-a \
--project devops-lab-cluster
# Verify connection
kubectl get nodes
# Use k9s for interactive management
k9s
```
---
## Security
### Secrets Management
- **Encryption**: SOPS with Age encryption
- **Key Storage**: `age.agekey` (do not commit unencrypted)
- **Flux Integration**: Automatic decryption during deployment
### Pod Security
- Non-root containers (UID 1000)
- Filesystem group enforcement
- Privilege escalation disabled
- Resource limits enforced
### Network Security
- Cilium network policies for pod-to-pod isolation
- TLS termination at load balancer
- Private cluster networking with NAT
### Database Security
- Managed roles with secret-based passwords
- Separate users per application (`customer1`, `news_app`)
- HA cluster with automatic failover
---
## Tool Reference
### Terraform Providers
```hcl
google = "~> 7.14" # GCP resources
helm = "~> 2.0" # Helm chart management
flux = "~> 1.7" # GitOps bootstrap
```
### Helm Charts
```yaml
cilium: 1.18.5 # CNI and service mesh
cloudnative-pg: 0.26.1 # PostgreSQL operator
```
### Container Images
```
docker.n8n.io/n8nio/n8n:2.1.4
ghcr.io/cloudnative-pg/postgresql:15.2
ollama/ollama:latest
siriussec/newsscraper:latest
siriussec/summarizer:latest
siriussec/news-messenger:latest
```
---
## Cost Optimization
- **SPOT GPU Instances**: 60-90% savings on LLM workloads
- **Autoscaling**: GPU nodes scale to 0 when idle
- **Resource Limits**: Prevents runaway costs
- **Scheduled Workloads**: CronJobs only run when needed
---
## License
Private repository - All rights reserved.
Reference in a new issue View git blame Copy permalink