GCloud-Lab DevOps Infrastructure
A cloud-native DevOps laboratory project showcasing modern infrastructure-as-code, GitOps practices, and Kubernetes orchestration on Google Cloud Platform. This project runs a news intelligence system with LLM-powered analysis and a workflow automation platform.
Table of Contents
Project Overview
This repository contains infrastructure and application configurations for:
- News Intelligence Pipeline: Automated web scraping, LLM-powered summarization, and Telegram distribution
- Workflow Automation: N8N platform for custom integrations
- DevOps Reference Architecture: Demonstrates GitOps, IaC, and cloud-native best practices
Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ Google Cloud Platform │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ GKE Cluster (devops-lab-cluster) │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │
│ │ │ Standard │ │ GPU Pool │ │ Flux CD (GitOps) │ │ │
│ │ │ Node Pool │ │ (SPOT L4) │ │ - Source Controller │ │ │
│ │ │ e2-std-2 │ │ g2-std-8 │ │ - Kustomize Controller│ │ │
│ │ │ 1-16 nodes │ │ 0-5 nodes │ │ - Helm Controller │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ Cilium CNI + Hubble │ │ │
│ │ │ (Network Policies + Observability) │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌───────────────────────┐ ┌─────────────────────────────────┐ │ │
│ │ │ customer1 namespace │ │ cnpg-system namespace │ │ │
│ │ │ ┌─────────────────┐ │ │ ┌───────────────────────────┐ │ │ │
│ │ │ │ N8N │ │ │ │ CloudNative PG Operator │ │ │ │
│ │ │ │ (Workflows) │ │ │ └───────────────────────────┘ │ │ │
│ │ │ └─────────────────┘ │ └─────────────────────────────────┘ │ │
│ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ News Scraper │ │ ┌─────────────────────────────────┐ │ │
│ │ │ │ (CronJob :00) │ │ │ PostgreSQL HA Cluster │ │ │
│ │ │ └─────────────────┘ │ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │ │
│ │ │ ┌─────────────────┐ │ │ │ DB1 │ │ DB2 │ │ DB3 │ │ │ │
│ │ │ │ News Analyst │◄─┼──┼──│(RW) │ │(RO) │ │(RO) │ │ │ │
│ │ │ │ (CronJob :15) │ │ │ └─────┘ └─────┘ └─────┘ │ │ │
│ │ │ │ + Ollama/Gemma2 │ │ └─────────────────────────────────┘ │ │
│ │ │ └─────────────────┘ │ │ │
│ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ Telegram Bot │ │ │ │
│ │ │ │ (CronJob :20) │ │ │ │
│ │ │ └─────────────────┘ │ │ │
│ │ └───────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ GCP L7 Global Load Balancer (HTTPS) │ │
│ │ n8n.sirius-sec.com │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
DevOps Tools & Technologies
Infrastructure as Code (IaC)
| Tool |
Version |
Purpose |
| Terraform |
1.7+ |
Infrastructure provisioning for GCP resources |
| Google Provider |
7.14.1 |
Terraform provider for GCP |
| Helm Provider |
Latest |
Terraform provider for Helm charts |
| Flux Provider |
1.7.6 |
Terraform provider for Flux bootstrap |
Container Orchestration & Networking
| Tool |
Version |
Purpose |
| Google Kubernetes Engine (GKE) |
Latest |
Managed Kubernetes cluster |
| Cilium |
1.18.5 |
CNI plugin with eBPF-based networking |
| Hubble |
1.18.5 |
Network observability and monitoring |
| Kubernetes Gateway API |
v1 |
Ingress routing and traffic management |
GitOps & Configuration Management
| Tool |
Version |
Purpose |
| Flux CD |
1.7.6 |
GitOps continuous delivery |
| Kustomize |
v1beta1 |
Kubernetes manifest customization |
| Helm |
3+ |
Kubernetes package manager |
| SOPS |
Latest |
Secrets encryption in Git |
| Age |
Latest |
Modern encryption for SOPS |
Database
| Tool |
Version |
Purpose |
| CloudNative PG |
0.26.1 |
PostgreSQL Kubernetes operator |
| PostgreSQL |
15.2 |
Relational database (3-node HA cluster) |
AI/ML Infrastructure
| Tool |
Version |
Purpose |
| Ollama |
Latest |
Local LLM inference server |
| Gemma2 |
Latest |
Open-source LLM for text summarization |
| NVIDIA L4 GPU |
- |
GPU acceleration for LLM workloads |
Development Environment
| Tool |
Version |
Purpose |
| Mise |
Latest |
Development tool version manager |
| Dev Containers |
Latest |
Consistent development environment |
| k9s |
Latest |
Kubernetes CLI dashboard |
Project Structure
gcloud-lab/
├── modules/ # Terraform IaC modules
│ ├── providers.tf # Provider configurations
│ ├── gke.tf # GKE cluster definition
│ ├── vpc.tf # VPC and subnet configuration
│ ├── nodepool.tf # Standard node pool
│ ├── nodepool-gpu.tf # GPU node pool (SPOT instances)
│ ├── flux.tf # Flux GitOps bootstrap
│ ├── helm.tf # Helm chart deployments (Cilium)
│ └── variables.tf # Input variables
│
├── clusters/ # Cluster configurations
│ └── devops-lab/
│ ├── flux-system/ # Flux CD components
│ │ ├── gotk-components.yaml # Flux controllers
│ │ ├── gotk-sync.yaml # Git repository sync
│ │ └── kustomization.yaml # Flux kustomization
│ ├── customer1.yaml # Customer1 Kustomization
│ ├── infra-controllers.yaml # Infrastructure controllers
│ └── infra-configs.yaml # Infrastructure configs
│
├── infrastructure/ # Infrastructure components
│ ├── controllers/
│ │ ├── base/
│ │ │ └── cnpg/ # CloudNative PG operator
│ │ │ ├── repository.yaml # Helm repository
│ │ │ └── release.yaml # Helm release
│ │ └── staging/
│ │ └── kustomization.yaml
│ └── configs/
│ └── staging/
│ └── kustomization.yaml
│
├── apps/ # Application deployments
│ ├── base/
│ │ └── customer1/
│ │ ├── namespace.yaml # Namespace definition
│ │ ├── deployment.yaml # N8N deployment
│ │ ├── service.yaml # ClusterIP service
│ │ ├── storage.yaml # PersistentVolumeClaim
│ │ ├── configmap.yaml # N8N configuration
│ │ ├── pg-cluster-customer1.yaml # PostgreSQL cluster
│ │ ├── apigateway.yaml # GCP Gateway
│ │ ├── http-route.yaml # HTTP routing
│ │ ├── healthcheck.yaml # Health check policy
│ │ └── news_bot/ # News bot microservices
│ │ ├── scraper-cronjob.yaml
│ │ ├── analyst-cronjob.yaml
│ │ ├── telebot-cronjob.yaml
│ │ ├── scrapy-configmap.yaml
│ │ └── scrapy-urls-configmap.yaml
│ └── staging/
│ └── customer1/
│ └── kustomization.yaml # Staging overlay
│
├── scripts/
│ └── setup # Development setup script
│
├── .devcontainer.json # Dev container configuration
├── mise.toml # Tool version management
├── age.agekey # SOPS encryption key
└── README.md # This file
Infrastructure Components
GKE Cluster
- Name:
devops-lab-cluster
- Region:
us-central1-a
- Network: Custom VPC with dual-stack IPv4/IPv6
Node Pools
| Pool |
Machine Type |
Scaling |
Purpose |
| Standard |
e2-standard-2 |
1-16 nodes |
General workloads |
| GPU (SPOT) |
g2-standard-8 + L4 |
0-5 nodes |
LLM inference |
Networking
- VPC:
devops-lab-network
- Primary CIDR:
10.0.0.0/16
- Pod CIDR:
192.168.32.0/20
- Service CIDR:
192.168.16.0/24
- CNI: Cilium with advanced datapath
- Ingress: GCP L7 Global Load Balancer
GitOps Flow
GitHub Repository
│
▼
Flux Source Controller (watches git, 1min interval)
│
▼
Flux Kustomize Controller (applies manifests)
│
├── infrastructure/controllers → CNPG Operator
├── infrastructure/configs → Cluster configs
└── apps/staging/customer1 → Applications
Applications
N8N Workflow Automation
- URL:
https://n8n.sirius-sec.com
- Image:
docker.n8n.io/n8nio/n8n:2.1.4
- Database: PostgreSQL (dedicated
n8n database)
- Storage: 1GB persistent volume
News Intelligence Pipeline
A three-stage data pipeline running as Kubernetes CronJobs:
| Stage |
Schedule |
Container |
Purpose |
| Scraper |
:00 hourly |
siriussec/newsscraper |
Scrapes 100+ global news sources |
| Analyst |
:15 hourly |
siriussec/summarizer + ollama/ollama |
LLM-powered summarization |
| Telegram |
:20 hourly |
siriussec/news-messenger |
Distributes summaries to Telegram |
News Sources Coverage:
- North America: NPR, AP News, CBC, etc.
- Europe: BBC, Reuters, The Guardian, etc.
- Asia: SCMP, Al Jazeera, Times of India, etc.
- Africa: BBC Africa, News24, etc.
- South America: Buenos Aires Herald, etc.
Getting Started
Prerequisites
- Google Cloud account with billing enabled
- GitHub account with repository access
gcloud CLI authenticated- Terraform 1.7+
Local Development Setup
# Install tools via mise
./scripts/setup
# Or manually
mise trust && mise install
Infrastructure Deployment
cd modules
# Initialize Terraform
terraform init
# Set required variables
export TF_VAR_github_token="your-token"
export TF_VAR_github_org="your-org"
export TF_VAR_github_repository="gcloud-lab"
# Plan and apply
terraform plan
terraform apply
Accessing the Cluster
# Configure kubectl
gcloud container clusters get-credentials devops-lab-cluster \
--zone us-central1-a \
--project devops-lab-cluster
# Verify connection
kubectl get nodes
# Use k9s for interactive management
k9s
Security
Secrets Management
- Encryption: SOPS with Age encryption
- Key Storage:
age.agekey (do not commit unencrypted)
- Flux Integration: Automatic decryption during deployment
Pod Security
- Non-root containers (UID 1000)
- Filesystem group enforcement
- Privilege escalation disabled
- Resource limits enforced
Network Security
- Cilium network policies for pod-to-pod isolation
- TLS termination at load balancer
- Private cluster networking with NAT
Database Security
- Managed roles with secret-based passwords
- Separate users per application (
customer1, news_app)
- HA cluster with automatic failover
Tool Reference
Terraform Providers
google = "~> 7.14" # GCP resources
helm = "~> 2.0" # Helm chart management
flux = "~> 1.7" # GitOps bootstrap
Helm Charts
cilium: 1.18.5 # CNI and service mesh
cloudnative-pg: 0.26.1 # PostgreSQL operator
Container Images
docker.n8n.io/n8nio/n8n:2.1.4
ghcr.io/cloudnative-pg/postgresql:15.2
ollama/ollama:latest
siriussec/newsscraper:latest
siriussec/summarizer:latest
siriussec/news-messenger:latest
Cost Optimization
- SPOT GPU Instances: 60-90% savings on LLM workloads
- Autoscaling: GPU nodes scale to 0 when idle
- Resource Limits: Prevents runaway costs
- Scheduled Workloads: CronJobs only run when needed
License
Private repository - All rights reserved.
