398 lines
16 KiB
Markdown
398 lines
16 KiB
Markdown
|
# GCloud-Lab DevOps Infrastructure
|
||
|
|
|
||
|
|
A cloud-native DevOps laboratory project showcasing modern infrastructure-as-code, GitOps practices, and Kubernetes orchestration on Google Cloud Platform. This project runs a news intelligence system with LLM-powered analysis and a workflow automation platform.
|
||
|
|
|
||
|
|
## Table of Contents
|
||
|
|
|
||
|
|
- [Project Overview](#project-overview)
|
||
|
|
- [Architecture](#architecture)
|
||
|
|
- [DevOps Tools & Technologies](#devops-tools--technologies)
|
||
|
|
- [Project Structure](#project-structure)
|
||
|
|
- [Infrastructure Components](#infrastructure-components)
|
||
|
|
- [Applications](#applications)
|
||
|
|
- [Getting Started](#getting-started)
|
||
|
|
- [Security](#security)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Project Overview
|
||
|
|
|
||
|
|
This repository contains infrastructure and application configurations for:
|
||
|
|
|
||
|
|
1. **News Intelligence Pipeline**: Automated web scraping, LLM-powered summarization, and Telegram distribution
|
||
|
|
2. **Workflow Automation**: N8N platform for custom integrations
|
||
|
|
3. **DevOps Reference Architecture**: Demonstrates GitOps, IaC, and cloud-native best practices
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||
|
|
│ Google Cloud Platform │
|
||
|
|
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ GKE Cluster (devops-lab-cluster) │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │
|
||
|
|
│ │ │ Standard │ │ GPU Pool │ │ Flux CD (GitOps) │ │ │
|
||
|
|
│ │ │ Node Pool │ │ (SPOT L4) │ │ - Source Controller │ │ │
|
||
|
|
│ │ │ e2-std-2 │ │ g2-std-8 │ │ - Kustomize Controller│ │ │
|
||
|
|
│ │ │ 1-16 nodes │ │ 0-5 nodes │ │ - Helm Controller │ │ │
|
||
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
|
||
|
|
│ │ │ Cilium CNI + Hubble │ │ │
|
||
|
|
│ │ │ (Network Policies + Observability) │ │ │
|
||
|
|
│ │ └─────────────────────────────────────────────────────────────┘ │ │
|
||
|
|
│ │ │ │
|
||
|
|
│ │ ┌───────────────────────┐ ┌─────────────────────────────────┐ │ │
|
||
|
|
│ │ │ customer1 namespace │ │ cnpg-system namespace │ │ │
|
||
|
|
│ │ │ ┌─────────────────┐ │ │ ┌───────────────────────────┐ │ │ │
|
||
|
|
│ │ │ │ N8N │ │ │ │ CloudNative PG Operator │ │ │ │
|
||
|
|
│ │ │ │ (Workflows) │ │ │ └───────────────────────────┘ │ │ │
|
||
|
|
│ │ │ └─────────────────┘ │ └─────────────────────────────────┘ │ │
|
||
|
|
│ │ │ ┌─────────────────┐ │ │ │
|
||
|
|
│ │ │ │ News Scraper │ │ ┌─────────────────────────────────┐ │ │
|
||
|
|
│ │ │ │ (CronJob :00) │ │ │ PostgreSQL HA Cluster │ │ │
|
||
|
|
│ │ │ └─────────────────┘ │ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │ │
|
||
|
|
│ │ │ ┌─────────────────┐ │ │ │ DB1 │ │ DB2 │ │ DB3 │ │ │ │
|
||
|
|
│ │ │ │ News Analyst │◄─┼──┼──│(RW) │ │(RO) │ │(RO) │ │ │ │
|
||
|
|
│ │ │ │ (CronJob :15) │ │ │ └─────┘ └─────┘ └─────┘ │ │ │
|
||
|
|
│ │ │ │ + Ollama/Gemma2 │ │ └─────────────────────────────────┘ │ │
|
||
|
|
│ │ │ └─────────────────┘ │ │ │
|
||
|
|
│ │ │ ┌─────────────────┐ │ │ │
|
||
|
|
│ │ │ │ Telegram Bot │ │ │ │
|
||
|
|
│ │ │ │ (CronJob :20) │ │ │ │
|
||
|
|
│ │ │ └─────────────────┘ │ │ │
|
||
|
|
│ │ └───────────────────────┘ │ │
|
||
|
|
│ └───────────────────────────────────────────────────────────────────┘ │
|
||
|
|
│ │
|
||
|
|
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
|
|
│ │ GCP L7 Global Load Balancer (HTTPS) │ │
|
||
|
|
│ │ n8n.sirius-sec.com │ │
|
||
|
|
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## DevOps Tools & Technologies
|
||
|
|
|
||
|
|
### Infrastructure as Code (IaC)
|
||
|
|
|
||
|
|
| Tool | Version | Purpose |
|
||
|
|
|------|---------|---------|
|
||
|
|
| **Terraform** | 1.7+ | Infrastructure provisioning for GCP resources |
|
||
|
|
| **Google Provider** | 7.14.1 | Terraform provider for GCP |
|
||
|
|
| **Helm Provider** | Latest | Terraform provider for Helm charts |
|
||
|
|
| **Flux Provider** | 1.7.6 | Terraform provider for Flux bootstrap |
|
||
|
|
|
||
|
|
### Container Orchestration & Networking
|
||
|
|
|
||
|
|
| Tool | Version | Purpose |
|
||
|
|
|------|---------|---------|
|
||
|
|
| **Google Kubernetes Engine (GKE)** | Latest | Managed Kubernetes cluster |
|
||
|
|
| **Cilium** | 1.18.5 | CNI plugin with eBPF-based networking |
|
||
|
|
| **Hubble** | 1.18.5 | Network observability and monitoring |
|
||
|
|
| **Kubernetes Gateway API** | v1 | Ingress routing and traffic management |
|
||
|
|
|
||
|
|
### GitOps & Configuration Management
|
||
|
|
|
||
|
|
| Tool | Version | Purpose |
|
||
|
|
|------|---------|---------|
|
||
|
|
| **Flux CD** | 1.7.6 | GitOps continuous delivery |
|
||
|
|
| **Kustomize** | v1beta1 | Kubernetes manifest customization |
|
||
|
|
| **Helm** | 3+ | Kubernetes package manager |
|
||
|
|
| **SOPS** | Latest | Secrets encryption in Git |
|
||
|
|
| **Age** | Latest | Modern encryption for SOPS |
|
||
|
|
|
||
|
|
### Database
|
||
|
|
|
||
|
|
| Tool | Version | Purpose |
|
||
|
|
|------|---------|---------|
|
||
|
|
| **CloudNative PG** | 0.26.1 | PostgreSQL Kubernetes operator |
|
||
|
|
| **PostgreSQL** | 15.2 | Relational database (3-node HA cluster) |
|
||
|
|
|
||
|
|
### AI/ML Infrastructure
|
||
|
|
|
||
|
|
| Tool | Version | Purpose |
|
||
|
|
|------|---------|---------|
|
||
|
|
| **Ollama** | Latest | Local LLM inference server |
|
||
|
|
| **Gemma2** | Latest | Open-source LLM for text summarization |
|
||
|
|
| **NVIDIA L4 GPU** | - | GPU acceleration for LLM workloads |
|
||
|
|
|
||
|
|
### Development Environment
|
||
|
|
|
||
|
|
| Tool | Version | Purpose |
|
||
|
|
|------|---------|---------|
|
||
|
|
| **Mise** | Latest | Development tool version manager |
|
||
|
|
| **Dev Containers** | Latest | Consistent development environment |
|
||
|
|
| **k9s** | Latest | Kubernetes CLI dashboard |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Project Structure
|
||
|
|
|
||
|
|
```
|
||
|
|
gcloud-lab/
|
||
|
|
├── modules/ # Terraform IaC modules
|
||
|
|
│ ├── providers.tf # Provider configurations
|
||
|
|
│ ├── gke.tf # GKE cluster definition
|
||
|
|
│ ├── vpc.tf # VPC and subnet configuration
|
||
|
|
│ ├── nodepool.tf # Standard node pool
|
||
|
|
│ ├── nodepool-gpu.tf # GPU node pool (SPOT instances)
|
||
|
|
│ ├── flux.tf # Flux GitOps bootstrap
|
||
|
|
│ ├── helm.tf # Helm chart deployments (Cilium)
|
||
|
|
│ └── variables.tf # Input variables
|
||
|
|
│
|
||
|
|
├── clusters/ # Cluster configurations
|
||
|
|
│ └── devops-lab/
|
||
|
|
│ ├── flux-system/ # Flux CD components
|
||
|
|
│ │ ├── gotk-components.yaml # Flux controllers
|
||
|
|
│ │ ├── gotk-sync.yaml # Git repository sync
|
||
|
|
│ │ └── kustomization.yaml # Flux kustomization
|
||
|
|
│ ├── customer1.yaml # Customer1 Kustomization
|
||
|
|
│ ├── infra-controllers.yaml # Infrastructure controllers
|
||
|
|
│ └── infra-configs.yaml # Infrastructure configs
|
||
|
|
│
|
||
|
|
├── infrastructure/ # Infrastructure components
|
||
|
|
│ ├── controllers/
|
||
|
|
│ │ ├── base/
|
||
|
|
│ │ │ └── cnpg/ # CloudNative PG operator
|
||
|
|
│ │ │ ├── repository.yaml # Helm repository
|
||
|
|
│ │ │ └── release.yaml # Helm release
|
||
|
|
│ │ └── staging/
|
||
|
|
│ │ └── kustomization.yaml
|
||
|
|
│ └── configs/
|
||
|
|
│ └── staging/
|
||
|
|
│ └── kustomization.yaml
|
||
|
|
│
|
||
|
|
├── apps/ # Application deployments
|
||
|
|
│ ├── base/
|
||
|
|
│ │ └── customer1/
|
||
|
|
│ │ ├── namespace.yaml # Namespace definition
|
||
|
|
│ │ ├── deployment.yaml # N8N deployment
|
||
|
|
│ │ ├── service.yaml # ClusterIP service
|
||
|
|
│ │ ├── storage.yaml # PersistentVolumeClaim
|
||
|
|
│ │ ├── configmap.yaml # N8N configuration
|
||
|
|
│ │ ├── pg-cluster-customer1.yaml # PostgreSQL cluster
|
||
|
|
│ │ ├── apigateway.yaml # GCP Gateway
|
||
|
|
│ │ ├── http-route.yaml # HTTP routing
|
||
|
|
│ │ ├── healthcheck.yaml # Health check policy
|
||
|
|
│ │ └── news_bot/ # News bot microservices
|
||
|
|
│ │ ├── scraper-cronjob.yaml
|
||
|
|
│ │ ├── analyst-cronjob.yaml
|
||
|
|
│ │ ├── telebot-cronjob.yaml
|
||
|
|
│ │ ├── scrapy-configmap.yaml
|
||
|
|
│ │ └── scrapy-urls-configmap.yaml
|
||
|
|
│ └── staging/
|
||
|
|
│ └── customer1/
|
||
|
|
│ └── kustomization.yaml # Staging overlay
|
||
|
|
│
|
||
|
|
├── scripts/
|
||
|
|
│ └── setup # Development setup script
|
||
|
|
│
|
||
|
|
├── .devcontainer.json # Dev container configuration
|
||
|
|
├── mise.toml # Tool version management
|
||
|
|
├── age.agekey # SOPS encryption key
|
||
|
|
└── README.md # This file
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Infrastructure Components
|
||
|
|
|
||
|
|
### GKE Cluster
|
||
|
|
|
||
|
|
- **Name**: `devops-lab-cluster`
|
||
|
|
- **Region**: `us-central1-a`
|
||
|
|
- **Network**: Custom VPC with dual-stack IPv4/IPv6
|
||
|
|
|
||
|
|
### Node Pools
|
||
|
|
|
||
|
|
| Pool | Machine Type | Scaling | Purpose |
|
||
|
|
|------|-------------|---------|---------|
|
||
|
|
| Standard | e2-standard-2 | 1-16 nodes | General workloads |
|
||
|
|
| GPU (SPOT) | g2-standard-8 + L4 | 0-5 nodes | LLM inference |
|
||
|
|
|
||
|
|
### Networking
|
||
|
|
|
||
|
|
- **VPC**: `devops-lab-network`
|
||
|
|
- **Primary CIDR**: `10.0.0.0/16`
|
||
|
|
- **Pod CIDR**: `192.168.32.0/20`
|
||
|
|
- **Service CIDR**: `192.168.16.0/24`
|
||
|
|
- **CNI**: Cilium with advanced datapath
|
||
|
|
- **Ingress**: GCP L7 Global Load Balancer
|
||
|
|
|
||
|
|
### GitOps Flow
|
||
|
|
|
||
|
|
```
|
||
|
|
GitHub Repository
|
||
|
|
│
|
||
|
|
▼
|
||
|
|
Flux Source Controller (watches git, 1min interval)
|
||
|
|
│
|
||
|
|
▼
|
||
|
|
Flux Kustomize Controller (applies manifests)
|
||
|
|
│
|
||
|
|
├── infrastructure/controllers → CNPG Operator
|
||
|
|
├── infrastructure/configs → Cluster configs
|
||
|
|
└── apps/staging/customer1 → Applications
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Applications
|
||
|
|
|
||
|
|
### N8N Workflow Automation
|
||
|
|
|
||
|
|
- **URL**: `https://n8n.sirius-sec.com`
|
||
|
|
- **Image**: `docker.n8n.io/n8nio/n8n:2.1.4`
|
||
|
|
- **Database**: PostgreSQL (dedicated `n8n` database)
|
||
|
|
- **Storage**: 1GB persistent volume
|
||
|
|
|
||
|
|
### News Intelligence Pipeline
|
||
|
|
|
||
|
|
A three-stage data pipeline running as Kubernetes CronJobs:
|
||
|
|
|
||
|
|
| Stage | Schedule | Container | Purpose |
|
||
|
|
|-------|----------|-----------|---------|
|
||
|
|
| Scraper | `:00` hourly | `siriussec/newsscraper` | Scrapes 100+ global news sources |
|
||
|
|
| Analyst | `:15` hourly | `siriussec/summarizer` + `ollama/ollama` | LLM-powered summarization |
|
||
|
|
| Telegram | `:20` hourly | `siriussec/news-messenger` | Distributes summaries to Telegram |
|
||
|
|
|
||
|
|
**News Sources Coverage**:
|
||
|
|
- North America: NPR, AP News, CBC, etc.
|
||
|
|
- Europe: BBC, Reuters, The Guardian, etc.
|
||
|
|
- Asia: SCMP, Al Jazeera, Times of India, etc.
|
||
|
|
- Africa: BBC Africa, News24, etc.
|
||
|
|
- South America: Buenos Aires Herald, etc.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Getting Started
|
||
|
|
|
||
|
|
### Prerequisites
|
||
|
|
|
||
|
|
- Google Cloud account with billing enabled
|
||
|
|
- GitHub account with repository access
|
||
|
|
- `gcloud` CLI authenticated
|
||
|
|
- Terraform 1.7+
|
||
|
|
|
||
|
|
### Local Development Setup
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Install tools via mise
|
||
|
|
./scripts/setup
|
||
|
|
|
||
|
|
# Or manually
|
||
|
|
mise trust && mise install
|
||
|
|
```
|
||
|
|
|
||
|
|
### Infrastructure Deployment
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd modules
|
||
|
|
|
||
|
|
# Initialize Terraform
|
||
|
|
terraform init
|
||
|
|
|
||
|
|
# Set required variables
|
||
|
|
export TF_VAR_github_token="your-token"
|
||
|
|
export TF_VAR_github_org="your-org"
|
||
|
|
export TF_VAR_github_repository="gcloud-lab"
|
||
|
|
|
||
|
|
# Plan and apply
|
||
|
|
terraform plan
|
||
|
|
terraform apply
|
||
|
|
```
|
||
|
|
|
||
|
|
### Accessing the Cluster
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Configure kubectl
|
||
|
|
gcloud container clusters get-credentials devops-lab-cluster \
|
||
|
|
--zone us-central1-a \
|
||
|
|
--project devops-lab-cluster
|
||
|
|
|
||
|
|
# Verify connection
|
||
|
|
kubectl get nodes
|
||
|
|
|
||
|
|
# Use k9s for interactive management
|
||
|
|
k9s
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Security
|
||
|
|
|
||
|
|
### Secrets Management
|
||
|
|
|
||
|
|
- **Encryption**: SOPS with Age encryption
|
||
|
|
- **Key Storage**: `age.agekey` (do not commit unencrypted)
|
||
|
|
- **Flux Integration**: Automatic decryption during deployment
|
||
|
|
|
||
|
|
### Pod Security
|
||
|
|
|
||
|
|
- Non-root containers (UID 1000)
|
||
|
|
- Filesystem group enforcement
|
||
|
|
- Privilege escalation disabled
|
||
|
|
- Resource limits enforced
|
||
|
|
|
||
|
|
### Network Security
|
||
|
|
|
||
|
|
- Cilium network policies for pod-to-pod isolation
|
||
|
|
- TLS termination at load balancer
|
||
|
|
- Private cluster networking with NAT
|
||
|
|
|
||
|
|
### Database Security
|
||
|
|
|
||
|
|
- Managed roles with secret-based passwords
|
||
|
|
- Separate users per application (`customer1`, `news_app`)
|
||
|
|
- HA cluster with automatic failover
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Tool Reference
|
||
|
|
|
||
|
|
### Terraform Providers
|
||
|
|
|
||
|
|
```hcl
|
||
|
|
google = "~> 7.14" # GCP resources
|
||
|
|
helm = "~> 2.0" # Helm chart management
|
||
|
|
flux = "~> 1.7" # GitOps bootstrap
|
||
|
|
```
|
||
|
|
|
||
|
|
### Helm Charts
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
cilium: 1.18.5 # CNI and service mesh
|
||
|
|
cloudnative-pg: 0.26.1 # PostgreSQL operator
|
||
|
|
```
|
||
|
|
|
||
|
|
### Container Images
|
||
|
|
|
||
|
|
```
|
||
|
|
docker.n8n.io/n8nio/n8n:2.1.4
|
||
|
|
ghcr.io/cloudnative-pg/postgresql:15.2
|
||
|
|
ollama/ollama:latest
|
||
|
|
siriussec/newsscraper:latest
|
||
|
|
siriussec/summarizer:latest
|
||
|
|
siriussec/news-messenger:latest
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Cost Optimization
|
||
|
|
|
||
|
|
- **SPOT GPU Instances**: 60-90% savings on LLM workloads
|
||
|
|
- **Autoscaling**: GPU nodes scale to 0 when idle
|
||
|
|
- **Resource Limits**: Prevents runaway costs
|
||
|
|
- **Scheduled Workloads**: CronJobs only run when needed
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
Private repository - All rights reserved.
|