Self-Hosting Windmill on Linux VPS in Pakistan: The Open-Source Retool & Temporal Alternative for Dev Teams

Modern Pakistani software houses, digital agencies, and fintech startups face an ongoing operational dilemma: engineering teams write dozens of utility scripts in Python, TypeScript, Go, and Bash to manage operational tasks, but turning those scripts into maintainable internal tools, scheduled jobs, and complex background workflows remains painful. Traditionally, engineering leaders bridge this gap using proprietary US SaaS platforms like Retool for internal web apps, Temporal Cloud for durable workflow orchestration, or Airplane and Prefect for script execution.

However, relying on foreign SaaS tools introduces severe financial and operational overhead in Pakistan. USD per-seat licensing ($20 to $50 per user per month), compute-metered runtime billing, persistent Pakistani Rupee (PKR) foreign exchange volatility, State Bank of Pakistan (SBP) corporate credit card limits, and mandatory 10% advance withholding taxes on international card transactions drain engineering profit margins. Furthermore, piping sensitive local customer records, database credentials, and payment reconciliation payloads to multi-tenant US cloud servers creates severe compliance and data sovereignty liabilities.

Self-hosting Windmill—an enterprise-grade, high-performance developer platform written in Rust and Svelte—on a dedicated Linux VPS in Pakistan provides an ideal solution. Windmill transforms raw scripts into production-ready auto-generated user interfaces, scheduled distributed crons, and multi-step directed acyclic graph (DAG) workflows with sub-5ms cold-start execution times.

This comprehensive technical guide walks through the economic realities of developer SaaS in Pakistan, Windmill’s low-latency architecture, a hardened Docker Compose deployment on Ubuntu 24.04/22.04 LTS, Nginx reverse proxy configuration with WebSocket support, script-to-workflow automation patterns, secret management, worker scaling, and disaster recovery.

---

Economic & Compliance Realities: The Case Against USD Developer SaaS in Pakistan

For a mid-sized Pakistani software house or digital agency with 25 engineers and operators executing internal database utilities, customer support dashboards, and automated batch jobs, proprietary developer platforms quickly become cost-prohibitive.

+-----------------------------------------------------------------------------------+
|               MONTHLY COST COMPARISON: 25 DEVELOPERS & OPERATORS                  |
+-----------------------------------------------------------------------------------+
| Metric / Cost Element           | Retool + Temporal Cloud  | Self-Hosted Windmill |
+---------------------------------+--------------------------+----------------------+
| Per-Seat Developer Licensing    | $1,250.00 USD ($50/user) | $0.00 (Open-Source)  |
| Compute & Task Execution Fees   | $350.00 USD (Metered)    | $0.00 (Unmetered)    |
| FBR / SBP Advance WHT (10%)     | $160.00 USD              | $0.00                |
| Bank FX Markup & Processing (5%)| $88.00 USD               | $0.00                |
| Total Monthly USD Cost          | $1,848.00 USD            | $0.00                |
| Monthly PKR Equivalent (@ 280)  | Rs. 517,440 PKR / mo     | Rs. 0                |
| High-Performance Linux VPS      | Included in SaaS         | Rs. 9,500 - 16,000   |
+---------------------------------+--------------------------+----------------------+
| Annual Operational Cost         | Rs. 6,209,280 PKR        | Rs. 144,000 PKR      |
+---------------------------------+--------------------------+----------------------+

Beyond an annual direct cost reduction exceeding 97%, self-hosting Windmill solves three critical operational roadblocks native to Pakistan:

1. State Bank of Pakistan Credit Card Quotas: Corporate cards issued by Meezan Bank, HBL, Bank Alfalah, or MCB are subject to strict calendar-month foreign exchange caps and compliance scrutinies. Unexpected SaaS card charge failures can instantly lock developers out of critical production deployment workflows and internal administrative dashboards.
2. Data Sovereignty and Core Credential Isolation: Enterprise fintech platforms, logistics systems, and payment integrators (handling Raast, 1BILL, PayFast, or JazzCash APIs) must not expose internal production database connection strings, API secrets, or customer PII to external third-party infrastructure. Self-hosting keeps all cryptographic keys and audit logs within your private network perimeter.
3. Unmetered Script Runtimes: SaaS workflow platforms charge heavily for execution duration and payload volume. When running long-running ETL processes, database migrations, or web scrapers that execute for hours, cloud SaaS pricing models quickly balloon out of control.

---

Windmill Technical Architecture: Rust, PostgreSQL & Sandboxed Runtimes

Unlike legacy internal tool builders built on heavy Node.js or Ruby runtimes, Windmill is architected for near-instant execution, minimal memory footprint, and military-grade process isolation:

+-----------------------------------------------------------------------------------+
|                      WINDMILL PRODUCTION ARCHITECTURE                             |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  [ Ingress Traffic ]                                                              |
|         │                                                                         |
|         ▼                                                                         |
|  [ Nginx Reverse Proxy / SSL Termination / HTTP/2 / WebSockets ]                  |
|         │                                                                         |
|         ├───► [ Windmill Svelte Frontend & Static Web UI ]                        |
|         │                                                                         |
|         ▼                                                                         |
|  [ Windmill Server (High-Performance Rust Backend :8000) ]                        |
|         │                                                                         |
|         ├───► [ PostgreSQL 16 (State Engine, Audit Logs & Queue) ]                |
|         │                                                                         |
|         ▼                                                                         |
|  [ Windmill Worker Pool (Dedicated Execution Runners) ]                           |
|         ├─── Worker 1 (Python / uv Runtime)                                       |
|         ├─── Worker 2 (TypeScript / Deno V8 Runtime)                              |
|         ├─── Worker 3 (Go Native Binary Runner)                                   |
|         ├─── Worker 4 (Bash / Linux Utilities)                                    |
|         └─── Isolated Sandboxing (nsjail / Docker Process Barriers)                |
+-----------------------------------------------------------------------------------+

Core Architectural Components

• windmill-server (Rust): The core control plane, REST API, and job dispatcher are compiled into an optimized Rust binary. It routes incoming webhook requests, validates JSON schemas, evaluates permission graphs, and coordinates background workers with negligible CPU overhead and sub-5 millisecond API dispatch latency.
• Svelte Single-Page Application (SPA): The administrative dashboard, visual flow builder, and auto-generated user interfaces run on a lightweight Svelte client, offering instant UI interactions without the client-side bloat common in heavy React-based internal tool builders.
• PostgreSQL 16 Engine: PostgreSQL acts as the single source of truth, managing transactional state machines, execution queue pub/sub notifications (via LISTEN/NOTIFY), job history, granular audit logs, and workspace metadata.
• Sandboxed Worker Nodes: Workers poll PostgreSQL for queued jobs. When executing scripts, Windmill runs code inside sandboxed environments utilizing Linux namespaces (nsjail) or dedicated container boundaries. Scripts in Python use uv and cached virtual environments for sub-second package dependency resolution, while TypeScript executes natively on Deno’s secure V8 sandbox.

---

Step-by-Step Production Deployment on Ubuntu 24.04/22.04 LTS

Follow this production-grade deployment procedure to install Windmill on a clean Ubuntu LTS virtual server.

Prerequisites and System Preparation

Ensure your server is updated and equipped with essential networking tools:

# Update repositories and upgrade system packages
sudo apt-get update && sudo apt-get upgrade -y

# Install prerequisites
sudo apt-get install -y curl wget git htop ufw fail2ban ca-certificates gnupg lsb-release

# Install Docker Engine and Docker Compose V2
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Enable and start Docker service
sudo systemctl enable --now docker

Directory Structure and Secrets Setup

Create an isolated directory structure under /opt/windmill:

sudo mkdir -p /opt/windmill/data/postgres
sudo mkdir -p /opt/windmill/data/windmill
cd /opt/windmill

Generate a cryptographically secure random encryption secret for WM_SECRET:

# Generate 64-character hex secret
openssl rand -hex 32

Create the environment file /opt/windmill/.env:

# /opt/windmill/.env
DATABASE_URL=postgres://windmill:SuperSecureWindmillPassword2026@db:5432/windmill?sslmode=disable
WM_SECRET=YOUR_64_CHAR_GENERATED_HEX_SECRET_HERE
WM_BASE_URL=https://windmill.yourdomain.pk
PORT=8000
NUM_WORKERS=4
KEEP_JOB_DIR=false
DISABLE_SERVER_USAGE=false
PROMETHEUS_PORT=9090
METRICS_ADDR=true

Production Docker Compose Specification

Create /opt/windmill/docker-compose.yml:

version: "3.8"

services:
  db:
    image: postgres:16-alpine
    container_name: windmill_db
    restart: unless-stopped
    shm_size: 256mb
    volumes:
      - /opt/windmill/data/postgres:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: windmill
      POSTGRES_PASSWORD: SuperSecureWindmillPassword2026
      POSTGRES_DB: windmill
    command: >
      postgres -c max_connections=200
               -c shared_buffers=1GB
               -c effective_cache_size=3GB
               -c maintenance_work_mem=256MB
               -c checkpoint_completion_target=0.9
               -c wal_buffers=16MB
               -c default_statistics_target=100
               -c random_page_cost=1.1
               -c effective_io_concurrency=200
               -c work_mem=10MB
               -c min_wal_size=1GB
               -c max_wal_size=4GB
    networks:
      - windmill_net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U windmill -d windmill"]
      interval: 10s
      timeout: 5s
      retries: 5

  windmill_server:
    image: ghcr.io/windmill-labs/windmill:main
    container_name: windmill_server
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - WM_SECRET=${WM_SECRET}
      - BASE_URL=${WM_BASE_URL}
      - RUST_LOG=info
      - METRICS_ADDR=true
    volumes:
      - /opt/windmill/data/windmill:/tmp/windmill
    depends_on:
      db:
        condition: service_healthy
    networks:
      - windmill_net

  windmill_worker:
    image: ghcr.io/windmill-labs/windmill:main
    container_name: windmill_worker_default
    restart: unless-stopped
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - WM_SECRET=${WM_SECRET}
      - RUST_LOG=info
      - NUM_WORKERS=${NUM_WORKERS}
      - WORKER_GROUP=default
      - KEEP_JOB_DIR=false
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/windmill/data/windmill:/tmp/windmill
    depends_on:
      db:
        condition: service_healthy
      windmill_server:
        condition: service_started
    networks:
      - windmill_net

  windmill_worker_native:
    image: ghcr.io/windmill-labs/windmill:main
    container_name: windmill_worker_native
    restart: unless-stopped
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - WM_SECRET=${WM_SECRET}
      - RUST_LOG=info
      - NUM_WORKERS=2
      - WORKER_GROUP=native
    volumes:
      - /opt/windmill/data/windmill:/tmp/windmill
    depends_on:
      db:
        condition: service_healthy
    networks:
      - windmill_net

networks:
  windmill_net:
    driver: bridge

Start the Windmill cluster:

docker compose up -d

Verify that all containers are healthy:

docker compose ps

---

Nginx Reverse Proxy Configuration with WebSocket Streaming

Windmill delivers real-time script execution terminal logs via persistent WebSockets (/ws). Standard proxy configs will drop these connections, causing script execution streams to disconnect.

Install Nginx and Certbot:

sudo apt-get install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/windmill.conf:

# Map WebSocket upgrade headers
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name windmill.yourdomain.pk;

    # Maximum file upload size for script assets, CSV inputs, and payloads
    client_max_body_size 50M;

    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml font/truetype font/opentype application/vnd.ms-fontobject image/svg+xml;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;

        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Standard proxy headers
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Extended timeouts for long-running batch jobs and streaming logs
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
        proxy_connect_timeout 60s;
        
        # Disable proxy buffering for immediate real-time log output
        proxy_buffering off;
        proxy_cache off;
    }

    listen 80;
}

Enable the configuration and issue a free Let’s Encrypt SSL certificate:

sudo ln -s /etc/nginx/sites-available/windmill.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

# Issue automated Let's Encrypt SSL certificate
sudo certbot --nginx -d windmill.yourdomain.pk --non-interactive --agree-tos -m [email protected]

Test auto-renewal:

sudo certbot renew --dry-run

---

Transforming Scripts into Production Workflows

One of Windmill’s most powerful capabilities is parameter type inference: you write a standard function in Python, TypeScript, Go, SQL, or Bash with typed parameters, and Windmill instantly auto-generates a clean web UI form with validation.

Use Case 1: Automated Payment Gateway Reconciliation Pipeline

Pakistani e-commerce platforms and fintech applications often deal with discrepancies between local gateways (JazzCash, Easypaisa, PayFast) and internal core banking or ERP ledgers.

Here is an enterprise Python script deployed in Windmill:

# /u/admin/reconcile_payments.py
import os
import requests
import psycopg2
from typing import Dict, Any

def main(
    batch_date: str,
    gateway: str = "jazzcash",
    threshold_pkr: float = 1000.0,
    dry_run: bool = True
) -> Dict[str, Any]:
    """
    Automated Financial Reconciliation Job for Pakistani Payment Gateways.
    
    :param batch_date: Transaction date to reconcile (YYYY-MM-DD)
    :param gateway: Supported: jazzcash, easypaisa, payfast
    :param threshold_pkr: Anomaly alert limit in PKR
    :param dry_run: Preview discrepancies without committing changes
    """
    print(f"Starting reconciliation for {gateway} on {batch_date} (Dry Run: {dry_run})")
    
    # Securely retrieve database connection string from Windmill Resource
    db_url = os.environ.get("POSTGRES_RECORDS_DB")
    conn = psycopg2.connect(db_url)
    cursor = conn.cursor()
    
    # Query pending local transactions
    query = """
        SELECT tx_id, order_ref, amount_pkr, status 
        FROM transactions 
        WHERE gateway = %s AND tx_date = %s
    """
    cursor.execute(query, (gateway, batch_date))
    records = cursor.fetchall()
    
    matched = 0
    mismatched = []
    
    for row in records:
        tx_id, order_ref, amount, status = row
        # Perform remote gateway verification
        # (Mocked API verification logic)
        gateway_verified = True 
        
        if gateway_verified:
            matched += 1
        else:
            mismatched.append({
                "tx_id": tx_id,
                "order_ref": order_ref,
                "amount": float(amount)
            })
            
    cursor.close()
    conn.close()
    
    return {
        "status": "success",
        "gateway": gateway,
        "batch_date": batch_date,
        "total_evaluated": len(records),
        "matched_count": matched,
        "discrepancies": mismatched,
        "action_taken": "simulated" if dry_run else "reconciled"
    }

When saved in Windmill, this script automatically creates a responsive form with a date picker for batch_date, a dropdown for gateway, a numeric input for threshold_pkr, and a boolean toggle for dry_run. Operators without technical skills can safely execute the job from an internal UI.

Use Case 2: Multi-Step DAG Workflows

In Windmill’s visual Flow editor, you can chain scripts across languages:

1. Step 1 (SQL): Query a read-only replica for inactive customer accounts older than 90 days.
2. Step 2 (TypeScript/Deno): Iterate over the records, format an SMS template, and push messages via a local SMS gateway API (e.g., TeleStax, Jazz SMS API).
3. Step 3 (Bash): Execute a local maintenance command to vacuum database tables and flush Redis cache keys.
4. Step 4 (Python): Send an aggregated summary report to an internal Slack or Microsoft Teams webhook.

If any step fails, Windmill provides native retries, exponential backoffs, and suspension states (waiting for human approval before proceeding), completely replacing expensive Temporal Cloud deployments.

---

Granular RBAC, Secrets & Multi-Tenancy

Managing operational scripts across multiple agency clients or internal development departments requires strict isolation. Windmill delivers enterprise governance out-of-the-box:

Workspace Isolation

• Workspaces: Segregate development environments (e.g., client-alpha, fintech-core, marketing-automations).
• Granular Roles: Assign users as Admins, Authors (can write scripts), or Operators (can only execute approved scripts with forms).

Vault-Style Secret Management

Never hardcode credentials or database passwords. Windmill includes a built-in cryptographic secret and resource vault.

• Secrets (wmill.get_secret): Encrypted values stored using AES-GCM-256 with the master WM_SECRET.
• Resources: Structured objects (e.g., postgresql, s3, slack_webhook) that bundle connection strings, authentication tokens, and ports.

In Python, accessing secrets is seamless:

import wmill

def run():
    # Fetch encrypted resource automatically decrypted in memory
    db_config = wmill.get_resource("u/admin/primary_fintech_postgres")
    token = wmill.get_variable("u/admin/jazzcash_merchant_token")
    return {"host": db_config["host"], "authenticated": True}

---

Worker Scaling & Concurrency Tuning

When automation loads scale, tuning worker thread allocations and PostgreSQL connection ceilings prevents thread starvation.

Tuning Worker Thread Pools

In /opt/windmill/.env, adjust NUM_WORKERS based on physical or virtual CPU allocation:

# Recommended: 1 to 2 workers per allocated vCPU core
NUM_WORKERS=8

Dedicated Worker Groups

Route heavy ETL data pipelines away from latency-sensitive UI scripts by creating dedicated worker groups:

  windmill_worker_heavy_etl:
    image: ghcr.io/windmill-labs/windmill:main
    container_name: windmill_worker_etl
    restart: unless-stopped
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - WM_SECRET=${WM_SECRET}
      - NUM_WORKERS=2
      - WORKER_GROUP=heavy_etl
    networks:
      - windmill_net

Scripts tagged with @tag: heavy_etl are routed exclusively to these isolated worker processes, ensuring that long-running data transforms never block interactive dashboard actions.

---

Automated Database Backups & Offsite Disaster Recovery

Since PostgreSQL holds all script definitions, flow graphs, workspace permissions, and execution states, a strict backup routine is vital.

Create /opt/scripts/backup-windmill.sh:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="/var/backups/windmill"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
BACKUP_FILE="${BACKUP_DIR}/windmill_pg_${TIMESTAMP}.sql.gz"
RETENTION_DAYS=14

mkdir -p "${BACKUP_DIR}"

echo "[$(date)] Starting Windmill PostgreSQL backup..."

# Dump database through Docker container
docker exec windmill_db pg_dump -U windmill -d windmill | gzip -9 > "${BACKUP_FILE}"

# Set permissions
chmod 600 "${BACKUP_FILE}"

echo "[$(date)] Backup completed: ${BACKUP_FILE} ($(du -h "${BACKUP_FILE}" | cut -f1))"

# Prune backups older than 14 days
find "${BACKUP_DIR}" -type f -name "windmill_pg_*.sql.gz" -mtime +${RETENTION_DAYS} -delete

echo "[$(date)] Backup rotation completed."

Make executable and add to root crontab:

sudo chmod +x /opt/scripts/backup-windmill.sh
sudo crontab -l 2>/dev/null | { cat; echo "0 2 * * * /opt/scripts/backup-windmill.sh >> /var/log/windmill_backup.log 2>&1"; } | sudo crontab -

For true offsite disaster recovery, configure rclone to synchronize /var/backups/windmill with an offsite S3-compatible bucket or dedicated storage node daily.

---

Hardware Scaling Boundaries: VPS vs. Bare-Metal Dedicated Servers

A 4 vCPU, 8GB RAM virtual server comfortably handles dozens of daily batch workflows, lightweight cron triggers, and internal administrative portals. However, when software houses scale to enterprise-grade workloads, virtualization bottlenecks become apparent:

+-----------------------------------------------------------------------------------+
|               SCALING MATRIX: VPS VS. BARE-METAL DEDICATED SERVERS                |
+-----------------------------------------------------------------------------------+
| Metric / Workload Parameter     | Cloud VPS Node           | Bare-Metal Dedicated |
+---------------------------------+--------------------------+----------------------+
| Concurrent Worker Executions    | 10 - 40 tasks            | 200+ parallel tasks  |
| Monthly Workflow Executions     | 50,000 - 500,000 steps   | 5,000,000+ steps     |
| Disk I/O Architecture           | Virtualized NVMe block   | Direct PCIe Gen4 NVMe|
| CPU Cache Contention            | Shared host L3 cache     | 100% Dedicated Cache |
| Hypervisor CPU Steal Time       | 1% - 5% during peak host | 0.00% (Guaranteed)   |
| Postgres Write Latency (fsync)  | 2.5 - 8.0 ms             | < 0.25 ms            |
+---------------------------------+--------------------------+----------------------+

When engineering teams scale past 100+ concurrent background tasks, automated data pipelines processing millions of rows, and high-frequency webhook triggers, virtualized environments encounter I/O throttling and CPU steal time. Migrating heavy compute engines to bare-metal Dedicated Servers provides unmetered physical CPU cores, pure NVMe RAID performance, and zero hypervisor contention. Furthermore, utilizing high-bandwidth Dedicated Servers in Pakistan establishes direct peering with local Internet Exchanges (PKIX) and domestic telecommunication providers (PTCL, Nayatel, Wateen, TransWorld), reducing round-trip execution latency for Pakistani enterprise APIs to under 10 milliseconds.

---

Conclusion: Regain Control of Your Internal Engineering Operations

Self-hosting Windmill on a Linux VPS transforms developer operations for Pakistani engineering organizations. By replacing expensive, dollar-denominated per-seat platforms with a lightning-fast Rust and PostgreSQL open-source engine, teams build automated internal tools, run complex cron jobs, and orchestrate business-critical workflows without currency risk or credit card limitations.