Headless Browser Automation: Puppeteer & Selenium on Linux VPS in Pakistan

From automated price monitoring across competitive Pakistani e-commerce stores (Daraz, PriceOye, Telemart) to automated PDF invoice generation, end-to-end regression testing, and large-scale web scraping, running Headless Browser Automation (Puppeteer, Playwright, Selenium) on Linux servers is a core engineering requirement.

However, operating headless Chromium or Firefox in production is notorious for crashing servers:

• Each headless browser process consumes 250MB to 500MB of RAM; spinning up 20 concurrent tasks triggers Linux kernel Out-of-Memory (OOM) killer panics.
• Cloudflare and Akamai bot-detection systems instantly flag default headless browser flags (navigator.webdriver = true), throwing Cloudflare Turnstile CAPTCHAs.
• Rogue, zombie Chromium processes linger indefinitely in the process table after script execution crashes.

Running reliable, high-throughput browser automation requires proper shared memory configuration (/dev/shm), virtual framebuffers (Xvfb), stealth evasions, and process supervision.

This guide provides an end-to-end engineering blueprint for deploying, optimizing, and scaling Puppeteer and Selenium on Linux VPS in Pakistan.

---

1. System Dependencies & Headless Chrome Installation on Ubuntu / Debian

A minimal headless Linux server lacks graphics libraries, font rendering engines, and sound daemons required by Chromium. Installing these base libraries is essential:

# 1. Update system packages
sudo apt update && sudo apt upgrade -y

# 2. Install Chromium and essential graphics/font libraries
sudo apt install -y chromium-browser chromium-chromedriver \
  xvfb libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 \
  libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 \
  libxrandr2 libgbm1 libasound2 fonts-liberation libappindicator3-1

---

2. Preventing Kernel OOM Crashes: Sizing /dev/shm

By default, Chromium uses the /dev/shm shared memory partition for inter-process communication and rendering buffers. On standard Docker containers and minimal Linux VPS, /dev/shm is often capped at a tiny 64MB:

# Check current shared memory size
df -h /dev/shm

When Chromium exhausts /dev/shm, the browser crashes silently with Error: Session closed. Most likely the page crashed.

The Two Solutions:

1. Pass the Flag in Puppeteer/Selenium: Launch the browser with --disable-dev-shm-usage. This instructs Chromium to fall back to the standard /tmp disk directory instead of RAM shared memory.
2. Resize /dev/shm in /etc/fstab:

   sudo mount -o remount,size=4G /dev/shm

---

3. High-Performance Puppeteer Script with Anti-Bot Stealth

Below is a production-hardened Node.js Puppeteer automation script incorporating memory leak recycling and Cloudflare Turnstile stealth bypasses:

// scraper.js
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

// Enable stealth plugin to mask headless indicators
puppeteer.use(StealthPlugin());

async function runScraper(targetUrl) {
    const browser = await puppeteer.launch({
        headless: 'new', // Modern Chrome headless implementation
        executablePath: '/usr/bin/chromium-browser',
        args: [
            '--no-sandbox',                      // Mandatory when running as root/daemon
            '--disable-setuid-sandbox',
            '--disable-dev-shm-usage',           // Prevents shared memory crashes
            '--disable-accelerated-2d-canvas',
            '--disable-gpu',
            '--window-size=1920,1080',
            '--single-process',                  // Combines threads into single process for lower RAM
            '--proxy-server=http://proxy.pk:8080' // Optional proxy rotation
        ]
    });

    try {
        const page = await browser.newPage();

        // 1. Block unnecessary assets (Images, CSS, Fonts) to save 85% bandwidth!
        await page.setRequestInterception(true);
        page.on('request', (req) => {
            const resourceType = req.resourceType();
            if (['image', 'stylesheet', 'font', 'media'].includes(resourceType)) {
                req.abort();
            } else {
                req.continue();
            }
        });

        // 2. Set realistic user agent
        await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36');

        // 3. Navigate with strict timeout
        await page.goto(targetUrl, { waitUntil: 'domcontentloaded', timeout: 25000 });

        // Extract required data
        const title = await page.title();
        console.log(`Successfully scraped: ${title}`);

    } catch (err) {
        console.error(`Scrape failed for ${targetUrl}:`, err.message);
    } finally {
        // CRITICAL: Always close browser in finally block to prevent zombie processes!
        await browser.close();
    }
}

runScraper('https://example.com');

---

4. Killing Zombie Chromium Processes via Systemd Watchdog

Even with strict error handling, browser automation scripts can occasionally leave orphaned processes that linger in RAM.

Create an automated cleanup script /usr/local/bin/cleanup-zombie-chrome.sh:

#!/bin/bash
# Find and terminate Chrome processes older than 15 minutes
killall -o 15m -9 chromium-browser || true
killall -o 15m -9 chromedriver || true

Schedule in crontab to run every 30 minutes:

*/30 * * * * /usr/local/bin/cleanup-zombie-chrome.sh > /dev/null 2>&1

For data engineering teams and web intelligence platforms in Pakistan running continuous multi-threaded scraping fleets across millions of URLs, deploying on Dedicated Servers in Pakistan provides physical hardware isolation, unthrottled CPU cores, and sub-10ms domestic ping times over the Pakistan Internet Exchange (PKIX).

---

5. Architectural Comparison: Headless Browser Frameworks

Framework	Execution Engine	Memory per Worker	Anti-Bot Stealth	Language Support
Selenium WebDriver	Traditional JSON Wire	~450MB	Moderate	Python, Java, C#, PHP
Puppeteer (Chrome DevTools)	Native CDP WebSocket	~280MB	High (Stealth Plugin)	Node.js / JavaScript
Playwright (Microsoft)	Unified Multi-Browser	~220MB	High	Node, Python, C#, Java
HTTP Request (Cheerio/BS4)	No Browser / Pure HTTP	< 25MB	High (API level)	Node, Python

For growing development teams seeking dedicated vCPU compute, private virtual networks, and pure NVMe performance for scraping and QA automation, our pure NVMe Cloud VPS instances deliver predictable execution and local sub-15ms domestic ping times across Pakistan.

For enterprise IT corporations operating distributed scraping clusters and international data pipelines across Europe, North America, and Asia, combining domestic nodes with our global Dedicated Servers provides unthrottled 10Gbps connectivity and enterprise hardware customization.

---

Related DevOps & Automation Engineering Guides

Continue advancing your server automation and container architecture:

• Docker Compose Production Best Practices: Secrets & Healthchecks
• Podman Rootless Containers on Production Linux VPS
• Admin RDP vs User RDP: Full Access vs Restricted Sessions