SpyWeb CDP Documentation

Spyweb does not bundle a browser. Instead, the built-in cdp module automatically detects and uses a Chromium-based browser (such as Chrome, Edge, or Brave) already installed on your system to execute JavaScript and render dynamic pages.

If you need JS rendering in a hook (to scrape a React site, bypass a Cloudflare challenge, click buttons, wait for DOM elements), you use the cdp module. That's it.

How It Works

Spyweb can get a browser two ways:

1. Launch one - cdp.launch() finds Chrome, Edge, or Brave on your system, starts it in headless mode, and connects to it automatically. Zero config. You can also pass { executable = "/path/to/browser" } to use a specific binary.
2. Connect to an existing one - cdp.connect("ws://...", [headers]) attaches to any browser or CDP-compatible server already running. The optional headers table allows for custom HTTP headers like Authorization.

local ws_url = "wss://api.cloudflare.com/client/v4/accounts/<ID>/browser-rendering/devtools/browser"
local browser = cdp.connect(ws_url, {
    ["Authorization"] = "Bearer <TOKEN>"
})

Once connected, you call browser:attach() to get a page/tab, then drive it with simple commands like page:open(url), page:wait_for_selector(), page:click(), and page:content() to get the rendered HTML.

Under the Hood

When you call cdp.launch(), Spyweb spawns the browser as a child process with --remote-debugging-port=0 (which picks a random free port). It reads the DevTools WebSocket URL from the browser's stderr output, connects to it, and gives you a Browser object. Every command and response is sent as JSON-RPC messages over WebSocket - the same protocol Chrome's own DevTools uses.

Each page/tab gets its own dedicated WebSocket connection. This is why browser:attach() doesn't just return a reference - it actually opens a new WebSocket to the page's specific endpoint.

By default, closing the browser kills the process and cleans up the profile. You can override this with keep_alive = true to keep the browser warm across multiple hook calls.

Basic Usage in override_fetch

CDP is available globally via the cdp table. While it can be used in any hook, it is most commonly used in override_fetch to replace the default HTTP client with a real browser.

function override_fetch(request)
    -- 1. Launch a temporary browser
    local browser = cdp.launch({})

    -- 2. Register automatic cleanup on exit (successful or error)
    defer(function() browser:close() end)

    -- 3. Attach to a page (tab)
    local page = browser:attach()

    -- 4. Navigate safely and check for errors
    local ok, err = page:open(request.url)
    if not ok then
        return { error = "Navigation failed: " .. tostring(err) }
    end

    -- 5. Wait for content before extraction
    local found, wait_err = page:wait_for_selector(".dynamic-content", 10000)
    if not found then
        return { error = "Selector timeout: " .. tostring(wait_err) }
    end

    -- 6. Get the rendered HTML (browser is closed automatically by defer)
    local html = page:content()

    -- 7. Pass the html back to the pipeline
    return {
        status = 200,
        body = html,
        url = request.url
    }
end

Persistent Browser Pattern

Launching a new browser for every fetch is slow. Because Spyweb persists the Lua state for each job, you can keep a browser instance "warm" across multiple scraper runs. Store the browser in a global variable (without local) to keep it alive between iterations.

-- This runs once when the job is loaded
if not browser then
    print("[CDP] Launching persistent browser...")
    browser = cdp.launch({
        headless = true,
        keep_alive = true
    })
end

function override_fetch(request)
    local page = browser:attach()
    local ok, err = page:open(request.url)
    if not ok then
        page:close()
        return { error = "Failed to load " .. request.url }
    end
    if page:wait_for_selector(".item", 5000) then
        local html = page:content()
        page:close()

        -- Pass the html back to the pipeline (triggers extraction or catches in after_fetch)
        return {
            status = 200,
            body = html,
            url = request.url
        }
    else
        page:close()
        return { error = "Content timeout" }
    end
end

These methods are provided directly by the Spyweb Core on the Browser object returned by cdp.launch() or cdp.connect().

-- Attach with a specific URL
local page = browser:attach({ url = "https://example.com" })

-- Attach within an isolated context
local ctx = browser:new_context()
local page = ctx:attach("https://example.com")

local ctx = browser:new_context()
local page = ctx:attach("https://example.com")
-- Cookies/storage in this context are isolated from the default context
ctx:close()

browser:close()

local dir = browser:get_user_data_dir()
print("Profile at: " .. tostring(dir))

-- List all targets (tabs/pages) in the browser
local targets = browser:call("Target.getTargets", {})
for _, t in ipairs(targets.targetInfos) do
    print(t.targetId, t.url)
end

-- Wait for a new target to be created (up to 5 seconds)
local event = browser:wait_event("Target.targetCreated", 5000)
print("New target: " .. event.params.targetInfo.targetId)

local result = browser:attach_session(target_id)
local session_id = result.sessionId

local result = browser:call_session(sid, "Page.navigate", {
    url = "https://example.com"
})

local ev = browser:wait_session_event(sid, "Page.loadEventFired", {
    timeout_ms = 10000
})

Returned by browser:new_context().

local page = ctx:attach("https://example.com")

ctx:close()

The Page object combines native transport methods with high-level Lua helpers. Returned by browser:attach() and context:attach().

Native Methods

Low-level CDP methods available directly on every page object.

local result = page:call("Runtime.evaluate", {
    expression = "document.title",
    returnByValue = true
})
print(result.result.value)

-- Download a PDF and save to disk
page:call_save("Page.printToPDF", {}, "/tmp/report.pdf")
-- Result includes { saved_to = "/tmp/report.pdf" } instead of the raw data

-- Wait for page load with default 30s timeout
local ev = page:wait_event("Page.loadEventFired")

-- With predicate and timeout
local ev = page:wait_event("Network.responseReceived", 5000, function(params)
    return params.response.status == 200
end)

page:close()

Higher-level CDP abstraction methods injected by cdp.lua for convenience. These are plain Lua functions - you can override or extend them on any page object.

-- Default: wait for full load (30s timeout)
local ok, err = page:open("https://example.com")

-- Wait for DOM content instead of full load
page:open("https://example.com", "Page.domContentEventFired")

-- Custom timeout
page:open("https://example.com", "Page.loadEventFired", 15000)

-- Fire-and-forget: navigate without waiting
page:open("https://example.com", false)

-- Simple timeout
local found, info = page:wait_for_selector(".content", 5000)

-- Full options
local found, info = page:wait_for_selector(".product", {
    timeout_ms = 15000,
    poll_ms = 200,
    visible = true,
    scroll = true
})
if found then
    print("Tag: " .. info.tagName)
    print("Text: " .. info.text)
end

-- Wait until the URL contains "/success"
local url = page:wait_for_url("/success", 10000)
if url then
    print("Redirected to: " .. url)
end

-- Wait for any response
local resp = page:wait_for_response(nil, 5000)

-- Wait for a specific API response
local resp = page:wait_for_response(function(params)
    return params.response.url:find("/api/data") and params.response.status == 200
end, 10000)

-- Wait up to 10s for the page to be idle
page:open("https://example.com", false)
page:wait_for_idle(10000, 500)

-- Scroll until an element appears or bottom is hit
page:scroll({
    max_scrolls = 50,
    delay_ms = 500,
    until_selector = ".load-more"
})

local html = page:content()
print(string.sub(html, 1, 500)) -- First 500 chars

local title = page:evaluate("document.title")
local count = page:evaluate("document.querySelectorAll('.item').length")
local data = page:evaluate("JSON.stringify(window.__INITIAL_STATE__)")

-- Standard click (JS dispatch)
page:click(".submit-btn")

-- Real hardware-level mouse events (better anti-bot)
page:click(".buy-now", { real = true })

-- Standard input
page:type("#search", "hello world")

-- Real keystroke simulation
page:type("#username", "admin", { real = true })

-- Basic screenshot (PNG)
page:screenshot("debug.png")

-- Full-page JPEG
page:screenshot("fullpage.jpg", {
    format = "jpeg",
    quality = 80,
    full_page = true
})

page:set_extra_headers({
    ["Accept-Language"] = "en-US",
    ["X-Custom"] = "my-value"
})

page:set_user_agent("Mozilla/5.0 ...", {
    accept_language = "en-US,en;q=0.9",
    platform = "Linux"
})

-- All cookies
local all = page:cookies()

-- Cookies for specific URLs
local filtered = page:cookies({ "https://example.com" })
for _, c in ipairs(filtered) do
    print(c.name, c.value)
end

page:set_cookies({
    { name = "session", value = "abc123", domain = "example.com", path = "/" },
    { name = "theme", value = "dark", domain = "example.com" }
})

-- Block images and fonts
page:block_resources({ "image", "font" })

-- Block specific URL patterns
page:block_resources({ "*.analytics.js", "*.tracking.com/*" })

While standard Chromium-based browsers are the most compatible, they are resource-intensive. A single Chrome instance can consume hundreds of megabytes of RAM, and its profile directory (user_data_dir) often balloons to 150MB+ immediately upon launch, For a lighter approach, consider these specialized alternatives designed for scraping and automation. They speak the same CDP protocol.

Lightpanda

A high-performance, lightweight browser written in Zig.

1. Launch the server:

   lightpanda serve --port 9222

2. Connect in Lua:

   local browser = cdp.connect("ws://127.0.0.1:9222")

Obscura

A headless browser built for AI agents and anti-detection.

1. Launch the server:

   obscura serve --port 9222 --stealth

2. Connect in Lua:

   local browser = cdp.connect("ws://127.0.0.1:9222/devtools/browser")

Compatibility Note

These browsers implement CDP but may not support every method. Based on our assessment:

Hybrid Human-in-the-Loop

For sites with aggressive bot detection (Cloudflare, CAPTCHAs, etc.), you can switch from a fast headless browser to a visible Chrome window when a block is detected.

1. Normal Mode: Scrape headlessly for maximum speed.
2. Detection: In override_fetch, check for a "block" selector (e.g., #captcha-container).
3. Transition: Close the headless browser and launch a visible Chrome instance using the same user_data_dir.
4. Notification: Use notify() to alert a human operator.
5. Intervention: Wait in a while loop, polling for a "success" selector.
6. Handback: Capture the HTML, close the visual browser, and return to headless mode.

See examples/hybrid-recovery/hooks.lua for a complete implementation.