File Structure

SpyWeb expects this directory layout for complete setup:

• Directoryexamples/ Sample configs

  • …
• Directorydocs/ Offline documentation

  • …
• Directoryjobs/ Per-job folders (advanced setups)

  • …
• Directoryserver/ Custom API server Lua scripts

  • …
• Directoryshared/ Cross-job/server shared files

  • …
• Directoryui/ Web dashboard files

  • …
• data Embedded database (created on first run)
• jobs.toml Single-file config (simple setups)
• spyweb Terminal executable
• spyweb-tray Tray executable (desktop)

Two Ways to Configure Jobs

Single File (jobs.toml)

All jobs in one file. Good for simple setups with few jobs.

[[jobs]]
name = "Job One"
url = "https://example.com"
selector = ".item"
fields = ["title:h2"]

[[jobs]]
name = "Job Two"
url = "https://other.com"
selector = ".post"
fields = ["name:h3"]

Per-Job Folders (jobs/)

Each job gets its own directory. Required when using Lua hooks or complex configs.

• Directoryjobs/

  • Directorymy-monitor/

    • config.toml Job configuration (required)
    • hooks.lua Lua hooks (optional)
    • defer.lua Lifecycle hooks (optional)
    • tests.lua Lua tests (optional)
    • data.csv Job-specific data files
  • Directoryanother-job/

    • config.toml
    • hooks.lua

Note

jobs.toml and jobs/ folder can coexist. Jobs in both are loaded.

Shared Directory (shared/)

Files in shared/ are accessible from any job via fs_read("shared/file.txt") or fs_overwrite("shared/file.txt", ...). Use this for:

• Config files loaded by multiple jobs
• Shared data or caches
• Assets used across jobs

• Directoryshared/

  • api-keys.json
  • proxy-list.txt
  • Directorytemplates/

    • …

Note

Write to shared/ from any job using the shared/ prefix: fs_append("shared/log.txt", "data").

Server Directory (server/)

For the programmable API server. Define custom HTTP endpoints here.

• Directoryserver/

  • init.lua Server entry point

See API Server for details.

Data Files

SpyWeb creates and manages these automatically:

File	Description
data	Embedded database (redb or SQLite)
hooks.log	Per-job log file (in job directory)
error.log	API server errors (in server/ directory)
*-response.html	Raw HTML from debug command
*-fields.json	Extracted fields from debug command

Lua File Lookup

When you require("utils"), SpyWeb searches:

1. Current job’s directory first
2. Project root as fallback

-- In jobs/my-job/hooks.lua
local utils = require("utils")  -- checks jobs/my-job/utils.lua, then ./utils.lua

Log Files

Context	File	Location
Job hooks	hooks.log	Job directory
API server	error.log	server/ directory