v0.1.0 In development — the desktop app has no published release yet

Running a crawl

Consuela walks a site from one start URL, records what it finds in SQLite, and audits the result. The same binary runs interactively and in CI.

Install

There is no published binary yet, and no way to install Consuela today. The commands below are what running it will look like; the download page is where a release will appear.

Your first pass

The URL is a positional argument; there are no subcommands. By default a crawl stops at 500 URLs and keeps nothing on disk, so pass --db when you want the results to survive the process.

# the whole site, from one start URL
consuela https://acme.com

# keep the results, and go wider than the 500-URL default
consuela https://acme.com --db acme.db --max-urls 20000

# stay off a client's server
consuela https://acme.com --concurrency 2 --rate 1

Robots rules are respected unless you pass --no-robots, and the per-host rate limit you set is never exceeded. You are responsible for not knocking over a client's server.

CLI

consuela <URL> [OPTIONS]

Flag	Default	What it does
--max-urls	500	Maximum number of URLs to crawl.
--max-depth	10	Maximum crawl depth.
--concurrency	5	Concurrent requests.
--rate	2.0	Requests per second.
--timeout	30	Per-request timeout, in seconds.
--user-agent	—	User-Agent header.
--include	—	Only crawl URLs matching this regex. Repeatable.
--exclude	—	Skip URLs matching this regex. Repeatable.
--no-robots	off	Ignore robots.txt.
--follow-external	off	Follow links off the starting host.
--format	text	text, json or junit.
--output, -o	stdout	Write output to a file instead of stdout.
--fail-on	none	Exit non-zero at this severity or worse.
--db	in-memory	Persist the crawl to this SQLite file.
--quiet, -q	off	Suppress progress output on stderr.

Exit codes and CI

--fail-on takes a severity and drives the exit code, which is what makes the crawler usable as a build step. --format junit emits XML that most CI systems read natively.

# fails the build when anything critical is found
consuela https://acme.com --format junit --output results.xml \
  --fail-on critical --quiet

The crawl database

Results stream into SQLite rather than accumulating in memory, so a large site does not grow until the machine gives up. The file is yours — open it with any SQLite client and query it directly. Nothing is uploaded unless you choose Consuela Cloud.