Skip to content
indice

Command-line reference

indice index           [--home <DIR>] [--name <NAME>] --collection <NAME> [-f|--from-file <FILE>] [--download] [--force] [--no-optimize] [--concurrency <N>] [-v|--verbose] <PATH|URL>...
indice reindex         [--home <DIR>] [--concurrency <N>] [-v|--verbose]
indice optimize        [--home <DIR>] [--max-segments <N>] [-v|--verbose]
indice stats           [--home <DIR>]
indice config          [--home <DIR>]
indice serve           [--home <DIR>] [--bind <ADDR>] [--manage] [--auth-proxy-header <HEADER> --auth-proxy-secret <SECRET>]
indice collection set  [--home <DIR>] <NAME> [--creator <TEXT>] [--dates <TEXT>] [--rights <TEXT>] [--subject <SUBJECT>]... [--narrative <MD> | --narrative-file <FILE>] [--thumbnail <FILE>] [--description <TEXT>] [--curator <TEXT>]
indice collection list [--home <DIR>]
indice crawl set       [--home <DIR>] <CRAWL_ID> [--image <FILE>] [--note <MD> | --note-file <FILE>]
indice crawl list      [--home <DIR>] [<COLLECTION>]
indice search-url      [--home <DIR>] <URL>
indice verify          [--home <DIR>]
indice import browsertrix [--home <DIR>] [--host <URL>] [--org <SLUG>] [--collection <ID|SLUG>] [--crawl <ID>] [--into <NAME>] [--include-unreviewed] [--min-review <N>] [--limit <N>] [--dry-run] [--stream] [--force] [-v]
indice import archive-it [--home <DIR>] [--host <URL>] [--collection <ID>] [--crawl <ID>] [--into <NAME>] [--crawl-time-after <DATE>] [--crawl-time-before <DATE>] [--limit <N>] [--dry-run] [--include-deleted] [--force] [-v]
indice wacz build      [--home <DIR>] --collection <NAME> [--name <NAME>] [--title <T> | --title-file <FILE>] [--description <D> | --description-file <FILE>] [--creator <TEXT>] [--software <TEXT>] [--main-page-url <URL>] [--keyword <K>]... [--license <L>]... [--yes] [-v] <WARC>...

Every command takes --home <DIR> (default .); archive/ and index/ are derived siblings under it.

  • index — indexes one or more archived WACZ files or http(s):// URLs (at least one). By default indice reads a WACZ through its internal CDX index, extracting only the page records (and falling back to a full WARC scan only when a WACZ can’t be read that way — see How indexing reads a WACZ). A remote URL is streamed over HTTP range requests, no download (see Remote WACZ files). A local WACZ may live anywhere — indice files it into <home>/archive/<slug>/ (moving it if already under archive/, else copying it), and a directory or non-.wacz path is an error. Index several with a shell glob. Extracts searchable text from each page (HTML, Browsertrix’s rendered urn:text records or pages/*.jsonl text, and PDFs), reads datapackage.json for collection metadata, and records everything in the manifest under <home>/index/, including the SHA-256 of each local WACZ. Local WACZ paths are stored relative to home so the folder is portable. The WACZ name comes from --name if given, otherwise the WACZ’s datapackage.json title, otherwise the filename. --collection <NAME> is required — every crawl belongs to a curated collection (created if new); there are no auto singletons. --download fetches a remote WACZ into <home>/archive/<collection-slug>/ for a durable local copy instead of streaming it in place. To index many at once, pass a newline-delimited list of files/URLs with --from-file <FILE> (or -f - to read from stdin); blank lines and # comments are ignored, and it combines with any positional args. --concurrency <N> sets how many records are fetched at once during CDX-guided (streaming) indexing (default: 4 for remote URLs — gentle on the host, raise for object stores like S3; CPU count for local files; capped at 64 per host). Indexing shows a progress bar on an interactive terminal; -v/--verbose replaces it with debug logs. A multi-WACZ (a WACZ that bundles other WACZs, e.g. a Browsertrix combined-collection download) is detected automatically and its inner crawls indexed too, into one entry. Each WACZ is committed as it finishes, so a re-run skips sources already indexed into the collection — an interrupted large ingest resumes where it stopped. --force re-indexes a source that’s already there (to refresh it). (A --download remote URL is stored under a local path whose id differs from the URL’s, so it’s re-fetched on a re-run rather than skipped.) When a batch ingest leaves the index fragmented into many segments (which slows every query), indice compacts it automatically at the end of the run — so you don’t have to remember to run optimize. --no-optimize skips that (it just prints a reminder to optimize later instead); a healthy index, or a single add to an already-tidy one, is left alone either way.
  • collection — collection list shows collections and their crawl counts; collection set <NAME> … writes a collection’s finding-aid metadata (creator, dates, rights, subjects, a Markdown narrative, and an optional --thumbnail) to a git-committable collections/<slug>/README.md you can also hand-edit.
  • crawl — crawl list [<COLLECTION>] lists individual crawls with their 8-char ids (optionally filtered to one collection) — the ids you pass to crawl set/crawl delete. crawl set <ID> --note adds a per-crawl Markdown note (collections/<slug>/crawls/<id>.md); crawl set <ID> --image pins a crawl thumbnail there too. (WACZ→collection membership is set when indexing, via index --collection <NAME>.)
  • reindex — rebuild the search index from the WACZs already in the manifest, preserving collection membership and metadata. Re-fetches remote URL sources and recreates the index from scratch, so it’s the way to migrate after an upgrade changes the index schema. It’s resilient: a source that can’t be indexed (a missing local file, or a remote source still failing after retries) is skipped with a warning rather than aborting the rebuild; the mostly-rebuilt index is still usable, and if anything was skipped the command exits non-zero with a summary count so you (or cron/CI) know to re-run it once fixed. Takes --concurrency <N> and shows the same progress bar as index (a full reindex re-streams every source, so it can take a while); -v/--verbose swaps the bar for debug logs. (If you try to index or serve against an index built by an older version, indice tells you to run this.) The rebuild is atomic: it builds a fresh index alongside the live one and swaps it in only once the rebuild finishes, so a crash, kill, or full disk mid-rebuild leaves your existing index intact — and a running serve keeps answering from the old index until the swap. (Transient cost: the old and new index coexist on disk until then, ~2× the index size.)
  • optimize — compacts the search index by merging its Tantivy segments down toward --max-segments (default 8), without re-fetching sources — so it’s much cheaper than reindex. Every search fans out across all segments, so an index that has fragmented into hundreds of tiny segments (which happens when Tantivy’s background merges fail — classically on a full disk) gets slow; optimize merges them back down. A lower --max-segments compacts more but needs more free disk during the merge (roughly index size ÷ target). It also reclaims disk from deleted crawls — a delete only tombstones documents; their bytes are freed when the segment is rewritten, which optimize now does for any segment still carrying deletes (regardless of --max-segments) — and sweeps orphaned segment files left by an interrupted (Ctrl-C’d) run. Reports the before → after segment count and disk reclaimed. index runs this automatically when a batch ingest leaves the index fragmented, so you mostly only reach for it by hand to reclaim space after deleting crawls.
  • stats — reports the search index’s on-disk footprint, broken down by Tantivy file type (.store doc store, .pos positions, .term/.idx inverted index, .fast columnar, …), with bytes-per-document and projected sizes at 1M / 100M docs. Use it to see the effect of the frugality knobs (config.yaml) and to size a large ingest before running it.
  • config — prints the resolved operator configuration for the home (<home>/config.yaml, plus the built-in defaults for anything unset): the stored-body cap (index.stored_body_cap_kb, 0 = full body) and the Tantivy writer heap (index.writer_heap_mb). See Operator configuration.
  • serve — opens the index read-only and starts the HTTP server (so you can index while it runs). Defaults to 127.0.0.1:8080. --manage adds an opt-in browser UI + write API for adding archives and curating collections; see Manage & curate for local vs. behind-a-proxy (--auth-proxy-*) use. On startup it warns if the index is fragmented (many segments — e.g. built by an older version, or a killed run), pointing you to optimize.
  • search-url — a debugging aid: reads the CDX index inside each WACZ and prints the records matching a URL. No separate CDX store is maintained; the WACZ’s own index is authoritative.
  • verify — re-hashes every registered WACZ and compares against the SHA-256 recorded at index time, reporting each as OK, MODIFIED, or MISSING. Exits non-zero if any collection fails, so it works in a cron job or CI. This is indice’s fixity check — a small guard against the archive quietly bit-rotting or being tampered with.
  • import browsertrix — imports WACZ files from a Browsertrix instance (Webrecorder’s hosted crawler) — the “index your own crawls” path. See Importing from Browsertrix.
  • import archive-it — imports crawls from an Archive-It account over WASAPI, building one WACZ per crawl. See Importing from Archive-It.
  • wacz build — the “I have WARCs, not WACZs” on-ramp: packages one or more .warc/.warc.gz files into a WACZ under <home>/archive/ and indexes it. --collection <NAME> is required. The original WARC bytes are stored verbatim (uncompressed in the zip) — indice only packages your crawl data, it never rewrites it — and a CDX index + datapackage.json are generated so the WACZ both indexes here and is shaped to replay in ReplayWeb.page. The CDX mirrors warcio.js’s indexer (verified line-for-line against it) and the packaging mirrors browsertrix-crawler, so the output matches what Webrecorder’s own tools produce. Metadata (--title, --description, --creator, --keyword, --license, …) comes from flags; on an interactive terminal, missing values are prompted for (--yes skips prompting for scripts/CI). Each input WARC is sniff-tested first (must parse as a WARC with at least one indexable record) so a bad file fails fast instead of producing a broken WACZ. This is also the building block for importing from services that serve WARCs rather than WACZs (e.g. Archive-It).