Crawl
Turns a scattered public record into a reference you can trust.
Crawl is the agentic web-crawling Tool, and its mandate is web crawling in the general case: scan, corroborate and organise any public web source. Where Frame and Forge create first-party supply from the field, Crawl reaches the public record: the registries, portals, reports, and datasets that exist but sit scattered, in incompatible shapes, at unknown freshness, behind no single front door. Its loop is plain, find the sources, prove them, organise them, and check them again, under human curation.
From one Tool come a Hub and a supply stream. The Hub, its leading solution today, is source-rated and current, for local stakeholders and researchers, and for the people using a digital tool who need context beside their own records. The supply stream is the same organised data, tradable once Scribe conforms it and Spine exchanges it, the way any external source is.
Crawl’s method is the one part of the suite that has already met real sectors: the reference hubs for cobalt, extractives transparency, artisanal mining, responsible gold and Global-South climate were built this way, by hand. What is being built now is the engine underneath, one that takes a new sector as configuration, not a rewrite.
Where it sits. A public-data supply source. Its Hub output is standalone; its market-facing output flows through Scribe to conform, then Spine to exchange, the same path any external source follows.
- 01ScanStart where the truth is filed
Work the authoritative registers first, then the periphery, never a keyword guess.
- 02VerifyCorroborate, or say you couldn’t
Rate every finding strong · moderate · single · contested, and keep the sources behind it.
- 03OrganiseLegible to a person and to Scribe
Shape findings against the suite primitives, so the record reads well and conforms without a later step.
- 04MaintainThe date matters as much as the fact
Every record shows when it was last checked and whether its links still resolve, re-scanned on a cadence.
- 05LearnEach pass covers more ground
Fold each run into the sector’s crawl memory; it tunes what to look for, never what counts as proven.
memory sharpens the next scan, never the bar a claim has to clear
every record rated strong · moderate · single · contested
each record carries the date it was last checked
dead links surfaced, not hidden
every claim records the sources behind it
trust is earned by survival, not assumed
Corroboration and freshness sit on every record, the difference between a reference you can lean on and a scraped list. Tool-level run telemetry reports through Console. No blended scores, no numbers we can’t back.
- Public data consolidationOne navigable public reference for the sector, built out of sources that were only ever scattered, and answering for what is true this week rather than last quarter.
- Targeted knowledge discoveryA local data supply reaches the markets that want it: discoverable, permissioned and payable at the holder’s own discretion, with nobody taking custody of it to make that true.