Public data consolidation
Piecing together everything public about a sector is a job someone starts over every time, and it goes stale again within the week they finish.
The knowledge a sector runs on is scattered across the whole internet. It is public, and it is still hard to know about, hard to reach, and hard to hold together in one place. Anyone who needs the whole picture assembles it by hand, and it goes stale the week after they finish.
One navigable public reference for the sector, built out of sources that were only ever scattered, and answering for what is true this week rather than last quarter.
One run, start to finish. A box with an accent tick is a tool acting and opens its own page; a rounded box is an artefact or a record; a diamond is a decision the work cannot pass without. A dashed line is a hand-off the suite has not mechanised yet.
- reads the declared contract
- Nothing is sent. The downstream tool reads the upstream tool’s declared contract at call time, which is why substrate tools have no invoke step.
- writes current state
- Current state, replacing the prior value. A registry holds what is true now, not how it got there.
Identify the public sources, crawl and corroborate them, build the front door, and keep crawling so the front door stays true.
Each step in the order it runs, and where one stops before the next begins.
- 01RECORDWHICH SECTORS NEED THIS A record the run reads, writes or hands on.Nothing in this run reaches it. Hands on to NOT MECHANISED YET.
- 02RECORD · decisionNOT MECHANISED YET The work cannot pass this point without a decision.Reached from WHICH SECTORS NEED THIS. Hands on to MASON (FORK AND BRAND).
- 03RUN · toolMASON (FORK AND BRAND) Fork hub-template, brand, declare structure.Reached from NOT MECHANISED YET. Hands on to CRAWL.
- 04RECORDPUBLIC SOURCES A record the run reads, writes or hands on.Nothing in this run reaches it. Hands on to CRAWL.
- 05RUN · toolCRAWL Seed the verified corpus.Reached from MASON (FORK AND BRAND), PUBLIC SOURCES. Hands on to STARTER DATA FOR THE HUB, HUB DATA.
- 06RECORDSTARTER DATA FOR THE HUB A record the run reads, writes or hands on.Reached from CRAWL. Hands on to MASON (BUILD AND SHIP).
- 07RECORDHUB DATA A record the run reads, writes or hands on.Reached from CRAWL. Hands on to MASON.
- 08RUN · toolMASON (BUILD AND SHIP) Fork hub-template, brand, declare structure.Reached from STARTER DATA FOR THE HUB. Hands on to BUILD RECORD, LIVE FRONT DOOR.
- 09RUN · toolMASON Fork hub-template, brand, declare structure.Reached from HUB DATA. Hands on to BUILD RECORD, LIVE FRONT DOOR.
- 10RECORDBUILD RECORD A record the run reads, writes or hands on.Reached from MASON (BUILD AND SHIP), MASON. The run parks here.
- 11RECORDLIVE FRONT DOOR A record the run reads, writes or hands on.Reached from MASON (BUILD AND SHIP), MASON. The run parks here.
- 01nothing crossesMASON (FORK AND BRAND) WHICH SECTORS NEED THIS.NOT MECHANISED YET. The work cannot pass this point without a decision.This hand-off is not mechanised yet. A person notices, or nobody does.
- 02reads the declared contractMASON (FORK AND BRAND) → CRAWL Template structure.Nothing is sent. The downstream tool reads the upstream tool’s declared contract at call time, which is why substrate tools have no invoke step.
- 03writes current stateCRAWL → MASON (BUILD AND SHIP) Hub record.Current state, replacing the prior value. A registry holds what is true now, not how it got there.
- 04parks hereMASON (BUILD AND SHIP) BUILD RECORD.Build record.Nothing later in this run reads it, so this is where that branch of the run stops.
- 05parks hereMASON (BUILD AND SHIP) LIVE FRONT DOOR.Nothing later in this run reads it, so this is where that branch of the run stops.
- 06picked upCRAWL PUBLIC SOURCES.Scan + corroborate.Matter the run starts from. No member of the suite put it there.
- 07writes current stateCRAWL → MASON Build input.Current state, replacing the prior value. A registry holds what is true now, not how it got there.
- 08parks hereMASON BUILD RECORD.Build record.Nothing later in this run reads it, so this is where that branch of the run stops.
- 09parks hereMASON LIVE FRONT DOOR.Deploy.Nothing later in this run reads it, so this is where that branch of the run stops.
Fork → seed is a contract read (the template's config declares the structure Crawl seeds into); seed → build is a state write (hub record); ship is a state write (build record).
Crawl → Mason is a state write: Crawl updates the hub's own hub record and source credibility record, Mason reads the hub data as its build input. Mason → ship is a state write again.