Ground · Chapter 04

Data as moat

Proprietary data and feedback loops against commodity models.

A collection of customer records becomes an advantage when it helps the business do something useful that would otherwise be difficult. The test is an outcome: a better recommendation, an earlier warning, or a more reliable decision. Data volume alone cannot establish that advantage.

Start with a question someone actually needs answered. If you want to know which customers are profitable, agree what revenue and service cost mean, which period you are measuring and how records identify the same customer. Joining on a name can produce an impressive answer about the wrong people.

Make one answer trustworthy

Try an authorized export before building a warehouse. Reconcile a known customer and the totals against their sources, recording currency, period and freshness. A warehouse is a measuring bench: putting the instruments together does not calibrate them.

When the question genuinely needs repeated data from several systems, replication tools such as Airbyte can connect them to an analytical destination such as ClickHouse. Select only the streams you need and give the assistant read-only access. Test schema changes on disposable data; the documented ClickHouse destination can remove historical column data when a source column disappears.

Find out whether the context helps

Compare the same task with and without the proprietary context, using an agreed measure of quality. Keep the observed outcome with the example so later work can learn from it. Check that you have the right to retain and use the records, and keep different customers’ information isolated.

You have a useful beginning when another person can reproduce the answer and explain its limits. You have evidence of an advantage when the added context improves the result. Those are separate achievements, and both need to be demonstrated.

Tools and setup

Existing implementation: Airbyte connections define sources, destinations, streams, timing and schema behavior. Its ClickHouse destination is a documented route for analytical replication, not an automatic fit for every business.

  1. Write the question and metric definition. Try an authorized read-only export first. Record source account, period, currency, customer key, data rights and expected freshness.
  2. If replication is justified, use an existing destination or create an approved ClickHouse service. Enter credentials only in the tool's secure configuration. Configure an Airbyte source and destination; test both. Select only required streams and a suitable sync mode; start with a manual sample sync.
  3. Reconcile source and destination totals and one known record. Airbyte's ClickHouse documentation warns that removing a source column drops that column's historical destination data; test schema-change handling on disposable data and preserve needed history before production changes. Give agents a separate read-only query identity.

Setup paths checked against linked documentation on 15 November 2026. Verify current account support during setup; reader integrations have not been tested here.

Work through this with your agent

Paste the request into your assistant. It will interview you and help produce the working artifact.

Interview me about ONE business question requiring operational data. Ask at most three questions per turn and wait. Identify source systems, entity IDs, owners, access rights, update needs and the decision the answer will change. Start with current exports or read-only queries; propose a warehouse only if the question needs it. Define each metric and reconcile one sample against its source. Draft DATA-MAP.md and a reproducible query or supported retrieval procedure. Ask which data we uniquely generate, whether we may retain/use it, and what outcome labels improve later work. Keep customer and tenant data isolated; cross-customer reuse needs explicit rights and an appropriate aggregation method. Propose a small comparison measuring whether extra context improves the outcome. Do not claim a moat or network effect from data volume alone. Finish with the verified result, freshness limits, permissions and next test.

Use these documented setup paths as references; check current account support before acting:
https://docs.airbyte.com/platform/using-airbyte/getting-started/set-up-a-connection
https://docs.airbyte.com/integrations/destinations/clickhouse

The Hypothesis Diff · weekly

Know when a hypothesis is stressed or changes.

One email a week: which hypotheses hold, which are stressed, which changed, and the tools worth trying. Nobody else publishes what stopped being true.