Crawler-Readable API Catalogs That Survive Ingestion

Crawler-readable API catalogs are the copies automated buyers actually evaluate. Your site can describe an endpoint perfectly for a person, but an agent may only see the structured document a crawler fetched yesterday and stored for later retrieval.

That changes the publishing job. The catalog cannot be a loose index of documentation links. It has to behave like a versioned operational artifact: available without a browser, complete enough to ingest once, and explicit about when every record was last changed.

Publish one authoritative retrieval surface

A crawler needs a predictable place to start. It should not have to render navigation, execute client-side JavaScript or follow a sales funnel to find the catalog. Serve the document from a stable public URL with a successful response, a consistent media type and no authentication requirement for the metadata itself.

Different documents can support different stages. An OpenAPI description explains HTTP operations. An x402 discovery document joins a payable resource to its current call and payment requirements. The emerging `llms.txt` convention can orient an agent toward important machine-readable sources. It should point to the authoritative catalog rather than becoming another copy of endpoint data that can drift.

This is the broader publishing surface behind how AI agents discover APIs. The agent often reads a cached artifact, so the first public retrieval has to produce the same operational meaning as the live provider route.

Choose one record as the source of truth for each fact. Generate other surfaces from it where possible. If the input schema comes from OpenAPI, do not retype it into three documents. If payment terms come from the gateway, link or embed the current gateway output instead of maintaining a separate price list.

Design records for independent ingestion

Crawlers split, rank and cache records. A provider introduction at the top of a file may be separated from the resource it qualifies. Every resource therefore needs enough local context to remain understandable on its own.

A durable record includes:

• A stable resource identifier and absolute request URL.
• A capability name written as the job performed, not the vendor category.
• The HTTP method and required input fields with locations and types.
• An output schema or representative response shape.
• The live route that returns current payment requirements.
• Provider identity and a canonical documentation URL.
• A record version or update time that changes with material edits.
• A status that distinguishes active, deprecated and removed resources.

The capability wording should match buyer tasks. “Company enrichment” is more retrievable than an internal plan name. “Return one normalized company profile from a domain” is better still because it states the unit of work and the required starting point.

Input names need the same precision. Two providers may perform the same job while accepting `domain`, `company_url` or a nested object. That difference does not make them incomparable, but it does require an explicit API input compatibility mapping before a buyer can substitute one call for another.

Treat freshness as part of the contract

A cached catalog is useful only when a crawler can tell whether it has changed. Publish a document update time, individual record update times when resources change independently, and stable validators such as an entity tag or last-modified value where your delivery stack supports them.

Do not change an identifier merely because copy improved. The identifier is how an ingestion system replaces an old record instead of creating a duplicate. Change it when the commercial resource itself changes identity, not when a description gains a clearer sentence.

Handle removal deliberately. If a route is being retired, mark it deprecated before it disappears and provide the replacement identifier where one exists. A hard deletion can leave cached copies looking active until the next full crawl. A tombstone or explicit inactive state gives the ingestion system something concrete to apply.

Payment terms need a shorter path to freshness than capability prose. Never state a fixed list of chains, tokens or prices in a catalog that is maintained by hand. Point to the payment terms the gateway currently returns. For Apiosk, the live gateway discovery document is authoritative, while the copy-paste integration page explains how to connect an endpoint.

Make catalog releases atomic

A crawler can arrive during a deployment. If you update the method first, the schema second and the payment reference last, it may cache a record that never represented a valid state of the endpoint.

Build the complete catalog before making it public, validate it, then replace the prior snapshot in one release. The document version and its records should describe the same deployed route set. When a catalog is large enough to paginate, keep the snapshot boundary consistent across pages so the crawler does not combine two releases.

Run release checks against the public URL, not only a generated file in the repository:

• Fetch without cookies, credentials or a browser user agent.
• Confirm redirects end at one canonical URL.
• Validate the response syntax and required fields.
• Match every active record to a deployed method and route.
• Validate examples against the same request rules as production.
• Confirm removed routes are deprecated or absent by design.
• Compare payment references with the gateway response.

These checks protect the provider from a quiet failure mode: the website deploy succeeds while the crawler receives an empty shell, an HTML error page or a stale edge-cached document.

Test from an empty cache

The strongest catalog test starts with no brand knowledge and forbids visits to the marketing site. Fetch the public document once. Select one resource. Ask whether a program can identify the job, construct a valid request, predict the response shape and find the live payment requirements.

Then repeat the test with an old copy. Change one input, retire one route and publish a new snapshot. The consumer should be able to match records by stable identifier, detect the newer version and remove or deactivate the old resource without guessing.

Do not confuse retrieval with eligibility. Apiosk can sweep external x402 sources through `/v1/discover`, but those results stay in `external_candidates`. They are unreviewed, unmeasured, carry no `candidate_id` and cannot enter `/v1/compare` or `/v1/decide`. A crawler-readable record can be found without yet being safe to compare or settle.

Where Apiosk fits

Apiosk turns reviewed catalog records into a purchasing path. `GET /v1/discover?q=…` returns candidates for a job with stable IDs. `GET /v1/compare` places price, settlement rail, input compatibility and sufficiently measured latency and success rate side by side. Unmeasured dimensions are dropped and named, never scored as zero. Result quality and provider terms are not measured.

`GET /v1/decide` then returns one provider, the rule that selected it, the exact constraint that removed every rejection and the runners-up in order. Providers can list an API on Apiosk to move from an externally found record toward a reviewed catalogue entry.

The crawler-readable catalog remains your responsibility at the provider boundary. Give it a stable public location, publish complete records, release changes atomically and make freshness visible. That is how the copy an agent reads later still describes the endpoint you operate now.