AI Schema Generation - Entity Enricher Documentation

AI Schema Generation

Generate structured JSON schemas from sample data using AI, with automatic self-correction and intelligent post-processing.

How It Works

Schema generation turns raw entity data into a typed, annotated JSON schema that defines exactly what information to extract during enrichment. Instead of manually writing schemas, you paste sample JSON and let AI analyze the structure, infer types, assign expertise domains, and suggest improvements.

The Generation Pipeline

Generation is not one big prompt. It is a sequence of small, single-concern calls, most of them running concurrently — which is what lets small and cheap models produce a usable schema, since each call answers one narrow question about material it can hold in view.

  1. Canonicalize the sample (no LLM) — a value carrying its own unit becomes a number with the unit in the name ("8.275 h" becomes half_life_seconds: 29790), and a date no native type can hold becomes an integer year. Every observed value has to prove the claim, or the property stays text. The rewritten sample is what gets saved.
  2. Identity scoping — one call, run before all others because it is the last one allowed to change your sample. Where an item in a related array mixes facts about that entity with facts about its pairing to the parent, the item is restructured: pairing facts stay put, the entity's own facts nest under a named subobject. Without this the two kinds of fact share one identity.
  3. Derive the skeleton (no LLM) — the property tree, JSON types and nullability come straight from the sample(s); repeated shapes and entity-like array items become reusable definitions. Localized objects (like {"en": "...", "fr": "..."}) collapse to a single multilingual value.
  4. Ask the parallel questions — separate concurrent calls settle the entity's identity and naming, the behavioural flags (key, preserve, multilingual, nullable, plus format proposals), whether whole-number fields are genuinely discrete, which strings come from a closed vocabulary, and how properties route to expertise domains.
  5. Write the documentation — one call per expertise domain, in that domain's persona, producing each property's description and examples — and its own second opinion on whether the value can genuinely be absent.
  6. Assemble, validate, save (no LLM) — the fragments are merged, the 8 validation rules run as a safety net, deterministic post-processing settles flag conflicts, and the schema is saved — deduplicated by content hash, so identical schemas are not duplicated.

Each step retries on its own (3 attempts) and its answers accumulate across attempts, so a model that replies in fragments still converges. After that the step accepts what it got and the gaps are filled deterministically — a weak model degrades the descriptions rather than failing the generation. Only identity and domain routing are allowed to fail the whole run. Every call is billed and logged as its own prompt, so the record shows exactly what was spent where.

You can pass several samples of the same entity type instead of one — the schema then covers the union of their fields, anything missing from a sample becomes nullable, and the values seen across them become real examples. Field names must match: samples describing different entity types are refused, and so are objects inside an array that share no field at all, since nothing would be left to identify their rows. The sample editor flags any such difference before you spend a generation.

Values that carry their own unit are converted to numbers before the schema is derived, because a column of "8.275 h" and "85 ms" cannot be sorted, filtered by range or aggregated. The unit moves into the property name (half_life_seconds), a non-numeric stand-in like "stable" becomes null, and dates reaching before year 1 become an integer year (negative for BCE), which no date type can store and text sorts wrongly. Your sample panel is updated to match, so it always shows the sample the schema describes. Anything the conversion cannot read with certainty is left exactly as you wrote it.

Self-Correction, Step by Step

Because each step answers one narrow question, correction can be narrow too: a step's validator keeps whatever came back usable and asks again only for what is missing. Nothing is regenerated from scratch, so a partially-correct answer is progress rather than a wasted attempt.

Example: the flags step on 30 properties

Attempt 1The model answers for 22 of them, and splits its reply across several tool calls — a common failure mode on small models. All 22 are kept.
RetryThe follow-up asks for the 8 remaining properties only — a shorter question, more likely to be answered whole.
Attempt 26 more arrive. The last 2 fall back to deterministic defaults, and the shortfall is recorded on the generation record instead of failing it.

The eight validation rules still run over the assembled schema as a final check — type correctness, expertise assignment, reference integrity, completeness. By then they are a safety net rather than the correction mechanism. Learn more about each rule in the Validation Rules guide.

What the Schema Contains

A generated schema is more than a simple type definition. Each property includes metadata that guides the enrichment process:

Type

JSON Schema type (string, number, integer, boolean, array, object)

Description

Contextual description that tells the AI what information to find

Expertise

Which expert domain (financial, regulatory, etc.) provides this value

Identifying

Whether this field is part of what identifies the instance. Identifying properties do both jobs at once: they focus the enrichment prompt on the right entity, and they are what fusion matches array items on. One may still be nullable — a qualifier that separates look-alike siblings stays identifying even when whole families genuinely lack it

Closed vocabulary

Where a string's values come from a small, completely enumerable set (statuses, grades, classification codes), generation proposes the members — spelled the way your samples spell them — so enrichment cannot drift to a synonym

Nullable

Whether the field can be null — non-nullable fields are required for database admission

Multilingual

Whether the field should be enriched across multiple languages

Preserve

Whether to keep the original value unchanged during enrichment

Examples

Realistic example values that guide the AI toward the right format

Format / Pattern

Machine-checkable shape for string values: malformed answers are rejected and retried, and stored values keep the canonical form. Generation only ever claims a named format (date, time, date-time, uuid, email, uri, ipv4, ipv6) its samples prove — a regex pattern is a prediction about values nobody has seen yet, and a wrong one fails every enrichment of the field, so you add that one yourself in the editor

Expertise Domain Detection

The AI groups schema properties into expertise domains based on their semantic meaning. For example, a pharmaceutical company schema might have domains like “Financial Analyst,” “Regulatory Expert,” and “Corporate Information.” These domains are used by the multi-expertise strategy to run parallel, specialized LLM calls for deeper results.

Domain Count Limits

The number of expertise domains is automatically limited based on your data's property count to prevent over-fragmentation:

5 properties
1 domain
12 properties
2 domains
30 properties
5 domains
60 properties
10 domains

Post-Processing

Once the fragments are assembled, deterministic steps settle everything that should not be left to a model — using your actual input data as the evidence:

Nullable widening

A field missing or null in any sample becomes nullable whatever the model answered, so an unknown value is an accepted answer rather than a data-quality failure. Samples can only widen: a handful of samples proves presence for those instances, never for every instance of the type — which is why the model gets a vote too, and the two are ORed.

Flag conflict resolution

Attributes that cannot coexist are reconciled by rule rather than by asking again: preserve wins over multilingual and nullable, a surviving closed vocabulary clears multilingual, and a key property never keeps an enum.

Array-item key repair

Every object inside an array is guaranteed at least one key property — it is the unit fusion deduplicates on, so an array item with no key would make merging two models' answers impossible.

Expertise collection

All unique expertise domains are gathered from the schema for metrics and strategy configuration.

The language a schema is written in

A schema describes itself in a language — its type names, property descriptions, expertise labels and suggestions. That is a different thing from the languages you enrich into. Name one when you generate, or leave it out and the language of your sample's own property names decides: a French sample stops producing an English schema. The choice is stored, so later AI edits keep writing in the same language instead of drifting back to English.

Generating the samples themselves

You do not need sample data to start. Describe the entity type — optionally with documents to ground it in, or web search to check it against reality — and the platform writes the samples for you. Ask for several and you get several different instances, not one instance restated.

All the instances are chosen at once

The first sample also decides who the others will be about, in the same call. Asking N times independently for “an example” reliably returns the same famous instance N times; naming the whole cast up front is what makes them distinct.

The first sample fixes the shape

The remaining samples are generated in parallel against sample 1's structure as a contract, not merely asked to match it — so a variant cannot rename, add or drop a field. Any that still come back duplicated or malformed are re-requested in a bounded retry wave, and if the full count is not reached you are told rather than silently given fewer.

Language, and your own instructions

Language defaults to auto, inferred from the words of your own request, then from any attached document. Extra instructions you add are binding: they are honoured, or the response tells you what could not be honoured and why — never quietly dropped.

Ambiguity Check

Read a property name in the context of its parent object and count the distinct things it could be asking for. One is clear. Two or more and each model settles on a different one, so the column ends up mixing answers to different questions — annual_revenue on a company can be the group or the entity, gross or net, in one of several currencies. None at all — a name for something this parent simply does not have — is worse: with nothing to look up, the model invents a value.

Generation fights this twice: the prompt itself demands names that admit a single reading, and a post-pass over the finished schema annotates the ones that still do not. At this stage the remedy is a rename — the descriptions were generated from the names, so a description cannot disambiguate the name it came from, and nothing depends on the schema yet. Sample generation runs the same check on the sample and applies the renames before you ever see it. Free-text prose — a description, a summary, notes — is never flagged: the wording varies, but the question asked is clear.

Flagged properties show an “ambiguous” badge in the Workflow Editor, listing the readings the name admits. Once the schema is live the remedy flips to a rewritten description, which pins one meaning without breaking the data contract. See the Ambiguity Check guide for the full rubric and its remedies.

When generating a sample entity from a description, you can enable “Use web search” to let the model look up current facts on the web instead of relying on its training data alone. This produces fresher, more accurate sample values — especially for fast-moving facts like prices, staff counts, or recent releases. The option only appears for models whose provider supports built-in web search, and search calls are billed by the provider like any other model usage.

AI Schema Editing

After generation, you can modify schemas using natural language instructions. Type a command and the AI applies the change while preserving your existing schema structure. Each edit also produces 5 suggestions for further improvements.

Example Edit Commands

Add an employee_count integer field
Create a nested address object with city and country
Add French descriptions to all text fields
Define a parent company reference using $defs
Mark the website field as nullable

AI edits are validated using a subset of the generation rules (type checking, reference integrity, expertise consistency) without comparing against input data, since you may intentionally add or remove fields.

AI Suggestions

Both schema generation and AI editing produce 5 targeted suggestions covering different improvement categories:

Data completenessMissing fields that could enrich your entity
Data qualityClosed vocabularies, nullability, type corrections
RelationshipsNested structures, entity references via $defs
InternationalizationMultilingual translations, locale support
Business contextDomain-specific fields and expertise groupings

Suggestions appear as clickable chips in the Workflow Editor — click one to auto-fill the AI edit input and apply it.

Next Steps