Truth Consolidation

Over time, Truths can accumulate semantic overlap. Multiple Truths saying essentially the same thing - like "has a son named Ellis," "father of Ellis," and "child named Ellis" - dilute your context window without adding value. Truth consolidation merges these redundant Truths into denser, more useful representations.

Why Consolidation Matters

• Reduces redundancy - Similar Truths merged into one
• Preserves context budget - Fewer Truths means more room for memories
• Improves coherence - Related facts combined into comprehensive statements

Unlike Memory Consolidation, which runs daily on new memories, Truth consolidation is more conservative. Truths are already distilled facts - they've earned their place through behavioral evidence. We consolidate only when there's clear semantic overlap.

Protection Model

Not all Truths are consolidatable. The system protects Truths based on their source and status:

Truth Type	Consolidatable?	Reason
Promoted (unpinned, under max generation)	Yes	System-created, can be refined
User Created	No	Explicit user intent must be preserved
Agent Created	No	Created during conversation for specific reasons
Pinned	No	User marked as important
At Max Generation	No	Already been refined multiple times

This protection model matches the refinement system - if a Truth can't be automatically refined, it can't be automatically consolidated.

How It Works

Consolidation uses a two-phase job architecture for efficient parallel processing:

Phase 1: Cluster Building

1. Find Truths with high similarity (≥0.75)
2. Group them into clusters using greedy clustering
3. Dispatch one job per cluster

Phase 2: Cluster Processing

1. Each cluster job asks the LLM if Truths should merge
2. If approved, create consolidated Truth and soft-delete originals
3. If rejected, Truths remain separate

The LLM may keep Truths separate if they contain genuinely distinct information, reference different people, or if merging would lose important nuance.

Example

Input Truths:

• "Has a son named Ellis" (Gen 0, Score: 45)
• "Father of a child named Ellis" (Gen 1, Score: 32)
• "Ellis is the user's son" (Gen 0, Score: 28)

Result:

• "Has a son named Ellis" (Gen 2, Score: 45)

The consolidated Truth:

• Takes the clearest, most informative wording
• Inherits the highest distillation score from sources
• Tracks its lineage for the history view

Generation Tracking

Consolidation tracks how many times a Truth has been merged through generation tracking. This enables organic evolution while preventing runaway over-consolidation.

How Generations Work

Gen 0 ─┬─► Gen 1 ─┬─► Gen 2 ─┬─► Gen 3 ─┬─► Gen 4 ─┬─► Gen 5 (max)
       │          │          │          │          │
  Original    First      Re-consolidation continues...
   Truths   merge

Generation	Description
Gen 0	Original Truths from promotion or manual creation
Gen 1	First consolidation (merging original Truths)
Gen 2-4	Re-consolidation of already-consolidated Truths
Gen 5	Maximum - these Truths won't be re-consolidated

When Truths are consolidated, the new Truth's generation is calculated as max(source generations) + 1.

Re-Consolidation

Unlike systems that only consolidate original items, Iris can re-consolidate already-merged Truths. This allows Truths to evolve naturally over time as more related information is gathered.

IMPORTANT

The LLM applies extra scrutiny when re-consolidating. High-generation Truths already pack dense information, so only truly redundant Truths are merged.

Generation Limits

Generation 5 is the hard limit. Truths reaching this generation are excluded from future consolidation to prevent:

• Over-abstraction losing important details
• Runaway consolidation chains
• Truth content becoming too generic

Running Consolidation

By default, consolidation dispatches jobs to the queue for parallel processing:

# Queue batch for all users (default behavior)
php artisan iris:consolidate-truths

# Run synchronously (useful for debugging)
php artisan iris:consolidate-truths --sync

# Preview without changes (forces sync)
php artisan iris:consolidate-truths --dry-run

# Single user
php artisan iris:consolidate-truths --user=1

# Custom similarity threshold
php artisan iris:consolidate-truths --threshold=0.70

Dry Run Mode

The --dry-run flag shows what would be consolidated without making changes:

php artisan iris:consolidate-truths --user=1 --dry-run

Output shows each cluster with the decision (MERGE or KEEP SEPARATE), the LLM's reasoning, and proposed content if merging.

Processing Batches

When running in queue mode, the command returns a batch ID:

php artisan iris:consolidate-truths
# Dispatched batch: 9c3b5f2a-...

# Process jobs (Horizon should already be running)
php artisan horizon

# Retry failed jobs in a batch
php artisan queue:retry-batch 9c3b5f2a-...

Use Laravel Horizon for monitoring batch progress in a UI.

Scheduled Runs

Iris automatically schedules consolidation in bootstrap/app.php:

// Weekly truth consolidation (after memory consolidation)
$schedule->command('iris:consolidate-truths')
    ->weeklyOn(Schedule::SUNDAY, '05:00')
    ->withoutOverlapping()
    ->runInBackground();

Truths accumulate slower than memories since they only come through distillation. Weekly consolidation is sufficient to keep redundancy in check.

Job Architecture

Consolidation uses two job types for efficient processing:

ConsolidateUserTruths

Handles Phase 1 for a single user:

• Filters to consolidatable Truths (promoted, unpinned, under max gen)
• Builds clusters based on semantic similarity
• Dispatches cluster jobs for Phase 2
• 60 second timeout

ConsolidateTruthCluster

Handles Phase 2 for a single cluster:

• Reconstructs cluster from Truth IDs
• Calls LLM to review and decide on merging
• Executes consolidation if approved
• 120 second timeout
• Rate-limited to prevent API overload

This architecture prevents timeout issues with large Truth sets by processing clusters independently.

Configuration

Setting	Default	Description
truth_consolidation.similarity_threshold	0.75	Minimum similarity to cluster
truth_consolidation.max_cluster_size	5	Max Truths per cluster
truth_consolidation.min_cluster_size	2	Min Truths to form a cluster
truth_consolidation.timeout	120	Job timeout in seconds
truth_consolidation.model	claude-sonnet-4-5	Model for merge decisions
truth_consolidation.jobs_per_minute	10	Rate limit for queued jobs

The 0.75 threshold is lower than memory consolidation (0.80) because Truths are already distilled - semantic overlap is more likely to be true redundancy rather than related-but-distinct information.

TIP

Use --dry-run with different --threshold values to find the right balance for your data before running actual consolidation.

Rate Limiting

Consolidation jobs are rate-limited to prevent overwhelming LLM APIs:

1. Preventive: Jobs throttled to jobs_per_minute limit
2. Reactive: If rate limited by the API, jobs automatically retry after the limit resets using Prism's resetsAt timing
3. Time-based retries: Jobs use retryUntil with a 2-hour window, allowing unlimited rate-limit releases without failing

Adjust truth_consolidation.jobs_per_minute in config/iris-custom.php based on your API tier.

NOTE

Only the cluster jobs are rate-limited since they make LLM calls. The parent user jobs run without throttling.