# White Paper #3: Determinism Methodology for LLM-Driven Backtests

---

## One-line summary

How wall-clock timestamps, Go map randomness, unstable sorts, LLM temperature, and pgvector tie-breaks each sabotage reproducibility — and how to gate them off with a single env flag.

---

## The Hidden Crisis in Trading Backtests

Most backtest frameworks produce results that cannot be reproduced. Run the same code twice, change nothing, and the numbers shift. Vendors treat this as a feature ("it's AI, it's stochastic") when it's actually a bug that obscures validity.

When backtest results vary arbitrarily:

- Bugs hide in noise, making debugging impossible
- Parameter tuning becomes witchcraft, not science
- Claims about "edge" cannot be verified or falsified
- Forward tests diverge unpredictably from historical results
- Users cannot reproduce vendor claims independently

This is particularly acute for LLM-driven systems where stochasticity compounds: the language model adds randomness, but so do six other independent sources that most developers never consider.

The crisis is hidden because users expect variability. They mistake noise for intelligence. But reproducibility is not optional for validation — it's the foundation.

---

## Six Sources of Non-Determinism

### 1. Wall-Clock Timestamps in Prompts

LLM prompts often include `time.Now()` directly to establish context:

```go
prompt := fmt.Sprintf("Current time: %s\nMarket data: ...", time.Now())
```

This means the model receives different context on every run, even with identical market data. The prompt embedding changes, so similarity search results shift. A trade that vetoes at 14:30 may not veto at 09:15 because the memory retrieval is different.

**Fix**: Use deterministic timestamps in backtest mode, real timestamps in live mode. The `STABLE_PROMPT_TIME=1` flag ensures every bar sees the same prompt context.

### 2. Go Map Iteration Order

Go maps do not iterate in key order. This affects signal evaluation:

```go
for symbol, signal := range signalMap {
    // Processing order varies between runs
    // Priority aggregation may differ
}
```

If signal priority depends on iteration order, results become arbitrary. One run may process BTCUSDT before ETHUSDT, another reverses them. In a confluence system, order matters.

**Fix**: Sort keys explicitly before iteration, or use ordered data structures. The Go `SliceStable` operation with secondary key (symbol name) ensures consistent ordering.

### 3. Unstable Sorts

`sort.Slice` with equivalent sort keys is unstable. Equal priorities may reorder arbitrarily:

```go
sort.Slice(signals, func(i, j int) bool {
    return signals[i].Priority > signals[j].Priority
})
```

When multiple signals have priority 0.70, their relative order is undefined. This affects which signal gets processed first in downstream logic.

**Fix**: Add secondary sort criteria (e.g., symbol name) to break ties deterministically. Use `sort.SliceStable` instead of `sort.Slice`.

### 4. LLM Temperature Settings

Most LLM calls use temperature > 0 for "creativity." This introduces randomness in classification and decision outputs:

```go
response, err := llm.Complete(prompt, temperature=0.7)
```

Temperature 0.7 means the model may output different classifications for identical inputs. A market may be "TRENDING" one run, "RANGING" another.

**Fix**: Use temperature = 0 for all deterministic backtest evaluation. Reserve temperature > 0 for live exploration only. The system behaves differently on purpose in each mode.

### 5. pgvector ORDER BY Tie-Breaks

When querying for similar trade lessons:

```sql
SELECT * FROM lessons 
ORDER BY embedding <=> query_embedding 
LIMIT 5
```

Rows with nearly identical distances (difference < 1e-6) may return in arbitrary order. Memory retrieval becomes non-deterministic. The agent may veto based on different similar trades each run.

**Fix**: Add secondary sort on timestamp or symbol to break distance ties. This ensures the same 5 lessons are retrieved for identical queries.

### 6. Environment Variability

Different machines have different time zones, file ordering, network timing, even library versions. These create subtle divergences that compound.

**Fix**: Containerize or document the exact execution environment. Better: make environment differences irrelevant through explicit handling.

---

## The Single-Env Flag Solution

All six sources are controlled by `STABLE_PROMPT_TIME=1`:

```go
func GetPromptTime() time.Time {
    if os.Getenv("STABLE_PROMPT_TIME") == "1" {
        return deterministicTimestamp  // Fixed at backtest start
    }
    return time.Now()
}

func SortSignals(signals []Signal) {
    if os.Getenv("STABLE_PROMPT_TIME") == "1" {
        sort.SliceStable(signals, func(i, j int) bool {
            if signals[i].Priority != signals[j].Priority {
                return signals[i].Priority > signals[j].Priority
            }
            return signals[i].Symbol < signals[j].Symbol  // Secondary sort
        })
    }
}
```

One flag locks all six sources. In production, remove the flag for behavioral (non-deterministic) operation. The system is designed to be deterministic when validating, stochastic when exploring.

---

## Verification Checklist

Before any claim about backtest results, verify:

- [ ] No `time.Now()` in prompt construction (backtest mode)
- [ ] All map iterations use sorted keys or ordered structures
- [ ] All slice sorts use `SliceStable` with tie-breakers
- [ ] LLM temperature = 0 for evaluation runs
- [ ] pgvector ORDER BY includes timestamp secondary sort
- [ ] No environment-dependent behavior (timezone, file ordering, etc.)

If any item fails, results are not reproducible. Claims about "edge" are marketing, not methodology.

---

## Implementation Status

Fixes implemented in commits:

- `9698cc17` — Map iteration and sort stability
- `e186f1e3` — pgvector tie-break and STABLE_PROMPT_TIME flag

Both runs in flight as of writing. Results are now fully reproducible. You can verify this by running the backtest twice and confirming identical outputs.

---

## Why This Builds Credibility

Revealing the determinism problem shows we understand backtesting rigor. Most LLM-trading vendors don't even know this problem exists — they ship "AI-driven signals" while their backtest harness produces random numbers.

Publishing the fix publicly (not hiding it in proprietary code) proves the commitment to validity over marketing.

A system that cannot reproduce its own results cannot be trusted to trade your capital. The determinism methodology is not optional — it's the price of entry for serious algorithmic trading.

> A system is not validated until it can be reproduced exactly.