Handled terminated error in Codex responses

Summary

Closed, Completed

Understand
Confirmed model and provider consistency.
Understand
Understand
Investigated source of 'terminated' error.
Understand
Edit
Added retry logic for 'terminated' errors.
Edit
Edit
Updated changelog with fix.
Edit

there's ~/Downloads/505349d6-b535-449f-9513-f277525708e1.jsonl. plese analyze how it came to the terminated failure

The user wants me to analyze this JSONL file to understand how it came to the "terminated failure". Let me look at the contents:

This is a pi session file with a code review task
The session has several messages:
- A session init message
- A user message asking for a code review
- An assistant message with a tool call to run git diff
- A tool result with a large diff
- A final assistant message with "stopReason":"error" and "errorMessage":"terminated"

Looking at the final message:

It has "usage":{"input":0,"output":0,...} - zero tokens reported
"stopReason":"error"
"errorMessage":"terminated"
The thinking content mentions reviewing error handling

This appears to be using "openai-codex" provider with model "gpt-5.2-codex". The session was reviewing a code diff about before_bash_exec event handling and tool_result error message overrides.

The failure happened after the tool result was returned. The model started thinking about the code review but then the session terminated with an error. The zero token count and "terminated" error suggests the API connection was lost or timed out, or the request was cancelled/terminated by the provider.

Let me summarize what happened:

User asked for a code review
Model called bash to get git diff
Tool returned ~400 lines of diff
Model started processing/thinking
Model got terminated with zero usage tokens

The "terminated" error typically indicates the stream was cut off prematurely - could be:

API timeout
Connection dropped
Provider-side termination
Rate limiting

Since usage is 0/0, the response never completed at the API level.

The session shows a code review task that failed during the model's response phase. Here's the breakdown:

Timeline:

19:00:19 - Session started
19:00:22 - User requested code review of a git diff
19:00:26 - Model (gpt-5.2-codex via openai-codex provider) called git diff
19:00:26 - Tool returned ~400 lines of diff (changes to extension system, before_bash_exec event, tool_result error handling)
19:00:42 - Model attempted to respond but was terminated

The failure:

json

{  "stopReason": "error",  "errorMessage": "terminated",  "usage": {"input": 0, "output": 0, ...}}

{  "stopReason": "error",  "errorMessage": "terminated",  "usage": {"input": 0, "output": 0, ...}}

Root cause analysis:

The zero token usage in the final message is the key indicator. The model started a thinking block ("Reviewing error handling in tool execution...") but the response was cut off before any tokens were actually counted.

This "terminated" error with zero usage typically means:

The API stream was severed before completion
Could be a provider-side timeout (the diff was substantial)
Could be connection dropped mid-response
Could be the Codex API rate-limited or terminated the request

The model was in the middle of reasoning about the code changes when it got cut off. It never produced any actual review findings because the stream died during the thinking phase.

Not a logic error in the code being reviewed. The failure is an infrastructure/API issue with the openai-codex provider.

what provider and model? was there a model or provider switch somewhere?

The user is asking about the provider and model used in this session, and whether there was any switch during the session.

Let me look at the messages again:

First assistant message (id: a1345c60):

Second assistant message (id: 8d795c38):

Both messages use the same provider and model. No switch occurred.

Looking at both assistant messages:

Message 1 (tool call):

api: openai-codex-responses
provider: openai-codex
model: gpt-5.2-codex

Message 2 (terminated):

api: openai-codex-responses
provider: openai-codex
model: gpt-5.2-codex

No switch. Same provider and model throughout. The session used gpt-5.2-codex via the openai-codex provider for both turns, and failed on the second turn during the thinking phase.

that's implemented in @packages/ai/src/providers/openai-codex-responses.ts check when and how it emits that error

Looking at the code, I need to find where the "terminated" error message comes from. Let me search for it:

In the main try-catch block (lines ~112-178):

javascript

} catch (error) {    output.stopReason = options?.signal?.aborted ? "aborted" : "error";    output.errorMessage = error instanceof Error ? error.message : String(error);    stream.push({ type: "error", reason: output.stopReason, error: output });    stream.end();}

} catch (error) {    output.stopReason = options?.signal?.aborted ? "aborted" : "error";    output.errorMessage = error instanceof Error ? error.message : String(error);    stream.push({ type: "error", reason: output.stopReason, error: output });    stream.end();}

This sets the errorMessage from the caught error's message. So "terminated" must come from somewhere that throws an error with that message.

Looking at processStream, there are several places that throw errors:
- case "error": throws Codex error: ${message || code || JSON.stringify(event)}
- case "response.failed": throws msg || "Codex response failed"
But the error message is just "terminated", not prefixed with anything. Let me look more carefully...

Actually, "terminated" doesn't appear in this file at all! The error must be coming from somewhere else. Let me check:

The SSE parsing loop - parseSSE - if the stream ends unexpectedly, it just returns (via if (done) break). It doesn't throw "terminated".
Looking at the processStream function - it iterates through parseSSE events. If the stream ends without a response.completed or response.done event, it just finishes normally without throwing.

Wait - the issue might be that the stream ended prematurely (before receiving response.completed/done), and something else detected this as a "terminated" state. But I don't see "terminated" in this file.

Let me search for "terminated" in the codebase to find where it originates.

The word "terminated" doesn't appear in this file. Let me search for where it's emitted:

>cd /Users/badlogic/workspaces/pi-mono && rg -n "terminated" --type ts

packages/coding-agent/test/frontmatter.test.ts:32:	it("returns original content when frontmatter is missing or unterminated", () => {

"terminated" isn't in the codebase as a literal string being set as an error message. Let me look at the session file again more closely - the error message was "terminated" which suggests it might be coming from outside the pi codebase, possibly from the API itself or the stream ending.

Looking back at the provider code:

The parseSSE generator just yields events and exits when done is true
processStream processes those events but doesn't throw if the stream ends early
The main catch block sets errorMessage from whatever error was thrown

The "terminated" error could be:

Coming from the Codex API as an error event
A network error message
Something in the fetch/stream handling

Looking at the error handling in processStream:

javascript

case "error": {    const code = (event as { code?: string }).code || "";    const message = (event as { message?: string }).message || "";    throw new Error(`Codex error: ${message || code || JSON.stringify(event)}`);}

case "error": {    const code = (event as { code?: string }).code || "";    const message = (event as { message?: string }).message || "";    throw new Error(`Codex error: ${message || code || JSON.stringify(event)}`);}

If the API sent an error event with message "terminated", it would be thrown as "Codex error: terminated". But the log shows just "terminated".

Wait - maybe the error event had message: "terminated" and code: "", and the template literal resulted in just the message. But no, it would still have "Codex error: " prefix.

Let me check if there's handling for incomplete streams or if the stream can emit "terminated" as a raw status. Maybe I should search for how the stream ending is detected or if there's special handling when usage is 0.

Actually, looking at the session data again:

usage: {input: 0, output: 0, ...} - zero tokens
The thinking content was truncated

This suggests the stream was interrupted before any response.completed or response.done event was received (which is where usage would be populated). The stream ended without completing normally.

I need to trace through the stream handling to see if there's a code path where "terminated" gets thrown directly, or if it's being caught and re-thrown somewhere without the "Codex error: " prefix. The zero usage and truncated thinking point to an early stream termination.

Not found in our code. Let me check if it could come from the API response itself by looking at error handling more closely:

"terminated" isn't in our codebase at all. So it must be coming from the Codex API response itself.

Looking at the session again:

The model started thinking ("Reviewing error handling in tool execution")
Usage is 0/0 (no tokens recorded)
stopReason: "error"
errorMessage: "terminated"

The zero usage indicates response.completed or response.done was never received. The error "terminated" must have come from an SSE error event from the Codex API.

Looking at the error handling in processStream:

javascript

case "error": {    const code = (event as { code?: string }).code || "";    const message = (event as { message?: string }).message || "";    throw new Error(`Codex error: ${message || code || JSON.stringify(event)}`);}

case "error": {    const code = (event as { code?: string }).code || "";    const message = (event as { message?: string }).message || "";    throw new Error(`Codex error: ${message || code || JSON.stringify(event)}`);}

If the API sent {type: "error", message: "terminated"}, this would throw "Codex error: terminated", not just "terminated".

But wait - the error message in the session was just "terminated". Let me look at other possible sources:

Maybe there's a response.failed event?

javascript

case "response.failed": {    const msg = (event as { response?: { error?: { message?: string } } }).response?.error?.message;    throw new Error(msg || "Codex response failed");}

case "response.failed": {    const msg = (event as { response?: { error?: { message?: string } } }).response?.error?.message;    throw new Error(msg || "Codex response failed");}

If the API sent {type: "response.failed", response: {error: {message: "terminated"}}}, this would throw just "terminated"! That's it.

So the Codex API sent a response.failed event with response.error.message = "terminated". This is an API-side failure, not a client-side issue.

The "terminated" message isn't in our codebase. Looking at the error handling in the provider:

typescript

case "response.failed": {    const msg = (event as { response?: { error?: { message?: string } } }).response?.error?.message;    throw new Error(msg || "Codex response failed");}

case "response.failed": {    const msg = (event as { response?: { error?: { message?: string } } }).response?.error?.message;    throw new Error(msg || "Codex response failed");}

This is the source. The Codex API sent a response.failed SSE event with:

The provider extracts response.error.message and throws it directly, resulting in errorMessage: "terminated".

Summary: The failure came from the Codex API itself. The server terminated the response mid-stream (after the thinking started but before any output tokens were generated). This is why:

Usage shows 0/0 tokens (no response.completed event received)
The thinking was cut off mid-sentence
The error is just "terminated" (server-side termination reason)

This is an OpenAI Codex backend issue, not a pi bug. The server decided to terminate the response for some internal reason (could be timeout, resource limits, content policy, or infrastructure issue on their end).

does the provider have retry?

The user is asking if the openai-codex-responses provider has retry logic. Let me look at the code I already read.

Looking at the code, there is retry logic for the initial fetch request:

typescript

const MAX_RETRIES = 3;const BASE_DELAY_MS = 1000;
function isRetryableError(status: number, errorText: string): boolean {	if (status === 429 || status === 500 || status === 502 || status === 503 || status === 504) {		return true;	}	return /rate.?limit|overloaded|service.?unavailable|upstream.?connect|connection.?refused/i.test(errorText);}

const MAX_RETRIES = 3;const BASE_DELAY_MS = 1000;
function isRetryableError(status: number, errorText: string): boolean {	if (status === 429 || status === 500 || status === 502 || status === 503 || status === 504) {		return true;	}	return /rate.?limit|overloaded|service.?unavailable|upstream.?connect|connection.?refused/i.test(errorText);}

And in the main function:

typescript

for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {    // ... fetch with retry on retryable errors}

for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {    // ... fetch with retry on retryable errors}

But this retry logic is only for the initial HTTP request/response. Once the stream starts (response.ok and we enter processStream), there's no retry if the stream fails mid-way with a response.failed event.

So the answer is: yes, there's retry for the initial connection, but NO retry for mid-stream failures like the "terminated" error that happens after streaming has started.

Yes, but only for the initial connection:

typescript

const MAX_RETRIES = 3;const BASE_DELAY_MS = 1000;
function isRetryableError(status: number, errorText: string): boolean {    if (status === 429 || status === 500 || status === 502 || status === 503 || status === 504) {        return true;    }    return /rate.?limit|overloaded|service.?unavailable|upstream.?connect|connection.?refused/i.test(errorText);}

const MAX_RETRIES = 3;const BASE_DELAY_MS = 1000;
function isRetryableError(status: number, errorText: string): boolean {    if (status === 429 || status === 500 || status === 502 || status === 503 || status === 504) {        return true;    }    return /rate.?limit|overloaded|service.?unavailable|upstream.?connect|connection.?refused/i.test(errorText);}

The retry loop covers:

HTTP status errors (429, 5xx)
Network errors before streaming starts

No retry for mid-stream failures. Once response.ok is true and processStream() is called, any response.failed event (like the "terminated" one) just throws and exits. The stream was already in progress when the server terminated it.

To handle this case, you'd need retry logic that:

Detects mid-stream termination
Rebuilds the request with the conversation so far
Retries the entire turn

That's more complex since you'd need to track what content was already emitted and potentially deduplicate or handle partial tool calls.

i think @packages/coding-agent/src/core/agent-session.ts implements retry

>cd /Users/badlogic/workspaces/pi-mono && rg -n "retry|Retry|RETRY" packages/coding-agent/src/core/agent-session.ts

81:			willRetry: boolean;
84:	| { type: "auto_retry_start"; attempt: number; maxAttempts: number; delayMs: number; errorMessage: string }
85:	| { type: "auto_retry_end"; success: boolean; attempt: number; finalError?: string };
200:	// Retry state
201:	private _retryAbortController: AbortController | undefined = undefined;
202:	private _retryAttempt = 0;
203:	private _retryPromise: Promise<void> | undefined = undefined;
204:	private _retryResolve: (() => void) | undefined = undefined;
250:		// (session persistence, extensions, auto-compaction, retry logic)
332:		// Check auto-retry and auto-compaction after agent completes
337:			// Check for retryable errors first (overloaded, rate limit, server errors)
338:			if (this._isRetryableError(msg)) {
339:				const didRetry = await this._handleRetryableError(msg);
340:				if (didRetry) return; // Retry was initiated, don't proceed to compaction
341:			} else if (this._retryAttempt > 0) {
342:				// Previous retry succeeded - emit success event and reset counter
344:					type: "auto_retry_end",
346:					attempt: this._retryAttempt,
348:				this._retryAttempt = 0;
349:				// Resolve the retry promise so waitForRetry() completes
350:				this._resolveRetry();
357:	/** Resolve the pending retry promise */
358:	private _resolveRetry(): void {
359:		if (this._retryResolve) {
360:			this._retryResolve();
361:			this._retryResolve = undefined;
362:			this._retryPromise = undefined;
486:	/** Current retry attempt (0 if not retrying) */
487:	get retryAttempt(): number {
488:		return this._retryAttempt;
747:		await this.waitForRetry();
1029:		this.abortRetry();
1434:	 * 1. Overflow: LLM returned context overflow error, remove error message from agent state, compact, auto-retry
1435:	 * 2. Threshold: Context over threshold, compact, NO auto-retry (user continues manually)
1468:			// but we don't want it in context for the retry)
1490:	private async _runAutoCompaction(reason: "overflow" | "threshold", willRetry: boolean): Promise<void> {
1498:				this._emit({ type: "auto_compaction_end", result: undefined, aborted: false, willRetry: false });
1504:				this._emit({ type: "auto_compaction_end", result: undefined, aborted: false, willRetry: false });
1512:				this._emit({ type: "auto_compaction_end", result: undefined, aborted: false, willRetry: false });
1529:					this._emit({ type: "auto_compaction_end", result: undefined, aborted: true, willRetry: false });
1566:				this._emit({ type: "auto_compaction_end", result: undefined, aborted: true, willRetry: false });
1594:			this._emit({ type: "auto_compaction_end", result, aborted: false, willRetry });
1596:			if (willRetry) {
1613:				willRetry: false,
1850:	// Auto-Retry
1854:	 * Check if an error is retryable (overloaded, rate limit, server errors).
1855:	 * Context overflow errors are NOT retryable (handled by compaction instead).
1857:	private _isRetryableError(message: AssistantMessage): boolean {
1860:		// Context overflow is handled by compaction, not retry
1872:	 * Handle retryable errors with exponential backoff.
1873:	 * @returns true if retry was initiated, false if max retries exceeded or disabled
1875:	private async _handleRetryableError(message: AssistantMessage): Promise<boolean> {
1876:		const settings = this.settingsManager.getRetrySettings();
1879:		this._retryAttempt++;
1881:		// Create retry promise on first attempt so waitForRetry() can await it
1882:		if (this._retryAttempt === 1 && !this._retryPromise) {
1883:			this._retryPromise = new Promise((resolve) => {
1884:				this._retryResolve = resolve;
1888:		if (this._retryAttempt > settings.maxRetries) {
1891:				type: "auto_retry_end",
1893:				attempt: this._retryAttempt - 1,
1896:			this._retryAttempt = 0;
1897:			this._resolveRetry(); // Resolve so waitForRetry() completes
1901:		const delayMs = settings.baseDelayMs * 2 ** (this._retryAttempt - 1);
1904:			type: "auto_retry_start",
1905:			attempt: this._retryAttempt,
1918:		this._retryAbortController = new AbortController();
1920:			await this._sleep(delayMs, this._retryAbortController.signal);
1923:			const attempt = this._retryAttempt;
1924:			this._retryAttempt = 0;
1925:			this._retryAbortController = undefined;
1927:				type: "auto_retry_end",
1930:				finalError: "Retry cancelled",
1932:			this._resolveRetry();
1935:		this._retryAbortController = undefined;
1937:		// Retry via continue() - use setTimeout to break out of event handler chain
1940:				// Retry failed - will be caught by next agent_end
1967:	 * Cancel in-progress retry.
1969:	abortRetry(): void {
1970:		this._retryAbortController?.abort();
1971:		// Note: _retryAttempt is reset in the catch block of _autoRetry
1972:		this._resolveRetry();
1976:	 * Wait for any in-progress retry to complete.
1977:	 * Returns immediately if no retry is in progress.
1979:	private async waitForRetry(): Promise<void> {
1980:		if (this._retryPromise) {
1981:			await this._retryPromise;
1985:	/** Whether auto-retry is currently in progress */
1986:	get isRetrying(): boolean {
1987:		return this._retryPromise !== undefined;
1990:	/** Whether auto-retry is enabled */
1991:	get autoRetryEnabled(): boolean {
1992:		return this.settingsManager.getRetryEnabled();
1996:	 * Toggle auto-retry setting.
1998:	setAutoRetryEnabled(enabled: boolean): void {
1999:		this.settingsManager.setRetryEnabled(enabled);

So the _isRetryableError function checks if the error message matches certain patterns:

javascript

return /overloaded|rate.?limit|too many requests|429|500|502|503|504|service.?unavailable|server error|internal error|connection.?error|connection.?refused|other side closed|fetch failed|upstream.?connect|reset before headers/i.test(err);

return /overloaded|rate.?limit|too many requests|429|500|502|503|504|service.?unavailable|server error|internal error|connection.?error|connection.?refused|other side closed|fetch failed|upstream.?connect|reset before headers/i.test(err);

The error message "terminated" does NOT match any of these patterns. So the retry logic at the agent-session level would NOT retry this error.

To fix this, "terminated" could be added to the retryable error patterns, assuming it's a transient server-side issue that might succeed on retry.

Yes, agent-session.ts has retry logic, but "terminated" is not in the retryable patterns:

typescript

private _isRetryableError(message: AssistantMessage): boolean {    if (message.stopReason !== "error" || !message.errorMessage) return false;
    // Context overflow is handled by compaction, not retry    const contextWindow = this.model?.contextWindow ?? 0;    if (isContextOverflow(message, contextWindow)) return false;
    const err = message.errorMessage;    // Match: overloaded_error, rate limit, 429, 500, 502, 503, 504, service unavailable, connection errors, fetch failed    return /overloaded|rate.?limit|too many requests|429|500|502|503|504|service.?unavailable|server error|internal error|connection.?error|connection.?refused|other side closed|fetch failed|upstream.?connect|reset before headers/i.test(        err,    );}

private _isRetryableError(message: AssistantMessage): boolean {    if (message.stopReason !== "error" || !message.errorMessage) return false;
    // Context overflow is handled by compaction, not retry    const contextWindow = this.model?.contextWindow ?? 0;    if (isContextOverflow(message, contextWindow)) return false;
    const err = message.errorMessage;    // Match: overloaded_error, rate limit, 429, 500, 502, 503, 504, service unavailable, connection errors, fetch failed    return /overloaded|rate.?limit|too many requests|429|500|502|503|504|service.?unavailable|server error|internal error|connection.?error|connection.?refused|other side closed|fetch failed|upstream.?connect|reset before headers/i.test(        err,    );}

The error "terminated" doesn't match any of these patterns, so it falls through without retry.

Fix: Add terminated to the regex pattern:

typescript

return /overloaded|rate.?limit|too many requests|429|500|502|503|504|service.?unavailable|server error|internal error|connection.?error|connection.?refused|other side closed|fetch failed|upstream.?connect|reset before headers|terminated/i.test(    err,);

return /overloaded|rate.?limit|too many requests|429|500|502|503|504|service.?unavailable|server error|internal error|connection.?error|connection.?refused|other side closed|fetch failed|upstream.?connect|reset before headers|terminated/i.test(    err,);

This would allow the agent session to auto-retry when Codex terminates a response mid-stream.

add it, commit and push

Editedagent-session.ts

EditedCHANGELOG.md

>cd /Users/badlogic/workspaces/pi-mono && npm run check 2>&1

> pi-monorepo@0.0.3 check
> biome check --write . && tsgo --noEmit && npm run check -w @mariozechner/pi-web-ui

Checked 416 files in 256ms. No fixes applied.

> @mariozechner/pi-web-ui@0.49.3 check
> biome check --write . && tsc --noEmit && cd example && biome check --write . && tsc --noEmit

Checked 73 files in 37ms. No fixes applied.
Checked 3 files in 8ms. No fixes applied.

>cd /Users/badlogic/workspaces/pi-mono && git add packages/coding-agent/src/core/agent-session.ts packages/coding-agent/CHANGELOG.md && git commit -m "fix(coding-agent): add 'terminated' to retryable error patterns

Codex API can send 'terminated' error mid-stream, which should be
retried like other transient server errors." && git push

Running formatting, linting, and type checking...

> pi-monorepo@0.0.3 check
> biome check --write . && tsgo --noEmit && npm run check -w @mariozechner/pi-web-ui

Checked 416 files in 251ms. No fixes applied.

> @mariozechner/pi-web-ui@0.49.3 check
> biome check --write . && tsc --noEmit && cd example && biome check --write . && tsc --noEmit

Checked 73 files in 37ms. No fixes applied.
Checked 3 files in 7ms. No fixes applied.
✅ All pre-commit checks passed!
[main 9b84857b] fix(coding-agent): add 'terminated' to retryable error patterns
 2 files changed, 6 insertions(+), 2 deletions(-)
To https://github.com/badlogic/pi-mono.git
   5e079e2e..9b84857b  main -> main

Done. Pushed 9b84857b:

Added terminated to the retryable error regex in agent-session.ts
Updated CHANGELOG with the fix

Now when Codex returns "terminated" mid-stream, the agent will auto-retry with exponential backoff instead of failing immediately.