Analyzed and fixed GitHub issue #967

Summary

Closed, Completed

Understand
Reviewed GitHub issue and related code.
Understand
Understand
Checked SDK cache control options.
Understand
Edit
Committed and pushed fix with auto-close message.
Edit

Analyze GitHub issue(s): https://github.com/badlogic/pi-mono/issues/967

For each issue:

Read the issue in full, including all comments and linked issues/PRs.
For bugs:
- Ignore any root cause analysis in the issue (likely wrong)
- Read all related code files in full (no truncation)
- Trace the code path and identify the actual root cause
- Propose a fix
For feature requests:
- Read all related code files in full (no truncation)
- Propose the most concise implementation approach
- List affected files and changes needed

Do NOT implement unless explicitly asked. Analyze and propose only.

Let me analyze this issue. It's a feature request to add configurable cache retention for Anthropic and OpenAI APIs.

The discussion concluded that instead of exposing it as a user-facing setting, it should be an environment variable that the providers check (to avoid confusion for users who don't understand the cost implications).

Let me read the relevant code files to understand how the streaming works for Anthropic and OpenAI providers.

Let me read the relevant code files to understand the current implementation.

Readanthropic.ts

Readopenai-responses.ts

Readtypes.ts

Now I have a good understanding of the current implementation. Let me look at where the cache control is set in the Anthropic provider. I can see it's using cache_control: { type: "ephemeral" } in several places:

In buildParams() for system prompts
In convertMessages() for the last user message

For OpenAI Responses, I see prompt_cache_key: options?.sessionId but I don't see any cache retention settings. Let me check what OpenAI supports for this.

The issue discussion concluded that:

Instead of exposing this as a user-facing setting, it should be an environment variable
Only changes to pi-ai package needed
Anthropic supports: 5m (ephemeral) or 1h
OpenAI supports: in_memory (default) or 24h

Let me check the Anthropic SDK types for cache_control to see what TTL options are available.

Let me check the Anthropic SDK types for cache control options and OpenAI's cache retention options:

Now let me check the env-api-keys.ts file to see the pattern for environment variable handling:

Readenv-api-keys.ts

Now I have a complete understanding of the issue and the codebase. Let me summarize my analysis:

Summary

Issue #967: Add configurable cache retention for Anthropic and OpenAI

This is a feature request to add support for extended cache retention:

Anthropic: 5m (default) or 1h TTL
OpenAI: in_memory (default) or 24h

Discussion outcome: Instead of exposing this as a user-facing setting (which could confuse users about cost implications), it should be controlled via environment variables that the providers check.

Proposed Implementation

Affected Files

packages/ai/src/providers/anthropic.ts
- Check for environment variable (e.g., PI_CACHE_RETENTION or ANTHROPIC_CACHE_TTL)
- Modify cache_control objects to include ttl: '1h' when persistent cache is enabled
- Locations to update:
  - Line ~335-340: buildParams() - system prompt cache_control
  - Line ~460-465: convertMessages() - last user message cache_control
packages/ai/src/providers/openai-responses.ts
- Check for environment variable (e.g., PI_CACHE_RETENTION or OPENAI_CACHE_RETENTION)
- Modify buildParams() to add prompt_cache_retention: '24h' when persistent cache is enabled
- Location to update:
  - Line ~101-115: buildParams() - add prompt_cache_retention field

Environment Variable Design

Options:

Single unified variable: PI_CACHE_RETENTION=persistent (maps to 1h for Anthropic, 24h for OpenAI)
Provider-specific variables: ANTHROPIC_CACHE_TTL=1h and OPENAI_CACHE_RETENTION=24h

A unified approach makes more sense since both providers distinguish between ephemeral (default, short-lived) and persistent (longer-lived) cache modes. The implementation can map a single PI_CACHE_RETENTION value to the appropriate TTL for each provider. The variable needs browser-safe access via typeof process !== "undefined" ? process.env.PI_CACHE_RETENTION : undefined, should default to current behavior (ephemeral), and should follow the pattern established in env-api-keys.ts for compatibility.

Analysis: Issue #967 - Add configurable cache retention

Summary

Type: Feature request

Request: Add support for extended cache retention for Anthropic and OpenAI APIs, controlled via environment variable (not user-facing setting).

Agreed approach (from discussion): Environment variable to keep it hidden from users who don't understand cost implications.

Provider Support

Provider	Default	Extended
Anthropic	`5m`	`1h` (costs more for cache writes)
OpenAI	`in_memory`	`24h` (no extra cost)

Current Implementation

Anthropic (anthropic.ts):

Uses cache_control: { type: "ephemeral" } without TTL (defaults to 5m)
Applied to: system prompt (line ~319-335) and last user message (line ~454-460)

OpenAI Responses (openai-responses.ts):

Uses prompt_cache_key: options?.sessionId (line 103)
No prompt_cache_retention field set (defaults to in_memory)

Proposed Implementation

Environment variable: PI_CACHE_RETENTION

Values: unset (default behavior) or persistent (extended retention)
Browser-safe check: typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent"

Files to modify:

packages/ai/src/providers/anthropic.ts
- Add helper function to get cache control TTL from env
- Update buildParams() (~line 319-335): Add ttl: '1h' to system prompt cache_control when persistent
- Update convertMessages() (~line 454-460): Add ttl: '1h' to last user message cache_control when persistent
packages/ai/src/providers/openai-responses.ts
- Update buildParams() (~line 97-127): Add prompt_cache_retention: '24h' when persistent
packages/ai/README.md
- Document PI_CACHE_RETENTION environment variable
packages/ai/CHANGELOG.md
- Add entry under [Unreleased]

Code Changes

anthropic.ts - Add helper and modify cache_control:

typescript

// Helper to get cache TTL based on envfunction getCacheTtl(): '5m' | '1h' {  if (typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent") {    return '1h';  }  return '5m';}
// In buildParams() and convertMessages():cache_control: { type: "ephemeral", ttl: getCacheTtl() }

// Helper to get cache TTL based on envfunction getCacheTtl(): '5m' | '1h' {  if (typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent") {    return '1h';  }  return '5m';}
// In buildParams() and convertMessages():cache_control: { type: "ephemeral", ttl: getCacheTtl() }

openai-responses.ts - Add cache retention:

typescript

// In buildParams():prompt_cache_key: options?.sessionId,prompt_cache_retention: (typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent")   ? '24h'   : undefined,

// In buildParams():prompt_cache_key: options?.sessionId,prompt_cache_retention: (typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent")   ? '24h'   : undefined,

Alternative: Provider-specific env vars

Could also support provider-specific overrides:

ANTHROPIC_CACHE_TTL=1h
OPENAI_CACHE_RETENTION=24h

With PI_CACHE_RETENTION as unified fallback. This gives power users more control.

Add prompt_cache_retention: '24h' when persistent what does this mean? store: true? can't do that

The user is asking about the OpenAI implementation. Let me re-check the OpenAI Responses API types to understand what prompt_cache_retention actually does and whether it requires store: true.

Looking at the earlier search results:

Let me check the OpenAI documentation or types more carefully to understand the relationship between prompt_cache_key, prompt_cache_retention, and store.

Let me check the OpenAI types more carefully:

prompt_cache_retention is independent from store. It's specifically about how long the prompt cache is retained. The docs say:

in-memory: default, short-lived
24h: keeps cached prefixes active for longer, up to 24 hours

This is separate from the store field which is about whether to store responses for later retrieval/fine-tuning.

So the implementation for OpenAI is straightforward - just add prompt_cache_retention: '24h' when the env var is set. No store: true needed.

You're right to question. prompt_cache_retention is independent from store. It's specifically about prompt caching duration:

in-memory (default): short-lived cache
24h: keeps cached prefixes active up to 24 hours

No store: true required. The implementation for OpenAI is just:

typescript

// In buildParams():prompt_cache_retention: (typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent")   ? '24h'   : undefined,

// In buildParams():prompt_cache_retention: (typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "persistent")   ? '24h'   : undefined,

The store field is about storing responses for later retrieval/fine-tuning, which is a different feature entirely.

PI_CACHE_RETENTION shoule be either "short" or "long". no provider has any more granular shit than that. then @packages/ai/src/providers/openai-responses-shared.ts (?) and anthropic should interpret that env var.

Readopenai-responses-shared.ts

I see - openai-responses-shared.ts is shared code for stream processing and message conversion, but the actual buildParams is in openai-responses.ts.

The prompt_cache_retention param needs to be added in openai-responses.ts in the buildParams function, not in the shared file.

Let me update my analysis:

Files to modify:

packages/ai/src/providers/anthropic.ts
- Check PI_CACHE_RETENTION env var
- When long, set ttl: '1h' on cache_control objects
packages/ai/src/providers/openai-responses.ts
- Check PI_CACHE_RETENTION env var
- When long, add prompt_cache_retention: '24h' to params

The shared file doesn't need to change since caching config is in the params building, not the message conversion or stream processing.

Got it. The cache retention param belongs in openai-responses.ts (in buildParams), not the shared file. The shared file is just for message conversion and stream processing.

Revised Analysis

Environment variable: PI_CACHE_RETENTION

Values: short (default behavior) or long (extended retention)
Unset = short

Files to modify:

packages/ai/src/providers/anthropic.ts
- Add helper to check env var
- When long: set cache_control: { type: "ephemeral", ttl: "1h" }
- When short/unset: keep current cache_control: { type: "ephemeral" } (defaults to 5m)
- Locations: buildParams() (system prompt) and convertMessages() (last user message)
packages/ai/src/providers/openai-responses.ts
- In buildParams(): add prompt_cache_retention: '24h' when env var is long
- When short/unset: don't set the field (defaults to in-memory)
packages/ai/README.md - Document PI_CACHE_RETENTION
packages/ai/CHANGELOG.md - Add entry

Helper pattern (browser-safe):

typescript

function useLongCacheRetention(): boolean {  return typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "long";}

function useLongCacheRetention(): boolean {  return typeof process !== "undefined" && process.env.PI_CACHE_RETENTION === "long";}

we should also only do this if the baseUrl matches the OG providers

Good point. Need to check the base URLs to avoid sending provider-specific params to proxies/third-parties.

Now I have the full picture:

Anthropic: https://api.anthropic.com
OpenAI: https://api.openai.com/v1

The helper should check:

PI_CACHE_RETENTION === "long"
model.baseUrl matches the OG provider URL

Final Analysis

Helper pattern:

typescript

function useLongCacheRetention(baseUrl: string, ogBaseUrl: string): boolean {  return (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.startsWith(ogBaseUrl)  );}

function useLongCacheRetention(baseUrl: string, ogBaseUrl: string): boolean {  return (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.startsWith(ogBaseUrl)  );}

For Anthropic: baseUrl.startsWith("https://api.anthropic.com") For OpenAI: baseUrl.startsWith("https://api.openai.com")

Final Analysis

Environment variable: PI_CACHE_RETENTION=long

Files to modify:

1. `packages/ai/src/providers/anthropic.ts`

Add helper and apply only for OG Anthropic:

typescript

function getCacheTtl(baseUrl: string): '5m' | '1h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.startsWith("https://api.anthropic.com")  ) {    return '1h';  }  return undefined; // SDK defaults to 5m}

function getCacheTtl(baseUrl: string): '5m' | '1h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.startsWith("https://api.anthropic.com")  ) {    return '1h';  }  return undefined; // SDK defaults to 5m}

In buildParams() and convertMessages():

typescript

cache_control: { type: "ephemeral", ttl: getCacheTtl(model.baseUrl) }// or if ttl is undefined, just { type: "ephemeral" }

cache_control: { type: "ephemeral", ttl: getCacheTtl(model.baseUrl) }// or if ttl is undefined, just { type: "ephemeral" }

2. `packages/ai/src/providers/openai-responses.ts`

In buildParams():

typescript

function getPromptCacheRetention(baseUrl: string): '24h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.startsWith("https://api.openai.com")  ) {    return '24h';  }  return undefined; // defaults to in-memory}
// In params:prompt_cache_retention: getPromptCacheRetention(model.baseUrl),

function getPromptCacheRetention(baseUrl: string): '24h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.startsWith("https://api.openai.com")  ) {    return '24h';  }  return undefined; // defaults to in-memory}
// In params:prompt_cache_retention: getPromptCacheRetention(model.baseUrl),

3. `packages/ai/README.md` - Document env var

4. `packages/ai/CHANGELOG.md` - Add entry

Behavior:

`PI_CACHE_RETENTION`	Anthropic (api.anthropic.com)	OpenAI (api.openai.com)	Other providers
unset / `short`	5m (default)	in-memory (default)	unchanged
`long`	1h	24h	unchanged

baseUrl.startsWith("https://api.anthropic.com") should be an includes without the https i think? do we check for that url in @packages/ai/src/providers/anthropic.ts already? sould mirror that then, same for openai

I see the pattern now. In openai-completions.ts, they use baseUrl.includes("domain") pattern, like:

baseUrl.includes("api.z.ai")
baseUrl.includes("cerebras.ai")
baseUrl.includes("api.x.ai")
baseUrl.includes("mistral.ai")
baseUrl.includes("openrouter.ai")

So for our case:

Anthropic: baseUrl.includes("api.anthropic.com")
OpenAI: baseUrl.includes("api.openai.com")

There's no existing check for these URLs in the anthropic.ts or openai-responses.ts files, but the pattern from openai-completions.ts is clear: use .includes() with the domain.

No existing checks in those files, but openai-completions.ts uses the pattern:

typescript

baseUrl.includes("api.z.ai")baseUrl.includes("cerebras.ai")baseUrl.includes("api.x.ai")baseUrl.includes("mistral.ai")baseUrl.includes("openrouter.ai")

baseUrl.includes("api.z.ai")baseUrl.includes("cerebras.ai")baseUrl.includes("api.x.ai")baseUrl.includes("mistral.ai")baseUrl.includes("openrouter.ai")

Updated Analysis

Use .includes() with domain:

typescript

// anthropic.tsbaseUrl.includes("api.anthropic.com")
// openai-responses.tsbaseUrl.includes("api.openai.com")

// anthropic.tsbaseUrl.includes("api.anthropic.com")
// openai-responses.tsbaseUrl.includes("api.openai.com")

Helper functions:

typescript

// anthropic.tsfunction getCacheTtl(baseUrl: string): '1h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.includes("api.anthropic.com")  ) {    return '1h';  }  return undefined;}
// openai-responses.ts  function getPromptCacheRetention(baseUrl: string): '24h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.includes("api.openai.com")  ) {    return '24h';  }  return undefined;}

// anthropic.tsfunction getCacheTtl(baseUrl: string): '1h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.includes("api.anthropic.com")  ) {    return '1h';  }  return undefined;}
// openai-responses.ts  function getPromptCacheRetention(baseUrl: string): '24h' | undefined {  if (    typeof process !== "undefined" &&    process.env.PI_CACHE_RETENTION === "long" &&    baseUrl.includes("api.openai.com")  ) {    return '24h';  }  return undefined;}

lgtm, implement, set up a simple etst in pi-ai package for both providers, see stream.test.ts as an example

Readstream.test.ts

Now I understand the test pattern. The tests:

Use describe.skipIf(!process.env.XXX_API_KEY) to conditionally run tests based on env vars
Use helper functions like basicTextGeneration, handleToolCall, etc.
Use getModel to get model configurations

For the cache retention test, I need to:

Implement the changes to anthropic.ts and openai-responses.ts
Create a test file that verifies the PI_CACHE_RETENTION env var is respected

Let me now implement the changes:

First, modify anthropic.ts to add cache TTL support
Then, modify openai-responses.ts to add cache retention support
Create a test file

Let me start with the implementation.

Now let me implement the changes. First, I'll update both provider files:

Editedanthropic.ts

Now let me update the buildParams function to use the cache TTL:

Readanthropic.ts100 lines

Readanthropic.ts150 lines

Readanthropic.ts80 lines

Now I have all the locations where cache_control is set. I need to update:

buildParams function - OAuth token Claude Code identity (~line 473)
buildParams function - OAuth token context.systemPrompt (~line 483)
buildParams function - Non-OAuth token context.systemPrompt (~line 493)
convertMessages function - last user message (~line 676)

The issue is that buildParams doesn't have access to model.baseUrl directly. I need to pass the model to check the baseUrl. Let me look at how buildParams is called.Looking at the code:

So model is available in buildParams. Good. Now I need to update all the cache_control locations.

Let me do the edits one by one.

Now I have all the locations. Let me update the buildParams function to use the cache TTL:

Editedanthropic.ts

Now I need to update the convertMessages function to also use the cache TTL for the last user message. The issue is that convertMessages doesn't have access to model.baseUrl. I need to pass the baseUrl or use a different approach.Looking at the function signature:```typescriptfunction convertMessages( messages: Message[], model: Model<"anthropic-messages">, isOAuthToken: boolean, ): MessageParam[] {


The model is available! So I can use `model.baseUrl` directly. Let me update the cache_control for the last user message.


The model is available! So I can use `model.baseUrl` directly. Let me update the cache_control for the last user message.

Now update the convertMessages function for the last user message:

Editedanthropic.ts

Now let me update the openai-responses.ts file:

Readopenai-responses.ts60 lines

Readopenai-responses.ts80 lines

Now I'll add the helper function and update buildParams in openai-responses.ts:

Editedopenai-responses.ts

Now update the buildParams function to include the cache retention:

Editedopenai-responses.ts

Now let me create a test file for cache retention:

Now let me run the check to ensure everything compiles:

Let me run the cache retention test:

Let me run the test directly from the ai package:

All tests pass. Now let me update the CHANGELOG and README:

ReadCHANGELOG.md50 lines

EditedCHANGELOG.md

Now let me update the README:

ReadREADME.md

EditedREADME.md

Let me run the check one more time to ensure everything is good:

Implementation complete. Here's a summary:

Changes Made

1. `packages/ai/src/providers/anthropic.ts`

Added getCacheTtl() helper function that returns '1h' when PI_CACHE_RETENTION=long and baseUrl includes api.anthropic.com
Updated buildParams() to add ttl to cache_control for system prompts (both OAuth and non-OAuth)
Updated convertMessages() to add ttl to cache_control for the last user message

2. `packages/ai/src/providers/openai-responses.ts`

Added getPromptCacheRetention() helper function that returns '24h' when PI_CACHE_RETENTION=long and baseUrl includes api.openai.com
Updated buildParams() to include prompt_cache_retention in the request params

3. `packages/ai/test/cache-retention.test.ts`

New test file with 6 tests covering:
- Anthropic: default TTL, long TTL, proxy URL (no TTL)
- OpenAI: default retention, 24h retention, proxy URL (no retention)

4. `packages/ai/CHANGELOG.md`

Added entry under [Unreleased] documenting the new feature

5. `packages/ai/README.md`

Added "Cache Retention" section under Environment Variables explaining the feature

Usage

bash

# Enable extended cache retentionexport PI_CACHE_RETENTION=long
# Run your application - Anthropic gets 1h TTL, OpenAI gets 24h retention

# Enable extended cache retentionexport PI_CACHE_RETENTION=long
# Run your application - Anthropic gets 1h TTL, OpenAI gets 24h retention

ok, please also pudate the env var section in @packages/coding-agent/README.md

ReadREADME.md

EditedREADME.md

Done. Added PI_CACHE_RETENTION to the Environment Variables table in packages/coding-agent/README.md.

commit and push, leave a comment on the issue explaining things concisely in my tone

>gh issue comment 967 --repo badlogic/pi-mono --body "Implemented via env var as discussed.

\`\`\`bash
export PI_CACHE_RETENTION=long
\`\`\`

- Anthropic: sets \`ttl: '1h'\` on cache_control (vs default 5m)
- OpenAI: sets \`prompt_cache_retention: '24h'\` (vs default in-memory)

Only kicks in for direct API calls (\`api.anthropic.com\`, \`api.openai.com\`). Proxies and third-party endpoints are unaffected.

@StaticShielding thanks for the suggestion. Should work for your extension use case too."

Summary

Analysis: Issue #967 - Add configurable cache retention

Summary

Provider Support

Current Implementation

Proposed Implementation

Code Changes

Alternative: Provider-specific env vars

Revised Analysis

Final Analysis

1. packages/ai/src/providers/anthropic.ts

2. packages/ai/src/providers/openai-responses.ts

3. packages/ai/README.md - Document env var

4. packages/ai/CHANGELOG.md - Add entry

Updated Analysis

Changes Made

1. packages/ai/src/providers/anthropic.ts

2. packages/ai/src/providers/openai-responses.ts

3. packages/ai/test/cache-retention.test.ts

4. packages/ai/CHANGELOG.md

5. packages/ai/README.md

Usage

1. `packages/ai/src/providers/anthropic.ts`

2. `packages/ai/src/providers/openai-responses.ts`

3. `packages/ai/README.md` - Document env var

4. `packages/ai/CHANGELOG.md` - Add entry

1. `packages/ai/src/providers/anthropic.ts`

2. `packages/ai/src/providers/openai-responses.ts`

3. `packages/ai/test/cache-retention.test.ts`

4. `packages/ai/CHANGELOG.md`

5. `packages/ai/README.md`