Simulated RGBA framebuffer with Kitty protocol
Public
Simulated RGBA framebuffer with Kitty protocol
Agent
Pi
Model
GPT-5.2 Codex
Started
Jan 22, 2:01 AM
Messages68
Downloads0
Explained Kitty protocol image support and limitations.
UnderstandCreated Node.js app demoing RGBA framebuffer at 60fps.
BuildRan demo and encountered runtime error with buffer copy.
ReproduceFixed buffer copy method to use correct API.
Edit
so, kitty protocol has image support, right? do you think we could simulate something like an rgba framewbuffer with that we can write to at 60fps?
Kitty has graphics protocol support, so you can push images. For 60fps RGBA framebuffer simulation, it depends on terminal performance. In practice, image uploads are heavy. You can stream by sending new images each frame, but throughput and latency will likely cap well below 60fps for large frames. Also, Kitty images are chunked and often re-encoded, so you do not get a true shared framebuffer with zero copy updates.
If you want a smoother effect, you can try smaller regions, lower resolution, or limited dirty rect updates. For a full screen 60fps RGBA buffer, it is not realistic in most terminal setups.
read @packages/tui/README.md create a standalone node app that tries to proof that out in @packages/tui/test/ using @packages/tui/src/components/image.ts in a simple tui
Planning initial reads
I'll start by reading the README fully to understand the project context, then read the image component file before any modifications to prepare for creating the standalone node app in packages/tui/test/.
Preparing to read image component
I'm confirming the plan to read the full image.ts file in packages/tui/src/components to understand it before creating the standalone node app in packages/tui/test, aligning with the user's explicit request to read first.
Planning standalone TUI test app
Planning to create a simple node app under packages/tui/test that uses the image component to simulate an RGBA framebuffer at 60fps by regularly updating image data, starting with checking existing test files for structure and naming patterns.
>ls /Users/badlogic/workspaces/pi-mono/packages/tui/testReviewing image-test file
Planning TUI image update test
I'm planning a new test that updates an image at 60fps in a simple TUI using a generated RGBA buffer, checking existing dependencies like pngjs or sharp in the tui package before implementation.
Considering custom PNG encoding
I'm exploring implementing a simple PNG encoder for RGBA using Node's built-in zlib to generate base64 images since no PNG libraries are available and the Image component requires base64 PNG data. Next, I'll search the repo for any existing PNG encoding utilities to avoid starting from scratch.
>rg -n "PNG" /Users/badlogic/workspaces/pi-mono/packages/tuiSwitching to grep due to missing rg
>grep -R "PNG" -n /Users/badlogic/workspaces/pi-mono/packages/tuiPlanning simple PNG encoder and animated image display
Since the existing Image component’s base64 data is private and immutable, I'll implement a simple PNG encoder in a test file and create a custom TUI component that updates animation frames by swapping new Image instances per frame despite some inefficiency, focusing on a workable test setup.
Designing custom framebuffer image component and PNG encoder
I'm planning a custom FramebufferImage component that wraps the existing Image component by recreating it each frame with new base64 PNG data generated from an RGBA buffer using a minimal PNG encoder. This encoder will implement PNG chunks, CRC32, and zlib compression at a moderate level, and I'll track frame timing and update an FPS text overlay within a standalone test Node app under packages/tui/test/image-framebuffer.ts.
Designing test file for framebuffer TUI
Planning a new test file that mimics a framebuffer using an Image component updated with base64 data from an RGBA buffer, displaying a simple time-based gradient pattern for demonstration at a modest resolution to test frame updates.
Planning comprehensive framebuffer test file with PNG encoding
Outlined a detailed design for a new test file that renders a time-based animated framebuffer as base64-encoded PNG images in a TUI using a custom component wrapping Image, updating frames at target FPS with gradient animations, CLI args for dimensions and FPS, and integration of Text components for stats and input handling to exit cleanly.
>ls /Users/badlogic/workspaces/pi-mono/packages/tui/srcPlanning to read keybindings for export names
Examining key exports
I'm checking keys.ts to understand the exported values, focusing on how MatchesKey and Key constants are defined for correct import paths.
Designing framebuffer image component and input handling
Planning to create a new test file with a FramebufferImage component implementing the Component interface, ensuring usage of the existing Image class. For input focus, planning to use tui.setFocus with an object having a handleInput method, mimicking patterns from existing image tests. Also considering clean exit handling with tui.stop() and input-based interrupt capture.
jesus stop fucking around so much, you should know enough by now, noß0
just write a simple self containted node ap as a .ts file invokable via npx tsx, like the other demos in that folder. it should create a pixel buffer (320x200) which we draw to, which then gets shown in the tui via an image component. it should update somehow, say a rectangle bouncing around from the edges of the buffer, red, background of buffer black. just to see if perfomance is there
Exploring PNG encoding options
I'm assessing how to generate PNG frames from raw buffers without adding dependencies, considering a custom minimal PNG encoder or repurposing existing repo helpers to produce base64 PNG data for the image component.
>grep -R "PNG" -n /Users/badlogic/workspaces/pi-mono/packages/tui/srcPlanning minimal PNG encoder
I'm deciding to implement a minimal PNG encoder for RGBA data using standard chunks and zlib.deflateSync, including a small crc32 function; this will produce base64 PNG data for the image component. To handle frame updates without relying on private base64 setters or caching issues, I'll consider creating a custom component or replacing the image instance each frame with removal and addition in the UI.
Designing frame update strategy
I'm planning to wrap the Image in a Container to replace its child each frame by removing the old Image and adding a new one, minimizing flicker and layout shifts. I'll handle 60fps updates with setInterval, manage stopping on Ctrl+C with key matching, and show FPS via a Text component updated with setText plus manual render requests. Next, I'll review the Text component file for its API to confirm usage.
Considering container for image update
I’m planning to use a container component to swap images dynamically since the Image component's caching blocks updates; this involves creating a new Image with the same base64 and replacing it within the container.
Considering component location
I'm evaluating placing the list in components/container/index based on the current structure.
>ls /Users/badlogic/workspaces/pi-mono/packages/tui/src/componentsSearching for container usage
I'm checking whether the Container class is defined or used in tui.ts or elsewhere since it’s mentioned in the README but not found in the components folder.
>grep -R "class Container" -n /Users/badlogic/workspaces/pi-mono/packages/tui/srcPlanning dynamic image update
I'll create a container for a dynamic image and update it each frame by clearing and adding a new Image instance with updated base64 data, avoiding hacks by replacing the entire Image component. The layout will have a root TUI with title, FPS text, and the image container, limiting width for scaling and redrawing a black-filled buffer with a red rectangle each frame.
Designing animated PNG framebuffer test
I'm outlining a new test file that draws a black-and-red rectangle animation into a RGBA buffer each frame, encodes it to PNG with a custom encoder using zlib deflate and CRC32, then updates a new Image component in a container every frame while tracking and displaying FPS. The file will handle input (ctrl+c) to stop cleanly, use setInterval for 60fps rendering, and import necessary TUI components and helpers.
Providing summary and run instructions
I'm summarizing the change, noting the file path, and including instructions to run without executing any tests since they were not requested.
Summary:
packages/tui/test/image-framebuffer.ts with a 320x200 RGBA buffer demo that encodes PNG frames and renders them via the Image component at ~60fps.Run:
npx tsx packages/tui/test/image-framebuffer.tsTests not run.
➜ pi-mono git:(main) ✗ npx tsx packages/tui/test/image-framebuffer.ts
Framebuffer Image Test
FPS: 0
Press Ctrl+C to exit /Users/badlogic/workspaces/pi-mono/packages/tui/test/image-framebuffer.ts:133 rgba.copy(raw, rowOffset + 1, y * stride, y * stride + stride); ^
TypeError: rgba.copy is not a function at encodePng (/Users/badlogic/workspaces/pi-mono/packages/tui/test/image-framebuffer.ts:133:8) at updateScene (/Users/badlogic/workspaces/pi-mono/packages/tui/test/image-framebuffer.ts:88:14) at Timeout._onTimeout (/Users/badlogic/workspaces/pi-mono/packages/tui/test/image-framebuffer.ts:55:2) at listOnTimeout (node:internal/timers:605:17) at process.processTimers (node:internal/timers:541:7)
Node.js v25.2.1 ➜ pi-mono git:(main) ✗
Correcting buffer usage and file read order
The rgba data should be handled using Buffer.from or copied via set instead of copy on Uint8Array. Also, files must be fully read before any edits; since a file was already written, I'll read it fully again before applying the edit.
Updated packages/tui/test/image-framebuffer.ts to copy RGBA rows with raw.set(...) instead of rgba.copy(...).
Run:
npx tsx packages/tui/test/image-framebuffer.tsTests not run.