Full support for responses API plus conversation api like functionality

[ockhamlabs]

We currently use responses API along with conversational api for full context engineered ai agentic conversations and actions (,openai). Will be great to enable same drop-in replacement using any model and using memory /custom memory solution to provide same result

[salmanap]

This is a well requested feature and we will need to use a short term cache or database to support that. Looking at options now

[salmanap]

High-level network diagram.

[salmanap]

Decision on storage is SQLLite. Its binary size is 4.7M and it can be bundled as a first-class object with archgw. The only thing we'll have to think about is conversational state pruning. What happens to conversations more than 30 days, especially if the upstream provider doesn't support storing messages for more than 30 days.

[salmanap]

Tracking PR: #622

[salmanap]

#622 - the basic support is ready. A new PR will be published in the next couple of days to add support for storing the conversations and populating them via SQLlite

[salmanap]

Adding designs for storage here, which will be built out today once we merge in 622

@adilhafeez - looking at the insertion points to store and retrieve resp_id + json, we have two options, both of which assume that we will create a conversation_persistent_store.rs trait and implement a SQLLite version that retrieves and stores information in a cluster that is locally defined in arch_config.yaml.

Option #1
Store and retrieve the resp_id + json in the request path via llm_gateway/stream_context.rs. Use the same pattern we used in prompt_gateway/stream_context.rs to make outbound calls. These calls aren't "decision" calls, meaning they aren't part of the control plane so they are essentially enriching the request with additional details. This would be an easy lift, basically detect the client API, add more context to the request payload, transform it just like today and make sure on the on_http_response_body on end=true write to disk the new resp_id + json. Failures would bubble up just like today.

Option #2
Store and retrieve the resp_id + json in the request path via bright_staff. This means make sure that bright_staff is aware of reconstructing the buffer (again), which requires refactoring the transformation logic out of llm_gateway/stream_context.rs into bright_staff. The one benefit of doing this in bright_staff is that you have more control over the request path. But I am not 100% sure what that gives us today. Previously, bright_staff was added to decouple decision making from request forwarding.

The size of the refactor in option #2 is very high imho. A lot of moving parts. Might be beneficial long-term but short-term I can't see immediate benefits.

[salmanap]

@adilhafeez - as post fact I also asked Claude 4.5 to evaluate this approach. Prompt: "Evaluate my above design notes". Be thorough. The model has an "agreeable" factor so we should take it with a grain of salt, but I think its evaluation was correct in this instance

---

Design Evaluation: ResponsesAPI Storage Implementation Overview

You're evaluating where to implement storage for resp_id + JSON response data for the OpenAI ResponsesAPI. Let me provide a thorough analysis.

Option #1: Storage in llm_gateway/stream_context.rs

✅ Strengths

• Minimal Surface Area: Leverages existing patterns from prompt_gateway/stream_context.rs
• Clear Separation: Storage is purely operational (enrichment), not control plane logic
• Low Risk: Failures bubble up through existing error handling
• Fast Implementation: Pattern already proven and understood
• Correct Layer: Response persistence is a gateway concern, not a routing concern

⚠️ Considerations

• Detection Point: You'll need to identify ResponsesAPI requests early
• Buffer Management: Must ensure complete response before storage (already handled by ResponsesAPIStreamBuffer)
• Error Handling: Storage failures need clear semantics - fail request or log and continue?

💡 Implementation Notes. Looking at your code:

• ResponsesAPIStreamBuffer already reconstructs full responses
• You have SseEvent::provider_stream_response available
• needs_buffering() can detect when ResponsesAPI is involved
• The into_bytes() pattern gives you the complete response

Option #2: Storage in bright_staff

❌ Major Issues

• Wrong Abstraction Level: bright_staff is for routing decisions, not data persistence
• Unnecessary Coupling: Forces bright_staff to understand streaming transformation details
• Duplication Risk: Buffer reconstruction logic would exist in two places
• Breaking Separation of Concerns: Mixes control plane (routing) with data plane (storage)

🚫 Anti-Patterns This Would Introduce

• bright_staff shouldn't care about SSE wire formats
• bright_staff shouldn't need ResponsesAPIStreamBuffer knowledge
• Storage is an infrastructural concern, not a policy concern

[adilhafeez]

I looked at both options the way I see llm_gateway is a translation layer and nothing else. It should stay and act like that and nothing else if we were to keep design and ownership simple. But the moment we add state and orchestration to llm_gateway it will become 1) difficult to test and 2) spill unnecessary abstractions to envoy/wasm-filter. We have much better control in brightstaff and we can evolve and test our changes better. My opinion is to keep wasm-filter simple and use it only for translation of request/response. And add any state management piece inside brightstaff.

[salmanap]

The ability to take a v1/responses api and previous_response_id and translate that to an upstream /v1/messages request is essentially a translation task. Agree we are pulling state and storing that via the WASM filter, but that is part of translation - not orchestration in my view.

I suppose we have to define the true purpose of bright_staff because if we refactor and/or repeat the functionality in two places then we will have to maintain two separate tests for the same thing vs. one concrete unit where all the functionality is housed.

On testing, I think we can have integration tests that test out the complete workflow by adding additional APIs that openai already expooses like v1/responses/{response_id}/GET that would return the same response based on the resp_id. This way we can have a complete end to end test and unit test the storage and retrieval of conversations separately.