[llm-gateway] Performance optimization: Inefficient string conversion in streaming response handler

[jolestar]

In the streaming response handler (paymentKit.ts:565-569), we're converting each Buffer chunk to string for usage processing, but then writing the original Buffer to the response. This creates unnecessary overhead for large streams.

upstream.data.on('data', (chunk: Buffer) => {
  const chunkStr = chunk.toString();  // Expensive conversion
  totalBytes += chunk.length;
  streamProcessor.processChunk(chunkStr);  // Only usage processing needs string
  res.write(chunk);  // Writing original buffer anyway
});

Impact

• Redundant Buffer.toString() calls on every chunk
• Performance degradation for high-throughput streaming scenarios
• Memory allocation overhead for temporary string objects

Potential Solutions

1. Smart processing: Only convert chunks that likely contain usage data ("usage" keyword detection)
2. TextDecoder: Use streaming text decoder for better UTF-8 handling
3. Batch processing: Accumulate chunks and process in batches
4. Async processing: Move usage processing off the critical path

Priority

Medium - Performance optimization that affects streaming workloads but doesn't break functionality.