Add Opt-In Concurrency to Service Endpoints #993
Description
Proposed change
Allow for developers to opt-in to processing Nats service requests concurrently.
This could potentially be achieved by wrapping the service message handler in the Parellel.ForEachAsync method here: https://github.com/nats-io/nats.net/blob/main/src/NATS.Client.Services/NatsSvcEndPoint.cs#L221C14-L221C113, e.g.
await Parallel.ForEachAsync(_channel.Reader.ReadAllAsync(_cancellationToken).ConfigureAwait(false), (svcMsg, cancellationToken) =>
// The rest of handler function
)There are probably other implications of this particular solution, but IMO it is somewhat limiting to only be able to service one request to a given endpoint at a time.
Use case
Context
If I understand correctly, Nats services are designed around the Request/Reply pattern and provide a compelling alternative to other protocols (e.g. HTTP/gRPC).
However, in the cases of HTTP/gRPC, endpoints are designed to handle multiple concurrent requests and you generally process them without regard for the order in which they were received. We are hoping to use Nats services in the same way.
If I understand it correctly, the current Nats services implementation is not set up this way since it's based on an IAsyncEnumerable<> backed by a channel which will process requests sequentially. If you have a slow request in the queue then it will back up all the other incoming requests. Additionally, it requires scaling applications horizontally just to be able accommodate concurrent requests to the same endpoint.
Our current workaround
To work around this limitation, we are wrapping our handlers in a fire-and-forget Task.Run so they don't block the service. e.g.
await group.AddEndpointAsync<string>(
name: "a.concurrent.subject",
handler: async msg =>
{
var _ = Task.Run(async () =>
{
// Some work which might take up to a second
});
}
);This feels like an imperfect solution, as it will skew the stats for the endpoint and requires us to manually manage exception handling which the library provides here: https://github.com/nats-io/nats.net/blob/main/src/NATS.Client.Services/NatsSvcEndPoint.cs#L229
Other ideas?
If folks have other ideas or best practices on how to allow service endpoints to handle multiple requests to the same endpoint currently, I'd love to know - thanks!
Contribution
Definitely interested in contributing code to a solution, if a suitable one can be agreed upon.