Introduce Worker-Pool for Containerd Event Handling to Prevent Blocking and Improve Throughput

[keshav78-78]

Problem

MonitorContainerdEvents currently processes events synchronously.
Since UpdateContainerdContainer involves containerd RPC calls, PID/NS lookups, and multiple locks, it can block the subscription loop during heavy workloads.

This leads to:

subscription loop blocking

reduced throughput under bursts

increased lock contention

delayed event handling

Proposed Enhancement

Introduce a bounded worker-pool to asynchronously process containerd events.

Key benefits:

prevents subscription loop from blocking

improves throughput and responsiveness

reduces lock contention inside event handlers

handles bursts of start/stop events more efficiently

Ordering Model

Depending on the project’s expectations, two approaches are possible:

Therefore, two modes are viable:

Full parallel mode (default)

Workers freely process events across all containers.

Maximum throughput.

Per-container ordering (optional)

Implemented using consistent hashing:
workerID = hash(containerID) % numWorkers

Ensures events for the same container are always processed by the same worker.

Full parallelism is safe unless ordering requirements change later.

Implementation Plan

Add a buffered job queue (e.g., size 100)

Spawn N worker goroutines (e.g., 5–10 workers)

Subscription loop only enqueues events

Workers call handleContainerdEvent → UpdateContainerdContainer

Add metrics: queue length, processed/sec, worker busy count

Add helper to dedupe PID/NS retrieval logic

Reduce lock hold-times by computing states outside locks

Graceful shutdown by closing queue on StopChan

Request

Please review the proposed enhancement approach.
Once approved, I will start implementation and open a PR.

Maintainer Visibility

Tagging maintainers for input:
kubearmor-maintainers-- @aryansharma , @rksharma95

[keshav78-78]

[keshav78-78]

upper are the screenshots of perfectly working worker pool