infra(helm): Allow concurrency pool settings via Helm

[chollinger93]

Resolves #31086

What

You can't configure concurrency pools via Helm, meaning it defaults to 1 and needs to be click ops'd, which is not a viable path for a production deployment (ask me how I know).

This allows you to do the following in Helm:

    type: QueuedRunCoordinator
    config:
      queuedRunCoordinator:
        maxConcurrentRuns: 10
# ...
        dequeueNumWorkers: 4
        pools:
          defaultLimit: 2
          granularity: op
          opGranularityRunBuffer: 1

How

Follows the standard pattern

Tests

• Added a unit test

Template test:

helm show values dagster/dagster > values.yaml
vim values.yaml # Edit 
cat values.yaml| yq '.dagsterDaemon.runCoordinator.config.queuedRunCoordinator.pools'
defaultLimit: 2
granularity: op
opGranularityRunBuffer: 1

helm template test ~/workspace/dagster/helm/dagster/ -f ./values.yaml | grep -A5 -B5 pools
      num_workers: 4
    schedules:
      use_threads: true
      num_workers: 4
    concurrency:
      pools:
        default_limit: 2
        granularity: op
        op_granularity_run_buffer: 1

    telemetry:

The same test on master:

dagster on  master [?] via 🐍 v3.12.3 (dagster) 
❯ helm template test ~/workspace/dagster/helm/dagster/ -f ./values.yaml
Error: values don't meet the specifications of the schema(s) in the following chart(s):
dagster:
- dagsterDaemon.runCoordinator.config.queuedRunCoordinator: Must validate at least one schema (anyOf)
- dagsterDaemon.runCoordinator.config.queuedRunCoordinator: Additional property pools is not allowed

Local deployment

uv venv --python 3.12
source .venv/bin/activate
make dev_install
pip install -e python_modules/dagster
dagster-image build-all --name dagster-celery-k8s-editable --dagster-version 1.9.6 
docker tag dagster/dagster-celery-k8s-editable:py3.10-2025-12-08T173515 docker.io/dagster/dagster-celery-k8s:0.0.1-dev

docker build . -t iris-pool
docker tag iris-pool iris-pool:0.0.1-dev

k3d cluster create test-cluster
k3d image import dagster/dagster-celery-k8s:0.0.1-dev -c test-cluster
k3d image import iris-pool:0.0.1-dev -c test-cluster

helm upgrade --install dagster ~/workspace/dagster/helm/dagster/ -f values.yaml

And we can see

and

Permutations

Also tested

• defaultLimit => 4
• defaultLimit unset
• pool unset

Which all worked.

Side Node: Test Setup

Side node, I tried to figure out how one is supposed to test this with local image builds and that's the best I could come up with. This was relevant too: #30427

So I needed

FROM python:3.11

# Copy your Dagster project. You may need to replace the filepath depending on your project structure
COPY examples/deploy_k8s/ /

# This makes sure that logs show up immediately instead of being buffered
ENV PYTHONUNBUFFERED=1

RUN pip install --upgrade pip

# Install dagster and any other dependencies your project requires
RUN \
    pip install \
        dagster \
        dagster-postgres \
        dagster-k8s \
        # add any other dependencies here
        pandas


WORKDIR /iris_analysis/

# Expose the port that your Dagster instance will run on
EXPOSE 80

With the following changes:

git diff examples/deploy_k8s/iris_analysis/definitions.py
-@dg.asset
+@dg.asset(pool="testing_pool")
 def iris_dataset_size(context: dg.AssetExecutionContext) -> None:
     df = pd.read_csv(
         "https://docs.dagster.io/assets/iris.csv",

And for the deployment:

❯ diff values.yaml values_default.yaml
38,39c38,39
<     tag: 0.0.1-dev
<     pullPolicy: IfNotPresent
---
>     tag: ~
>     pullPolicy: Always
293,294c293,294
<         repository: "iris-pool"
<         tag: 0.0.1-dev
---
>         repository: "docker.io/dagster/user-code-example"
>         tag: ~
299c299
<         pullPolicy: IfNotPresent
---
>         pullPolicy: Always
316c316
<         - "/iris_analysis/definitions.py"
---
>         - "/example_project/example_repo/repo.py"
406c406
<         failureThreshold: 15 # Allow roughly 300 seconds to start up by default
---
>         failureThreshold: 15  # Allow roughly 300 seconds to start up by default
429,431c429,431
<     repository: "iris-pool"
<     tag: 0.0.1-dev
<     pullPolicy: IfNotPresent
---
>     repository: "docker.io/dagster/user-code-example"
>     tag: ~
>     pullPolicy: Always
483c483
<       #   pullPolicy: IfNotPresent
---
>       #   pullPolicy: Always
609,610c609,610
<         tag: 0.0.1-dev
<         pullPolicy: IfNotPresent
---
>         tag: ~
>         pullPolicy: Always
874c874
<     pullPolicy: IfNotPresent
---
>     pullPolicy: Always
1059,1060c1059,1060
<     tag: 0.0.1-dev
<     pullPolicy: IfNotPresent
---
>     tag: ~
>     pullPolicy: Always
1077c1077
<         maxConcurrentRuns: 10
---
>         maxConcurrentRuns: ~
1091,1094d1090
<         pools:
<           defaultLimit: 2
<           granularity: op
<           opGranularityRunBuffer: 1
1352a1349
> 

Caveats

• I put the Helm config under dagsterDaemon.runCoordinator.config, since that's where maxConcurrentRuns sits (rather than concurrency.runs.max_concurrent_runs as in the dagster.yaml)
• I used the default values and types from the existing schema in config.py. I'd probably use literals, but that doesn't seem to be the pattern here.
• There is a opGranularityRunBuffer field which doesn't really show up in the docs, but it does show up here:

  dagster/python_modules/dagster/dagster/_core/instance/config.py

  Line 459 in d538aed

  "op_granularity_run_buffer": Field(

  . I have no idea if we want this in here.
• I'm not sure what other tests we should run to validate

Changelog

Allows controlling concurrency pool setting dagsterDaemon.runCoordinator.config.queuedRunCoordinator.pools via Helm