Problem you have encountered:

We have a large number of independent build steps, but they execute very slowly when they are all running at the same time (all clamoring for the same CPU + disk).

There is no way to specify how many jobs should be executing in parallel in a build config; all we can specify are the job dependencies.

What you expected to happen:

There should be some way to control the maximum number of steps to execute in parallel, e.g.,

options:
  max_parallelism: 10  # run at most 10 jobs in parallel

Steps to reproduce:

Example config:

steps:
  - name: do-something-disk-intensive1
    waitFor: -
  - name: do-something-disk-intensive2
    waitFor: -
  - name: do-something-disk-intensive3
    waitFor: -
...
  - name: do-something-disk-intensive100
    waitFor: -

This configuration will cause all 100 jobs to execute simultaneously, and since the jobs are all thrashing the same resources (be it disk or CPU or memory), running them all at the same time will go much slower than if they ran in chunks.

Other information (workarounds you have tried, documentation consulted, etc):

We can induce some parallel structure in the dependency graph itself, e.g., for parallelism of 10, we can make 10 separate chains of work:

steps:
  - name: do-something-disk-intensive1
    waitFor: -
  - name: do-something-disk-intensive2
    waitFor: -
  - name: do-something-disk-intensive3
    waitFor: -
...
  - name: do-something-disk-intensive10
    waitFor: do-something-disk-intensive1
  - name: do-something-disk-intensive11
    waitFor: do-something-disk-intensive2
  - name: do-something-disk-intensive12
    waitFor: do-something-disk-intensive3
...
  - name: do-something-disk-intensive100
    waitFor: do-something-disk-intensive91

but it's a pretty annoying hack and doesn't account for the possibility of lots of larger jobs happening to show up in the same chain.