Upgrade Concourse pipelines

This pattern helps with upgrading Concourse pipelines to version 7.

It handles these cases:

Basic input

BEFORE
jobs:
  - across:
    - var: function
      values:
      - file1
      - file2
      - file3
    - var: x
      values:
      - one
      - two
    - var: y
      values:
      - a
      - b
    task: create-file
    params:
      FUNCTION: ((.:function))-js
    input_mapping:
      code: ((.:x))-js
    output_mapping:
      code: ((.:y))-js
AFTER
jobs:
  - in_parallel:
      steps:
        - task: create-file-1
          params:
              FUNCTION: file1-js
          input_mapping:
              code: one-js
          output_mapping:
              code: a-js

        - task: create-file-2
          params:
              FUNCTION: file2-js
          input_mapping:
              code: one-js
          output_mapping:
              code: a-js

        - task: create-file-3
          params:
              FUNCTION: file3-js
          input_mapping:
              code: one-js
          output_mapping:
              code: a-js

        - task: create-file-4
          params:
              FUNCTION: file1-js
          input_mapping:
              code: two-js
          output_mapping:
              code: a-js

        - task: create-file-5
          params:
              FUNCTION: file2-js
          input_mapping:
              code: two-js
          output_mapping:
              code: a-js

        - task: create-file-6
          params:
              FUNCTION: file3-js
          input_mapping:
              code: two-js
          output_mapping:
              code: a-js

        - task: create-file-7
          params:
              FUNCTION: file1-js
          input_mapping:
              code: one-js
          output_mapping:
              code: b-js

        - task: create-file-8
          params:
              FUNCTION: file2-js
          input_mapping:
              code: one-js
          output_mapping:
              code: b-js

        - task: create-file-9
          params:
              FUNCTION: file3-js
          input_mapping:
              code: one-js
          output_mapping:
              code: b-js

        - task: create-file-10
          params:
              FUNCTION: file1-js
          input_mapping:
              code: two-js
          output_mapping:
              code: b-js

        - task: create-file-11
          params:
              FUNCTION: file2-js
          input_mapping:
              code: two-js
          output_mapping:
              code: b-js

        - task: create-file-12
          params:
              FUNCTION: file3-js
          input_mapping:
              code: two-js
          output_mapping:
              code: b-js

From the concourse github issue

On GitHub, this case was reported.

BEFORE
jobs:
  - across:
      - var: name
        values:
          - file1
          - file2
          - file3
    task: create-file
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      run:
        path: touch
        args:
          - manifests/((.:name))
      outputs:
        - name: manifests
  - task: list-file
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests
AFTER
jobs:
  - in_parallel:
      steps:
        - task: create-file-1
          config:
              platform: linux
              image_resource:
                type: registry-image
                source: { repository: busybox }
              run:
                path: touch
                args:
                  - manifests/file1
              outputs:
                - name: manifests
        - task: create-file-2
          config:
              platform: linux
              image_resource:
                type: registry-image
                source: { repository: busybox }
              run:
                path: touch
                args:
                  - manifests/file2
              outputs:
                - name: manifests
        - task: create-file-3
          config:
              platform: linux
              image_resource:
                type: registry-image
                source: { repository: busybox }
              run:
                path: touch
                args:
                  - manifests/file3
              outputs:
                - name: manifests
  - task: list-file
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests

Out of order

It still processes correctly even if the task is defined after the params and var comes after task.

BEFORE
jobs:
  - across:
      - values:
          - eu-west-1
          - us-east-1
        var: target
    file: deploy.yml
    task: deploy
    params:
      TARGET: ((.:target))
      other_value: 42
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests
AFTER
jobs:
  - in_parallel:
      steps:
        - file: deploy.yml
          task: deploy-1
          params:
              TARGET: eu-west-1
              other_value: 42
        - file: deploy.yml
          task: deploy-2
          params:
              TARGET: us-east-1
              other_value: 42
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests

Using simple do syntax

The do syntax for steps is also supported.

BEFORE
jobs:
  - across:
      - var: target
        values:
          - eu-west-1
          - us-east-1
    do:
    - file: deploy.yml
      task: deploy
      params:
        TARGET: ((.:target))
        other_value: 42
    - file: test.yml
      task: smoke-test
      params:
        TARGET: ((.:target))
        only_test: true
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests
AFTER
jobs:
  - in_parallel:
      steps:
        - do:
            - file: deploy.yml
              task: deploy-1
              params:
                TARGET: eu-west-1
                other_value: 42
            - file: test.yml
              task: smoke-test-1
              params:
                TARGET: eu-west-1
                only_test: true
        - do:
            - file: deploy.yml
              task: deploy-2
              params:
                TARGET: us-east-1
                other_value: 42
            - file: test.yml
              task: smoke-test-2
              params:
                TARGET: us-east-1
                only_test: true
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests

Respect max_in_flight

The max_in_flight parameter is respected.

BEFORE
jobs:
  - across:
      - var: target
        values:
          - eu-west-1
          - us-east-1
        max_in_flight: 5
    do:
    - file: deploy.yml
      task: deploy
      params:
        TARGET: ((.:target))
        other_value: 42
    - file: test.yml
      task: smoke-test
      params:
        TARGET: ((.:target))
        only_test: true
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests
AFTER
jobs:
  - in_parallel:
      limit: 5

      steps:
        - do:
            - file: deploy.yml
              task: deploy-1
              params:
                TARGET: eu-west-1
                other_value: 42
            - file: test.yml
              task: smoke-test-1
              params:
                TARGET: eu-west-1
                only_test: true
        - do:
            - file: deploy.yml
              task: deploy-2
              params:
                TARGET: us-east-1
                other_value: 42
            - file: test.yml
              task: smoke-test-2
              params:
                TARGET: us-east-1
                only_test: true
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests

Multiple max_in_flight variables

The semantics of max_in_flight are for multiple variables are complicated to replicate, since across can dynamically choose which tasks to parallelize.

To keep things simple, this change just keeps the first max_in_flight value.

BEFORE
jobs:
  - across:
      - var: target
        values:
          - eu-west-1
          - us-east-1
          - us-central-1
        max_in_flight: 3
      - var: middle_value
        values:
          - "a"
          - "b"
          - "c"
        max_in_flight: 1
      - var: namer
        values:
          - susan
          - alice
        max_in_flight: 2
    file: deploy.yml
    task: deploy
    params:
      TARGET: ((.:target))
      other_value: 42
      name: ((.:namer))
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests
AFTER
jobs:
  - in_parallel:
      limit: 3

      steps:
        - file: deploy.yml
          task: deploy-1
          params:
              TARGET: eu-west-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-2
          params:
              TARGET: eu-west-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-3
          params:
              TARGET: eu-west-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-4
          params:
              TARGET: eu-west-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-5
          params:
              TARGET: eu-west-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-6
          params:
              TARGET: eu-west-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-7
          params:
              TARGET: us-east-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-8
          params:
              TARGET: us-east-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-9
          params:
              TARGET: us-east-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-10
          params:
              TARGET: us-east-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-11
          params:
              TARGET: us-east-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-12
          params:
              TARGET: us-east-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-13
          params:
              TARGET: us-central-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-14
          params:
              TARGET: us-central-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-15
          params:
              TARGET: us-central-1
              other_value: 42
              name: susan
        - file: deploy.yml
          task: deploy-16
          params:
              TARGET: us-central-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-17
          params:
              TARGET: us-central-1
              other_value: 42
              name: alice
        - file: deploy.yml
          task: deploy-18
          params:
              TARGET: us-central-1
              other_value: 42
              name: alice
  - task: other-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      inputs:
        - name: manifests
      run:
        path: ls
        args:
          - manifests