This pattern helps with upgrading Concourse pipelines to version 7.
It handles these cases:
- across outputs, by converting from across steps to in_parallel steps.
Apply with the Grit CLI
grit apply concourse_v7
Basic input
BEFORE
jobs:
- across:
- var: function
values:
- file1
- file2
- file3
- var: x
values:
- one
- two
- var: y
values:
- a
- b
task: create-file
params:
FUNCTION: ((.:function))-js
input_mapping:
code: ((.:x))-js
output_mapping:
code: ((.:y))-jsAFTER
jobs:
- in_parallel:
steps:
- task: create-file-1
params:
FUNCTION: file1-js
input_mapping:
code: one-js
output_mapping:
code: a-js
- task: create-file-2
params:
FUNCTION: file2-js
input_mapping:
code: one-js
output_mapping:
code: a-js
- task: create-file-3
params:
FUNCTION: file3-js
input_mapping:
code: one-js
output_mapping:
code: a-js
- task: create-file-4
params:
FUNCTION: file1-js
input_mapping:
code: two-js
output_mapping:
code: a-js
- task: create-file-5
params:
FUNCTION: file2-js
input_mapping:
code: two-js
output_mapping:
code: a-js
- task: create-file-6
params:
FUNCTION: file3-js
input_mapping:
code: two-js
output_mapping:
code: a-js
- task: create-file-7
params:
FUNCTION: file1-js
input_mapping:
code: one-js
output_mapping:
code: b-js
- task: create-file-8
params:
FUNCTION: file2-js
input_mapping:
code: one-js
output_mapping:
code: b-js
- task: create-file-9
params:
FUNCTION: file3-js
input_mapping:
code: one-js
output_mapping:
code: b-js
- task: create-file-10
params:
FUNCTION: file1-js
input_mapping:
code: two-js
output_mapping:
code: b-js
- task: create-file-11
params:
FUNCTION: file2-js
input_mapping:
code: two-js
output_mapping:
code: b-js
- task: create-file-12
params:
FUNCTION: file3-js
input_mapping:
code: two-js
output_mapping:
code: b-jsFrom the concourse github issue
On GitHub, this case was reported.
BEFORE
jobs:
- across:
- var: name
values:
- file1
- file2
- file3
task: create-file
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
run:
path: touch
args:
- manifests/((.:name))
outputs:
- name: manifests
- task: list-file
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsAFTER
jobs:
- in_parallel:
steps:
- task: create-file-1
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
run:
path: touch
args:
- manifests/file1
outputs:
- name: manifests
- task: create-file-2
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
run:
path: touch
args:
- manifests/file2
outputs:
- name: manifests
- task: create-file-3
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
run:
path: touch
args:
- manifests/file3
outputs:
- name: manifests
- task: list-file
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsOut of order
It still processes correctly even if the task is defined after the params and var comes after task.
BEFORE
jobs:
- across:
- values:
- eu-west-1
- us-east-1
var: target
file: deploy.yml
task: deploy
params:
TARGET: ((.:target))
other_value: 42
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsAFTER
jobs:
- in_parallel:
steps:
- file: deploy.yml
task: deploy-1
params:
TARGET: eu-west-1
other_value: 42
- file: deploy.yml
task: deploy-2
params:
TARGET: us-east-1
other_value: 42
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsUsing simple do syntax
The do syntax for steps is also supported.
BEFORE
jobs:
- across:
- var: target
values:
- eu-west-1
- us-east-1
do:
- file: deploy.yml
task: deploy
params:
TARGET: ((.:target))
other_value: 42
- file: test.yml
task: smoke-test
params:
TARGET: ((.:target))
only_test: true
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsAFTER
jobs:
- in_parallel:
steps:
- do:
- file: deploy.yml
task: deploy-1
params:
TARGET: eu-west-1
other_value: 42
- file: test.yml
task: smoke-test-1
params:
TARGET: eu-west-1
only_test: true
- do:
- file: deploy.yml
task: deploy-2
params:
TARGET: us-east-1
other_value: 42
- file: test.yml
task: smoke-test-2
params:
TARGET: us-east-1
only_test: true
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsRespect max_in_flight
The max_in_flight parameter is respected.
BEFORE
jobs:
- across:
- var: target
values:
- eu-west-1
- us-east-1
max_in_flight: 5
do:
- file: deploy.yml
task: deploy
params:
TARGET: ((.:target))
other_value: 42
- file: test.yml
task: smoke-test
params:
TARGET: ((.:target))
only_test: true
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsAFTER
jobs:
- in_parallel:
limit: 5
steps:
- do:
- file: deploy.yml
task: deploy-1
params:
TARGET: eu-west-1
other_value: 42
- file: test.yml
task: smoke-test-1
params:
TARGET: eu-west-1
only_test: true
- do:
- file: deploy.yml
task: deploy-2
params:
TARGET: us-east-1
other_value: 42
- file: test.yml
task: smoke-test-2
params:
TARGET: us-east-1
only_test: true
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsMultiple max_in_flight variables
The semantics of max_in_flight are for multiple variables are complicated to replicate, since across can dynamically choose which tasks to parallelize.
To keep things simple, this change just keeps the first max_in_flight value.
BEFORE
jobs:
- across:
- var: target
values:
- eu-west-1
- us-east-1
- us-central-1
max_in_flight: 3
- var: middle_value
values:
- "a"
- "b"
- "c"
max_in_flight: 1
- var: namer
values:
- susan
- alice
max_in_flight: 2
file: deploy.yml
task: deploy
params:
TARGET: ((.:target))
other_value: 42
name: ((.:namer))
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifestsAFTER
jobs:
- in_parallel:
limit: 3
steps:
- file: deploy.yml
task: deploy-1
params:
TARGET: eu-west-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-2
params:
TARGET: eu-west-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-3
params:
TARGET: eu-west-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-4
params:
TARGET: eu-west-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-5
params:
TARGET: eu-west-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-6
params:
TARGET: eu-west-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-7
params:
TARGET: us-east-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-8
params:
TARGET: us-east-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-9
params:
TARGET: us-east-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-10
params:
TARGET: us-east-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-11
params:
TARGET: us-east-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-12
params:
TARGET: us-east-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-13
params:
TARGET: us-central-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-14
params:
TARGET: us-central-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-15
params:
TARGET: us-central-1
other_value: 42
name: susan
- file: deploy.yml
task: deploy-16
params:
TARGET: us-central-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-17
params:
TARGET: us-central-1
other_value: 42
name: alice
- file: deploy.yml
task: deploy-18
params:
TARGET: us-central-1
other_value: 42
name: alice
- task: other-task
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: manifests
run:
path: ls
args:
- manifests