Developing CWL Workflows with VSCode

These lessons give step by step instructions for using Visual Studio Code (abbreviated “vscode”) to develop CWL workflows on Arvados.

Set up SSH
Install vscode and necessary extensions, then use vscode to connect to an Arvados shell node for development
Register a workflow, run it on workbench, and view the log
Upload input, run a workflow on it, and view the output
Register a workflow with default inputs
Run a workflow without registering it

1. SSH Setup

(Windows only) Install Git for Windows https://git-scm.com/download/win
1. Choose “64-bit Git for Windows Setup”. It does not require admin privileges to install.
2. Hit “Next” a bunch of times to accept the defaults
3. The most important things is that “install git bash” and “install OpenSSH” are enabled (this is the default).
4. At the end of the installation, you can launch tick a box to git bash directly.
5. Open “Git Bash” (installed in the “Git” folder of the start menu)
(All operating systems) Starting from bash shell (on MacOS or Linux you will open “Terminal”)
1. Shell: Run ssh-keygen
  1. Hit enter to save to a default location
  2. You can choose to protect the key with a password, or just hit enter for no password.
2. Shell: Look for a message like Your public key has been saved in /c/Users/MyUsername/.ssh/id_rsa.pub (Windows git bash example, on MacOS or Linux this will probably start with /Users or /home)
  1. Shell: Run cat /c/Users/MyUsername/.ssh/id_rsa.pub
3. Shell: Use the pointer to highlight and copy the lines starting with ssh-rsa … up to the next blank line. Right click and select “Copy”
Open Arvados workbench 2. If necessary, go to the user menu and select “Go to Workbench 2”
1. Workbench: Go to SSH keys in the user menu
2. Workbench:Click +Add new ssh key
3. Workbench: Paste the key into Public key and enter something for name
4. Workbench: Go to Virtual Machines in the user menu
5. Workbench: Highlight and copy the value in in the Command line column.
At the git bash command line
1. Shell: paste the ssh shell… command line you got from workbench.
2. Shell: type “yes” if it asks Are you sure you want to continue connecting.
3. Note: it can take up to two minutes for the SSH key to be copied to the shell node. If you get “Permission denied” the first time, wait 60 seconds and try again.
4. Shell: You should now be logged into the Arvados shell node.
5. Shell: Log out by typing exit

2. VSCode setup

Install Visual Studio Code and start it up
Vscode: On the left sidebar, select Extensions
1. In Search Extensions in Marketplace enter “remote development”.
2. Choose and install the “Remote Development” extension pack from Microsoft
Vscode: On the left sidebar, choose Remote Explorer
1. At the top of the Remote Explorer panel choose SSH targets
2. Click Add New
3. Enter the ssh shell… command line you used in the previous section, step 1.4.1
  1. If it asks you Select SSH configuration file to update choose the first one in the list.
4. Right click the newly added ssh target in the list and select “connect to host in current window`
5. If it asks Select platform of the remote host select Linux.
Vscode: On the left sidebar, go back to Extensions
1. Search for “benten”, then look for CWL (Rabix/Benten) and click Install
2. On the information page for CWL (Rabix/Benten)
  1. If you see a warning Install the extension on 'SSH: ...' to enable then click the button Install in SSH: ...
3. You should now see a message Extension is enabled on 'SSH: ...' and disabled locally.
Vscode: On the left sidebar, choose Explorer
1. Select Clone Repository and enter https://github.com/arvados/arvados-vscode-cwl-training, then click Open
2. If asked Would you like to open the cloned repository? choose Open
Go to Arvados Workbench
1. Workbench: In the user menu, select Current token
2. Workbench: Click on Copy to Clipboard.
3. Workbench: You should see a notification Token copied to clipboard.
4. Go to Vscode
5. Vscode: Click on the Terminal menu
6. Vscode: Click Run Task…
7. Vscode: Select Set Arvados Host
8. Vscode: Paste the value of API Host from the Workbench Get API Token dialog (found in the User menu) at the prompt
9. Vscode: Next, run task Set Arvados Token
10. Vscode: Paste the value of API Token from the Workbench Get API Token dialog
11. Vscode: These will create files called API_HOST and API_TOKEN

3. Register & run a workflow

Vscode: Click on the lesson1/main.cwl file
1. Click on the Terminal menu
2. Click Run Task…
3. Select Register or update CWL workflow on Arvados Workbench
4. This will create a file called WORKFLOW_UUID
Workbench: Go to +NEW and select New project
1. Enter a name for the project like “Lesson 1”
2. You should arrive at the panel for the new project
Workbench: With Lesson 1 selected
1. Click on +NEW and select Run a process
2. Select CWL training lesson 1 from the list and click Next
3. Enter a name for this run like First training run
4. Enter a message (under #main/message) like “Hello world”
5. Click Run process
6. This should take you to a panel showing the workflow run status
Workbench: workflow run status panel
1. Wait for the badge in the upper right to say Completed
2. In the lower panel, double click on the echo workflow step
3. This will take you to the status panel for the echo step
4. Click on the three vertical dots in the top-right corner next to Completed
5. Choose Log
6. This will take you to the log viewer panel
7. Under Event Type choose stdout
8. You should see your message

4. Working with input and output files

Vscode: Click on the lesson2/main.cwl file
1. Click on the Terminal menu
2. Click Run Task…
3. Select Register or update CWL workflow on Arvados Workbench
Go to your desktop
1. Right click on the desktop, select New > Text Document
2. Name the file message
3. Enter a message like “Hello earth” and save
Workbench: Go to +NEW and select New project
1. Enter a name for the project like “Lesson 2”
2. You should arrive at the panel for the new project
Arvados workbench: With Lesson 2 project selected
1. Click on +NEW and select New collection
2. For Collection Name enter “my message”
3. Drag and drop message.txt into the browser
4. Click Create a collection
5. The file should be uploaded and then you will be on the collection page
Workbench: Select the Lesson 2 project
1. Click on +NEW and select Run a process
2. Select CWL training lesson 2 from the list and click Next
3. Enter a name for this run like “Second training run”
4. Click on #main/message
5. A selection dialog box will appear
6. Navigate to the collection you created in step (4.4.4) and choose message.txt
7. Click Run process
8. This should take you to a panel showing the workflow run status
Workbench: workflow run status panel
1. Wait for the process to complete
2. Click on the dot menu
3. Choose Outputs
4. Right click on reverse.txt
5. Click on Open in new tab
6. The results should be visible in a new browser tab.

5. Register a workflow with default inputs

The default value for the message parameter will taken from the lesson3/defaults.yaml file

Vscode: Click on the lesson3/main.cwl file
1. Click on the Terminal menu
2. Click Run Task…
3. Select Register or update CWL workflow on Arvados Workbench
Workbench: Go to +NEW and select New project
1. Enter a name for the project like “Lesson 3”
2. You should arrive at the panel for the new project
Workbench: With Lesson 3 selected
1. Click on +NEW and select Run a process
2. Select CWL training lesson 3 from the list and click Next
3. Enter a name for this run like “Third training run”
4. The #main/message parameter will be pre-filled with your default value. You can choose to change it or use the default.
5. Click Run process
6. This should take you to the status page for this workflow
7. The greeting will appear in the Log of the echo task, which can be found the same way as described earlier in section 3.

6. Run a workflow without registering it

The message parameter will be taken from the file lesson4/main-input.yaml. This is useful during development.

Workbench: Go to +NEW and select New project
1. Enter a name for the project like “Lesson 4”
2. You should arrive at the panel for the new project
3. Click on Additional info in the upper right to expand the info panel
4. Under Project UUID click the Copy to clipboard button
Vscode: Select the file lesson4/main.cwl
1. Click on the Terminal menu
2. Click Run Task…
3. Select Set Arvados project UUID
4. Paste the project UUID from workbench at the prompt
Vscode: Select the file lesson4/main.cwl
1. Click on the Terminal menu
2. Click Run Task…
3. Select Run CWL workflow on Arvados
Vscode: In the bottom panel select the Terminal tab
1. In the upper right corner of the Terminal tab select Task - Run CWL Workflow from the drop-down
2. Look for logging text like submitted container_request zzzzz-xvhdp-0123456789abcde
3. Highlight and copy the workflow identifier (this the string containing -xvhdp- in the middle)
4. The results of this run will appear in the terminal when the run completes.
Workbench: Paste the workflow identifier into the search box
1. This will take you to the status page for this workflow

Notes

If you need to change something about the environment of the user on the remote host (for example, the user has been added to a new unix group) you need to restart the vscode server that runs on the remote host. Do this in vscode:

ctrl+shift+p: Remote-SSH: Kill VS Code Server on Host

This is because the vscode server remains running on the remote host even after you disconnect, so exiting/restarting vscode on the desktop has no effect.

Previous: Getting Started with CWL Next: Working with container images