How to submit a pipeline from the command line (finngen-cli)
Last updated
Last updated
The other way to submit a pipeline, besides the Pipeline tool, is by command line. Finngen-cli is a command line tool to run FinnGen applications in Sandbox. The command line is accessible through Terminal Emulator from the Application menu in the Sandbox.
!! NB !! Please be cautious with how many GWAS you create and the number of phenotypes you include. Submitting more than ten GWAS jobs simultaneously or GWAS with more than 15 phenotypes may jam the process and can make your organization's pipeline unusable for others. If you are going to launch more than 5 GWASs or GWAS with tens of phenotypes please contact the humgen-servicedesk@helsinki.fi and we can temporarily increase the resources of your organization's Sandbox and downscale afterward.
To open the finngen-cli help page and to list all finngen-cli commands in Terminal Emulator type
To see the help page for each command type
In Sandbox any pipeline can be run using the finngen-cli request-workflow. You just need to provide wdl, json, and sub-wdl files packed in a zip. Pipeline input files are run from /finngen/red/ folder.
For example, to run a workflow with user defined WDL, JSON, and dependencies:
You will get the following notification if the workflow request creation succeeded
Note: If your WDL uses sub-workflow, this is the only way to submit your job. If you are using this, your WDL and JSON needs to be in /finngen/red. You can import the WDL and JSON using the green bucket (eg. from the git repository). See instructions on how to upload files to your own IVM via /finngen/green.
The third dependence is the Docker images used by the WDL. You can check what is currently available in Sandbox container repository using finngen-cli list-images.
After the workflow is submitted successfully you can check the status of your run from the Pipeline tool menu (Applications -> Pipelines)
The finngen-cli request-gwas is an automatic system where you don’t need to handle the input JSON or WDL but only the phenofile or lists of phenotypes. Detailed instructions on how to run finngen-cli request-gwas are in Custom GWAS command line (CLI) tool. See also How to run SAIGE (GWAS). Note that the status of workflows submitted with command finngen-cli request-gwas can be checked from the "RECENTS" menu in the custom GWAS command line (CLI) tool, not from FinnGen Pipeline tools.
See tutorial video on how to run pipelines in Terminal:
