How to use the Pipelines tool
PreviousHow to create custom endpoint using bigquery: exampleNextHow to submit a pipeline from the command line (finngen-cli)
Last updated
Last updated
You can find the Pipelines in sandbox from Applications -> FinnGen -> Pipelines
It will open a new browser page with a view like this:
New front page:
Old front page:
You can submit a new job in 2 different ways from pipelines (under Create a new job):
You can submit a New job by copy+pasting your .wdl and .json files from New job button:
That will open open a new page where you need to copy-paste your wdl (workflow definition language) script (the easiest way to do this is to write it as a separate file, cat file and copy-paste) and press the ‘Validate’ button at the bottom of the wdl.
This will enable you to define your inputs.
You can either define the inputs manually or click the Input as JSON button and copy-paste your own .json file. If your inputs are ok and valid compared to wdl, you will see JSON validity: valid and you can press Accept. If your job has no Sub WLDs select Submit.
From Sandbox v.10.3 onwards, the Pipeline tool supports sub-workflow inputs. To include sub-workflow tick the Sub WDLs mark. Click Browse and select the sub.wld.zip file for your job. Then select Submit.
Your run will go either as the first on the list (if it goes directly to the ‘Running’ state). If there is a queue, it might go to the ‘Submitted’ state first, in which case you might not see it at the top right away. To monitor your run, it is good to save your jobs ‘ID’ [workflow_ID] (the long string of numbers and letters), and also the workflow_name of your job (this is the string defined in your wdl as ‘workflow’, for example, if your wdl’s first row would be workflow saige_pipeline_ps, ‘saige_pipeline_ps’ would be your job’s name).
For some of the most used pipelines, you can get the .wdl and example inputs (.json) directly from the standard pipelines.
You can also submit a new standard job by selecting New standard job, which will open a selection of implemented standard pipelines to use:
Currently standard pipeline includes:
Regenie pipeline (DF10)
GATE pipeline (DF6 and DF10)
Finemapping pipeline (DF7)
Conditional GWAS pipeline using regenie (DF10)
Example pipeline (hello wdl)
When chosen the standard pipeline you want to use, click Create. This will open a view with ready implemented .wdl, sub-wdl(s) and .json, which all you can edit before submitting the pipelines, from the button below the inputs:
You can also save your changes as draft for later (Save Draft), and then Load Draft later.
Open the Pipelines tool from the Sandbox Application menu, and click on the Submitted jobs on the right. You can click on either all jobs in your sandbox (Show recent jobs), or only Running, Succeeded or Failed jobs.
When you click on any of them, you'll see a list of jobs to match your selection. You can further filter listed jobs in the Search window. You can for example list only gate/regenie/saige runs, only succeeded runs, or by your WORKFLOW_ID. Remember to press '+' sign to add the filter.
Clicking one of the jobs will open the information page of that run. On the sub-pages, you can see Info, WLD, sub-WDLs, and Calls & Steps information for the run. Save your Pipeline ID as you may need it later.
When/if your job (standard or non-standatd) is successfully done (state is Succeeded), the state of your run changes to ‘Succeeded’, and you can check your results from:
/finngen/pipeline/cromwell/workflows/[workflow_name]/[workflow_ID]
Related:
