By default, Nextflow will run locally on the node you are on. It can also be configured to send each task as a SLURM job submission.
Add the following lines to your ~/.bashrc.
# ~/.bashrc
export AWS_DEFAULT_REGION="us-east-1" # or your default region
export NXF_ANSI_LOG="False"
export NXF_CONDA_CACHEDIR="${HOME}/.conda/envs"
export NXF_SINGULARITY_CACHEDIR="${HOME}/.singularity"
module load Miniconda3
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --set channel_priority strict
conda config --add create_default_packages ipykernel
conda config --add create_default_packages ipython
Nextflow can work with multiple execution backends. It can run locally on a node/workstation/laptop, or spread tasks out on a cluster.
Let's configure to use Nextflow as its execution backend.
module load pcluster-helpers
pcluster-helper --help
Usage: pcluster-helper [OPTIONS] COMMAND [ARGS]...
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --install-completion [bash|zsh|fish|powershe Install completion for │
│ ll|pwsh] the specified shell. │
│ [default: None] │
│ --show-completion [bash|zsh|fish|powershe Show completion for the │
│ ll|pwsh] specified shell, to │
│ copy it or customize │
│ the installation. │
│ [default: None] │
│ --help Show this message and │
│ exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ gen-nxf-slurm-config Generate a slurm.config for nextflow that is │
│ compatible with your cluster. │
│ sinfo A more helpful sinfo │
╰──────────────────────────────────────────────────────────────────────────────╯
The pcluster-helper library has utilities to translate the cluster configuration to a configuration format Nextflow understands.
pcluster-helper \
gen-nxf-slurm-config \
--include-memory \
--output ~/slurm.config \
--overwrite
Now, open up your ~/slurm.config and give it a once over. Make sure the nodes the default process_low, process_medium, and process_high labels are nodes you want to commonly use, and not, for example GPU nodes.