Snakemake and Slurm: How to Manage Workflow with Resource Constraint on HPC

Note: This is an update to my previous post: How to Run Snakemake pipeline on HPC. In my previous post, I disucessed some tips on how to effectively manage workflow using Snakemake on an HPC system. However, I have recently noticed that Snakemake support for --cluster-config is offcially deprecated in favor of --profile. I spent most of today digging into this feature and now I’m happy to share with you my latest setup.

/dev/stderr vs >&2: a dive into Linux STDIO

I was assisting a bioinformatics workshop for FAANG group at Plant and Animal Genome (PAG) and I ran into an interesting question with an example provided by one of the instructors on the topic of Linux output redirecting. Take a look at this simplified example script #!/usr/bin/bash echo "first line of stderr" > /dev/stderr echo "second line of stderr" > /dev/stderr echo "third line of stderr" > /dev/stderr echo "A line of stdout" Now if we redirect the stderr of this script to a file, instead of all three lines of stderr messages, we only see the last line!