posts

Snakemake has great support for most HPC systems, via custom submission command you can specify in profile and it works amazingly most of the time. However, occasionally it would fail to correctly recognize job status in some edge cases.

Note: This is an update to my previous post: How to Run Snakemake pipeline on HPC.

In my previous post, I disucessed some tips on how to effectively manage workflow using Snakemake on an HPC system. However, I have recently noticed that Snakemake support for --cluster-config is offcially deprecated in favor of --profile. I spent most of today digging into this feature and now I’m happy to share with you my latest setup.

I was assisting a bioinformatics workshop for FAANG group at Plant and Animal Genome (PAG) and I ran into an interesting question with an example provided by one of the instructors on the topic of Linux output redirecting. Take a look at this simplified example script print_stderr.sh:

For years I have been sticking with R and RStudio, primarily because RStudio’s useful and user-friendly GUI design (and my own comfort with Tidyverse). Indeed, my biggest complain against Jupyter Notebook whenever somebody introduced it to me was its bare-bone functionality. Until I discovered Jupyter Lab!

Snakemake is a handy workflow manager written in Python. It handles workflow based on predefined job dependencies. One of the great features of Snakemake is that it can manage a workflow on both a standalone computer, or a clustered HPC system. HPC, or “cluster” as it’s often referred to, requires additional considerations.

If you have ever written a scientific paper in MS Word (or any other word processing software) then you’re probably experienced frustrations like me: Word constantly crashing when loaded with too many high-resolution figures, citations getting messed up when working with others that use different citation managers, or hard to keep track of hundreds of versions of the draft, etc.

NASA uploads a selected astronomy picture daily as their Astronomy Picture of the Day (APOD). For a space sci-fi junkie myself, I find these pictures extremely pretty and I want to use them as my wallpapers! But obviously downloading them from NASA everyday is a huge hassle. Luckily, NASA provides open APIs that we can make use of to automate this process.