Workflows with Snakemake

University of Edinburgh

Online

8 - 10 February 2022

09:00 - 17:00

Instructors: Lana Talmane

Helpers: Tim Booth, Ezra Herman, Frances Turner

Overview

Workflows with Snakemake

Researchers needing to implement data analysis workflows face a number of common challenges, including the need to organise tasks, make effective use of compute resources, handle any errors in processing, and document and share their methods. The Snakemake workflow system provides effective solutions to these problems. By the end of the course, you will be confident in using Snakemake to run real workflows in your day-to-day research.

Snakemake workflows are described by special scripts that define steps in the workflow as rules, and these are then used by Snakemake to construct and execute a sequence of shell commands to yield the desired output. Re-calculation of existing results is avoided where possible, so you can add or update input data, then efficiently generate an updated result. Workflows can be seamlessly scaled to server, cluster, grid and cloud environments without the need to modify the workflow definition.

This course is primarily intended for researchers who need to automate data analysis tasks for biological research involving next-generation sequence data, for example RNA-seq analysis, variant calling, CHIP-Seq, bacterial genome assembly, etc. However, Snakemake has many uses beyond this and the course does not assume any specialist biological knowledge. The language used to write Snakemake workflows is Python-based, but no prior knowledge of Python is required or assumed either. We do require that attendees must have familiarity with using the Linux command line (pipes, redirects, variables, …).

Ed-DaSH

Ed-DaSH is a Data Science training programme for Health and Biosciences. The team has developed new lessons using The Carpentries platform. See workshops for dates and registration details. All Ed-DaSH workshops are delivered remotely.

General Information

Where: This training will take place online. The instructors will provide you with the information you will need to connect to this meeting.

When: 8 - 10 February 2022. Add to your Google Calendar.

Requirements: Participants must have access to a computer with a Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) that they have administrative privileges on. They should have a few specific software packages installed (listed below).

Accessibility: We are dedicated to providing a positive and accessible learning environment for all. Please notify the instructors in advance of the workshop if you require any accommodations or if there is anything we can do to make this workshop more accessible to you.

Contact: Please email to-be-announced for more information.

Roles: To learn more about the roles at the workshop (who will be doing what), refer to our Workshop FAQ.


Code of Conduct

Everyone who participates in Carpentries activities is required to conform to the Code of Conduct. This document also outlines how to report an incident if needed.


Collaborative Notes

We will use this collaborative document for chatting, taking notes, and sharing URLs and bits of code.


Surveys

Please be sure to complete the workshop survey (can be completed before you attend).

Ed-DaSH Survey


Schedule

The lesson taught in this workshop is being piloted and a precise schedule is yet to be established.

Day 1 : 8th February (09:15 - 17:00)

Lessons
Introduction to Snakemake Running commands with Snakemake
Placeholders and Wildcards
Chaining Rules
How Snakemake plans what jobs to run
Breaks
Morning 10:30 - 11:00
Lunch 12:30 - 13:30
Afternoon 15:00 - 15:30

Day 2 : 9th February (09:15 - 17:00)

Lessons
Introduction to Snakemake - Cont. Processing lists of inputs
Handling awkward programs
Configuration workflows
Optimising workflow performance
Breaks
Morning 10:30 - 11:00
Lunch 12:30 - 13:30
Afternoon 15:00 - 15:30

Day 3 : 10th February (09:15 - 17:00)

Lessons
Introduction to Snakemake Input functions
Conda integration
Cleaning up
Constructing a whole new workflow
Breaks
Morning 10:30 - 11:00
Lunch 12:30 - 13:30
Afternoon 15:00 - 15:30

Setup

To participate in a workshop, you will need access to software as described below. In addition, you will need an up-to-date web browser.

We maintain a list of common issues that occur during installation as a reference for instructors that may be useful on the Configuration Problems and Solutions wiki page.

Install the videoconferencing client

If you haven't used Zoom before, go to the official website to download and install the Zoom client for your computer.

Set up your workspace

Like other Carpentries workshops, you will be learning by "coding along" with the Instructors. To do this, you will need to have both the window for the tool you will be learning about (a terminal, RStudio, your web browser, etc..) and the window for the Zoom video conference client open. In order to see both at once, we recommend using one of the following set up options:

This blog post includes detailed information on how to set up your screen to follow along during the workshop.

Install a VNC client

The software you will be using is all pre-installed onto virtual machines, which you will access via a VNC (Virtual Network Computing) client. We recommend TigerVNC, available from sourceforge.net. For Windows users, the file is "vncviewer64-1.11.0.exe", and for Mac users you need "TigerVNC-1.11.0.dmg". Linux users may simply install tigervnc-viewer via the package manager. RealVNC VNC Viewer is another cross-platform client, which can be downloaded from realvnc.com.

Set-up notes for VNC

You will be provided with a connection address, user name and password at the start of the course. The following info may help with users of RealVNC if you are experiencing poor image quality. With TigerVNC, the default image settings are normally good.

If the resolution or colour quality of the RealVNC desktop looks poor and patchy, you can close the connection by clicking on the close icon at the top right of the VNC Viewer connection window, and then back in the VNC Viewer connections window, right-click on the connection icon and select "Properties..." from the context menu.

In the Properties window that opens, click the "Options" tab and change the following settings:

Click the "OK" button to confirm the changes, then re-initiate the VNC connection to the VM. The colour quality and clarity of the display should be much improved, and you can also maximise the VNC Viewer connection window to provide a larger interactive workspace by clicking on the maximise window button at the top right.