alevin-fry-tutorials

Processing 10x Flex Gene Expression data with simpleaf

Thu, 19 Mar 2026 00:00:00 +0000

Simpleaf is a command line toolkit written in rust that exposes a unified, simplified interface for processing single-cell and single-nucleus sequencing data using the alevin-fry pipeline. In this tutorial, you will learn how to use simpleaf multiplex-quant to process 10x Genomics Flex Gene Expression (Fixed RNA Profiling) data.

What is 10x Flex?

10x Chromium Fixed RNA Profiling (commonly called “Flex”) is a probe-based single-cell gene expression assay designed for FFPE (formalin-fixed, paraffin-embedded) and fixed frozen samples. Unlike standard 3’ or 5’ scRNA-seq, which sequences poly-A captured mRNA fragments against a full transcriptome reference, Flex uses a panel of short (~50bp) probes that hybridize directly to target transcripts.

A key feature of Flex is sample multiplexing: each sample in a multiplexed experiment is labeled with a unique probe barcode (8bp for Flex v1, 10bp for Flex v2), allowing multiple samples to be pooled in a single GEM well and sequenced together. Each read therefore carries two barcodes: a cell barcode (16bp, identifying the cell) and a sample/probe barcode (identifying which sample the cell belongs to). The simpleaf Flex pipeline handles demultiplexing, barcode correction, and quantification in a single command.

Installation

The recommended way to install simpleaf is via conda:

conda create -n simpleaf -y -c bioconda -c conda-forge simpleaf
conda activate simpleaf

This tutorial requires simpleaf version 0.22.0 or later, which includes the multiplex-quant command. (In earlier versions, this command was called flex-quant.)

Configuration

Simpleaf requires the ALEVIN_FRY_HOME environment variable to point to a directory where it stores configuration and cached resources.

# Create a working directory
AF_SAMPLE_DIR=$PWD/flex_tutorial
mkdir -p $AF_SAMPLE_DIR
cd $AF_SAMPLE_DIR

# Set the required environment variable
export ALEVIN_FRY_HOME="$AF_SAMPLE_DIR/af_home"

# Configure simpleaf (discovers piscem, alevin-fry, salmon in your PATH)
simpleaf set-paths

# Increase open file limit for piscem's intermediate files
ulimit -n 2048

Downloading the test data

For this tutorial, we use the 4-plex Human Colorectal Cancer and Normal Adjacent Colon dataset from 10x Genomics. This is a 4-plex Flex v1 experiment containing four samples:

Sample	Tissue	Prep	Probe BC
BC001	Colorectal	Manual	ACTTTAGG
BC002	Colorectal	Octo	AACGGGAA
BC003	Kidney	Manual	AGTAGGCT
BC004	Kidney	Octo	ATGTTGAC

The compressed FASTQ files are approximately 25GB. Please ensure you have about 50GB of free disk space to complete this tutorial (FASTQs + index + output).

# Download the FASTQ files
FASTQ_DIR="$AF_SAMPLE_DIR/data/fastqs"
mkdir -p $FASTQ_DIR

wget -qO- https://cf.10xgenomics.com/samples/cell-exp/9.0.0/4plex_human_colorectal_kidney_scFFPE_multiplex/4plex_human_colorectal_kidney_scFFPE_multiplex_fastqs.tar \
  | tar xf - --strip-components=1 -C $FASTQ_DIR

Running simpleaf multiplex-quant

The simpleaf multiplex-quant command handles the entire pipeline in a single invocation: building a probe index, mapping reads, generating the permit list with barcode correction, collating, and quantifying.

First, gather the FASTQ file paths:

# Collect R1 and R2 files (comma-separated, sorted)
reads1_pat="_R1_"
reads2_pat="_R2_"

reads1="$(find -L ${FASTQ_DIR} -name "*${reads1_pat}*" -type f | sort | paste -sd, -)"
reads2="$(find -L ${FASTQ_DIR} -name "*${reads2_pat}*" -type f | sort | paste -sd, -)"

Now run the pipeline:

simpleaf multiplex-quant \
  --chemistry 10x-flexv1-gex-3p \
  --organism human \
  --reads1 $reads1 \
  --reads2 $reads2 \
  --output $AF_SAMPLE_DIR/flex_output \
  -t 8

Here is what each argument does:

--chemistry 10x-flexv1-gex-3p selects the Flex v1 protocol. This tells simpleaf the read geometry (1{b[16]u[12]x:}2{r[50]x[18]s[8]x:}), and enables automatic fetching of the cell barcode whitelist, sample barcode list (with rotation collapsing), and probe set.
--organism human selects the human probe set (Chromium_Human_Transcriptome_Probe_Set_v1.1.0_GRCh38-2024-A). Use mouse for the mouse probe set.
--reads1 / --reads2 are comma-separated lists of R1 and R2 FASTQ files.
--output is the output directory.
-t 8 uses 8 threads.

What happens under the hood

The multiplex-quant command executes the following steps:

Probe set conversion: Downloads the probe set CSV from the 10x registry, converts it to a FASTA reference file and a transcript-to-gene (t2g) mapping.
Index building: Builds a piscem index from the probe FASTA (k=23 by default).
Read mapping: Maps reads with piscem using the Flex geometry, which extracts the cell barcode (R1:1-16), UMI (R1:17-28), biological sequence (R2:1-50), and sample barcode (R2:69-76).
Permit list generation: Loads the 737K cell barcode whitelist, counts cell barcodes per sample (only whitelist matches), applies a minimum read threshold, and rescues non-matching barcodes via 1-edit-distance correction.
Collation: Groups records by (sample, cell barcode) for efficient per-cell processing.
Quantification: Resolves UMIs per gene using the cr-like algorithm, producing a combined count matrix for all samples.

On the 4-plex dataset with 8 threads, expect the pipeline to take approximately 5-10 minutes (depending on hardware), with mapping being the longest step.

Understanding the output

The output directory has the following structure:

flex_output/
  af_map/                  # Mapping output (RAD file)
  af_quant/
    alevin/
      quants_mat.mtx       # Count matrix (Matrix Market format)
      quants_mat_rows.txt  # Barcode labels (SAMPLE_cellbarcode)
      quants_mat_cols.txt  # Gene IDs
    featureDump.txt        # Per-cell QC metrics with sample_name column
    collation_manifest.bin # Sample-to-chunk mapping
    ...
  probe_conversion/        # Probe FASTA, t2g map, metadata
  simpleaf_multiplex_quant_info.json  # Pipeline metadata and timing

The count matrix contains all samples combined. Each barcode in quants_mat_rows.txt is prefixed with the sample name:

BC001_GGAGCGAAGCGAATCG
BC001_GCATGCTTCGGCTAGA
...
BC002_GGGAGTGAGGAAGGTA
...

The featureDump.txt file includes a sample_name column for easy per-sample filtering:

CB              sample_name  CorrectedReads  MappedReads  ...
GGAGCGAAGCGAATCG  BC001       551890          551890      ...

You can quickly check how many cells were found per sample:

awk -F'_' '{print $1}' flex_output/af_quant/alevin/quants_mat_rows.txt | sort | uniq -c | sort -rn

Using a pre-built index

If you have already built a probe index (or want to use a custom probe set or sample barcode list), you can skip the automatic resource fetching:

simpleaf multiplex-quant \
  --chemistry 10x-flexv1-gex-3p \
  --organism human \
  --index /path/to/probe_index/index \
  --probe-set /path/to/probe_set.csv \
  --sample-bc-list /path/to/sample_barcodes.txt \
  --reads1 $reads1 \
  --reads2 $reads2 \
  --output $AF_SAMPLE_DIR/flex_output_custom \
  -t 8

The --index flag points to the piscem index prefix (not the directory). The --probe-set provides a probe set CSV (which is also used to generate the t2g mapping). The --sample-bc-list is a 3-column TSV mapping observed probe barcodes to canonical barcodes and sample names (accounting for barcode rotation).

Splitting into per-sample matrices

The combined matrix is convenient for joint analysis, but many downstream workflows expect one matrix per sample. We provide a helper script that splits the combined output:

# Download the script
wget -qO split_flex_samples.py \
  https://combine-lab.github.io/alevin-fry-tutorials/scripts/split_flex_samples.py

# Split into per-sample directories
python split_flex_samples.py \
  --input flex_output/af_quant/alevin \
  --output per_sample_matrices

This creates one subdirectory per sample, each containing its own quants_mat.mtx, quants_mat_rows.txt (with plain cell barcodes, no sample prefix), and quants_mat_cols.txt.

A note on sample counts: Because alevin-fry processes the raw (unfiltered) permit list, the output includes all cell barcodes passing the minimum read threshold — not just those from the expected multiplexed samples. For this 4-plex dataset, you will see output directories for more than 4 sample barcode groups (up to 15 or so). This is expected: the Flex probe barcode panel contains 16 BC codes and 16 AB codes, and low levels of background or index-hopping signal can produce a small number of barcodes assigned to non-target sample groups.

The four true samples — BC001 (~165K cells), BC002 (~174K cells), BC003 (~112K cells), and BC004 (~154K cells) — dominate overwhelmingly. These correspond to the four samples for which 10x provides CellRanger output. The remaining sample groups typically contain only a handful of cells (often fewer than 100) and can be safely filtered out in downstream analysis. Providing the raw output allows users to apply their own filtering criteria and to inspect cross-sample contamination rates if desired.

Loading results in Python

Here is how to load the combined matrix in scanpy and annotate cells with their sample identity:

import scanpy as sc
import pandas as pd

# Read the combined count matrix
adata = sc.read_mtx("flex_output/af_quant/alevin/quants_mat.mtx").T

# Load gene and barcode names
genes = pd.read_csv("flex_output/af_quant/alevin/quants_mat_cols.txt",
                     header=None)[0].values
barcodes = pd.read_csv("flex_output/af_quant/alevin/quants_mat_rows.txt",
                        header=None)[0].values

adata.var_names = genes
adata.obs_names = barcodes

# Parse sample identity from barcode prefixes
adata.obs["sample"] = [bc.split("_")[0] for bc in adata.obs_names]
adata.obs["cell_barcode"] = [bc.split("_", 1)[1] for bc in adata.obs_names]

# Basic QC filtering
sc.pp.filter_cells(adata, min_counts=50)

print(adata)
print(adata.obs["sample"].value_counts())

If you prefer to work with individual sample matrices, use the split_flex_samples.py script from the previous section, then load each sample separately with sc.read_mtx().

Conclusion

In this tutorial, you learned how to process 10x Flex Gene Expression data using the simpleaf multiplex-quant command. The pipeline handles the complete workflow from FASTQs to a combined, sample-annotated count matrix in a single command, including probe barcode demultiplexing, cell barcode correction against the 10x whitelist, and UMI-level gene quantification.

For more information, see:

Generating a spatial transcriptomics count matrix with simpleaf

Mon, 30 Dec 2024 00:00:00 +0000

Simpleaf is a command line toolkit written in rust that exposes a unified, simplified interface for processing scRNA-seq datasets using the alevin-fry pipeline.

In this tutorial, you will learn how to use simpleaf to process spatial transcriptomics data. You can find more information about the available commands in the official simpleaf documentation. If you are new to alevin-fry and simpleaf or want to learn about processing scRNA-seq data, check out the other tutorials.

Installation

The easiest way to get started with simpleaf is by installing it in a conda environment:

conda create -n simpleaf -y -c bioconda -c conda-forge simpleaf
conda activate simpleaf

Configuration

simpleaf needs the environment variable ALEVIN_FRY_HOME to point to a directory where it can store information. Create a working directory and set the environment variable like so:

# Make a working directory
AF_SAMPLE_DIR=$PWD/af_test_workdir
mkdir $AF_SAMPLE_DIR
cd $AF_SAMPLE_DIR

# Define the required env variable
export ALEVIN_FRY_HOME="$AF_SAMPLE_DIR/af_home"

# Simpleaf configuration
simpleaf set-paths

# Increase the number of open files for piscem
ulimit -n 2048

If you use simpleaf often, you can choose to set it as a folder in your home folder and adding ALEVIN_FRY_HOME to your profile file to set up the environment automatically. For example, if you use bash, you can add the following lines to your ~/.bashrc file:

echo 'export ALEVIN_FRY_HOME="$HOME/af_home"' >> ~/.bashrc

Downloading the data

For this tutorial, we will download a small spatial transcriptomics dataset from the 10x Genomics website. The dataset is generated using a 10X Visium V5 slide and the Human Whole Transcriptome probe set. Note that although this is one of the smallest visium datasets, the compressed FASTQ tar ball is still 12GB. Please make sure you have about 40 GB free disk space to complete this tutorial.

First, we download the FASTQ files.

# Download the fastq files
DATA_DIR="$AF_SAMPLE_DIR/data"
FASTQ_DIR="$DATA_DIR/CytAssist_FFPE_Human_Colon_Post_Xenium_Rep1_fastqs"
mkdir -p $FASTQ_DIR

wget -qO- https://cf.10xgenomics.com/samples/spatial-exp/2.1.0/CytAssist_FFPE_Human_Colon_Post_Xenium_Rep1/CytAssist_FFPE_Human_Colon_Post_Xenium_Rep1_fastqs.tar \
  | tar xf - --strip-components=1 -C $FASTQ_DIR

Second, we download the probe set file and generate a FASTA file and a transcript-to-gene mapping file from it.

PROBES_FILE="$DATA_DIR/CytAssist_FFPE_Human_Colon_Post_Xenium_Rep1_probe_set.csv"

# Download the probe set file
wget -qO $PROBES_FILE https://cf.10xgenomics.com/samples/spatial-exp/2.1.0/CytAssist_FFPE_Human_Colon_Post_Xenium_Rep1/CytAssist_FFPE_Human_Colon_Post_Xenium_Rep1_probe_set.csv

Running simpleaf index

Using the fetchedprobe_set.csv, we can now run the simpleaf index command to generate a reference index. Because piscem requires a working directory for storing intermediate files that need to be read and written frequently, if you are on a cloud computing platform, you might want to set --work-dir to a scratch folder (e.g., /tmp/workdir.noindex) to avoid high I/O burden. The reason is that, in cloud computing, CPUs and storage are often separate resources, and I/O speed can be much slower than local storage.

simpleaf index --probe-csv $PROBES_FILE --work-dir ./workdir.noindex --output simpleaf_index

The resulting index will be stored in the simpleaf_index/index directory. We will use this index in the next step to generate the count matrix.

Running simpleaf quant

Next, run the simpleaf quant command to generate the count matrix. The --output option specifies the output directory for the count matrix. The --andata-out flag tells simpleaf to also output the count matrix in the AnnData format, which is a common format for storing single-cell data.

# Define the reads1 and reads2 patterns
reads1_pat="_R1_"
reads2_pat="_R2_"

# Obtain and sort filenames
reads1="$(find -L ${FASTQ_DIR} -name "*$reads1_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -)"
reads2="$(find -L ${FASTQ_DIR} -name "*$reads2_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -)"

# Run simpleaf quant
simpleaf quant --chemistry visiumv5-probe \
               --unfiltered-pl \
               --reads1 $reads1 \
               --reads2 $reads2 \
               --index simpleaf_index/index \
               --output simpleaf_quant \
               --resolution cr-like \
               --anndata-out

The resulting count matrix will be stored in the simpleaf_quant/af_quant/alevin directory, and you can use it for downstream analysis. The quants.h5ad file contains the count matrix in the AnnData format, and all affiliated information you need to start your analysis. If you want to start with the matrix file, quants_mat.mtx, you can find the rownames (cell barcodee and tbhe corresponding xy coordinate) in quants_mat_rows.txt, and gene IDs in the quants_mat_cols.txt . The first few lines in the quants_mat_rows.txt file look like this:

GGCGCCTACGAATGAA        69      83
CAATCTGCAATGAACA        36      50
CGCTGTATCCGGTTCT        80      102
GTCTCCATTAGATGAA        37      53
GACTGATTGGCCATGT        38      48

Conclusion

Because simpleaf supports permit lists with auxiliary information, you can easily use simpleaf to process sequencing-based spatial transcriptomics data, including 10X Visium assays and Stereo-seq.

Operating on the Registry of Custom Chemistries in Simpleaf

Mon, 30 Dec 2024 00:00:00 +0000

Simpleaf is a command line toolkit written in rust that exposes a unified and simplified interface for processing scRNA-seq datasets using the alevin-fry pipeline.

In this tutorial, we will focus on the simpleaf chemistry commands for operations on the registry of simpleaf’s custom chemistries. The detailed description of each command can be found at simpleaf’s online documentation. For more information about using alevin-fry and simpleaf to process single-cell data, including spatial transcriptomics data, please refer to other tutorials.

Installation

The recommended way of using simpleaf is from a conda environment.

# Remove piscem if on Apple silicon
conda create -n simpleaf -y -c bioconda -c conda-forge simpleaf
conda activate simpleaf

Configuration

simpleaf requires the environment variable ALEVIN_FRY_HOME to be set to a directory where it can cache useful information. To configure simpleaf, please run the following shell commands.

# make a working directory
AF_SAMPLE_DIR=$PWD/af_test_workdir
mkdir $AF_SAMPLE_DIR
cd $AF_SAMPLE_DIR

# define the required env variable
export ALEVIN_FRY_HOME="$AF_SAMPLE_DIR/af_home"
# simpleaf configuration
simpleaf set-paths

Custom Chemistries in Simpleaf

In the single-cell world, a “chemistry” is defined as the library configuration of a single-cell assay, i.e., the position of the cell barcode, unique molecule identifier, and biological sequence relative to the sequenced reads. Single-cell protocols usually have their own unique chemistries. If you are unsure about the chemistry of your data, the Single Cell Genomics Library Structure repository describes in detail the chemistries of dozens of single-cell assays.

Chemistry definitions are essential for simpleaf to process any single-cell data. In simpleaf’s quantification stage, there are three types of chemistry definitions that the chemistry argument, --chemistry, can take:

the name of an internally defined chemistry,
a custom geometry description, or
the name of a registered custom geometry.

Internally, simpleaf only defines a few popular chemistries, such as

10xv2: the 10x Genomics Chromium 3’ v2 protocol
10xv3: the 10x Genomics Chromium 3’ v3 protocol
10xv4: the 10x Genomics Chromium 3’ v4 protocol
visiumv1: the 10x Genomics Visium Spatial Gene Expression Slide (v1, 6.5 mm)
visiumv1-probe: Same as visiumv1, but using a Visium probe set to target protein-coding genes instead of priming the polyA stretches of arbitrary RNAs
visiumv4: the 10x Genomics Visium Spatial Gene Expression Slide (v2, 6.5 mm)
visiumv4-probe: Same as visiumv4, but using a Visium probe set
visiumv5: the 10x Genomics Visium Spatial Gene Expression Slide (v2, 11 mm)
visiumv5-probe: Same as visiumv5, but using a Visium probe set

However, users can flexibly define and register any custom chemistries with the help of the simpleaf chemistry commands, and the Sequence Fragment Geometry Description Language. In the following text, we will show how to define and register a custom chemistry for the visiumv1 protocol.

Define the geometry description for custom chemistries

The most important part of defining a custom chemistry is to describe the geometry of the single-cell assay. In simpleaf, this is done by writing a geometry description in the Sequence Fragment Geometry Description Language. At a high level:

b[N]: barcode of length N
u[N]: UMI of length N
r: or r[N]: biological sequence, either everything left (r:) or a fixed number of bases (for example, r[50])
f[...]: fixed sequence (e.g., an adapter or known base sequence)
x: or x[N]: the sequence to discard, either everything left (x:) or a fixed number of bases (for example, x[10])

Here we show examples for writing the geometry description for some 10x Genomics protocols.

10X Chromium 3’ 3 protocol

Let’s start with a widely used protocol, 10x Genomics Chromium 3’ v3 protocol, 10xv3. A geometry usually consists of the following components: the barcode definition (b), the UMI definition (u), and the biological sequence definition. Sometimes we need to define some fixed fragment sequences (f) or the sequence that should be excluded (x).

When describing a geometry, we first need to figure out which component is at what position on which read, and what’s the length of each component. For reads from the 10x Genomics Chromium 3’ v3 protocol, the cell barcode is 16 bp long and is at the beginning of read1. The UMI is 12 bp long and is adjacent to the cell barcode. The entire read2 represents a cDNA fragment, i.e., the biological sequence.

Next, we want to assemble the information into a geometry description. We use {} to denote the content of each read, and use [] to denote the length of each component or : to denote “anything left.” Combining all these together, for the 10x Genomics Chromium 3’ v2 protocol (as described here), the geometry description is as follows: 1{b[16]u[12]x:}2{r:}, where

1{} and 2{} denote read1 and read2, respectively.
b[16] denotes the barcode takes the first 16 bases of read1.
u[12] denotes the UMI takes the next 12 bases after the barcode.
x: denotes anything left in read1 should be discarded. If using the recommended sequencing format, i.e., read1s are sequenced for 28 cycles (barcode+UMI), we can ignore the x: part, because the entire read1 is used for cell barcode and UMI.
r: denotes anything left in read2 should be treated as biological sequence. In this case, as it appears at the beginning of read2, it means the entire read2 is the biological sequence.

10X Chromium 5’ v3 protocol

The 10X Chromium 5’ v3 protocol, 10xv3-5p, as described here, does a round of reverse transcription at the beginning of library preparation. Regarding the geometry description, if using the recommended sequencing format, i.e., read1s are sequenced for 28 cycles, the geometry description is the same as 10xv3: 1{b[16]u[12]}2{r:}.

However, when read1s and read2s are sequenced for the same number of cycles, such as 91 or 150, and we want to utilize the extra bases in read1s during mapping, we need to consider the bases after the UMI in read1. Specifically, after the UMI sequence, there is a fixed fragment sequence, AAAGAATATACCC, and the rest of read1 is a biological sequence representing the 5’ end of the RNA. In this case, the geometry description should be 1{b[16]u[12]f[AAAGAATATACCC]r:}2{r:}, where the f[] denotes the fixed fragment sequence, and the r: in the read1 description denotes the biological sequence in read1. Interestingly, because there is a biological sequence in both read1 and read2, if we want to use the extra bases in read1s, we need to carefully define the direction, i.e., the expected orientation, of the reads. We will explain this later in detail in the expected orientation section.

In summary, if we want to follow the “default” 10X way to process 10xv3-5p data, we can use the same geometry description as 10xv3, 1{b[16]u[12]}2{r:}. If we want to utilize the extra bases in read1s, we should use 1{b[16]u[12]f[AAAGAATATACCC]r:}2{r:}, together with a carefully set direction.

10X Visium protocols

There are two types of reads that can be generated using the 10X Visium V1 protocol, visiumv1. The first type is the same as 10xv3, by priming the polyA stretches of arbitrary RNAs. The geometry description for this type of reads is 1{b[16]u[12]}2{r:}. The second type is generated by using a Visium probe set to target protein-coding genes. Because all probes are 50 bp in length, we only take the first 50 bases in read2 as the biological sequence and discard the rest. Therefore, the geometry description for this type of reads is 1{b[16]u[12]}2{r[50]}.

Expected Orientation

In the above examples, we have mentioned the expected orientation of the reads. The expected orientation, --expected-ori or -e, is a parameter that tells simpleaf, in the quantification stage, which read directions should be used to generate gene counts. It can be set to fw (forward), rc (reverse complement), or both (both forward and reverse complement). It describes the direction relative to the first (most upstream) mappable biological sequence.

For example, for a read from 10xv3 or a 10X Visium protocol, because its read2 is generated by sequencing the cDNA strand that is reversely transcribed from the corresponding mRNA (i.e., read2 is in the same direction as the mRNA), and it is the first (and only) mappable sequence, we should set the expected orientation to forward, fw.

For a read from 10xv3-5p, because its read2 is generated by sequencing the cDNA strand that is in the same direction as the mRNA, when we use only the read2 for mapping, we should set the expected orientation to reverse complement, rc. However, if we have extra bases in read1 and we want to use it for mapping, because the biological sequence in read1 is in the same direction as the mRNA (the reverse complement of read2), we should still set the expected orientation to forward, fw, to reflect the direction of the first mappable sequence, the biological sequence in read1.

Barcode Permit List

In the quantification stage, simpleaf can take a barcode permit list to correct observed barcodes against the permitted barcodes in the list. This cell barcode correction strategy is recommended for all protocols that have a permit list.

A barcode permit list is a tab-separated file without a header, where each line contains a permitted barcode in the first column and, optionally, some auxiliary information in the following columns. Usually, the permit list file for single-cell RNA-seq assays has a single column for the permitted barcode.

The permit list files for spatial transcriptomics assays (often called “coordinate files”) usually have two additional columns for x and y coordinates of the spot. For example, the following is a snippet of a permit list file for the 10X Genomics Visium Spatial Gene Expression Slide (v1, 6.5 mm) protocol:

AAACAACGAATAGTTC	17	1
AAACAAGTATCTCCCA	103	51
AAACAATCTACTAGCA	44	4
AAACACCAATAACTGC	20	60
AAACAGAGCGACTCCT	95	15
AAACAGCTTTCAGAAG	10	44
AAACAGGGTCTATATT	14	48
AAACAGTGTTCCTGGG	44	74
AAACATGGTGAGAGGA	1	63
AAACATTTCCCGGATT	98	62
AAACCACTACACAGAT	118	4
AAACCCGAACGAAATC	116	46
AAACCGGAAATGTTAA	125	55

Simpleaf Chemistry Commands

The chemistry command provides functionality to manage and inspect custom chemistries in simpleaf’s registry of recognized custom chemistries. It supports the following operations:

Add new custom chemistries.
Remove existing custom chemistries.
Add or refresh chemistry definitions from the upstream repository.
Lookup details of a specific chemistry.
Download corresponding permit lists for chemistries.
Search for unused permit lists and remove them from the cache.

Register a Custom Chemistry in Simpleaf

Once we figure out the geometry description, the expected orientation of the reads, and the permit list information of our custom chemistry, we can register it in simpleaf using the simpleaf chemistry add command.

Before we start, it is a good idea to refresh the local registry of chemistries, in case our local one is outdated.

simpleaf chemistry add \
--name 10xv3-test \
--geometry "1{b[16]u[12]}2{r:}" \
--expected-ori fw \
--remote-url "https://umd.box.com/shared/static/vc9zd4qyjj581gvtolw5kj638wmg4f3s" \
--version 0.1.0

In the above command, we have specified the name of the custom chemistry 10xv3-test, the geometry description 1{b[16]u[12]}2{r:}, and the expected orientation fw. We also versioned the chemistry as 0.1.0. If your chemistry has a corresponding permit list file and you want simpleaf to use it, there are two ways to tell simpleaf where to find the permit list file. One is to specify the --remote-url argument, which is a remote URL to download the permit list file. The other is to specify the --local-url argument, which is a path to the permit list file on the local machine. If either of these two arguments is specified, simpleaf will download or copy the permit list file to the defined ALEVIN_FRY_HOME directory. This also means that if your chemistry has an extra large permit list file, such as those generated from the mask files of Stereo-seq using their ST_BarcodeMap tool, you might want to pass them directly to the quantification command to avoid this copying step.

Update a Custom Chemistry in Simpleaf

Note that the versioning we use in simpleaf chemistry is for the custom chemistry definition itself, not for the version or revision of the protocol (like 10x Genomics Chromium 3’ V2 and V3). The purpose of versioning is to allow users to update the chemistry definition by providing a newer version number, for example, 0.2.0. In the following command, we update the geometry description of the chemistry 10xv3-test we defined above to add the x: part for read1. Please be sure to provide all information when you update the chemistry definition, including the permit list file, if any.

simpleaf chemistry add \
--name 10xv3-test \
--geometry "1{b[16]u[12]x:}2{r:}" \
--expected-ori fw \
--remote-url "https://umd.box.com/shared/static/vc9zd4qyjj581gvtolw5kj638wmg4f3s" \
--version 0.2.0

Remove Custom Chemistries in Simpleaf

If we want to remove a custom chemistry from the registry, we can use the simpleaf chemistry remove command. We just need to provide the name of the custom chemistry we want to remove, or a regex pattern that can match the chemistries’ names. If we are unsure about the behavior of the regex pattern, we can use the --dry-run argument to see which chemistries will be removed.

simpleaf chemistry remove --name 10xv3-test --dry-run

Once we are sure about the removal, we can remove the custom chemistry by removing the --dry-run argument.

simpleaf chemistry remove --name 10xv3-test

Look Up Custom Chemistries in Simpleaf

If we want to look up the details of a custom chemistry, we can use the simpleaf chemistry info command. We just need to provide the name of the custom chemistry we want to look up.

simpleaf chemistry lookup --name 10xv3-test

The --name argument can also take a regex pattern that can match the names of chemistries if we want to look up multiple chemistries at once. For example, if we want to look up all chemistries, just do .*

simpleaf chemistry lookup --name ".*"

You might notice that, for some internally defined chemistries, their geometry description is __builtin, which means they are built-in chemistries in simpleaf and their details are not stored in the registry.

Fetch the Permit List of Custom Chemistries in Simpleaf

If we want to fetch the permit list of a custom chemistry, we can use the simpleaf chemistry fetch command. We just need to provide the name of the custom chemistry from which we want to fetch the permit list. For example, if we want to fetch the permit list of all chemistries that have a defined remote permit list, we can use the following command.

simpleaf chemistry fetch --name ".*"

You might notice that for some chemistries, simpleaf will report that their permit list exists, even if we started with an empty registry. This is because some chemistries share the same permit list file, and simpleaf only downloads the permit list file once, identifying them by their hash value in the future.

Refresh the Local Registry of Chemistries in Simpleaf

The simpleaf team is constantly updating the registry of chemistries. If you want to refresh your local registry of chemistries, you can use the simpleaf chemistry refresh command.

simpleaf chemistry refresh

Summary

In conclusion, simpleaf provides the simpleaf chemistry commands to manage and inspect custom chemistries in its registry of recognized custom chemistries. We can use these commands to add new custom chemistries, remove existing custom chemistries, add or refresh chemistry definitions from the upstream repository, look up details of a specific chemistry, download corresponding permit lists for chemistries, and search for unused permit lists and remove them from the cache.

Developing a custom simpleaf workflow from scratch

Sun, 09 Apr 2023 00:00:00 +0000

Simpleaf is a command-line toolkit written in Rust that exposes a unified and simplified interface for processing scRNA-seq datasets using the alevin-fry ecosystem of tools. Since Simpleaf version 0.15.0, we introduce the re-designed simpleaf workflow sub-program, which provides the ability to execute complex and highly-configurable single-cell data processing workflows consisting of Simpleaf commands and shell commands described by a simple user-provided Jsonnet program. One can fetch ready-made Simpleaf workflow templates from our protocol library, protocol estuary, using the simpleaf workflow get program, or develop custom workflows to achieve specific tasks. This tutorial will discuss how to build a valid Simpleaf workflow template from scratch. If you are interested in running an existing workflow, please check our tutorial about running Simpleaf workflows.

Here, we will assume that Simpleaf is available in our operating environment. If you still need to, you can follow this tutorial to install and set up Simpleaf. Besides, we might also need a Jsonnet executable to debug in real-time in practice. You can install it by following these instructions or use Jrsonnet, a Rust implementation of Jsonnet.

Preparation

Before we start, let’s set up the environment.

export AF_SAMPLE_DIR=$PWD/simpleaf_workdir
mkdir $AF_SAMPLE_DIR
cd $AF_SAMPLE_DIR

Basics

First of all, let’s talk about some basic terminology in Simpleaf Workflow:

There are three phases of a Simpleaf Workflow:

Workflow Template: A Jsonnet program that has the potential to generate a valid Simpleaf Workflow but with required fields missing.
Instantiated Workflow Template or just Instantiate Template: A Workflow Template that contains enough information to generate a valid Workflow Manifest.
Workflow Manifest: A JSON record that contains the definition of each command in the workflow and the information needed to invoke them.

We call a Jsonnet program that has the potential to generate a valid Simpleaf Workflow but with required fields missing a Workflow Template. All published workflows in protocol estuary are in this phase. Users must fill in enough information to make the template an Instantiated Template. An Instantiated Template is a Jsonnet program that contains enough information to generate a valid Workflow Manifest. In a valid Workflow Manifest, all command records must be parameterized completely and correctly.

Now, let’s talk about how Simpleaf Workflow works:

The program simpleaf workflow run takes an Instantiated Template as the input and generates a Workflow Manifest from it. Any Workflow Manifest is also a valid Instantiated Template of itself and can be used as the input of simpleaf workflow run.
Simpleaf requires the command records in the resulting Workflow Manifest, including Simpleaf commands and external shell commands, to follow a specific format because it always parses the Workflow Manifest evaluated from the provided Workflow Template or the Workflow Manifest itself to find and assemble the actual commands included in a workflow.
When parsing an Instantiated Template, Simpleaf automatically passes many external variables and library searching paths to the Jsonnet engine. One important variable is the Simpleaf Workflow utility library, which contains many useful functions for developing a Simpleaf Workflow template. We should take advantage of these variables when developing our own Workflow Templates.

Developing a Workflow Template

This section will discuss developing a Simpleaf Workflow template using no, essential, and advanced features.

1. Define a Basic Workflow Template

As we discussed, Simpleaf Workflow requires that in the Workflow Manifest, the command records, including Simpleaf commands and external shell commands, must follow a specific format. The format consists of two parts: identity fields and argument field(s). The identity fields of these two types of commands are the same, but the argument fields are not.

There are three identity fields.

program_name: This field has to record a valid program name in the user’s operating environment.
- For Simpleaf programs, this will be the name we used to call the Simpleaf programs.
  - For simpleaf index commands, this field must be program_name: "simpleaf index".
  - For simpleaf quant commands, this field must be program_name: "simpleaf quant".
- For external shell commands, this field must represent a valid executable. For example, for a record representing an awk command:
  - if the awk program we want to call is not in our PATH variable, this field must be the (quoted) path to the desired awk executable, for example program_name: "/usr/bin/awk".
  - if the awk is in our PATH, this can be the path to the executable, or just program_name: "awk", because in this case, awk is accessible in both ways.
step: This field records which step this command constitutes in the workflow. This is the only allowed integer in the Simpleaf Workflow Manifest. Simpleaf will invoke the commands by order of their step.
active: This field indicates if the command is active in the workflow. Simpleaf will regard all commands without this field as active commands. This is the only allowed boolean field in the Workflow Manifest. Simpleaf will skip (neither parse nor invoke) all inactive commands (with "active": false).

As for the argument field(s), they are in a different format in Simpleaf command records and external command records:

In a Simpleaf command record, each argument, value pair must be provided as a field. The field name represents the argument name, and the value represents the argument value (or true or an empty string "" if the argument doesn’t take a value). For example, {"--threads": 16, "--use-piscem": true, "--ref-type": "splici"}. We recommend using the full name of arguments. For example, we suggest using --threads instead of -t.
In an external command record, all arguments must be listed in an ordered array as the arguments field. For example, the argument of the command ls -l -h . has to be provided as {"arguments": ["-l", "-h", "."]}.

With the format in mind, we can now create a simple workflow template! Here we will define a workflow that first creates a toy reference sequence file and calls simpleaf index to build a salmon index on this reference. The actual workflow template, instantiated template, and the resulting workflow manifest are identical and are shown in the following code chunk.

In the following code chunk, we create a workflow template Jsonnet program file and call simpleaf workflow run to evaluate it.

cd $AF_SAMPLE_DIR

# write the _workflow manifest_ into a file
cat <<EOF > workflow_template.jsonnet
{
    "create_ref" : {
        "program_name": "echo",
        "step": 1,
        "arguments": ["\">a\\nAACCAACACAAAC\\n>b\\nCCACAAACAACACAAC\"", ">", "./workflow_output/toy_ref.fastq"],
    },
    "simpleaf_index": {
        "program_name": "simpleaf index",
        step: 2,
        "--ref-seq": "toy_ref.fa",
        "--output": "./workflow_output",
        "--kmer-length": "5"
    },
}
EOF

simpleaf workflow run --template workflow_template.jsonnet -o ./workflow_output

The above workflow template (or instantiated template, or workflow manifest) defines a workflow that consists of two steps:

We create a FASTA file with two sequence records using the echo command. The first record is a, and its sequence is AACCAACACAAAC, and the second record is b, and its sequence is CCACAAACAACACAAC. We use the > symbol to redirect the output of the echo command to a file named toy_ref.fa.
We call simpleaf index to build a Salmon index on the toy_ref.fa file. We set the kmer length as 5 and the output directory as ./workflow_output.

# print toy reference
$cat $AF_SAMPLE_DIR/toy_ref.fa
>a
AACCAACACAAAC
>b
CCACAAACAACACAAC

# simpleaf index output directory
$ls $AF_SAMPLE_DIR/workflow_output
index  index_info.json  simpleaf_index_log.json  toy_ref.fastq

2. The Recommended Layout in a Simpleaf Workflow Template

Although any Jsonnet program is a valid input for Simpleaf Workflow as long as it can generate a valid workflow manifest, we recommend dividing a workflow template into four sections:

meta_info: This section contains the meta information of the workflow, such as the name of the workflow, the version of the workflow, etc. It also contains the meta-variables, such as threads, output, and use_piscem, which are used by the utility library to assign values to some arguments in the workflow.
fast_config: This section contains the minimum information needed to generate a valid workflow manifest in the standard and recommended way. We call this section fast_config because it is the fastest way to generate a valid workflow manifest. However, the resulting workflow manifest might not be the most appropriate one for the users’ purpose.
advanced_config: This section contains the information for generating a valid workflow manifest in a more advanced and comprehensive way. Usually, this section contains the alternative configurations defined in the fast_config and allows the users to fine-tune the workflow.
workflow: This section contains the logics for generating the workflow manifest using the information provided in fast_config and advanced_config. Usually, it includes function calls to the utility library functions to combine the above three sections or some external command records for preprocessing the data.

3. A Workflow Template with More Jsonnet Features

Following the best practices, we can improve the above workflow template from multiple aspects:

For the meta-variables that will be used throughout the workflow, such as threads, output, and use_piscem, we can define them in the meta_info section and use the utility library to assign them to the appropriate arguments in the workflow.
To ease the users’ burden, we can include the most essential arguments, such as the path to the reference FASTA file, in the fast_config section and put it at the beginning of the workflow template. By doing this, the users can generate a valid workflow manifest by only completing this section.
For some optional but important arguments, such as the --ker-length argument, we can put them in the advanced_config section and assign them a default value.
Finally, we can assemble the above three sections in the workflow section to generate the command records of the workflow.

Here, we assume you have some basic knowledge about Jsonnet. Uncertain? Check their excellent tutorial! Let’s see how the redesigned workflow template looks like. Notice that the following example represents an uninstantiated workflow because the required field, --ref-seq is missing (with null). To instantiate the template, we need to fill in the output field in meta_info, and replace the null with the path to the toy_ref.fa file generated from the previous step, or any other valid FASTA file. Notice that Jsonnet doesn’t require quotes around field names, as long as the field names are valid identifiers.

# local variable definition starts with the `local` keyword 
local template = {
    // meta_info
    meta_info : {
        template_name: "example workflow",
        output : null,
    },
    fast_config : {
        ref_seq : null,
    }
    advanced_config : {
        arguments : {
            "--kmer–length" : 5,
        }
    },
    workflow : {
        simpleaf_index : {
            program_name: "simpleaf index",
            step : 2,
            "--output" : meta_info.output + "/simpleaf_index", # the + operator merges the two strings # the . operator accesses the field of an object
            "--ref-seq" : std.get(fast_config, "ref_seq"), # std.get() is a function call
        } + arguments # the + operator merges the two objects
        ,
        create_ref : {
            program_name : "echo",
            step : 1,
            arguments : ["\">a\\nAACCAACACAAAC\\n>b\\nCCACAAACAACACAAC\"", ">", meta_info.output + "/toy_ref.fastq"],
        },
    }
}; # local variable definition ends with a semicolon

# We output the template object
template 

In this example:

We define a local variable template to store the workflow template. The template variable is a Jsonnet object that contains four fields: meta_info, fast_config, advanced_config, and workflow. Each of these fields is also a Jsonnet object.
We specify meta-information in the meta_info field, such as the name of the workflow and the output directory. Notice that we use the null value to represent the missing value of the output field. We will fill in this field when instantiating the template.
We specify essential parameters in the fast_config field. In this example, we only need to provide the path to the reference FASTA file (other than meta_info.output) to generate a valid workflow manifest.
To make the workflow more flexible, we put some optional but important arguments in the advanced_config field. In this example, we assign a default value to the --kmer-length argument.
In the workflow section, we assemble the above three sections to generate the command records of the workflow. We used the following features from Jsonnet:
- We used the dot operator, ., to access the fields of an object like in Python. For example, meta_info.output accesses the output field in the meta_info object. This can also be achieved by calling the get() function from the Jsonnet standard library, std.get(meta_info, "output"). For details about the Jsonnet std library, please check this page.
- We used the + operator to combine the base object defined on the LHS and the arguments field in advanced_config. This operator can also be used to concatenate strings and calculate the sum of two numbers.

There are many more useful features in Jsonnet. To have a systematic understanding of Jsonnet, please check the tutorials provided by Jsonnet.

4. A Workflow Template Using the Simpleaf Workflow Utility Library

To ease the development of a workflow template, the Simpleaf team provides a utility library that contains many useful functions for building Simpleaf’s commands in the manifest at different resolutions. This library is passed to the template evaluation process as an external variable __utils, and can be loaded in any template by adding local utils = std.extVar("__utils"); at the beginning of the template. Then, we can access any function in the library by calling utils.function_name().

In this section, we will show an example of calling some functions from utils to build a simpleaf index command smoothly. Because this example is a little bit long, we will divide it into multiple parts, each representing a section discussed in the previous section.

Firstly, we will define the layout of the template file. We first load the utils library, and then we define a local variable template to store the workflow template. The template variable is a Jsonnet object that contains five fields: meta_info, fast_config, advanced_config, intermediate_steps, and workflow. Each of these fields is also a Jsonnet object. Here we use a section called intermediate_steps to take some intermediate configurations that will be used in the workflow section. We will show the usage of this section later.

local utils = std.extVar("__utils"); # import the utility library

# The actual definition of each section is omitted here
local template = {
    meta_info : {},
    fast_config : {},
    advanced_config : {},
    intermediate_steps : {}
    workflow : {},
}

Next, we will define the meta_info section. In this section, we will define the meta information of the workflow, such as the name of the workflow, the version of the workflow, etc. It also contains the meta-variables, such as threads, output, and use_piscem, which are used in later sections. The ... in the following code chunk means we omit the lines before and after the meta_info section.

meta_info : {
    template_name : "example workflow",
    output : null,
    threads : 16,
    use_piscem : false,
}

We then design the fast_config section. In this section, we will ask for a genome FASTA file and a gene annotation GTF file, which is the minimum input of making a spliced+intronic (splici) reference, which contains spliced transcripts’ sequence and intronic sequences of each gene. Details about splici can be found in the supplementary section S2 of the alevin-fry paper. We will also list a rlen field taking the numeric read length value of the sequencing data as the value, which is an optional argument for building the splici reference.

fast_config : {
    splici : {
        fasta : null,
        gtf : null,
        rlen : 91,
    }
}

Now we turn to the advanced_config section, we will list other reference types supported by simpleaf index, and all optional arguments and flags used for fine-tuning the behavior of simpleaf index. Specifically, in addition to splici, we will also support spliced+unspliced (spliceu) reference, which contains the spliced transcripts and the gene body (unspliced transcript) of each gene, direct_ref, which will build the index directly from the provided (usually transcriptome) FASTA file , and existing_index reference type, which tells the template to use an existing index instead of building a new one by calling simpleaf index .

advanced_config : {
    simpleaf_index : {
        ref_type : {
            type : "splici", # splici arguments are in the `fast_config` section.
            spliceu : {
                gtf : null, 
                fasta : null,
            },
            direct_ref : {
                ref_seq : null, 
            },
            existing_index : {
                index : null,
                t2g_map : null,
            },
        },
        arguments : {	
            active : true,
            "--spliced" : null,
            "--unspliced" : null,
            "--dedup" : false,
            "--sparse" : false,
            "--keep-duplicates" : false,
            "--gff3-fomrat" : false,
            "--threads" : $.meta_info.threads,
            "--use-pisem" : $.meta_info.use_piscem, 
            "--overwrite" : $.meta_info.use_piscem,
            "--kmer-length" :  31,
            "--minimizer-length" : 7,
            "--decoy-paths" : null,
        },
        output : $.meta_info.output + "/simpleaf_index",
    }
}

Finally, we will define the workflow section containing a simpleaf command record called simpleaf_index.

workflow : {
    simpleaf_index : utils.simpleaf_index(
        1, 
        utils.ref_type($.advanced_config.ref_type + $.fast_config), 
        $.advanced_config.arguments, 
        $.advanced_config.output,
    ),
}

In this section, we utilized two utils functions, utils.ref_type and utils.simpleaf_index. The utils.ref_type function takes a reference type object, like the one defined in the advanced_config section and returns a valid reference type object. This object must has a type field specifying which reference type will be used, and a field named by the value of the type field including the required arguments for building the reference. Here we merged advanced_config with fast_config because it contains the arguments used for building a splici reference. If we specify splici : {gtf: "genes.gtf", fasta: "genome.fa", rlen: 91}, after merging, the input object to utils.ref_type looks like the following:

{
    type : "splici", 
    splici : {
        fasta : "genome.fa",
        gtf : "genes.gtf",
        rlen : 91,
    },
    spliceu : {
        gtf : null, 
        fasta : null,
    },
    direct_ref : {
        ref_seq : null, 
    },
    existing_index : {
        index : null,
        t2g_map : null,
    },
},

The output of the utils.ref_type function will be passed to the utils.simpleaf_index function as the second argument. It looks like the following:

{
    type :: "splici",
    arguments :: {gtf: "genes.gtf", fasta: "genome.fa", rlen: 91},
    "--ref-type" : "splici",
    "--fasta" : "genome.fa",
    "--gtf" : "genes.gtf",
    "--rlen" : 91,
}

Notice that the type and arguments fields are defined using double colons. This is the syntax in Jsonnet for defining hidden fields. Those hidden fields will not be included in the output manifest, but are accessiable during the manifesting process.

Finally, the utils.simpleaf_index function takes 1. a step number, 2. a reference type object returned by utils.ref_type(), 3. optional arguments, and 4. the output path, and returns a valid simpleaf index command record, or an empty record if we use existing_index as the reference type. If we instantiate this template using meat_info.output : "./workflow_output" and the splici parameters showed above , our final manfest will looks like the following:

{
    "meta_info" : {
        "template_name" : "example workflow",
        "output" : "./workflow_output",
        "threads" : 16,
        "use_piscem" : false,
    },
    "workflow" : {
        "simpleaf_index" : {
            "program_name" : "simpleaf_index",
            "step" : 1,
            "active" : true,
            "--ref-type" : "splici",
            "--fasta" : "genome.fa",
            "--gtf" : "genes.gtf",
            "--rlen" : 91,
            "--dedup" : false,
            "--sparse" : false,
            "--keep-duplicates" : false,
            "--gff3-fomrat" : false,
            "--threads" : 16,
            "--use-pisem" : false, 
            "--overwrite" : false,
            "--kmer-length" :  31,
            "--minimizer-length" : 7
        }
    }
}

Notice that all flags with false will be ignored by Simpleaf when invoking the commands.

4. Utilizing Built-in Variables and Custom Library Search Paths in Custom Templates

When parsing a workflow template, Simpleaf automatically provides useful external variables to it. We can easily utilize these variables in our templates by receiving them using the std.extVar() function from the Jsonnet std library. Currently, Simpleaf provides the following variables and library searching paths when parsing a template, and for variables, we will show the code to receive them in our template directly:

std.extVar("__utils"): the __utils variable represents the Simpleaf workflow utility library, which contains useful functions for developing a Simpleaf workflow template. Usually, a template will take this variable at the beginning by adding local utils = std.extVar("__utils");.
std.extVar("__output"): The __output variable represents the --output argument provided from the command line. To use this variable, we need to add local output = std.extVar("__output"); at the beginning of the template.
std.extVar("__validate"): The __validate variable is set to false in simpleaf workflow get and true in other programs, like simpleaf workflow run. Usaually, this variable is used to turn on/off the validation of the template, for example, missing values. To use this variable, we need to add local validate = std.extVar("__validate"); at the beginning of the template.

Build a Workflow Template for Processing 10X Chromium 3’ v3 Data

In the above example, we discussed how to build a Simpleaf workflow template containing a simpleaf index command. In this section, we will show how to add a simpleaf quant command to the workflow template to process 10X Chromium 3’ v3 data. As we apply the same idea as in the previous section, we will briefly discuss the procedure of building a simpleaf quant command record using the utility library, and then show the final workflow template.

Overall, simpleaf quant requires at least five file/directory paths:

The path to the index directory.
The path to the transcript-to-gene mapping file.
The path to the output directory.
The comma-separated paths to the Reads1 FASTQ files.
The comma-separated paths to the Reads2 FASTQ files.

As we can predict the first two paths using the output of the simpleaf index command and set the output directory according to the meta_info.output variable, we only need to ask for the paths to the FASTQ files in the fast_config section.

1. Fast Configuration Section

In this section, we need to ask for read1 and read2 FASTQ files. Therefore, we add one field in addition to the splici field discussed above:

fast_config: {
    splici: {fasta: null, gtf: null, rlen: 91},
    map_reads: {
        reads1: null, # comma-separated paths to the Reads1 FASTQ files 
        reads2: null, # comma-separated paths to the Reads2 FASTQ files
    }
}

2. Advanced Configuration Section

In this section, we’ll cover two crucial argument groups for running simpleaf quant: mapping mode (map_type) and cell filtration strategy (cell_filt_type). map_type offers two choices: providing paths to Reads1 and Reads2 FASTQ files, along with the index and transcript-to-gene-mapping file for read mapping, or simply supplying the path to existing mapping results. We include existing_map in the map_type field, along with a type field indicating the chosen mode. Similarly, we define a cell_filt_type field with five possible cell filtration options.

advanced_config : {
    simpleaf_quant : {
        map_type : {
            type : "map_reads",
            existing_mappings : {
                map_dir : null,
                t2g_map : null,
            },
        },
        cell_filt_type : {
            type : "cell_filt",

            unfiltered_pl : true,
            knee : false,
            expect_cells : null,
            forced_cells : null,
            explicit_pl : null,
        },
        argumetns : {
            active : true,
            "--min-reads" : 10,
            "--resolution" :  "cr-like",
            "--expected-ori" :  "fw",
            "--threads" :  $.meta_info.threads,
            "--chemistry" :  "10xv3",
            "--use-selective-alignment" : false, 
            "--use-piscem" : $.meta_info.use_piscem,
            "--struct-constraints" : false,
            "--ignore-ambig-hits" : false,
            "--no-poison" : false,
            "--skipping-strategy" : null,
            "--max-ec-card" : null,
            "--max-hit-occ" : null,
            "--max-hit-occ-recover" : null, 
            "--max-read-occ" : null,
        },
        output : $.meta_info.output + "/simpleaf_quant",
    }
}

3. workflow section

Finally, with all the information in the above three sections, we can call the utils.simpleaf_quant function, together with the utils.map_type and utils.cell_filt_type functions to generate a valid simpleaf quant command record.

workflow : {
    simpleaf_quant : utils.simpleaf_quant(
        2, 
        utils.map_type($.advanced_config.simpleaf_quant.map_type + $.fast_config, $.workflow.simpleaf_index),
        utils.cell_filt_type($.advanced_config.simpleaf_quant.cell_filt_type),
        $.advanced_config.simpleaf_quant.arguments, 
        $.advanced_config.simpleaf_quant.output,
    ),
}

In this section, we utilized three utils functions, utils.map_type, utils.cell_filt_type, and utils.simpleaf_quant. The utils.map_type function takes a map type object, like the one defined in the advanced_config section and returns a valid map type object. This object must has a type field specifying which map type will be used, and a field named by the value of the type field including the required arguments for mapping. Here we merged advanced_config with fast_config because it contains the arguments used for mapping. If we specify map_reads : {reads1: "reads1.fastq.gz", reads2: "reads2.fastq.gz"}, after merging, the input object to utils.map_type looks like the following:

{
    type : "map_reads", 
    map_reads : {
        reads1 : "reads1.fastq.gz",
        reads2 : "reads2.fastq.gz",
    },
    existing_mappings : {
        map_dir : null,
        t2g_map : null,
    },
},

The output of the utils.map_type function will be passed to the utils.simpleaf_quant function as the second argument. It looks like the following:

{
    type :: "map_reads",
    arguments :: {reads1: "reads1.fastq.gz", reads2: "reads2.fastq.gz"},
    "--reads1" : "reads1.fastq.gz",
    "--reads2" : "reads2.fastq.gz",
    "--index" : "./workflow_output/simpleaf_index/index",
    "--t2g-map": ./workflow_output/simpleaf_index/index/t2g_3col.tsv",
}

Notice that the path to the index directory and the transcript-to-gene mapping file are included in the simpleaf_index command record, which is the second argument of utils.map_reads.

Then, we use map_type, together with the simpleaf index command record, to call the utils.cell_filt_type function to generate a cell_filt_type object containing the simpleaf quant argument/flag for the selected cell filtering method. We will not show the detail here as it is very similar to map_type showed above.

Finally, we call the utils.simpleaf_quant function to generate a simpleaf quant command record using the results from the function calls introduced above. After evaluation, the manifest will look like the following :

"simpleaf_quant": {
"program_name": "simpleaf quant",
"step": 2,
"active": true,
"--chemistry": "10xv3",
"--expected-ori": "fw",
"--index": "./workflow_output/simpleaf_index/index",
"--min-reads": 10,
"--output": "./workflow_output/simpleaf_quant",
"--reads1": "reads1.fastq.gz",
"--reads2": "reads2.fastq.gz",
"--resolution": "cr-like",
"--t2g-map": "./workflow_output/simpleaf_index/index/t2g_3col.tsv",
"--threads": 16,
"--unfiltered-pl": true,
"--use-piscem": false
}

Finally, if we assemble the above sections, including those for simpleaf index and simpleaf quant together, we will get the final workflow template for processing 10X Chromium 3’ v3 data as shown in the protocol estuary GitHub repository.

Summary

In conclusion, we can quickly build complex and highly-configurable simpleaf workflow templates to guide simpleaf to run complicated single-cell data processing workflows. We can take advantage of the cool features and the standard library in Jsonnet, as well as the simpleaf workflow utility library when building a simpleaf workflow template. If you have further questions, please do not hesitate to start a GitHub issue!

Executing single-cell data processing workflows with simpleaf

Sun, 09 Apr 2023 00:00:00 +0000

Simpleaf is a command line toolkit written in rust that exposes a unified and simplified interface for processing scRNA-seq datasets using the alevin-fry ecosystem of tools. In simpleaf version 0.15.0, we introduce the re-designed simpleaf workflow module, which provides the ability to execute complex and highly-configurable single-cell data processing workflows consisting of simpleaf commands and shell commands, described by simple user-provided Jsonnet programs. One can fetch published simpleaf workflow templates from our protocol library, protocol estuary, using the simpleaf workflow get program or develop custom templates to achieve specific tasks.

In this tutorial, we will walk you through the procedure of obtaining a published workflow template, instantiating it by filling in the information, and executing the workflow, using a workflow designed for processing 10x feature barcode data. If you are interested in developing a custom workflow template, please check this tutorial.

A real example for processing 10X Feature Barcode data

10X Feature barcoding technology is a powerful method for adding extra layers of information to our single-cell gene expression data. Currently, 10X supports antibody capture and CRISPR guide capture via feature barcoding. However, adding additional information also means that the data processing of single-cell data with feature barcoding is more complex than single-cell gene expression data alone. For example, to process a feature barcode antibodies experiment from 10x Genomics, one usually needs to invoke 10 extra commands, including preprocessing, barcode translation, and the actual data processing steps, comparing with gene expression data alone.

simpleaf workflow is designed to turn these complicated but fixed workflows into a simple fill-and-go process: In most cases, one just needs to provide the path to the input files in a workflow template in Jsonnet Jsonnet format, so that the functions and logics encoded in the template can automatically complete the details of the workflow and generate a workflow manifest. In this example, we will show how to fetch, fill, execute and patch the workflow designed for processing 10X feature barcode antibodies data using the 1k PBMCs dataset with a panel of feature barcode antibodies dataset. Other published workflow templates, for example, for processing data from CITE-seq, 10X feature barcode antibody + CRISPR, and standard 10X Chromium 3’ v2 and v3 gene expression, are all available for fetching using simpleaf workflow get.

0. Install simpleaf

Before everything, we should install simpleaf and the underlying tools. Here we assume you are on a Linux machine and you use conda. For more complicated cases, such as Apple silicon or conda is not available, you can read this tutorial to properly set up simpleaf.

conda create -n simpleaf -y -c bioconda -c conda-forge simpleaf piscem
conda activate simpleaf

This command usually will install the latest version of simpleaf. To use the re-designed simpleaf workflow module, please ensure that the installed simpleaf is at least in version v0.15.0.

$ simpleaf --version
simpleaf 0.15.0

1. Set up simpleaf

We need to set it up properly to make simpleaf ready to use. In the following command, we define the ALEVIN_FRY_HOME environment variable, configure simpleaf, and allow simpleaf to create a large number of intermediate files. A detailed description of these steps can be found in this tutorial.

# create a working directory
export AF_SAMPLE_DIR=$PWD/simpleaf_workdir
mkdir $AF_SAMPLE_DIR
cd $AF_SAMPLE_DIR

# define env variable
export ALEVIN_FRY_HOME="$AF_SAMPLE_DIR/af_home"

# simpleaf configuration
simpleaf set-paths

# update the maximum number of file descriptors a program can create
ulimit -n 2048

2. Download the dataset

Usually, a 10X feature barcode dataset contains two parts:

The FASTQ files of gene expression reads and feature barcode reads.
the feature reference file containing the barcode sequence of all the Feature Barcode reagents used in an experiment.

Here we download these files, plus the human reference files, into the working directory.

# define the path to the input file and folder
FASTQ_DIR="$AF_SAMPLE_DIR/data/pbmc_1k_protein_v3_fastqs"
FEATURE_REF_PATH="$AF_SAMPLE_DIR/data/feature_reference.csv"
REF_DIR="$AF_SAMPLE_DIR/data/refdata-gex-GRCh38-2020-A"
mkdir -p $FASTQ_DIR
mkdir -p $REF_DIR

# download FASTQ files
wget -qO- https://cf.10xgenomics.com/samples/cell-exp/3.0.0/pbmc_1k_protein_v3/pbmc_1k_protein_v3_fastqs.tar | tar xf - --strip-components=1 -C $FASTQ_DIR

# Download feature reference path
wget -v -O $FEATURE_REF_PATH -L https://cf.10xgenomics.com/samples/cell-exp/3.0.0/pbmc_1k_protein_v3/pbmc_1k_protein_v3_feature_ref.csv

# Download human reference genome build and gene annotations
wget -qO- https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCh38-2020-A.tar.gz | tar xzf - --strip-components=1 -C $REF_DIR

3. Get the workflow configuration file

We need to call the simpleaf workflow get program to fetch the workflow template. This program takes a workflow name via the --name flag and an output directory via the --output flag, and dumps the workflow-related files, including the workflow template, of the queried workflow into the output directory. In this example, we will query the 10x-feature-barcode-antibody workflow for processing 10x feature barcode antibody data. One can ask simpleaf to list all available templates by calling the simpleaf workflow list program.

# Make sure you have the latest protocol estuary
simpleaf workflow refresh
simpleaf workflow get --name 10x-feature-barcode-antibody -o $AF_SAMPLE_DIR

This command will dump all files into a directory named by the workflow name plus _template, under the output directory passed to --output. For this example, those files locate in $AF_SAMPLE_DIR/10x-feature-barcode-antibody_template.

$ ls $AF_SAMPLE_DIR/10x-feature-barcode-antibody_template

10x-feature-barcode-antibody.jsonnet

The 10x-feature-barcode-antibody.jsonnet is the actual workflow template that should be completed and passed to the --template flag in the simpleaf workflow run. This template, after instantiated, can produce a valid simpleaf workflow manifest to instruct simpleaf to invoke the correct commands for processing our data. Like all other simpleaf workflow templates, this file is in Jsonnet format. You can think of it as a JSON configuration file, but with cool features enabled, such as variables, functions, and useful operators.

4. Instantiate the workflow template by filling in the required information

In most cases, the only thing we need to do to instantiate the workflow template is to fill in the missing fields in its fast_config section. These missing fields usually represent the path to the required files.

The fast_config section in the 10x-feature-barcode-antibody.jsonnet looks like the following but with all comments, which will help us to understand the meaning of each field, removed.

fast_config : {
    gene_expression : {
        splici : {
            gtf : null,
            fasta : null,
            rlen : 91,
        },
        map_reads : {
            reads1 : null,
            reads2 : null,
        },
    },
    antibody_capture : {
        feature_barcode_csv : null,
        map_reads : {
            reads1 : null,
            reads2 : null,
        },
    },
},

There are 7 missing (null) fields in the section, and here we talk about what they are and how to fill them in. Basically, We need to replace those nulls with the return string of the following code chunks. Please remember to “quote” them!

In the gene_expression section:
1. fasta: This field takes the path to the genome fasta file in the human reference set. This file is used for building the reference index. To get this, run the following command:
```
 echo $REF_DIR/fasta/genome.fa
```
2. gtf: This is the path to the gene annotation gtf file in the human reference set. This file is used for building the reference index. To get this, run the following command in the terminal:
```
 echo $REF_DIR/genes/genes.gtf
```
3. reads1: This field takes the path to the gene expression reads1 FASTQ files (technical reads). To get this, run the following command in the terminal:
```
 reads1_pat="_R1_"
 find -L ${FASTQ_DIR}/pbmc_1k_protein_v3_gex_fastqs -name "*$reads1_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -
```
4. reads2: This field takes the path to the gene expression reads2 FASTQ files (biological reads. To get this, run the following command in the terminal:
```
 reads2_pat="_R2_"
 find -L ${FASTQ_DIR}/pbmc_1k_protein_v3_gex_fastqs -name "*$reads2_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -
```
In the antibody_capture section:
1. feature_barcode_csv: This file takes the antibody feature reference file. To get this, run the following command in the terminal:
```
 echo $FEATURE_REF_PATH
```
2. reads1: This field takes the path to the feature barcode reads1 FASTQ files (technical reads). To get this, run the following command in the terminal:
```
 reads1_pat="_R1_"
 find -L ${FASTQ_DIR}/pbmc_1k_protein_v3_antibody_fastqs -name "*$reads1_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -
```
3. reads2: This field takes the path to the feature barcode reads2 FASTQ files (biological reads). To get this, run the following command in the terminal:
```
 reads2_pat="_R2_"
 find -L ${FASTQ_DIR}/pbmc_1k_protein_v3_antibody_fastqs -name "*$reads2_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -
```

As the above commands will return the absolute path to files, which are different in different machines, here we talk about how to correctly update the information in the configuration file instead of showing the complete version of that section. To update the fields, we need to replace the nulls in the file with valid values. For example, to complete the "fasta": null field, we need to replace the null with the “quoted” path to the human genome FASTA file we downloaded (returned by running echo $REF_DIR/fasta/genome.fa), such as

"--fasta": "/path/to/workdir/data/refdata-gex-GRCh38-2020-A/fasta/genome.fa"

Of course, in practice, we need to change /path/to/workdir to the absolute path to our working directory or use the path to another genome FASTA file we want to use. Details about other sections in a configuration Jsonnet program can be found in this section.

5. execute the workflow

With the instantiated template in hands, we can now call simpleaf workflow run to process our data: we pass the instantiated template via the --template flag and (optioanlly) an output directory via the --output flag. The output directory of the workflow can be set in two ways: the --output command line argument and the output field in the meta_info section of the workflow template. If both are set, the output meta-variable in the workflow template will be used.

simpleaf workflow run \
--template $AF_SAMPLE_DIR/10x-feature-barcode-antibody_template/10x-feature-barcode-antibody.jsonnet \
--output $AF_SAMPLE_DIR/workflow_output

When the execution is done, we are done with processing that complicated single-cell gene expression and feature barcode dataset, which consists of a total 12 commands, 4 simpleaf commands, and 7 external shell commands! Are you asking where the outputs are? They are in the $AF_SAMPLE_DIR/workflow_output. If not specified explicitly in the command records in the instantiated template, the actual layout of the output directory will follow the layout of the workflow manifest:

All results generated from the commands defined in the gene_expression section are in $AF_SAMPLE_DIR/workflow_output/gene_experssion. The gene expression count matrix can be found at $AF_SAMPLE_DIR/workflow_output/gene_experssion/simpleaf_quant/af_quant/alevin
All results generated from the commands defined in the antibody_capture section are in $AF_SAMPLE_DIR/workflow_output/antibody_capture.The feature barcode count matrix can be found at $AF_SAMPLE_DIR/workflow_output/antibody_capture/simpleaf_quant/af_quant/alevin.
The evaluted workflow manifest is at $AF_SAMPLE_DIR/workflow_output/workflow_execution_log.json. For more information about the workflow manifest, please check this section.

7. Clustering analysis

In this section, we follow the Seurat’s tutorial Using Seurat with multimodal data to analyze the quantification we generated from the previous steps.

Before we start, let’s first move to our working directory in our terminal.

cd $AF_SAMPLE_DIR

Now, let’s load the count matrix from our gene expression and feature barcode data, filtering high-quality cells, and run the standard Seurat pipeline on our gene expression count matrix. The following code chunk should be executed in R. If you are using your IDE, please ensure that you are in our working directory, or the loadFry function is run using correct paths.

suppressPackageStartupMessages({
    library(fishpond)
    library(ggplot2)
    library(SingleCellExperiment)
    library(Seurat)
    library(DropletUtils)
})

# set the seed 
set.seed(271828)

gex_q <- loadFry('workflow_output/gene_expression/simpleaf_quant/af_quant')
fb_q <- loadFry('workflow_output/antibody_capture/simpleaf_quant/af_quant')

e.out <- emptyDrops(counts(gex_q))
is.cell = e.out$FDR <= 0.01
is.cell[is.na(is.cell)] = FALSE

gid2name_df = read.csv("workflow_output/gene_expression/simpleaf_index/ref/gene_id_to_name.tsv", sep="\t", header = FALSE)
gid2name = gid2name_df[,2]
names(gid2name) = gid2name_df[,1]

gex_q.filt = gex_q[, is.cell]
fb_q.filt = fb_q[, colnames(gex_q.filt)]

rownames(gex_q.filt) = gid2name[rownames(gex_q.filt)]

pbmc <- CreateSeuratObject(counts = counts(gex_q.filt))

adt_assay <- CreateAssayObject(counts = counts(fb_q.filt))

# add this assay to the previously created Seurat object
pbmc[["ADT"]] <- adt_assay

library(RColorBrewer)
my_cols = brewer.pal(8,"Dark2")

DefaultAssay(pbmc) <- "RNA"

pbmc <- NormalizeData(pbmc)
pbmc <- FindVariableFeatures(pbmc)
pbmc <- ScaleData(pbmc)
pbmc <- RunPCA(pbmc, verbose = FALSE)
pbmc <- FindNeighbors(pbmc, dims = 1:30)
pbmc <- FindClusters(pbmc, resolution = 0.8, verbose = FALSE)
pbmc <- RunUMAP(pbmc, dims = 1:30)
# place plots side-by-side
pdf("rna_clusters.pdf",height=6, width=8)
DimPlot(pbmc, label = TRUE,cols=alpha(my_cols,0.66))
dev.off()


Fig 1. Cell Clusters UMAP Plot using gene expression count matrix.

Next, we want to use our feature barcoding data, which tells us the expression of some cell surface marker proteins, to help us assigning the cell types to the clusters we found. Here we will use CD19, which is the surface marker protein of B cells, to find which cluster, among the five clusters we found, is for B cells.

DefaultAssay(pbmc) <- "ADT"
pbmc <- NormalizeData(pbmc, normalization.method = "CLR", margin = 2)
DefaultAssay(pbmc) <- "RNA"

# Note that the following command is an alternative but returns the same result
pbmc <- NormalizeData(pbmc, normalization.method = "CLR", margin = 2, assay = "ADT")

# Now, we will visualize CD14 levels for RNA and protein By setting the default assay, we can
# visualize one or the other
DefaultAssay(pbmc) <- "ADT"
p1 <- FeaturePlot(pbmc, "CD19", cols = c("lightgrey", "darkgreen")) + ggtitle("CD19 protein")
DefaultAssay(pbmc) <- "RNA"
p2 <- FeaturePlot(pbmc, "CD19") + ggtitle("CD19 RNA")

# place plots side-by-side
p1 | p2


Fig 2. CD19 Protin(left) and RNA(right) expression in the RNA UMAP plot as Fig 1.

It is clear that Cluster 3 in Fig 1 represents the B cells, as CD19 is exclusively highly expressed in this cluster.

Patching

In the re-designed simpleaf workflow module, we introduce the simpleaf workflow patch program to help users to modify the templates (in Jsonnet) or manifest (in Json). This program takes either a workflow template via the --template or a workflow manifest via the --manifest flag, together with a patch file via the --patch flag, and patches the provided template or workflow. The patch file is a semicolon-separated values file (ssv?) that contains the modifications to the template or manifest.

Firstly, let’s see an example of a patch file, which should be in a semicolon-separated values file format.

name;<s>/meta_info/output;<n>/meta_info/threads;
patch1;"patch1_output";6
patch2;"patch2_output";8

In the above example, we see three rows and three columns in the patch file.

The first row contains column names. We require that, except a system column called name, all other column names should follow the format of value type followed by the path to the field to be patched, represented as <value type>/path/to/target/field/to/be/patched.
- The name column decides the name of the patch. This name will be used as the suffix of the output file name. For example, if we have a patch named patch1, the output manifest file name will be original_file_name_patch1.json.
- <value type> can be either <s>, <n> or <a>, which represent string, number, and array respectively.
- /path/to/target/field/to/be/patched should be a valid path in the provided template or manifest. For example, in the above example, <s>/meta_info/output means that we want to patch a string value to the output field in the meta_info section of the provided template or manifest. <n>/meta_info/threads means that we want to patch a number value to the threads field in the meta_info section of the template/manifest.
All following rows contain the patch name (or say id) and the values to be patched to the fields defined in the column name. In the above example, we have two patch row, the new values in each each patch will be patched to the provided template/manifest to generate a new manifest accordingly. Specifically, when processing the second row, simpleaf workflow patch will generate a new manifest with the output field in the meta_info section being updated as "patch1_output" and the threads field in the meta_info section as 6; When processing the third row, it will generate a new manifest with the output field in the meta_info section being updated as "patch1_output" and the threads field in the meta_info section as 6.

Secondly, we want to explain the behavioral difference between patching a workflow template with patching a workflow manifest.

When patching a manifest by a patch record (a row in the patch file), because a manifest is just a JSON record with no variables or functions, the simpleaf workflow patch program override the value of the fields defined in the patch record to the manifest. For example, if we have a patch record like the following:

  name;<s>/meta_info/output;<n>/meta_info/threads;
  patch1;"patch1_output";6

and we have a manifest like the following:

  {
      "meta_info": {
          "output": "original_output",
          "threads": 8
      },
      "command1": {
          "output" : "original_output/command1_output",
          "threads" : 12
      },
  }

After patching, we will have a new manifest like the following:

  {
      "meta_info": {
          "output": "patch1_output",
          "threads": 6,
      },
      "command1": {
          "output" : "original_output/command1_output",
          "threads" : 12
      },
  }

All fields other than the patched fields in the manifest will remain the same because there is no variable dependeny in the manifest.

When patching a template by a patch record, because a template is a Jsonnet program with variables, functions and other features, the simpleaf workflow patch program will override the value of the fields defined in the patch record to the template and re-evaluate the template to generate a new manifest. For example, if we have a patch record like the following:

  name;<s>/meta_info/output;<n>/meta_info/threads;
  patch1;"patch1_output";6

and we have a template like the following:

  {
      meta_info: {
          output: "original_output",
          threads: 8,
          use_piscem: true,
      },
      workflow: {
          command1: {
              output : meta_info.output + "/command1_output",
              threads : 12
              use_piscem: meta_info.use_piscem,
          },
      },
  }

This template will be evaluated to a manifest like the following:

  {
      "meta_info": {
          "output" : "original_output",
          "threads" : 8,
          "use_piscem" : true,
      },
      "workflow" : {
          "command1": {
              "output" : "original_output/command1_output",
              "threads" : 12,
              "use_piscem" : true,
          },
      },
  }

After patching, we will have a new template like the following:

  {
      meta_info: {
          output: "patch1_output",
          threads: 6,
          use_piscem: true,
      },
      workflow: {
          command1: {
              output : meta_info.output + "/command1_output",
              threads : 12,
              use_piscem: meta_info.use_piscem,
          },
      },
  }

and a new manifest like the following:

  {
      "meta_info": {
          "output": "patch1_output",
          "threads": 6,
          "use_piscem": true,
      },
      "workflow" : {
          "command1": {
              "output" : "patch1_output/command1_output",
              "threads" : 12,
              "use_piscem": true,
          },
      },
  }

Notice that in the final manifest, the workflow/command1/output field, which is not defined in the patch record, is updated because it depends on the meta_info/output patch field. On the contrary, the workflow/command1/threads field is not updated because itself is a number and does not depend on the any field, and the workflow/command1/use_piscem field is not updated because its dependency stay the same in the patched template.

Details about simpleaf workflow

The `simpleaf workflow run` program

simpleaf workflow run takes an instantiated simpleaf workflow template (a template jsonnet program with enough information for evaluating the workflow) as the input. During execution, simpleaf will first convert the instantiated template to a workflow manifest in JSON format with the help of Jrsonnet. simpleaf will then search the workflow manifest to find the command records, assemble them and push them into an execution queue. The execution order of the commands depends on the step field of the command record.

simpleaf workflow exposes multiple flags for controlling the execution flow when invoking the commands. If none of these flags is set, simpleaf will invoke all commands in the execution queue.

If setting the --no-execution flag, simpleaf workflow will parse the file passed to the --template, write the log files, and return without invoking any command.
If setting the --start-at flag with a step number, simpleaf workflow will ignore previous steps (commands) and begin the invocation from the commands in the execution queue whose step is larger than or equal to that specific step.
If setting the --resume flag, simpleaf workflow will find the log file of a previous run in the provided output folder to decide which step to begin with.
If setting the --skip-step flag with a set of comma-separated step numbers, simpleaf workflow will ignore the commands whose step is in those numbers.

2. The workflow template

Although simpleaf sets no hard format requirment on the workflow template (it will only use this file to produce a workflow manifest), we recommend a workflow template, and most of them do, contain four sections for best practices:

fast_config: This section should contain the minimum configuration that can trigger the default behavior of the workflow. Most users only need to fill in this section to instantiate the workflow template.
advanced_config: This seciton should define the alternative behaviors of the workflow. Usually, this section contains the flags of the simpleaf commands that are not in the fast_config section.
meta_info: This section should contain the meta-variables and meta-information of the workflow.
workflow: This section should contain the logics that build the actual workflow commands.

Later, in the evaluated manifest, we require two sections:

meta_info: This section should contain the meta-variables and meta-information of the workflow.
workflow: This section should contain the command records of the workflow.

fast_config

This section contains the minimum information to instantiate a workflow template. Each field in this section with a null value will have a detailed description of what the field represents, why we need it, and how to fill it in. In most cases, one can complete this section to instantiate the workflow in the standard and recommended manner. Of course, the users can fill in or modify the value of the fields in other sections if they would like to change the default behavior of the workflow.

meta_info

This section records the meta-variables and meta-information of a template. Usually, the meta-variables defined in this section, for example, output and threads, will be applied to all applicable commands. This might vary because the actual behavior of a Jsonnet program is defined in itself, not in the simpleaf workflow codebase. For example, in the example template we used above:

if one sets the threads meta-variable as "16" (remember that simpleaf workflow templates only take quoted values for arguments), this value will be assigned to all simpleaf commands in the workflow that don’t have this flag set.
As for output, simpleaf automatically pass a __output external variable to the template during evaluation corresponding to the value passed to the --output command line argument. In all protocol-estuary templates, we ask the users to at least set the output once, either in the meta_info section or via the --output flag, to ensure that the output directory of each workflow command is set correctly.

This section also contains valuable meta information, such as the workflow name and version.

advanced_config

This section contains the options that can be used to overwrite the default (and recommended) behavior of a workflow. Usually, this section lists the flags of each simpleaf command that are not in the fast_config section. One can modify the fields in this section to overwrite the defaults.

For example, in the example configuration program discussed above, if we have an existing human splici reference salmon index and we would like to use it instead of building a new one, we can set the reference type as "existing_index" in for the advanced_config/ref_type/type field, and provide the path to the index directory and the transcript-to-gene-map TSV file to the advanced_config/ref_type/existing_index/index and advanced_config/ref_type/existing_index/tx2gene fields respectively.

The modified configuration file will look like the following (with all other fields ignored):

advanced_config: {
    gene_expression: {
        simpleaf_index: {
            ref_type : {
                type : "existing_index", # changed from "splici" to "existing_index"
                existing_index : {
                    index : "/path/to/existing/index", # filled with the path to the existing index
                    t2g_map : "/path/to/t2g_3col.tsv", # filled with the path to the tx2gene.tsv file
                },
            },
        },
    },
}

By doing this, the workflow will use the existing index instead of building a new index by executing the simpleaf index command.

3. The workflow manifest

A workflow manifest refers to the JSON record that is manifested from an instantiated workflow template Jsonnet program. Simpleaf requires that each workflow manifest must be able to generate a manifest with a meta_info section and a workflow section. The meta_info section contains the meta-variables and meta-information of the workflow, and the workflow section contains the command records of the workflow.

Usually, the meta_info section in a workflow manifest contain at least three fields:

template_name: The name of the workflow template.
template_version: The version of the workflow template.
output: The output directory of the workflow.

The workflow section in a workflow manifest contains the command records of the workflow. Each command record must contains the following system fields:

program_name : The name of the command program.
step : The step number of the command. This field is used to control the execution order of the commands.
active : The active status of the command. This field is used to control the execution of the commands. If this field is set to false, the command will be ignored during execution. Then, depending on the command type, the command record may contain other fields.

For external commands, the command record must contain the arguments field, which is an array of string listing all the arguments of the command.

{
    "program_name": "echo",
    "step": 1,
    "active": true,
    "arguments": [
        "Hello World!"
    ]
}

For simpleaf commands, each field, except the three system fields listed above, must represent a valid argument, value pair of the command. For example, the following command record is for the simpleaf index command:

"simpleaf_index": {
    "program_name": "simpleaf index",
    "active": true,
    "step": 1,
    "--fasta": "/mnt/scratch3/alevin_fry_submission/refs/refdata-gex-GRCh38-2020-A/fasta/genome.fa",
    "--gtf": "/mnt/scratch3/alevin_fry_submission/refs/refdata-gex-GRCh38-2020-A/genes/genes.gtf",
    "--output": "workflow_output/gene_expression/simpleaf_index",
    "--ref-type": "spliceu",
    "--threads": 16
}

When executing the workflow, simpleaf workflow run will search the workflow manifest to find the command record with the three system fields, assemble them and push them into an execution queue. Therefore, the structure of the workflow manifest is not fixed, and the users can put the command records in nested layers as long as the three system fields are in the command record.

4. The validity of a workflow manifest and debugging

Although simpleaf workflow can take any valid Jsonnet program as the input, it will complain if the resulting JSON configuration is not a proper simpleaf workflow manifest according to some defined fields.

Usually, for a well-designed workflow template, like those published in protocol esturary, a valid workflow manifest can be generated as long as the missing fields in the fast_config section are all filled in. For users who want to enable some advanced flags in the simpleaf commands, if bugs occur, we would suggest the users follow the running logging we provided, as it will show the encountered errors are caused by what command in which step, and will provide a detailed error message to help you debug.

Suppose the error occurs when invoking a command. In that case, we suggest debugging according to the detailed error messages reported by simpleaf, fixing the bug in the instantiated template, and rerunning the workflow using simpleaf workflow run --resume. By setting the --resume flag, simpleaf will not rerun the commands that returned successfully. If you have further questions, please do not hesitate to ask by starting a GitHub issue!

Summary

In conclusion, simpleaf workflow provides the ability to execute complex and highly-configurable single-cell data processing workflows consisting of simpleaf commands and shell commands, described by simple user-provided configuration files. Many well-designed workflows are available on protocol estuary and can be easily obtained from the simpleaf workflow get program.

Generating a scRNA-seq count matrix with simpleaf

Tue, 24 Jan 2023 00:00:00 +0000

Simpleaf is a command line toolkit written in rust that exposes a unified and simplified interface for processing scRNA-seq datasets using the alevin-fry pipeline. It encapsulates the whole alevin-fry pipeline into two steps:

simpleaf index indexes the provided reference or makes an expanded reference and indexes it.
simpleaf quant maps the sequencing reads in provided FASTQ files against the indexed reference and quantifies the mapping records to generate a gene count matrix.

For more information about the processing pipeline conceptually, and the alevin-fry commands that correspond to each simpleaf command, see the raw data processing chapter in the Single-cell best practices book.

In this tutorial, we will show you how to generate a scRNA-seq count matrix from the raw input data – FASTQ files – using simpleaf, as well as describe some of the different options that simpleaf provides and what they do.

Installation

The recommended way of using simpleaf is from a conda environment. While you can also access it from a docker/singularity image or compile it manually.

conda create -n af -y -c bioconda -c conda-forge simpleaf
conda activate af

Download FASTQ Files

We create a working directory with sufficient space to download all the input data and hold the output (50GB should be sufficient). We alias this directory and use the alias below so that you can easily set it to something else and still copy and paste the later shell commands.

export AF_SAMPLE_DIR=$PWD/af_test_workdir
mkdir $AF_SAMPLE_DIR
cd $AF_SAMPLE_DIR

Then, we download the FASTQ files from the PBMC1k (v3) healthy donor samples.

FASTQ_DIR="$AF_SAMPLE_DIR/data/pbmc_1k_v3_fastqs"
mkdir -p $FASTQ_DIR

wget -qO- https://cf.10xgenomics.com/samples/cell-exp/3.0.0/pbmc_1k_v3/pbmc_1k_v3_fastqs.tar | tar xf - --strip-components=1 -C $FASTQ_DIR

A Quick Simpleaf Run Through Example

Step 0 : Configuration

In order to operate properly, simpleaf requires that you set the environment variable ALEVIN_FRY_HOME. It will use the directory pointed to by this variable to cache useful information. To configure simpleaf, please run the following shell commands. Please check section Configure simpleaf if you want to specify the path to the executable of its dependencies.

# define env variable
export ALEVIN_FRY_HOME="$AF_SAMPLE_DIR/af_home"

# simpleaf configuration
simpleaf set-paths

# Increase the number of open files for piscem's intemediate files
ulimit -n 2048

Step 1 : Build the reference index

Simpleaf maps the sequencing reads recorded in FASTQ files against a reference index. The index usually needs only to be built once for one set of the reference genome and gene annotations. In this example, we show the shell command of building a piscem spliced+intronic (splici) index, which contains the spliced transcripts and intronic sequences, for Human reference (GRCh38) dataset version 2024-A from 10x Genomics. For more options about indexing and mapping, check secion Indexing and Mapping Options.

cd $AF_SAMPLE_DIR
REF_DIR="$AF_SAMPLE_DIR/data/refdata-gex-GRCh38-2024-A"
mkdir -p $REF_DIR
IDX_DIR="$AF_SAMPLE_DIR/human-2024-A_splici"

# Download reference genome and gene annotations
wget -qO- https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCh38-2024-A.tar.gz | tar xzf - --strip-components=1 -C $REF_DIR

# simpleaf index
simpleaf index \
--output $IDX_DIR \
--fasta $REF_DIR/fasta/genome.fa \
--gtf $REF_DIR/genes/genes.gtf.gz \
--threads 8 \
--work-dir ./workdir.noindex

This will generate a piscem index in the $IDX_DIR/index directory. The index will be used in the next step, simpleaf quant, to map the sequencing reads in the FASTQ files against the reference. You can see that in the above command, genes.gtf.gz file is compressed but genome.fa is not. In general, simpleaf can work on both compressed and uncompressed files.

Step 2 : Quantify the sample

Given a reference index, a count matrix can be generated from the FASTQ files of a scRNA-seq dataset by the simpleaf quant program. In the following shell code chunk, in addition to the simpleaf quant program, we show how to get the expected order of the FASTQ filenames in a directory (and its subdirectory) recursively and store them in the reads1 and reads2 shell variables. If typing those filenames manually instead, please note that FASTQ filenames have to be comma-separated.

# Define filename pattern
reads1_pat="_R1_"
reads2_pat="_R2_"

# Obtain and sort filenames
reads1="$(find -L ${FASTQ_DIR} -name "*$reads1_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -)"
reads2="$(find -L ${FASTQ_DIR} -name "*$reads2_pat*" -type f | sort | awk -v OFS=, '{$1=$1;print}' | paste -sd, -)"

# simpleaf quantfication
simpleaf quant \
--reads1 $reads1 \
--reads2 $reads2 \
--threads 16 \
--index $IDX_DIR/index \
--chemistry 10xv3 --resolution cr-like \
--unfiltered-pl --anndata-out \
--output $AF_SAMPLE_DIR/pbmc1k_quant

Step 3 : Load output

simpleaf quant writes the count matrix to the directory $AF_SAMPLE_DIR/pbmc1k_quant/af_quant. There are two versions of the count matrix, (i) the count matrix in a mtx format, and the corresponding column names and row names in two separated files and (ii), an h5ad version containing the count matrix and all its affilicated information from the simpleaf quant run. Below, we show how to load the count matrix in R or Python.

R : In R, you can load the h5ad file using anndata, zellkonverter or SeuratDisk, or the count matrix using the loadFry() function from the fishpond Bioconductor package to load the count matrix as a SingleCellExperiment object, which can be easily turned into a seurat object.

# Run in R
library(SingleCellExperiment)
library(Seurat)

# define count matrix format
custom_format <- list("counts" = c("U", "S", "A"),
                      "spliced" = c("S"),
                      "unspliced" = c("U"),
                      "ambiguous" = c("A"))

# load count matrix
sce <- fishpond::loadFry("pbmc1k_quant/af_quant",
                         outputFormat = custom_format)

# create seurat object
seurat_obj <- CreateSeuratObject(counts(sce))

This will return a SingleCellExperiment object containing the counts for this experiment. The counts field contains the total counts of unspliced, spliced and ambiguous UMIs. Each type of counts is also stored in a separate layer.

For other pre-defined output formats and materials about the custom format, please see its reference page.

Python : In python, you can directly load the quants.h5ad file using scanpy.

# Run in python
import scanpy

adata = scanpy.read_h5ad("pbmc1k_quant/af_quant/quants.h5ad")

Simpleaf uses the sum of spliced, unspliced and ambgious counts as the X layer, and store each type of counts as a separate layer.

From here, you can use your favorite downstream analysis packages (like Seurat in R or Scanpy in python) to perform quality control, filtering, and analysis of your data.

Advanced Usage

1. Indexing and Mapping Options

Simpleaf has two underlying indexing and mapping engines, defaultly, piscem, and Salmon if specifying –no-piscem. Piscem is an improved index for the compacted reference de Bruijn graph that admits the same set of operations as the pufferfish index used by salmon, but that is substantially more space efficient. For example, the full spliceu indices of the human 2024-A reference, which contains all nascent and spliced transcripts, are 1.8GB. This allows the piscem single-cell mapper to operate in a very low memory footprint, ≤3GB for all the data evaluated in He et al., 2023.

Piscem is available on bioconda x86_64 and will be installed when installing simpleaf. For Non-conda users and Apple silicon users, piscem can be compiled and then specified in simpleaf set-paths or downloaded from the GitHub release page. See section Set Up Simpleaf Manually for details.

Suppose the piscem index was built by directly calling the salmon index program instead of simpleaf index. In that case, one needs to give the path to the piscem index prefix to the --index argument instead of the path to the index directory because piscem adopts a different (better) parsing strategy. For example, a piscem index directory is in the following structure:

$ ls $IDX_DIR/index
piscem_idx_cfish.json  piscem_idx.ectab    piscem_idx.sshash    t2g_3col.tsv
piscem_idx.ctab        piscem_idx.refinfo  simpleaf_index.json

Then, the path to the piscem index prefix should be $REF_DIR/index/piscem_idx. Again, if the piscem index was built by simpleaf index, simpleaf quant will automatically figure out the prefix of the index, so one just needs to specify the index folder. In this case, $REF_DIR/index. In conclusion,

If the piscem index was generated by simpleaf index, you should use --index $REF_DIR/index/.
If the piscem index was generated by piscem build, you should use --index $REF_DIR/index/piscem_idx

2. References

Currently, simpleaf index provides three strategies to build a reference index:

If passing a genome FASTA file to --fasta (or -f for short) and a gene annotation GTF file to --gtf (or -g for short), simpleaf will make the spliced + intronic (splici) reference, which contains the spliced transcripts and intronic sequences of each gene, and build a splici index on it. This is what we did in the above example.
```
simpleaf index \
--use-piscem \
--output $IDX_DIR \
--fasta $REF_DIR/fasta/genome.fa \
--gtf $REF_DIR/genes/genes.gtf \
--threads 16 
```
If passing --ref-type spliced+unspliced or --ref-type spliceu in addition to the above arguments, simpleaf will make the spliceu (spliced + unspliced) reference, which contains the spliced transcripts and the unspliced transcript of each gene, and build a spliceu index on it. Notice that here the unspliced transcript of a gene contains the whole genomic interval of the gene defined in the GTF file.
```
simpleaf index \
--use-piscem \
--output $IDX_DIR \
--fasta $REF_DIR/fasta/genome.fa \
--gtf $REF_DIR/genes/genes.gtf \
--threads 16 \
--ref-type spliced+unspliced # or spliceu
```
If passing the --ref-seq argument with a path to an existing reference FASTA file, for example, the transcript sequences downloaded from GENCODE, simpleaf will directly build a reference index for the sequences in the FASTA file. For example, we can directly pass the splici reference FASTA file created from the above example:
```
simpleaf index \
--use-piscem \
--output $IDX_DIR \
--threads 16 \
--ref-seq $IDX_DIR/ref/splici_fl86.fa
```
Probeset-based references are also supported. This is useful when processing reads genearted from feature barcoding, such as CITE-seq or 10X feature barcoding, or from probe sets, such as the 10X Flex Assay. Because feature barcoding reference usually has a different format with probe sets, we provide separate flags, --feature-csv and --probe-csv, to take them. For standardization, the provided CSV files must follow the format defined on 10x Genomics website. The feature barcoding CSV file format is defined at here. Currently, Currently, only three columns are used: id, name, and sequence. The probe set CSV file format is defined here. It should contain four mandatory columns: gene_id, probe_seq, probe_id, and included (TRUE or FALSE), and an optional column: region (spliced or unspliced).
```
simpleaf index \
--use-piscem \
--output $IDX_DIR \
--threads 16 \
--feature-csv path/to/features.csv \
# --probe-csv patj/to/probes.csv 
```

3. Custom Chemistries

Simpleaf currently supports the following chemistries: 10X Chromium 3’ and 5’ assays, 10xv2, 10xv2-5p, 10xv3, 10xv3-5, 10xv4-3p, and smartseq, and 10x Visium assays (reads from probes or transcripts), visiumv1, visiumv1-probe, visiumv4, visiumv4-probe, visiumv5, and visiumv5-probe. If your chemistry is not listed here, you can either specify the geometry specifications as we did in the runthrough, or you can add your custom chemistries using the simpleaf chemistry subcommands. This command provides many subcommands for chemistry oerations, including add, remove, clean, loolup, and fetch. For more information about how to use this command, please see the simpleaf chemistry documentation. Here, we show how to add the 10X Chromium V3 chemistry as a custom chemistry named mychem to simpleaf.

simpleaf chemistry add \
    --name mychem \
    --geometry "1{b[16]u[12]x:}2{r:}" \
    --expected-ori fw \
    --remote-url https://umd.box.com/shared/static/vc9zd4qyjj581gvtolw5kj638wmg4f3s

To note that if you have a local file instead of a remote URL, you can use the --local-url argument instead of --remote-url. Once added, you can call it from --chemistry mychem in the simpleaf quant command.

3. The Docker Hub Container

For users with difficulty installing conda packages or piscem, simpleaf also has its dedicated docker container (from biocontainers) and singularity container from the Galaxy project available. The following will show how to perform the above example in a singularity image.

First, we pull the singularity container that contains all of the software we’ll need to do our processing.

singularity pull https://depot.galaxyproject.org/singularity/simpleaf:0.19.4--ha6fb395_0

docker pull quay.io/biocontainers/simpleaf:0.19.4--ha6fb395_0

4. Set Up Simpleaf Manually

If neither conda nor usefulaf container works, you can also manually set up simpleaf and its dependencies and then configure the path to the executable of the dependencies in the simpleaf set-paths program. Before start, please ensure that you have rust and pip in your local environment.

Simpleaf and alevin-fry

As simpleaf and alevin-fry are 100% in rust, they can be installed from crate.io by running cargo install simpleaf alevin-fry. By doing so, simpleaf and alevin-fry will be in your path, and you won’t need to worry about setting their paths.

For people enjoying compiling source code, you can run the following shell commands:

For simpleaf:

cd $AF_SAMPLE_DIR
# clone GitHub repo
git clone https://github.com/COMBINE-lab/simpleaf.git && cd simpleaf

# build from source
cargo build --release

# add to PATH environment variable
export PATH="$AF_SAMPLE_DIR/simpleaf/target/release/simpleaf:$PATH"

# print help message
simpleaf --help
cd ..

For alevin-fry:

cd $AF_SAMPLE_DIR
# clone GitHub repo
git clone https://github.com/COMBINE-lab/alevin-fry.git && cd alevin-fry

# build from source
cargo build --release

# print help message
target/release/alevin-fry --help
cd ..

piscem

As piscem relies on cuttlefish and piscem-cpp, which are written in C++, one needs to clone piscem GitHub repository with submodules to install it.

cd $AF_SAMPLE_DIR

# clone repo with submodules
git clone --recursive https://github.com/COMBINE-lab/piscem.git 
cd piscem

# call rust compiler
cargo build --release

# print help message
target/release/piscem --help
cd ..

If you got some weird error messages talking about cmake, it means you have to specify your C++17-compatible C++ compiler.

# call rust compiler
CXX=<path_to_c++_compiler> CC=<path_to_c_compiler> cargo build --release

pyroe

The pyroe python package is used for constructing extended references. It could be installed via pip using pip install pyroe[scanpy] or being compiled by running the following shell command, where scanpy is an extra dependency and is only required if using the load_fry() python function.

cd $AF_SAMPLE_DIR

git clone https://github.com/COMBINE-lab/pyroe.git && cd pyroe

# pip install
pip install ".[scanpy]"

# print help page
bin/pyroe --help
cd ..

salmon

The latest version of salmon binary is provided here. One can download it directly to avoid compiling it (a giant beast). At the time we wrote this tutorial, the latest version of salmon was salmon v1.10.1, released on March 11, 2023. Here we show how to obtain its binary:

cd $AF_SAMPLE_DIR
mkdir salmon

# download and decompress salmon binary 
wget -qO- https://github.com/COMBINE-lab/salmon/archive/refs/tags/v1.10.1.tar.gz | tar xzf - --strip-components=1 -C salmon

# give permission
chmod +x salmon/bin/salmon

salmon/bin/salmon --help

To use other versions of salmon, just replace the link to the executable in the above command with yours.

Configure simpleaf

To tell simpleaf that its dependencies are ready to use, we need to give the path to the executables in simpleaf set-paths. If the paths to some dependencies are in your PATH environment variable, you can just skip setting them, as simpleaf set-paths will check your PATH to find them. Here we assume that alevin-fry was installed by calling cargo install alevin-fry and its path is in the PATH variable, and everything else was compiled from source code, and their path is not in your PATH.

cd $AF_SAMPLE_DIR

# simpleaf configuration
simpleaf set-paths \
--pyroe $AF_SAMPLE_DIR/pyroe/bin/pyroe \
--salmon $AF_SAMPLE_DIR/salmon/bin/salmon \
--piscem $AF_SAMPLE_DIR/piscem/target/release/piscem

If alevin-fry is not in your PATH, you can add its path to the above command.

cd $AF_SAMPLE_DIR

# simpleaf configuration
simpleaf set-paths \
--pyroe $AF_SAMPLE_DIR/pyroe/bin/pyroe \
--salmon $AF_SAMPLE_DIR/salmon/bin/salmon \
--alevin-fry $AF_SAMPLE_DIR/alevin-fry/target/release/alevin-fry \
--piscem $AF_SAMPLE_DIR/piscem/target/release/piscem

Summary

In conclusion, simpleaf drastically reduces the number of shell commands one must run to generate a gene count matrix from scRNA-seq FASTQ files. It can be quickly accessed from conda and the usefulaf docker/singularity image.

Analysing split-seq data using salmon and alevin-fry pipeline

Tue, 08 Feb 2022 00:00:00 +0000

Authors: Gaurav Sharma, Computational Biologist at Ocean Genomics; Jeremy Simon, Assistant Professor at UNC Chapel Hill; Rob Patro, Cofounder at Ocean Genomics
This project has been made possible by the team at Ocean Genomics, and by a grant from the Chan Zuckerberg Initiative.

In this tutorial we will use salmon and alevin-fry based pipeline to analyse SPLiT-seq data. We will also use a new tool splitp that modifies the fastq file to deal with the scenario where paired barcodes are used during the first round of combinatorial barcoding. We start by downloading the data. Post quantification, we will perfom single-cell analysis.

Introduction

Split-pool ligation-based transcriptome sequencing, or SPLiT-seq, is a single-cell RNA-seq method that uses combinatorial barcoding to label the cell or nucleus source of RNA. This method was introduced in the article “Single-cell profiling of the developing mouse brain and spinal cord with split-pool barcoding” by Rosenberg et al., Science 2018. As described in the article, different barcodes are ligated during different rounds of barcoding. There are three 8 bp cellbarcodes and a 10 bp UMI. Currently, SPLiT-seq can be performed with two chemistries — v1 and v2. Both v1 and v2 have 24 bp barcode and 10 bp UMI.

In this tutorial, we will use the data from the aforementioned article. The sample used in this tutorial, available for download on GEO, is a mix of cell lines HEK293, HelaS3, and NIH/3T3.

Download the input data

Create a new directory for the analysis

$ mkdir split-seq_tutorial
$ cd split-seq_tutorial

The SRA run corresponding to the data is SRR6750057 and can be downloaded from SRA.

$ mkdir -p data
$ fasterq-dump SRR6750057 -S -O ./data/ -e 16 -m 10GB -b 10MB

The other necessary files can be downloaded from here and saved in the data directory. Running fastp to trim and remove bad quality reads is helpful.

Paired barcoding case

In some cases, barcodes with known pairing are used during first round of combinatorial barcoding. This results in the same cell getting different barcode during the first round but the same barcode during later rounds. For example, if GATAGACA is paired with AACGTGAT, then the barcodes
CTGTAGCC-ACACAGAA-AACGTGAT, and
CTGTAGCC-ACACAGAA-GATAGACA
denote the same cell. The sequencing order is opposite of the order of ligation. For the first round, a transcript may get amplified by either an oligo-dT primer or a random hexamer, the pairing between the barcodes corresponds to the primer of amplification.

To deal with this case, splitp can be used. Splitp is a new tool that should eventually support a variety of complex read pre-processing steps necessary to process reads from various single-cell protocols, though for now the focus is only on the pre-processing necessary to process the SPLiT-seq protocol described here. The mapping file between oligo-dT primers and random hexamers is available at the link provided earlier. The command would be:

$ splitp -r SRR6750057_2.fastq -b ./data/oligo_hex_bc_mapping.txt -s 87 -e 94 -o > SRR6750057_corrected_2.fastq

splitp usage:

USAGE:
    splitp [OPTIONS] --read-file <READ_FILE> --bc-map <BC_MAP> --start <START> --end <END>

OPTIONS:
    -b, --bc-map <BC_MAP>          the map of oligo-dT to random hexamers
    -e, --end <END>                end position of the random barcode
    -h, --help                     Print help information
    -o, --one-hamming              consider 1-hamming distance neighbors of random hexamers
    -r, --read-file <READ_FILE>    the input R2 file
    -s, --start <START>            start position of the random barcode
    -V, --version                  Print version information

The bc-map, which maps barcodes corresponding to oligo-dT amplification to those corresponding to random hexamers, should be a tab separated text file with oligo-dT primer barcodes in the first column. The file must have a header comment line.

Creating the index

For this tutorial, we will need salmon(>=v1.7.0) and alevin-fry(>=v0.4.2). For the index, we will use a splici (spliced + intron) index, that contains both spliced mRNA transcripts and intronic sequences. This allows us to use the unspliced, spliced, ambiguous (USA) quantification mode of alevin-fry and generates these three types of counts for each gene. It can be used to compute RNA-velocity as well.

Since the fastq is contains a mix of human and mouse cell lines, first we will create a combined index of both species. For details on how to create splici index, please refer to the splici tutorial. For human, we will use GENCODE v31 and for mouse GENCODE vM25. The files needed to generate the splici index can be downloaded from the zenodo link.

Following sequence of commands was used to generate the index. First combine the references, then generate combined splici reference, and finally create the splici index.

Commands to combine the references

$ sed 's/chr/Mchr/g' mouse.gencode.vM25.genome.fa > mouse_gencode_vM25_genome.fa
$ sed 's/chr/Mchr/g' mouse.gencode.vM25.gene_annotations.gtf > mouse_gencode_vM25_gene_annotations.gtf
$ tail -n+6 mouse_gencode_vM25_gene_annotations.gtf > mgg
$ mv mgg mouse_gencode_vM25_gene_annotations.gtf
$ cat human.gencode.v35.genome.fa mouse_gencode_vM25_genome.fa > combined_genome.fa
$ cat human.gencode.v35.gene_annotations.gtf mouse_gencode_vM25_gene_annotations.gtf > combined_gene_annotations.gtf

The following R code was used to generate the reference for indexing (using the roe package; one could also use pyroe instead):

library(roe)

genome_path <- "combined_genome.fa"
gtf_path <- "combined_gene_annotations.gtf"
filename_prefix = "transcriptome_splici"
output_dir = "splici_index_reference"
read_length = 66
flank_trim_length = 5

make_splici_txome(
  genome_path = genome_path,
  gtf_path = gtf_path,
  read_length = read_length,
  output_dir = output_dir,
  flank_trim_length = flank_trim_length,
  filename_prefix = filename_prefix,
  dedup_seqs = FALSE,
  no_flanking_merge = FALSE
)

Finally, salmon index is used to generate the index

salmon index -t splici_index_reference/transcriptome_splici_fl61.fa -i human_mouse_index -p 32

Quantification

The quantification process is as follows:

salmon alevin peforms mapping of the fragments to the index to generate a RAD file.
alevin-fry uses that output to generate-permit list, collate the RAD file and perform quantification of the collated RAD file.

To read more about these steps please refer to alevin-fry documentation.

For the first step, we use salmon alevin to generate a RAD (Reduced Alignment Data) file. To use SPLiT-seq protocol, we can use either --splitseqV1 or --splitseqV2 flags for v1 and v2 chemistries, respectively.

The chemistry used by Rosenberg et al. is v1, whereas v2 is used for the commercial data from Parse biosciences. Both are similar, with the only difference being in the position of the third barcode. In v1 it occurs at position 78 and in v2 it occurs at position 86 of the sequenced read. The salmon alevin implementation assumes that, like the files from Rosenberg et al., the biological reads are in R1 fastq file and the barcodes and umi are present in R2. If your data has the opposite order, i.e., barcode and umi in R1, then in the command below use -1 for R2 and -2 for R1.

Here we use the --sketch flag to perform pseudoalignment with structural constraints, though selective-alignment could also be used (by passing --rad instead of --sketch). For viewing a detailed list of help options please use salmon alevin -h.

$ salmon alevin -i ./data/splici_index_human.v31_mouse.vM25 -l A -1 ./data/SRR6750057_1.fastq.gz -2 ./data/SRR6750057_corrected_2.fastq.gz -p 32 --splitseqV1 -o ./SRR6750057_run --sketch

To correct and associate sequenced barcodes to the most likely “corrected” barcodes, alevin-fry generate-permit-list is used. Here we use the flag -k, which attempts to estimate the number of high quality barcodes to include based on the “knee” of the cumulative frequency distribution of the reads associated with barcodes. Then the RAD file is collated using alevin-fry collate and quantification is performed using alevin-fry quant. The counts are generated as mtx files with cells as rows and genes as columns.

$ alevin-fry generate-permit-list -d both -i ./SRR6750057_run --output-dir ./SRR6750057_out_permit_knee -k
$ alevin-fry collate -r ./SRR6750057_run -t 16 -i ./SRR6750057_out_permit_knee
$ alevin-fry quant -m splici_index_reference/transcriptome_splici_fl61_t2g_3col.tsv -i ./SRR6750057_out_permit_knee -o ./SRR6750057_counts -t 16 -r cr-like-em --use-mtx

Post-quantification analysis

library(fishpond)
library(data.table)
library(Seurat)
library(miQC)
library(SeuratWrappers)
library(flexmix)
library(SingleCellExperiment)
library(Matrix)
library(stringr)

Since SPLiT-seq/ParseBio amplifies transcripts based on both oligo-dT and random hexamer priming, we recommend using outputFormat=“snRNA” to maximally capture signal occurring anywhere in the body of the mature or immature transcript. More quantitative details and benefits of this to come in a future manuscript.

# Read in entire output directory from alevin-fry using fishpond, will create SingleCellExperiment object
sce <- loadFry(fryDir = "SRR6750057_counts/", outputFormat = "snRNA")

## locating quant file

## Reading meta data

## USA mode: TRUE

## Processing 116057 genes and 301 barcodes

## Using pre-defined output format: snRNA

## Building the 'counts' assay, which contains U S A

## Constructing output SingleCellExperiment object

## Done

# Convert gene ids to gene symbols
gene_annotation_table <- fread("./data/combined_geneid_genesymbols.txt")
geneNames <- gene_annotation_table$GeneSymbol[match(rownames(sce),gene_annotation_table$Geneid)]
rownames(sce) <- geneNames

# Some gene names are duplicated such as the same gene in chromosomes X and Y. Those are merged here. 
exp.gene.grp <- t(sparse.model.matrix(~ 0 + geneNames))
exp.summarized <- exp.gene.grp %*% counts(sce)
rownames(exp.summarized) <- rownames(exp.summarized) %>% str_replace_all("geneNames","")

# Create Seurat object
rosenberg300ubc.seurat  <- CreateSeuratObject(counts = exp.summarized)

## Warning: Feature names cannot have underscores ('_'), replacing with dashes
## ('-')

# Compute mitochondrial contamination and filter out low quality cells using miQC
rosenberg300ubc.seurat <- subset(rosenberg300ubc.seurat, subset = nCount_RNA > 750 & nFeature_RNA > 375)
rosenberg300ubc.seurat <- PercentageFeatureSet(object = rosenberg300ubc.seurat, pattern = "^MT|^Mouse-mt", col.name = "percent.mt")
rosenberg300ubc.seurat <- RunMiQC(rosenberg300ubc.seurat, percent.mt = "percent.mt", nFeature_RNA = "nFeature_RNA", posterior.cutoff = 0.75, model.slot = "flexmix_model")

## Warning in RunMiQC(rosenberg300ubc.seurat, percent.mt = "percent.mt",
## nFeature_RNA = "nFeature_RNA", : flexmix returned only 1 cluster

## defaulting to backup.percentile for filtering

## Warning: Adding a command log without an assay associated with it

rosenberg300ubc.seurat.filtered <- subset(rosenberg300ubc.seurat, miQC.keep == "keep")

dim(rosenberg300ubc.seurat.filtered)

## [1] 114900    297

# Normalize and scale data
rosenberg300ubc.seurat.filtered <- NormalizeData(rosenberg300ubc.seurat.filtered)
rosenberg300ubc.seurat.filtered <- FindVariableFeatures(rosenberg300ubc.seurat.filtered, nfeatures = 5000)
all.features <- rownames(rosenberg300ubc.seurat.filtered@assays$RNA@counts)
rosenberg300ubc.seurat.filtered <- ScaleData(rosenberg300ubc.seurat.filtered, features = all.features)

## Centering and scaling data matrix

# Run PCA, then determine how many PCs are informative
rosenberg300ubc.seurat.filtered <- RunPCA(rosenberg300ubc.seurat.filtered, verbose = FALSE, npcs = 100)
ElbowPlot(rosenberg300ubc.seurat.filtered, ndims = 30, reduction = "pca")

# Run UMAP, and identify cell clusters
rosenberg300ubc.seurat.filtered <- RunUMAP(rosenberg300ubc.seurat.filtered, dims = 1:10)

## Warning: The default method for RunUMAP has changed from calling Python UMAP via reticulate to the R-native UWOT using the cosine metric
## To use Python UMAP via reticulate, set umap.method to 'umap-learn' and metric to 'correlation'
## This message will be shown once per session

## 12:27:11 UMAP embedding parameters a = 0.9922 b = 1.112

## 12:27:11 Read 297 rows and found 10 numeric columns

## 12:27:11 Using Annoy for neighbor search, n_neighbors = 30

## 12:27:11 Building Annoy index with metric = cosine, n_trees = 50

## 0%   10   20   30   40   50   60   70   80   90   100%

## [----|----|----|----|----|----|----|----|----|----|

## **************************************************|
## 12:27:11 Writing NN index file to temp file /tmp/RtmpnqRw7m/file13908e7ceec965
## 12:27:11 Searching Annoy index using 1 thread, search_k = 3000
## 12:27:11 Annoy recall = 100%
## 12:27:12 Commencing smooth kNN distance calibration using 1 thread
## 12:27:13 Initializing from normalized Laplacian + noise
## 12:27:13 Commencing optimization for 500 epochs, with 11226 positive edges
## 12:27:15 Optimization finished

rosenberg300ubc.seurat.filtered <- FindNeighbors(rosenberg300ubc.seurat.filtered, dims = 1:10, verbose = FALSE)
rosenberg300ubc.seurat.filtered <- FindClusters(rosenberg300ubc.seurat.filtered, verbose = FALSE, resolution = 0.1, algorithm=2)

# Plot UMAP labeled by clusters
DimPlot(rosenberg300ubc.seurat.filtered,reduction = "umap", label = TRUE)

# Find markers of the 3 clusters
markers = FindAllMarkers(rosenberg300ubc.seurat.filtered, assay="RNA", slot="scale.data", only.pos=T)

## Calculating cluster 0

## Calculating cluster 1

## Calculating cluster 2

# Cross-referenced top markers with Human Protein Atlas RNA expression in cell lines
HEK293.markers = c("SLIT2", "ZNF829", "KPNA5", "BRD2", "SPEN")
HELA.markers = c("NPR3", "PDE2A", "COL7A1", "FOLR1")
NIH3T3.markers = c("Mouse-Srrm2", "Mouse-Hspa5", "Mouse-Scd2", "Mouse-Huwe1") # These are the only mouse cells in the mix, so any mouse gene symbols will do
goi = c(HEK293.markers, HELA.markers, NIH3T3.markers)

# Create violin plot
VlnPlot(rosenberg300ubc.seurat.filtered, features = goi, stack=T, flip=T, sort=T, slot="scale.data")

sessionInfo()

## R version 4.1.2 (2021-11-01)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.4 LTS
## 
## Matrix products: default
## BLAS:   /opt/R-4.1.2/lib/R/lib/libRblas.so
## LAPACK: /opt/R-4.1.2/lib/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats4    stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] stringr_1.4.0               Matrix_1.4-0               
##  [3] SingleCellExperiment_1.16.0 SummarizedExperiment_1.24.0
##  [5] Biobase_2.54.0              GenomicRanges_1.46.1       
##  [7] GenomeInfoDb_1.30.1         IRanges_2.28.0             
##  [9] S4Vectors_0.32.3            BiocGenerics_0.40.0        
## [11] MatrixGenerics_1.6.0        matrixStats_0.61.0         
## [13] flexmix_2.3-17              lattice_0.20-45            
## [15] SeuratWrappers_0.3.0        miQC_1.2.0                 
## [17] SeuratObject_4.0.4          Seurat_4.1.0               
## [19] data.table_1.14.2           fishpond_2.0.1             
## 
## loaded via a namespace (and not attached):
##   [1] plyr_1.8.6             igraph_1.2.11          lazyeval_0.2.2        
##   [4] splines_4.1.2          listenv_0.8.0          scattermore_0.7       
##   [7] ggplot2_3.3.5          digest_0.6.29          htmltools_0.5.2       
##  [10] fansi_1.0.2            magrittr_2.0.2         tensor_1.5            
##  [13] cluster_2.1.2          ROCR_1.0-11            limma_3.50.3          
##  [16] remotes_2.4.2          globals_0.14.0         R.utils_2.11.0        
##  [19] spatstat.sparse_2.1-0  colorspace_2.0-2       ggrepel_0.9.1         
##  [22] xfun_0.29              dplyr_1.0.7            crayon_1.4.2          
##  [25] RCurl_1.98-1.5         jsonlite_1.7.3         spatstat.data_2.1-2   
##  [28] survival_3.2-13        zoo_1.8-9              glue_1.6.1            
##  [31] polyclip_1.10-0        gtable_0.3.0           zlibbioc_1.40.0       
##  [34] XVector_0.34.0         leiden_0.3.9           DelayedArray_0.20.0   
##  [37] future.apply_1.8.1     abind_1.4-5            scales_1.1.1          
##  [40] DBI_1.1.2              miniUI_0.1.1.1         Rcpp_1.0.8            
##  [43] viridisLite_0.4.0      xtable_1.8-4           reticulate_1.24       
##  [46] spatstat.core_2.3-2    rsvd_1.0.5             htmlwidgets_1.5.4     
##  [49] httr_1.4.2             RColorBrewer_1.1-2     ellipsis_0.3.2        
##  [52] modeltools_0.2-23      ica_1.0-2              farver_2.1.0          
##  [55] R.methodsS3_1.8.1      pkgconfig_2.0.3        nnet_7.3-16           
##  [58] sass_0.4.0             uwot_0.1.11            deldir_1.0-6          
##  [61] utf8_1.2.2             labeling_0.4.2         tidyselect_1.1.1      
##  [64] rlang_1.0.0            reshape2_1.4.4         later_1.3.0           
##  [67] munsell_0.5.0          tools_4.1.2            cli_3.1.1             
##  [70] generics_0.1.2         ggridges_0.5.3         evaluate_0.14         
##  [73] fastmap_1.1.0          yaml_2.2.2             goftest_1.2-3         
##  [76] knitr_1.37             fitdistrplus_1.1-6     purrr_0.3.4           
##  [79] RANN_2.6.1             pbapply_1.5-0          future_1.23.0         
##  [82] nlme_3.1-153           mime_0.12              R.oo_1.24.0           
##  [85] compiler_4.1.2         rstudioapi_0.13        plotly_4.10.0         
##  [88] png_0.1-7              spatstat.utils_2.3-0   tibble_3.1.6          
##  [91] bslib_0.3.1            stringi_1.7.6          highr_0.9             
##  [94] RSpectra_0.16-0        vctrs_0.3.8            pillar_1.6.5          
##  [97] lifecycle_1.0.1        BiocManager_1.30.16    spatstat.geom_2.3-1   
## [100] lmtest_0.9-39          jquerylib_0.1.4        RcppAnnoy_0.0.19      
## [103] cowplot_1.1.1          bitops_1.0-7           irlba_2.3.5           
## [106] httpuv_1.6.5           patchwork_1.1.1        R6_2.5.1              
## [109] promises_1.2.0.1       KernSmooth_2.23-20     gridExtra_2.3         
## [112] parallelly_1.30.0      codetools_0.2-18       MASS_7.3-54           
## [115] gtools_3.9.2           assertthat_0.2.1       sctransform_0.3.3     
## [118] GenomeInfoDbData_1.2.7 mgcv_1.8-38            parallel_4.1.2        
## [121] grid_4.1.2             rpart_4.1-15           tidyr_1.2.0           
## [124] rmarkdown_2.11         Rtsne_0.15             shiny_1.7.1

How to run many different sc-RNA-seq protocols using salmon alevin's custom geometry

Mon, 20 Dec 2021 00:00:00 +0000

Author: Gaurav Sharma, Computational Biologist at Ocean Genomics
This project has been made possible by the team at Ocean Genomics, and by a grant from the Chan Zuckerberg Initiative.

Nota Bene: The expanded custom geometry support described in this document is currently under a PR for inclusion into salmon in the near future. For the time being, to use / test this feature, you will have to build salmon from the custom_geometry branch. This documented will be updated again when the feature has been integrated into a tagged release of salmon.

The number of single-cell RNA sequencing technologies has been increasing, and new technologies continue to emerge. This leads to new protocols that need to be handled differently. To support a wide variety of protocols, salmon allows user to define the protocol in terms of cellular barcode and UMI geometry. This tutorial explains how to use this feature. Henceforth, the terms barcode and umi refer to the cell/nucleus barcode and UMI, respectively.

Types of custom protocols

There are two kinds of custom geometry protocols that are supported:

Fixed length: When the lengths and positions of the barcode and UMI are known
Variable length: When the lengths of the barcode and UMI may vary, but the limits are known. Furthermore, fixed sequences in the reads are supported.

1. Fixed length protocols

For single-cell protocols that involve fixed length of barcodes and reads, this feature should be used. There are 3 flags that need to be provided:

--bc-geometry
--umi-geometry
--read-geometry

Each of these needs to be supplied with arguments that specify barcode, umi and read geometry, respectively. The argument syntax is same for all flags. It is: ReadNumber[PositionList]. The ReadNumber can be either 1 or 2. For PositionList, here are some examples:

[1-5]
[1-10, 21-30]
[1-end]
[11-15, 21-25, 31-end]

As is clear from the examples, PositionList is a comma separated list of 1-based index positions. The keyword end denotes that the feature (barcode, umi or reads) spans till the end of the read.

Example

If you are working with SPLiT-seq data, here’s how you would specify the flags:

--bc-geometry 2[11-18,49-56,87-94] --umi-geometry 2[1-10] --read-geometry 1[1-end]

This means that the barcode occurs in read 2 from positions 11 to 18, 49 to 56 and 87 to 94. The final barcode would be a concatenation of the sequences in these positions. Similarly, the umi occurs in read 2 from positions 1 to 10 and the biological read is in read 1, spanning it entirely.

The full command will be something like:

salmon alevin -i <index> -l A -1 <fastq1> -2 <fastq2> -p <#Threads> --bc-geo 2[11-18,49-56,87-94] --umi-geo 2[1-10] --read-geo 1[1-end] -o <output_path> --justAlign

However, SPLiT-seq is supported by alevin, and to quantify its data one can use --splitseqV1 and --splitseqV2 switches, respectively for versions 1 and 2.

2. Custom geometry protocol

The fixed length flags well support multiple protocols, however, there are protocols that can’t be supported with these flags. For all of those protocols, you can use the --custom-geometry flag. Cases where it can be used:

The barcode or umi spans both reads
The barcode or umi has variable length
There is a fixed sequence in the read(s)

Let’s understand the argument that can be supplied to the --custom-geometry flag through an example:

1{b[10-12]f[ATCATC]u[10]}2{r}

Here, the first read starts with a barcode of variable length 10-12 bp, followed by a fixed sequence “ATCATC”, followed by a umi of length 10 bp. The second read is all biological sequence.

So, the way to specify the argument is — Read1{DescriptionList}Read2{DescriptionList}, where Read1 is 1 and Read2 is 2. DescriptionList is a list of descriptions of barcode, umi, biological read, fixed sequence and exclude lengths. A description is a description identifier followed by its accompanying information in square brackets. The possible description identifiers are:

b for barcode. Usage: b[Length] or b[Length range]. Eg. b[10] or b[9-11].
u for umi. Usage: u[Length] or u[Length range]. Eg. u[8] or u[8-10].
f for fixed sequence. Usage: f[sequence]. Eg. f[CAGAGC].
x for exclude. Usage: x[Length] or x[Length range]. Eg. x[4] or x[3-5].
r for biological read. It is special as it is not followed by square brackets.

Example

If you’re working with sci-RNA-seq3 data, here’s how you would specify the flag:

--custom-geometry 1{b[9-10]f[CAGAGC]u[8]b[10]}2{r}

This says that read 1 starts with a variable length barcode of length 9-10 bp, followed by a fixed sequence “CAGAGC”, followed by umi and barcode of lengths 8 and 10 bp, respectively. The second read is all biological sequence.

In case there is a variable length barcode or umi¹ in the protocol, there are padding sequences added to make the output barcode lengths equal, and to avoid spurious matches between barcodes. For example, if a protocol has barcodes of length 3-4 bp, and two of the barcodes are ATG and ATGA, after processing with salmon alevin using the --custom-geometry flag, these will be ATGAC and ATGAA, respectively. So, in case of variable length barcode, the length of barcodes in output is 1 more than maximum length of the barcode. For padding, the nucleotides ACGT are added in this order until the lengths are same.

Comparison between the two types

There are some key differences between the usage of flags for fixed length protocols (type 1) and custom geometry protocols (type 2) protocols:

Custom geometry protocols use just one flag, --custom-geometry, instead of three.
While positions need to be specified for type 1, type 2 needs lengths instead.
Variable length and fixed sequences are not supported in type 1.
Type 2 can be used for all type 1 protocols, but not vice-versa.

Having mentioned 4., it is important to note that type 2 uses regex parsing unlike type 1. Thus, it is about 30% slower. Therefore, if type 1 can be used for a protocol, it should be preferred if faster quantification is desired. Nonetheless, type 2 will be improved in future updates and will become faster.

Final thoughts

The field of single-cell transcriptomics is moving fast. The aforementioned features are provided with the goal of supporting almost all imaginable protocols. While salmon alevin already supports major protocols, it takes some time to add support for a new protocol. This is where the features in this tutorial come handy. It is possible to run a wide variety of protocols using the --custom-geometry flag, and thus, the users can run their data even when the explicit flag for their protocol is not present.

The author is not aware of any protocols that use variable length umi. ↩

alevin-fry-tutorials

Processing 10x Flex Gene Expression data with simpleaf

What is 10x Flex?

Installation

Configuration

Downloading the test data

Running simpleaf multiplex-quant

What happens under the hood

Understanding the output

Using a pre-built index

Splitting into per-sample matrices

Loading results in Python

Conclusion

Generating a spatial transcriptomics count matrix with simpleaf

Installation

Configuration

Downloading the data

Running simpleaf index

Running simpleaf quant

Conclusion

Operating on the Registry of Custom Chemistries in Simpleaf

Installation

Configuration

Custom Chemistries in Simpleaf

Define the geometry description for custom chemistries

10X Chromium 3’ 3 protocol

10X Chromium 5’ v3 protocol

10X Visium protocols

Expected Orientation

Barcode Permit List

Simpleaf Chemistry Commands

Register a Custom Chemistry in Simpleaf

Update a Custom Chemistry in Simpleaf

Remove Custom Chemistries in Simpleaf

Look Up Custom Chemistries in Simpleaf

Fetch the Permit List of Custom Chemistries in Simpleaf

Refresh the Local Registry of Chemistries in Simpleaf

Summary

Developing a custom simpleaf workflow from scratch

Preparation

Basics

Developing a Workflow Template

1. Define a Basic Workflow Template

2. The Recommended Layout in a Simpleaf Workflow Template

3. A Workflow Template with More Jsonnet Features

4. A Workflow Template Using the Simpleaf Workflow Utility Library

4. Utilizing Built-in Variables and Custom Library Search Paths in Custom Templates

Build a Workflow Template for Processing 10X Chromium 3’ v3 Data

1. Fast Configuration Section

2. Advanced Configuration Section

3. workflow section

Summary

Executing single-cell data processing workflows with simpleaf

A real example for processing 10X Feature Barcode data

0. Install simpleaf

1. Set up simpleaf

2. Download the dataset

3. Get the workflow configuration file

4. Instantiate the workflow template by filling in the required information

5. execute the workflow

7. Clustering analysis

Patching

Details about simpleaf workflow

The simpleaf workflow run program

2. The workflow template

fast_config

meta_info

advanced_config

3. The workflow manifest

4. The validity of a workflow manifest and debugging

Summary

Generating a scRNA-seq count matrix with simpleaf

Installation

Download FASTQ Files

A Quick Simpleaf Run Through Example

Step 0 : Configuration

Step 1 : Build the reference index

Step 2 : Quantify the sample

Step 3 : Load output

Advanced Usage

The `simpleaf workflow run` program