Building an index
spearfish index transcripts.fa.gz -o transcripts.sfi -t 16The input is a transcript FASTA, plain or compressed.
Gene membership
Section titled “Gene membership”spearfish collapses seeds within a gene, which is where isoforms share their exons, and it picks candidate genes for each read before scoring their transcripts. So it needs to know which transcripts belong together. There are three ways to tell it:
- GENCODE-style headers (
transcript|gene|...): used automatically. - A transcript-to-gene table,
--t2g t2g.tsv, with onetranscript genepair per line (whitespace-separated; further columns are ignored). Use this for Ensembl, RefSeq or any other annotation. - Inferred from sequence,
--infer-genes, for transcriptomes without gene IDs (other species, de novo assemblies). Transcripts that share a run of 12 consecutive seeds (about 300 bases) are grouped, ignoring runs found in more than 64 transcripts, which are repeats; then short transcripts left on their own join the one group they share shorter runs (about 100 bases) with. The groups merge gene families whose copies are near-identical, which reads cannot tell apart either.
If your transcripts carry no gene IDs and you have no table, use --infer-genes: on human
GENCODE, inferred groups do as well as the annotated genes. On simulated ONT reads (95%
accuracy), 98.7% of reads rank their true transcript first and 99.4% keep it among the reported
transcripts with either, and the index is the same size (1.3 GB without the rescue tables).
Without any of these, every transcript is its own gene: the index has no gene groups. This
happens, for example, with a RefSeq FASTA indexed without --t2g or --infer-genes. Isoforms
are then not collapsed: the GENCODE index grows from 1.3 to 3.1 GB (rescue tables
left out of both), mapping takes more
time and memory, and fewer reads are assigned to the right transcript (top-1 falls from 98.7% to
97.6% on the same reads, and the true transcript is among those reported for 98.2% instead of
99.4%), because the per-read candidate set holds a few transcripts instead of a few genes. The isoforms
of a gene then compete for those places one by one, and ties among them are broken by their order
in the FASTA. --write-t2g out.tsv writes the grouping that was used.
So spearfish index warns when an index has no gene groups, and spearfish map warns again,
once per run, when it loads such an index. Both warnings show under -q too, and the map run
record lists it under warnings (code no_gene_groups):
warning: the index has no gene groups: every transcript is its own gene (the FASTA headerscarry no gene IDs, and neither --t2g nor --infer-genes was given). spearfish keeps a fewcandidate genes per read, so isoforms then compete for those places one by one and ties amongthem go by FASTA order, which costs accuracy; give --t2g FILE (a transcript-to-gene table) or--infer-genesWhen you see it, rebuild the index with --t2g if you have a transcript-to-gene table for the
annotation, and with --infer-genes otherwise. A program using spearfish as a library can check
Index::has_gene_groups() (in Python, Index.has_gene_groups).
For human GENCODE v50 (670,670 transcripts) the index is 2.8 GB and builds in about 7 seconds on
32 threads. A little over half of the file is the rescue tables, which the ont and drna
presets map doubtful reads with (Doubtful reads);
--no-rescue-tables leaves them out, and mapping with those presets then makes them, at every
start and as reads need them. Map against an index built from the same transcript FASTA you quantify against.
What the index holds
Section titled “What the index holds”- The core graph: the transcriptome’s seeds, collapsed so that each exon interior is stored once per gene instead of once per isoform, and joined into unitigs.
- For every unitig, the transcripts that contain it and where.
- The homopolymer-compressed sequence of every transcript, and its raw sequence, used to align read ends where isoforms differ.
- The rescue tables of the
ontanddrnapresets: every transcript’s rescue seeds, a table of their runs of three, and each transcript’s antisense mates (the transcripts that hold stretches of its reverse complement). Mapping reads them in place from the file.
Inside spearfish explains why this structure is small.
What was built, and from what
Section titled “What was built, and from what”spearfish index writes a manifest into the index and next to it, as transcripts.sfi.json:
the spearfish version and git revision, the command line, the build options, a checksum of the
input transcripts (and of the --t2g table), where the genes came from (fasta, t2g,
inferred or none) and the construction statistics. spearfish inspect shows it:
spearfish inspect transcripts.sfi # key/value summaryspearfish inspect transcripts.sfi --json # everything, as JSONspearfish inspect transcripts.sfi --sections # plus each section's size and checksumThe index’s identifier (index_id) is a hash of its input transcripts, their gene grouping
(from the headers, --t2g or --infer-genes) and the build parameters: a rebuild from the
same inputs and grouping gets the same identifier. Mapping outputs record it, so a mapping
can always be traced to the index it came from. If an index was built by a spearfish whose
sequence parser differs from the running one, mapping stops and asks for a rebuild instead of
giving wrong results.
Options
Section titled “Options”| Option | Default | Meaning |
|---|---|---|
-o, --output | Index file (the manifest goes to <file>.json). | |
-t, --threads | all CPUs | Worker threads. |
-s, --scheme | lcp:K=3 | Seed scheme. LCP level-3 cores are the default; minimizers (minimizer:k=19,w=23) give similar accuracy. |
--no-hpc | Don’t homopolymer-compress transcripts (and reads at mapping time). | |
--collapse | gene | Collapse seeds per (label, gene) or globally (global, larger and slower). |
--mask-genes | 20 | Mask unitigs shared by more than this many genes (repeats). |
--no-rescue-tables | Leave the rescue tables out of the index (a little over half its size); the ont and drna presets then make them, at every start and as reads need them. | |
--t2g | Transcript-to-gene table (see above). | |
--infer-genes [N] | off (N = 64) | Infer genes from shared seed runs, ignoring runs in more than N transcripts. |
--infer-span | 12 | Consecutive seeds a shared run must span. |
--infer-attach-span | 4 | Shorter runs by which small groups attach to a larger one (0 = off). |
--infer-attach-max | 3 | Largest group, in transcripts, that may attach. |
--write-t2g | Write the transcript-to-gene grouping used. |