Skip to content

Building an index

Terminal window
spearfish index transcripts.fa.gz -o transcripts.sfi -t 16

The input is a transcript FASTA, plain or compressed.

spearfish collapses seeds within a gene, which is where isoforms share their exons, and it picks candidate genes for each read before scoring their transcripts. So it needs to know which transcripts belong together. There are three ways to tell it:

  • GENCODE-style headers (transcript|gene|...): used automatically.
  • A transcript-to-gene table, --t2g t2g.tsv, with one transcript gene pair per line (whitespace-separated; further columns are ignored). Use this for Ensembl, RefSeq or any other annotation.
  • Inferred from sequence, --infer-genes, for transcriptomes without gene IDs (other species, de novo assemblies). Transcripts that share a run of 12 consecutive seeds (about 300 bases) are grouped, ignoring runs found in more than 64 transcripts, which are repeats; then short transcripts left on their own join the one group they share shorter runs (about 100 bases) with. The groups merge gene families whose copies are near-identical, which reads cannot tell apart either.

If your transcripts carry no gene IDs and you have no table, use --infer-genes: on human GENCODE, inferred groups do as well as the annotated genes. On simulated ONT reads (95% accuracy), 98.7% of reads rank their true transcript first and 99.4% keep it among the reported transcripts with either, and the index is the same size (1.3 GB without the rescue tables).

Without any of these, every transcript is its own gene: the index has no gene groups. This happens, for example, with a RefSeq FASTA indexed without --t2g or --infer-genes. Isoforms are then not collapsed: the GENCODE index grows from 1.3 to 3.1 GB (rescue tables left out of both), mapping takes more time and memory, and fewer reads are assigned to the right transcript (top-1 falls from 98.7% to 97.6% on the same reads, and the true transcript is among those reported for 98.2% instead of 99.4%), because the per-read candidate set holds a few transcripts instead of a few genes. The isoforms of a gene then compete for those places one by one, and ties among them are broken by their order in the FASTA. --write-t2g out.tsv writes the grouping that was used.

So spearfish index warns when an index has no gene groups, and spearfish map warns again, once per run, when it loads such an index. Both warnings show under -q too, and the map run record lists it under warnings (code no_gene_groups):

warning: the index has no gene groups: every transcript is its own gene (the FASTA headers
carry no gene IDs, and neither --t2g nor --infer-genes was given). spearfish keeps a few
candidate genes per read, so isoforms then compete for those places one by one and ties among
them go by FASTA order, which costs accuracy; give --t2g FILE (a transcript-to-gene table) or
--infer-genes

When you see it, rebuild the index with --t2g if you have a transcript-to-gene table for the annotation, and with --infer-genes otherwise. A program using spearfish as a library can check Index::has_gene_groups() (in Python, Index.has_gene_groups).

For human GENCODE v50 (670,670 transcripts) the index is 2.8 GB and builds in about 7 seconds on 32 threads. A little over half of the file is the rescue tables, which the ont and drna presets map doubtful reads with (Doubtful reads); --no-rescue-tables leaves them out, and mapping with those presets then makes them, at every start and as reads need them. Map against an index built from the same transcript FASTA you quantify against.

  • The core graph: the transcriptome’s seeds, collapsed so that each exon interior is stored once per gene instead of once per isoform, and joined into unitigs.
  • For every unitig, the transcripts that contain it and where.
  • The homopolymer-compressed sequence of every transcript, and its raw sequence, used to align read ends where isoforms differ.
  • The rescue tables of the ont and drna presets: every transcript’s rescue seeds, a table of their runs of three, and each transcript’s antisense mates (the transcripts that hold stretches of its reverse complement). Mapping reads them in place from the file.

Inside spearfish explains why this structure is small.

spearfish index writes a manifest into the index and next to it, as transcripts.sfi.json: the spearfish version and git revision, the command line, the build options, a checksum of the input transcripts (and of the --t2g table), where the genes came from (fasta, t2g, inferred or none) and the construction statistics. spearfish inspect shows it:

Terminal window
spearfish inspect transcripts.sfi # key/value summary
spearfish inspect transcripts.sfi --json # everything, as JSON
spearfish inspect transcripts.sfi --sections # plus each section's size and checksum

The index’s identifier (index_id) is a hash of its input transcripts, their gene grouping (from the headers, --t2g or --infer-genes) and the build parameters: a rebuild from the same inputs and grouping gets the same identifier. Mapping outputs record it, so a mapping can always be traced to the index it came from. If an index was built by a spearfish whose sequence parser differs from the running one, mapping stops and asks for a rebuild instead of giving wrong results.

OptionDefaultMeaning
-o, --outputIndex file (the manifest goes to <file>.json).
-t, --threadsall CPUsWorker threads.
-s, --schemelcp:K=3Seed scheme. LCP level-3 cores are the default; minimizers (minimizer:k=19,w=23) give similar accuracy.
--no-hpcDon’t homopolymer-compress transcripts (and reads at mapping time).
--collapsegeneCollapse seeds per (label, gene) or globally (global, larger and slower).
--mask-genes20Mask unitigs shared by more than this many genes (repeats).
--no-rescue-tablesLeave the rescue tables out of the index (a little over half its size); the ont and drna presets then make them, at every start and as reads need them.
--t2gTranscript-to-gene table (see above).
--infer-genes [N]off (N = 64)Infer genes from shared seed runs, ignoring runs in more than N transcripts.
--infer-span12Consecutive seeds a shared run must span.
--infer-attach-span4Shorter runs by which small groups attach to a larger one (0 = off).
--infer-attach-max3Largest group, in transcripts, that may attach.
--write-t2gWrite the transcript-to-gene grouping used.