Output files

The --output option passed to oarfish corresponds to a path prefix (this prefix can contain the path separator character and if it refers to a directory that does not yet exist, that directory will be created). Based on this path prefix, say P, oarfish will create the following files:

• P.meta_info.json - a JSON format file containing information about relevant parameters with which oarfish was run, and other relevant information from the processed sample apart from the actual transcript quantifications. As in salmon, it records the version that produced the output (oarfish_version) and when the run started and ended (start_time, end_time).
• P.quant - a tab separated file listing the quantified targets, as well as information about their length and other metadata. The num_reads column provides the estimate of the number of reads originating from each target.
• P.infreps.pq - a Parquet table where each row is a transcript and each column is an inferential replicate, containing the estimated counts for each transcript under each computed inferential replicate.
• P.ambig_info.tsv - a tab separated file listing, for each transcript (in the same order in which they appear in P.quant) the number of uniquely mapped, ambiguously mapped, and total reads. The quantification estimate for each transcript, in general, should reside between the number of uniquely aligned reads and the total number of reads (i.e. these provide, respectively lower and upper bounds for the number of reads assigned to each transcript). Note that the total in this file is the total number of reads that align to this transcript with a sufficiently high alignment score — it is not, in general, an estimate of the number of reads originating from this transcript as many of those reads can be multimapping and, in fact, potentially better described by other transcripts.
• P.prob[.lz4] - a file encoding the assignment probability of each read to each transcript to which it had a valid alignment (optionally compressed using lz4). This file is optional and is generated only if --write-assignment-probs is passed to oarfish.

Run information (meta_info.json)

Besides the options of the run, P.meta_info.json records what happened to the reads.

Discard table

discard_table counts the alignments each filter removed (discard_5p, discard_3p, discard_score, discard_aln_len, discard_ori, discard_supp), and, for reads:

• no_mapping: reads without any mapping;
• no_valid_aln: mapped reads with no alignment left after the alignment filters;
• discard_ori_reads: those among no_valid_aln all of whose alignments were removed by the strand filter (-d);
• discard_aln_frac: reads dropped because the fraction of the read in its best alignment is below --min-aligned-fraction;
• valid_best_aln: reads that were kept and quantified.

Run warnings

warnings is an array, empty for a run that gave none, with one object for each warning about the run as a whole:

"warnings": [
  {
    "code": "aligned_fraction_filter",
    "fraction": 0.257,
    "threshold": 0.1,
    "message": "..."
  }
]

code is a stable name for the condition, fraction the share of reads that triggered it, threshold the share above which it is issued, and message the text that was also written to the log. For em_not_converged (the default or a per-read EM) and truncation_em_not_converged (the truncation EM), fraction is instead the EM’s last convergence statistic (the largest relative change of a transcript’s abundance) and threshold is --convergence-thresh. For index_without_gene_groups (a spearfish index without gene groups), fraction is the index’s genes per transcript (1) and threshold is 1. The codes, and what to do about each, are listed in Recommended practices by sequencing technology. The single-cell mode does not record warnings.

EM (em)

em records which EM ran and how it stopped, whether or not it converged:

"em": {
  "kind": "truncation",
  "iterations": 1209,
  "max_iter": 5000,
  "converged": true,
  "last_rel_change": 0.00077,
  "convergence_thresh": 0.001
}

kind is truncation (--model-truncation) or standard (the equivalence-class EM or a per-read EM). last_rel_change is the convergence statistic of the last iteration, as that EM measures it. The truncation EM counts each accelerated (SQUAREM) step as three iterations, so its iterations can exceed max_iter by up to two. Bootstrap replicates are not recorded here.

spearfish settings (spearfish_settings)

Runs with --mapper spearfish, and runs on a RAD file, record the spearfish settings their alignments were made with:

"spearfish_settings": {
  "source": "in_process",
  "preset": "drna",
  "extend_ends": true,
  "extend_ends_reason": "model_truncation",
  "strands": "fwd",
  "quant_scores": true,
  "end_fit": 5.0,
  "end_fit_tol": 4,
  "end_fit_from": "alignment",
  "orient_quant": true,
  "orient_evidence": "quant",
  "index_genes": {
    "source": "fasta",
    "genes": 2,
    "transcripts": 4,
    "has_gene_groups": true
  }
}

• source: in_process (oarfish mapped the reads; the settings are resolved from the preset, --seq-tech, and the options) or rad_header (read from the RAD file, which then also gives mapper and mapper_version).

• preset: the mapping preset.

• extend_ends: whether alignment spans were extended to the read ends, and extend_ends_reason why: preset (the preset extends spans), model_truncation (the preset leaves them at the outermost seeds, and --model-truncation turned extension on), preset_off (the preset leaves them at the outermost seeds), or rad_header (a RAD file: as its mapper recorded). The span decides how much of a read counts as aligned, and so which reads --min-aligned-fraction drops.

• strands: the read orientations mapped (both, fwd or rev).

• quant_scores: whether the alignments carry quantification scores that can differ from their mapping scores. oarfish weighs alignments by the quantification score; without one it is the mapping score. A RAD file from spearfish before 0.3.0 has none.

• end_fit (points; 0 when off), end_fit_tol and end_fit_from (chain or alignment): the end fit, orient_quant: whether spearfish’s orientation evidence acts through the quantification score, and orient_evidence: what the evidence does, quant (moves only the quantification score), mapping (moves the mapping score and the orientation) or off (none, as spearfish 0.2.1 mapped; see --orient-evidence). See Mapping and quantification scores. For a RAD file they are read from the mapper’s parameters in its header, and are null where the header does not have them.

• index_genes: the gene labels of the spearfish index the reads were mapped against.

  • source: where the genes came from, as the index manifest records it: fasta (gene IDs in the FASTA headers, GENCODE-style transcript|gene|...) and t2g (a transcript-to-gene table) are provided labels; inferred means spearfish grouped transcripts by shared sequence; provided means the genes were given when the index was built through spearfish’s Python API (given in the RAD file); none means every transcript is its own gene; unknown is an index without a manifest, or a RAD file written before spearfish 0.3.1.
  • genes and transcripts: their numbers in the index.
  • has_gene_groups: whether the index groups isoforms into genes. When it is false, the run also has an index_without_gene_groups warning.
  • For a RAD file, all four are read from its file tags (index_genes_from, index_num_genes, index_num_transcripts, index_gene_groups), which spearfish map --rad writes from 0.3.1 on; an index without gene groups gets the same warning as in process. A RAD file from spearfish 0.3.0 has no such tags: source is then unknown, genes and has_gene_groups are null, transcripts is the number of targets in the file, and there is no warning, since oarfish cannot tell.

Runs on a BAM file have no spearfish_settings.

Read-level assignment probabilities

oarfish has the ability to output read-level assignment probabilities. That is, for each input read, what is the probability, conditioned on the final estimate of transcript abundances, that the read was sequenced from each transcript to which it aligned. By default, this information is not recorded (as it’s not required, or commonly used, for most standard analyses). To enable this output, you should pass the --write-assignment-probs option to oarfish. Optionally, you may also pass --write-assignment-probs=compressed to write the output to a compressed (lz4) stream — the default output is to an uncompressed text file.

Format

The format of the read assignment probabilities is as follows — where all fields below on a single line are \t delimited:

<m> <n>
<tname_1>
<tname_2>
...
<tname_m>
<rname_1> <k_1> <tid_11> <tid_21> ... <tid_{k_1}1> <p_11> <p_21> ... <p_{k_1}1>
...
<rname_n> <k_1> <tid_1n> <tid_2n> ... <tid_{k_n}n> <p_1n> <p_2n> ... <p_{k_n}n>

Here, <m> is the number of transcripts in the reference, <n> is the number of mapped reads. The following m lines consist of the names of the transcripts in the order they will be referred to in the file. The following n lines after that provide the actual alignment information and assignment probabilities for each read.

The format of each of these lines is as follows; <rname> is the name of the read, <k> is the number of transcripts for which probabilities are reported (a transcript whose probability is below --display-thresh is not reported). A read with several alignments to the same transcript has one entry for that transcript: the probabilities of those alignments are added, so each value is the probability that the read comes from the transcript. Subsequently, there is a list of k integers, and k floating point numbers. Each of the k integers is the index of some transcript in the list of m transcripts given at the start of the file, and the subsequent list of k floating point numbers are the assignment probabilities of this read to each of the transcripts.

The reported probabilities are renormalized over the alignments that pass --display-thresh, so the k values on a line sum to 1. They are printed with enough decimal places that an assignment passing --display-thresh is never rounded to 0: the precision tracks the threshold (the default --display-thresh 1e-6 prints 6 decimal places, and thresholds of 1e-3 or larger keep 3), so lowering the threshold both reports more alignments and prints them at a resolution that can represent them.

Example

5 3
txpA
txpB
txpC
txpD
txpE
readX 2 0 2 0.4 0.6
readY 3 1 3 4 0.1 0.2 0.7
readZ 1 4 1.0

This file provides an example with 5 reference transcripts and 3 mapped reads. The first read (readX) maps to transcripts 0 and 2 (so txpA and txpC) with probabilities 0.4 and 0.6 respectively. The next read (readY) maps to 3 transcripts, txpB, txpD and txpE with probabilities 0.1, 0.2, and 0.7 respectively. Finally, the last read (readZ) maps uniquely to transcript txpE with probability 1.

Compressed output

The compressed output (i.e. what is generated if one passes --write-assignment-probs=compressed) is exactly the same format, except instead of residing in a plain text file, it is written to an lz4 compressed text file. You can either decompress this file first with an lz4 decompressor, or decompress it on-the-fly as you are parsing the file using the lz4 library in your favorite language.