To predict the effects of perturbations, there are many ongoing efforts to create a “virtual cell”. For molecules that interact directly with DNA, RNA and protein, such as siRNAs and other nucleic acid drugs, training a model to predict the overall perturbation response seems like a good idea. Surprisingly, we found that in fact it's the wrong problem to attempt solving.
The direct effect of an siRNA is its sequence-specific, RISC-mediated, RNA knockdown. A good virtual cell model should figure this out before going on to predict indirect effects, many of which measure how the cell reacts to the direct effects (and other non-specific drug responses). The problem is that direct effects are rare (~1% of all siRNA-RNA interactions result in at least a 50% RNA reduction), and we found that uncalibrated virtual cell objectives treat them as rounding errors. We inverted the problem by treating the perturbation profile solved by traditional virtual cell models as a noise term. For siRNA, the “rounding error” is actually the relevant learning signal.
As explained in Scott’s recent post, we generated RNA-seq data for 15 million siRNA-transcript pairs (~2200 siRNAs and ~7000 transcripts). Defining log2 fold change (log2FC) as the logarithm of a transcript’s expression with and without the perturbation, the traditionally processed RNA-seq data from the lab looks like this:
We see roughly the same number of up- and down-regulated transcripts, which makes sense given that the perturbations cause a lot of different things to happen. However, when it comes to direct effects, we know that siRNAs perform sequence-specific knockdown. So, the vast majority of the effects cannot be attributed to direct, hybridization-dependent off-target interactions. Given the recurring stereotypical patterns (even with diverse siRNAs targeting different transcripts), we can view the whole-transcriptome perturbation response as a nuisance variable to be removed before learning a model of direct effects.
In the above plot, if we leave just one square out, what would you expect the missing colour to be? If you didn’t know anything about the siRNA or transcript, the best you could do would be to look at the patterns of colors for the neighbors of your square. This naive “fill in the blanks” idea creates a surprisingly powerful perturbation response model - so much so that the leftover training signal might look like a rounding error if you don’t know what you’re looking for:
Aren’t we throwing out most of the data? I had encountered a similar problem before when I was studying the effects of combinatorial gene knockouts in yeast. Here, it is known that an additive model (i.e. predicting that the perturbation response of multiple genes is just the sum of its parts) explains 99% of the data, but the rare genetic interaction effects actually tell you that two genes may be functionally dependent. Direct effects (siRNA-transcript interactions) behave the same way.
To convince ourselves that this approach was right, we looked to see whether the leftover signal was enriched for two of the most important kinds of siRNA-transcript interactions, near-perfect matches (NPMs - consisting of an edit distance of <2 between the 19-mer siRNA and the transcript) and seed matches (a match between positions 2 and 7 of the siRNA). We found strong enrichment for seed matches and NPMs, even among other knockdown events, as shown below, demonstrating that the “rounding error” was enriched for direct effects and suitable for training a sequence-specific knockdown model.
We trained a model, called DeepRNAi, to predict the direct effects isolated as described above from an siRNA sequence and the target transcript sequence (see my first blog post). One way to demonstrate the benefit of modeling only direct effects in pursuit of a causal model is to demonstrate generalization capability. We expect the model to perform across held out siRNAs, but what about held-out transcripts? This split is important both in a practical sense (e.g. we need our model to work across different animal genomes, and take into account human genetic variation), but it is also an important sanity check for generalizability. A held out transcript challenges the model because it has never seen batch effects and other patterns in the training data, and tests whether it truly has learnt the rules of RISC biology.
We were excited to see that in our ablation studies of DeepRNAi, we found no substantial difference in the performance for seen versus unseen transcripts. In contrast, a traditional ML approach that was trained without a focus on direct effects saw a sizable and significant decrease, dropping roughly 50% in its above-baseline performance from an already-poor starting point.
The lesson we learned is that sometimes it’s important to set up the machine learning task in a way that hurts traditional performance metrics, but leads to a much more powerful and generalizable model. Here, we had the domain insight to treat siRNA modeling in a special way, both with the above causal outlier focus and how we set up our neural network.
We also found that indirect effects essentially had very little reproducibility across cell lines, whereas the direct effects predicted by DeepRNAi reproduced much better. In his next post, Scott is going to talk about why this insight could make using DeepRNAi complementary to (and in some ways better than) conducting an RNA-seq experiment. And, in a future post, I’ll talk about how important the biological foundation model (BioFM) features were for achieving the exceptional performance of DeepRNAi.
To read the previous pieces in this series, visit the Deep Genomics blog here.