By: Scott Findlay, Principal Scientist at Deep Genomics
When an AI prediction disagrees with an experiment, the model usually gets the blame. But how often does that experiment truly get repeated, and if it does, what happens when the two experiments disagree with each other? While developing DeepRNAi to predict siRNA off-target effects, we found ourselves confronting exactly this question. What we found led us to consider a different role for AI: helping us predict and confirm truly reproducible effects.
We often hear refrains such as “the model achieves 70% accuracy on the test data—there’s room for improvement”. But it’s important to ponder what’s implied when people say this. We think there’s a deeper story to be told here.
To set the stage, let’s discuss where variability in RNA-seq based quantifications comes from. Perhaps most obviously, there’s sampling variability: when a random partition of an RNA sample is pipetted out and carried through an RNA-seq workflow, you won’t always get the same number of molecules of any given transcript. The lower a transcript’s expression, and the lower the volume used, the higher this variability will be. This is well appreciated and can be mitigated by ensuring ample sample loading, and filtering transcripts with very low expression. Of course there are a slew of other technical variables that can also contribute to variability in RNA-seq estimates, but we contend that there is a bigger more impactful aspect at play in differential expression analyses—variability in biological responses, and we think we can back that statement up.
Reproducibility is front of mind when we are conducting our siRNA screens. For us, this typically means including a handful of the same siRNAs in each of our screens, and performing qPCR to assess consistency in the extent of on-target knockdown. The results are reassuring: correlations between screens typically hover around R=0.9 (e.g. Figure 1).
But, we were intrigued as to whether that consistency would hold up beyond the intended target, and when we turned our attention to off-target effects, the answer surprised us!
It turned out that the strongest putative off-target transcripts, those with Near-Perfect Matches (NPMs) to the siRNA, demonstrated a notable drop in inter-screen experimental reproducibility relative to on-target effects, down to R=0.67 (i.e. <50% of the variance in observed log2 fold-changes in one screen can be predicted by screening the same siRNAs in another screen). And miRNA-like seed matches? They came in even lower at R=0.55 (Figure 2). Importantly, these inter-screen correlations are not simply across transfection replicates. They are across independent screens, conducted weeks apart, on different sub-cultures of the same cell line, where the observed values from each screen are the average of three transfection replicates. The strong on-target reproducibility rules out simple technical explanations for poor off-target reproducibility such as differential transfection efficiencies across screens.
Given this observation that off-target effects were much less reproducible than on-target activity, an intriguing hypothesis emerged: each differential expression value reflects a mixture of a more reproducible direct effect and less reproducible indirect effects.
Let’s walk through it: In an oligo screening context, one can think of the response of any given transcript to any given oligo as the combined result of several components: the direct, hybridization-dependent impact of the oligo binding to the transcript, secondary indirect effects that can result from modulations of on- or off-target transcript levels and subsequent cascades through gene regulatory networks, and “other” effects—think innate immune pathway activation in response to exogenously delivered oligos. It is reasonable to hypothesize that direct, hybridization-dependent effects are highly reproducible across cellular contexts, while indirect or other effects are much less so—perhaps due to particular sensitivity to small differences in gene expression with responses that are more stochastic than deterministic.
As we move along the continuum of complementarity between a given siRNA and a given transcript, from (fully complementary) intended targets, to NPMs, to seed matches, the decreasing magnitude of direct down-regulation is accompanied by an increasing “contamination” of sorts of these more variable indirect effects. With complementarity being the best available proxy for degree of direct targeting, we are conventionally left with little ability to interpret, let alone predict, the differences in liability across transcripts within a complementarity class (e.g. 7-mer seed matches).
Until now: enter DeepRNAi.
As Albi introduced last week, we have gone to substantial lengths to isolate, and enable DeepRNAi to learn, these ostensibly more reproducible, direct, hybridization-dependent effects. In the context of the above hypothesis, we quickly became hopeful that DeepRNAi could cut through the stochasticity of the indirect effects in a way that simple measures of complementarity could not. And so, we put it to the test on data from two completely separate screens, each with its own three transfection replicates, conducted weeks apart, on different sub-cultures of the same cell line. We were about to find out how well DeepRNAi could predict truly reproducible off-target liabilities. Just as importantly, we also set out to ask: how well can biology predict itself?
What came next changed the game for us: Those ~3,000 siRNA-transcript pairs with Near-Perfect Matches that we introduced above? Recall they had an inter-screen log2 fold-change correlation of 0.67. Corresponding predictions from DeepRNAi achieved a correlation of 0.46 to observed values—not that much worse than an experiment predicting another experiment, but done in minutes without ever having seen any data for these siRNAs.
Although we were excited our predictions were approaching the “limits” of experimental reproducibility, we quickly asked ourselves a more daring question: could our predictions possibly be even more accurate and reproducible than using an experiment to predict itself?
This idea isn’t unprecedented in machine learning and artificial intelligence. We hear often about models being able to effectively “denoise” experimental data, and we had explicitly engineered DeepRNAi to isolate direct effects.
Now for the results: Among all ~100,000 siRNA-transcript pairs in this dataset, we used an observed log2 fold-change threshold of <-0.5 (~30% knockdown) to identify hits from one screen and assess how well they reproduced or “predicted” results in the other screen. The outcome was a surprisingly moderate average correlation (R=0.44) with only 36% of experimentally nominated hits validating across screens. That is, do the experiment again, and only a minority of off-target hits would validate, with others offering a proverbial “nothing to see here.” In contrast, the off-targets nominated by DeepRNAi (at the same threshold) were found to correlate very well across screens (R=0.82) and validated at almost double the rate—62% (Figure 3).
We had the evidence we’d been looking for: DeepRNAi’s strongest predictions had much higher agreement across screens. We were seeing the impact of our efforts to enable DeepRNAi to learn and predict direct effects in action. Furthermore, when we looked at off-target liabilities supported by screening evidence and DeepRNAi predictions, these validated at an extremely high rate of 85%. Add it all up and we’re excited that DeepRNAi can help predict and confirm the directly regulated off-target liabilities that are more likely to hold up across screens, and have impact across therapeutic programs, from our lab to yours.
However, our findings do raise some tough questions: If we repeat an experiment, which one should we trust? How many times do we have to fully reproduce the experiment to believe the result? Is there a more abstract ground truth off-target risk that a given compound poses, and how well is it being captured by differential expression analysis based on RNA-Seq? Perhaps most importantly, how do we prospectively deploy and evaluate a model in a therapeutic program setting given this context? More on that in our next post!
Let us know your thoughts—does this affect how you’re thinking about reproducibility in your screens?
To read the previous pieces in this series, visit the Deep Genomics blog here.