For a few years now, I have been pondering how to combat the marketing of false hope — in the form of marginal therapies — to vulnerable people (and their caregivers) grappling with severe degenerative diseases. Back in July 2024, I started this thread on Frank Harrell’s Datamethods forum. Recently I returned with a couple of follow-up posts outlining a Bayesian approach to inferring the marginal character of a therapy. Here, I draw freely on my posts from that thread, in order to develop a smoother flow of the ideas.
Background
In Sep 2016, FDA granted accelerated approval to exon-skipping drug eteplirsen for the subset of Duchenne muscular dystrophy (DMD) with a mutation amenable to exon 51 skipping. The basis for this approval was that initial studies had “demonstrated an increase in dystrophin production that is reasonably likely to predict clinical benefit in some patients.”1
In the nearly 10 years since approval, however, the required confirmatory clinical trials have still not been completed (Bendicksen, Kesselheim, and Rome 2024). The sponsor has instead produced ‘real-world evidence’ of benefit using claims and EHR data (Iff et al. 2023). To date, trials of eteplirsen have treated several hundred participants.
A Question
If the “demonstrated … increase in dystrophin production” still remains “reasonably likely to predict clinical benefit in some patients”, then among the hundreds treated so far isn’t it reasonable to expect one exceptional response (Marx 2015) to have been observed and reported by now in some patient?
I see a tacit recognition of this intuition in a report by Alfano et al. (2019) on identical-twin boys who lost ambulation about 36 weeks after randomization to eteplirsen, but then kept pace with the remaining 10 study participants on upper-extremity, cardiac and ventilation measures. But this offers merely a hint at the sort of thing I would have hoped to see by way of a case report by now. What would be persuasive to me as an ‘exceptional response’ would be a participant who demonstrated loss of ambulation at a relatively early age, either before initiating eteplirsen or shortly afterward [presumably, before the drug had exerted its full effect on dystrophin production], but then exhibited a distinctly Becker-like clinical course that left no doubt in an experienced clinician’s mind that the very character of the patient’s disease had been altered. (The clinician would say e.g., “I have never seen a boy with DMD lose ambulation at such an early age, then go on to preserve so much upper-extremity functioning for this long.”)
Typical patterns of clinical reasoning around pediatric growth charts illustrate how characteristic developmental trajectories are seen to have been altered. Just as a child with onset of growth restriction is said to ‘fall off their growth curve’, a boy with DMD who initiated a successful treatment might conversely be seen to ‘escape’ upward from his (downward) disease trajectory.
The challenge I’m putting to the sponsor here is far less stringent than the requirement to measure response in every individual patient. I’m asking if they can demonstrate a definite response in any individual patient.
A Model
Let’s generalize the discussion to degenerative disease, where we have a concept of possibly fluctuating disease activity \(A(t)\) as a function of time. In DMD, one could think of \(A(t)\) as the rate of fibrofatty replacement (FFR) of inflamed muscle tissue. In Alzheimer dementia, this could be2 the rate of amyloid β deposition. Lysosomal storage diseases may lend themselves to a similar concept, and so forth.
I do think it important that this picture could be elaborated substantively in terms of stochastic processes, with activity \(A_t\) modeled as (say) a mean-reverting Ornstein-Uhlenbeck process, and disease progression captured in its time-integral \(S_t = \int_0^t A_u\text{d}u\). But I won’t pursue that here.
What I’d like to focus on is characterizing the putative effect of a therapy as reduction of disease activity. Certainly, this accords with the concept of corticosteroid use in DMD, targeting the inflammation that presumably drives the FFR process.
As a first approximation, let’s treat this reduction as multiplication by a factor \(1-\theta\), for \(\theta \in [0,1]\). (Mnemonic: theta for therapy.) Thus, \(\theta = 0\) is a null therapy, while \(\theta = 1\) corresponds to a cure. In a heterogeneous degenerative disease — muscular dystrophy makes an excellent example here, with its many different mutations generating a wide spectrum of severity — we have to expect substantial heterogeneity of treatment effect (HTE). So we must consider the population distribution of \(\theta\). For simplicity, I’ll posit that a randomly selected individual \(i\) has \(\theta_i\) with the 1-parameter distribution,
\[ \text{P}(\theta_i < \theta) = \theta^\lambda. \]
Thus, for \(\lambda=1\) we have \(\theta \sim \text{U}[0,1]\), while for \(\lambda \rightarrow \infty\) we get increasing concentration of probability mass near \(\theta \approx 1\) (a universal cure), and for \(\lambda \rightarrow 0\) we concentrate the mass near \(\theta \approx 0\) (a marginal therapy).3
In a situation where a sponsor thrashes through a bunch of trials that never get reported, outsiders can nevertheless draw some inferences about \(\lambda\) by using the lack of any case-report of exceptional response as a censored observation of \(\theta\).
Suppose that any activity reduction exceeding \(\theta_c\) would be clinically evident (c for clinical or critical, say). Then every participant treated in one of these trials and never subsequently highlighted in a favorable case report contributes a factor \(\theta_c^\lambda\) to the likelihood, and the likelihood for \(n\) such participants is \((\theta_c^\lambda)^n = e^{n \ln\theta_c \cdot \lambda}.\)
Now \(\lambda \sim \text{Gamma}(\alpha,\beta)\) would yield a conjugate prior. That is, if we chose the prior
\[ p(\lambda) = \frac{\beta^\alpha}{\Gamma(\alpha)} \lambda^{\alpha-1} e^{-\beta\lambda}, \]
then our posterior is also Gamma-distributed:
\[ \begin{aligned} p(\lambda\!\mid\!n) = p(\lambda)\cdot\theta_c^{n\lambda} & \propto \lambda^{\alpha-1}e^{-\beta \lambda}\cdot e^{-n\ln(1/\theta_c)\lambda} \\ & = \lambda^{\alpha-1} e^{-[\beta +n\ln(1/\theta_c)]\lambda} \end{aligned} \]
Indeed, we see \(\lambda \sim \text{Gamma}(\alpha,\beta_n)\) with \(\beta_n = \beta+n\ln(1/\theta_c)\).
Since the \(\ln(1/\theta_c)\) coefficient on \(n\) here is positive (because \(\theta_c < 1\)), we have here a \(\beta_n\) parameter increasing linearly with the number \(n\) of participants treated in these trials. Moreover, this constant \(\ln(1/\theta_c)\) should lie generally in the range \(\frac{1}{5}\) to \(\frac{1}{2}\), depending on how sensitive and informative our clinical assessments are:4
| \(\theta_c\) | \(\ln(1/\theta_c)\) |
|---|---|
| 0.8 | 0.22 |
| 0.7 | 0.36 |
| 0.6 | 0.51 |
Because the Gamma distribution’s \(\beta\) is an inverse scale parameter, \(\beta_n\rightarrow\infty\) shifts the distribution to the left, concentrating the posterior mass toward \(\lambda \rightarrow 0\) (marginality).
Measuring Marginality
To render the marginality concept accessible to analysis, let us write \(\theta_p\) for the \(p\)-quantile of \(\theta\):
\[ p = P(\theta < \theta_p) = \theta_p^\lambda. \]
(Thus \(\theta_{0.5}\) would be median \(\theta\), \(\theta_{0.75}\) would be the top quartile, and \(\theta_{0.9}\) the top decile.)
We can rearrange the equation \(p = \theta_p^\lambda\) to obtain
\[ \frac{\ln(1/\theta_p)}{\ln(1/p)} = \frac{1}{\lambda}. \]
Now, since \(\lambda \sim \text{Gamma}(\alpha,\beta_n)\), we know that \[ \frac{\ln(1/\theta_p)}{\ln(1/p)} = \frac{1}{\lambda} \sim \text{Inv-Gamma}(\alpha,\beta_n). \]
Because the \(\beta\) parameter of \(\text{Inv-Gamma}\) is a scale parameter (rather than an inverse-scale, as with the \(\text{Gamma}\) distribution), we now have that \(\beta_n \rightarrow \infty\) shifts our distribution to the right. This drives \(\ln(1/\theta_p) \rightarrow \infty\) and consequently \(\theta_p \rightarrow 0\).
We can get a quantitative handle on this shift by focusing on the mode of \(\text{Inv-Gamma}(\alpha,\beta_n)\):
\[ \frac{\text{mode}\left(\ln\frac{1}{\theta_p}\right)}{\ln(1/p)} = \frac{\beta_n}{\alpha+1} = \frac{\beta + n\ln\frac{1}{\theta_c}}{\alpha+1} > \frac{n\ln\frac{1}{\theta_c}}{\alpha+1} = \frac{n}{\alpha+1}\cdot\ln(1/\theta_c). \]
Furthermore, using the fact that \(\ln(1/x) \ge 1-x\) for all \(x>0\), we obtain:
\[ \text{mode}\left(\ln\frac{1}{\theta_p}\right) > \frac{n}{\alpha+1}\cdot\ln(1/\theta_c)\cdot\ln(1/p) > \frac{n}{\alpha+1}\cdot(1-\theta_c)\cdot(1-p). \]
Looking for some reasonable numbers to plug in here, consider first that a low-information prior will have small \(\alpha \sim \mathcal{O}(1)\). Accordingly, let’s take \(\alpha = 1\). If we generously (to the sponsor) set a high threshold \(\theta_c = 0.8\) for clinically detectable efficacy, and choose \(p = 0.9\) to focus on the therapeutic effect at the top decile of responses, then \(n\approx 200\) treated to date in eteplirsen trials yields:
\[ \text{mode}\left(\ln\frac{1}{\theta_{0.9}}\right) > \frac{200}{1+1}(0.2)(0.1) = 2 \doteq \ln\frac{1}{0.135}, \]
corresponding to \(\theta_{0.9} < 0.135\) — a quite dismal bound on efficacy.
Now it should be said that the mode (unlike the median) is not invariant under transformations. So this ‘correspondence’ doesn’t directly bound the modal \(\theta_{0.9}\) (on the \(\theta\) scale). Still, since the rightward skew of the \(\text{Inv-Gamma}\) distribution guarantees that \[ \text{median}\left(\ln\frac{1}{\theta_p}\right) > \text{mode}\left(\ln\frac{1}{\theta_p}\right), \] we can at least state that median \(\theta_{0.9} < 0.135.\)
Thus, we conclude there’s a below-50% (Bayesian) chance the top decile of responses achieves better than a 13.5% reduction in disease activity — a strong suggestion that this is a truly marginal drug.
A Simple Formula
It seems entirely reasonable to standardize this type of analysis on the top decile; so let’s fix \(p = 0.9\). Pending any rationale the sponsor might offer for higher \(\alpha\), let’s keep it at \(\alpha = 1\) for now. Then we obtain the simple formula,
\[ \text{median}(\theta_{0.9}) < e^{-(1-\theta_c)\cdot n/20}. \]
This has the advantage of highlighting the most important parameter in this analysis: \(\theta_c\). As a measure of how sensitive our clinical assessments are (against the background of typical intraindividual fluctuation in disease activity over time), this parameter says a lot about how efficient or wasteful our trials can be.
Conclusion
Superficially, Accelerated Approval by FDA looks like setting a lower standard. But an inverted understanding may better serve patients’ interests: AA creates high expectations that are fully capable of being disappointed. If by now we have indeed not seen any exceptional responders among hundreds treated with eteplirsen, we should heed this signal that some robust phenomenon is at work dashing our hopes for the drug.
References
Footnotes
Although this increase was to the order of only ~1% of normal levels, this is nevertheless comparable to the quantities of dystrophin found in patients with the considerably milder Becker form of muscular dystrophy (De Feraudy et al. 2021). So the demonstrated levels of dystrophin production would on their face seem capable of converting DMD phenotypically to BMD.↩︎
Conceptually, at least; the amyloid cascade hypothesis is not incontroversial.↩︎
Observe how this term ‘marginal’ seems appropriate on the dual grounds of its colloquial meaning “limited in extent, significance or stature” and its more formal statistical meaning — that any efficacy will only ever be detected in a marginal analysis averaging over many patients, and never clinically in any given individual.↩︎
Another factor is how much the disease course fluctuates over time within individuals.↩︎