Robustness Testing

Introduction

In developing statistical inference applications, it is essential to test the robustness of one’s software to errors and noise of various kinds. Thus, QInfer provides tools to do so by corrupting likelihood calculations in various realistic ways.

Modeling Statistical and Sampling Error

A basic kind of robustness testing can be performed by using PoisonedModel, which adds noise to a model’s likelihood() method in such a way as to simulate sampling errors incurred in LFPE approaches [FG13]. The noise that PoisonedModel adds can be specified as the tolerance of an adaptive likelihood estimation (ALE) step [FG13], or as the number of samples and hedging used for a hedged maximum likelihood estimator of the likelihood [FB12]. In either case, the requested noise is added to the likelihood reported by the underlying model, such that

\[\widehat{\Pr}(d | \vec{x}; \vec{e}) = \Pr(d | \vec{x}; \vec{e}) + \epsilon,\]

where \(\widehat{\Pr}\) is the reported estimate of the true likelihood.

For example, to simulate using adaptive likelihood estimation to reach a threshold tolerance of 0.01:

>>> from qinfer.test_models import SimplePrecessionModel
>>> from qinfer.derived_models import PoisonedModel
>>> model = PoisonedModel(SimplePrecessionModel(), tol=0.01)

Testing Faulty Simulators

TODO