Performance Testing¶
Introduction¶
These functions provide performance testing support, allowing for the quick comparison of models, experiment design heuristics and quality parameters. QInfer’s performance testing functionality can be quickly applied without writing lots of boilerplate code every time. For instance:
>>> import qinfer
>>> n_particles = int(1e5)
>>> perf = qinfer.perf_test(
... qinfer.SimplePrecessionModel(), n_particles,
... qinfer.UniformDistribution([0, 1]), 200,
... qinfer.ExpSparseHeuristic
... )
Function Reference¶

qinfer.
perf_test
(model, n_particles, prior, n_exp, heuristic_class, true_model=None, true_prior=None, true_mps=None, extra_updater_args=None)[source]¶ Runs a trial of using SMC to estimate the parameters of a model, given a number of particles, a prior distribution and an experiment design heuristic.
Parameters:  model (qinfer.Model) – Model whose parameters are to be estimated.
 n_particles (int) – Number of SMC particles to use.
 prior (qinfer.Distribution) – Prior to use in selecting SMC particles.
 n_exp (int) – Number of experimental data points to draw from the model.
 heuristic_class (qinfer.Heuristic) – Constructor function for the experiment design heuristic to be used.
 true_model (qinfer.Model) – Model to be used in
generating experimental data. If
None
, assumed to bemodel
.  true_prior (qinfer.Distribution) – Prior to be used in
selecting the true model parameters. If
None
, assumed to beprior
.  true_mps (np.ndarray) – The true model parameters. If
None
, it will be sampled fromtrue_prior
. Note that the performance record can only handle one outcome and therefore ONLY ONE TRUE MODEL. An error will occur iftrue_mps.shape[0] > 1
returnsTrue
.  extra_updater_args (dict) – Extra keyword arguments for the updater, such as resampling and zeroweight policies.
Rtype np.ndarray: See Performance Results Structure for more details on the type returned by this function.
Returns: A record array of performance metrics, indexed by the number of experiments performed.
Performance Results Structure¶
Perfromance results, as collected by perf_test()
, are returned as
a record array with several fields, each describing a different
metric collected by QInfer about the performance. For a single performance
trial, the shape of this array is (n_exp, )
, such that perf[idx_exp]
returns metrics describing the performance immediately following collecting
the datum idx_exp
.
Field  Type  Description 

elapsed_time  float  Time (in seconds) elapsed during the SMC update for this experiment. Includes resampling, but excludes experiment design, generation of “true” data and calculation of performance metrics. 
loss  float  Decisiontheoretic loss incured by the estimate after updating with this experiment, given by the quadratic loss \(\Tr(Q (\hat{\vec{x}}  \vec{x}) (\hat{\vec{x}}  \vec{x})^{\mathrm{T}})\). 
resample_count  int  Number of times that resampling was performed on the SMC updater. 