Context: We were invited to assess the behaviour of a statistical model of failure of a given defence related mechanism. This model was already at an advanced stage of development, having been used for several years. Given the nature of the mechanism, there was very little experimental data available (70 data points) and it was infeasibly costly to recommend new experimental data points. The model itself was developed with the objective of recommending new experiments to learn about the sensitivity of the given mechanism to failure subject to a certain stimulus.
Our Assessment: On static analysis of the supplied code and related report, we identified that the model was developed was a non-linear regression model subject to constraints to maintain physical consistency.
- We identified issues with how the constraints were imposed, which was being done so by introducing large volumes of synthetic data which were used to “teach” the model to fail immediately if the constraint was not satisfied. This approach, while commonplace gives rise to several critical issues, which we identified. More crucially, this was biasing the output of the experimental design used to recommend new experiments.
- We recommended an alternative approach making use of implicitly satisfied constraints to ensure that they are satisfied without needing to introduce new data.
- Given the lack of data we recommend a Bayesian approach to allow integration of expert knowledge into the model through priors.
Follow-On Based on Recommendations: Based on the above recommendations we were asked to develop a new model for this system. We implemented a Bayesian statistical model which does not rely on artificial data to impose boundary conditions. The resulting model was far more stable to uncertainty and provided far superior predictions of outcomes. We provided a new experimental design scheme based on this model, which is currently being used to recommend new experiments for this system.
Timescales and Effort: Static analysis of the code and report required 1 man-month of effort, overall.
