Neural Predictive Monitoring
Virtual seminars Computer Science: Joint ICE-TCS and GSSI - Nicola Paoletti
Schedule: 24 June, 13:30 GMT/15:30 Italian time
Virtual link: https://accounts.eyeson.team/rooms/XpWCVrLlnc1BsF
Speaker: Nicola Paoletti Lecturer (Assistant Professor) @ Royal Holloway, University of London, UK https://nicolapaoletti.com
Title: Neural Predictive Monitoring
Abstract: In past work [1], we proposed Neural State Classification (NSC), a method for runtime predictive monitoring of Hybrid Automata (HA) using deep neural networks (DNNs). NSC trains a DNN as an approximate reachability predictor that labels an HA state x as positive if an unsafe state is reachable from x within a given time bound, and labels x as negative otherwise. Even though we demonstrated that NSC predictors have very high accuracy, these are prone to prediction errors that can negatively impact reliability and jeopardize the safety of the system.
To overcome this limitation, we present Neural Predictive Monitoring (NPM) [2], a technique that complements NSC predictions with estimates of the predictive uncertainty. These measures yield principled criteria for the rejection of predictions likely to be incorrect, without knowing the true reachability values. We also present an active learning method that significantly reduces the NSC predictor's error rate and the proportion of rejected predictions. We develop two versions of NPM based respectively on the use of frequentist and Bayesian techniques to learn the predictor and the rejection rule. The frequentist approach builds on the framework of conformal prediction (which allows to construct prediction regions with probabilistic guarantees); the Bayesian approach derives uncertainty estimates using Bayesian neural networks.
Our approach is highly efficient, with computation times on the order of milliseconds, and effective, managing in our experimental evaluation to successfully reject almost all incorrect predictions. In our experiments on a benchmark suite of six hybrid systems, we found that the frequentist approach consistently outperforms the Bayesian one. We also observed that the Bayesian approach is less practical, requiring a careful and problem-specific choice of hyperparameters.
[1] Bortolussi L, Cairoli F, Paoletti N, Smolka SA, Stoller SD. Neural Predictive Monitoring. In International Conference on Runtime Verification 2019 (pp. 129-147).
[2] Phan D, Paoletti N, Zhang T, Grosu R, Smolka SA, Stoller SD. Neural state classification for hybrid systems. In International Symposium on Automated Technology for Verification and Analysis 2018 (pp. 422-440).
--
Seminars Computer Science
Gran Sasso Science Institute (GSSI), L'Aquila, Italy
URL: http://cs.gssi.it
