Ricerca Resilient Computing LabOur LabSoftware

Software and datasets

Confidence Ensembles

https://github.com/tommyippoz/confidence-ensembles

Confidence ensembles, either Confidence Bagging (ConfBag) or Confidence Boosting (ConfBoost), can use any existing classifier as a base estimator, and build a meta-classifier through a training process that does not require any additional information with respect to those that are already used to train the base estimator. ConfBag reworks the traditional bagging by using only the most confident base-learners to assign the label to input data, whereas ConfBoost creates base-learners that are more and more specialized to classify train data items for which other base-learners cannot make a confident prediction. This is radically different from the typical boosting strategies, which always require a labeled training set as base-learners are typically trained using train data that gets misclassified by other base-learners.

Paper: Zoppi, T., & Popov, P. (2025). Confidence ensembles: Tabular data classifiers on steroids. Information Fusion, 120, 103126.

HOWLATE - Attack latency measurements

https://github.com/TommasoPuccetti/howlate

Anomaly-based Network Intrusion Detectors (NIDs) are effective and viable solutions to defend against cyberattacks. Such NIDs are machine learners trained on datasets that describe the behavior of a system in nominal operating conditions and under attack. Most often, the evaluation of an NID focuses on assessing its ability to correctly classify data as normal or anomalous. Recently, it has been repeatedly shown that the attack latency, i.e., the time from the start of an attack until its detection, is a relevant metric. However, measuring attack latency is most often complicated, if not impossible, because of the way datasets are built. This paper proposes HOWLATE, a tool designed to evaluate the attack latency of intrusion detection systems alongside state-of-the-art classification metrics. In addition, the tool constructs NID datasets from raw network traces in a format that enables the computation of attack latency. The tool, realized in Python and released open source, is currently the only one able to perform these tasks. 

 

SPROUT – a Safety wraPper thROugh ensembles of UncertainTy measures

https://github.com/tommyippoz/SPROUT

SPROUT implements quantitative uncertainty/confidence measures and integrates well with existing frameworks (e.g., Pandas, Scikit-Learn, PYOD, AutoGluon, and many more) that are commonly used in the machine learning domain for classification. While designing, implementing and testing such library we made sure it would work with supervised classifiers, as well as unsupervised classifiers. Also, we created connectors for tabular datasets as well as image datasets such that those classifiers can be fed with different inputs and provide confidence measures related to the execution of many classifiers on datasets with a different structure.

Paper: Zoppi, T., Ceccarelli, A., & Bondavalli, A. (2023). Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers. FRONTIERS IN ARTIFICIAL INTELLIGENCE AND APPLICATIONS, 372, 3156-3164.

Intrusion Detection Dataset for ROS2 services

Most of the intrusion detection datasets to research machine learning-based intrusion detection systems (IDSs) are devoted to cyber-only systems, and they typically collect data from one architectural layer. Often the attacks are generated in dedicated attack sessions, without reproducing the realistic alternation and overlap of normal and attack actions. We present a dataset for intrusion detection by performing penetration testing on an embedded cyber-physical system built over Robot Operating System 2 (ROS2). Features are monitored from three architectural layers: the Linux operating system, the network, and the ROS2 services. The dataset is structured as a time series and describes the expected behavior of the system and its response to ROS2-specific attacks: it repeatedly alternates periods of attack-free operation with periods when a specific attack is being performed. This allows measuring the time to detect an attacker and the number of malicious activities performed before detection. Also, it allows training an intrusion detector to minimize both, by taking advantage of the numerous alternating periods of normal and attack operations.

Puccetti, T., Nardi, S., Cinquilli, C. et al. ROSPaCe: Intrusion Detection Dataset for a ROS2-Based Cyber-Physical System and IoT Networks. Sci Data 11, 481 (2024).

https://github.com/TommasoPuccetti/rospace_dataset

Emulation of Camera Failures

A python library that aims to simulate failures that may occur in a camera during the acquisition/processing phase.

To support the definition of safe and robust vehicle architectures and intelligent systems, we define the failures model of a vehicle camera, together with an analysis of effects and known mitigations. As a natural consequence, here we present a software library for the generation of the corresponding failed images. These images are then fed the trained agent of an autonomous driving simulator: the misbehavior of the trained agent allows a better understanding of failures effects and especially of the resulting safety risk.

Github is available at: https://github.com/francescosecci/Python_Image_Failures

More information:

• Secci, Francesco, and Andrea Ceccarelli. "On failures of RGB cameras and their effects in autonomous driving applications." 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020.
• Ceccarelli, Andrea, and Francesco Secci. "RGB cameras failures and their effects in autonomous driving applications." IEEE Transactions on Dependable and Secure Computing (2022).

RELOAD - Rapid EvaLuation Of Anomaly Detectors

RELOAD (Rapid EvaLuation Of Anomaly Detectors) is a tool that allows to easily compare different algorithms for anomaly detection. It is written in Java, wrapping algorithms coming from other Java-based frameworks such as ELKI or WEKA.

For further information, please refer to the Github WIKI.

Information about the tool can be found in the following papers:

• Zoppi, T., Ceccarelli, A., & Bondavalli, A. (2019). Evaluation of Anomaly Detectors Made Easy with RELOAD. To appear at 30th International Symposium on Software Reliability Engineering (ISSRE 2019), Oct 2019.
• Zoppi, T., Ceccarelli, A., & Bondavalli, A. (2019). MADneSs: a Multi-layer Anomaly Detection Framework for Complex Dynamic Systems. IEEE Transactions on Dependable and Secure Computing, DOI: 10.1109/TDSC.2019.2908366 (2019, May)
• Falcão, F., Zoppi, T., Silva, C. B. V., Santos, A., Fonseca, B., Ceccarelli, A., & Bondavalli, A. (2019, April). Quantitative comparison of unsupervised anomaly detection algorithms for intrusion detection. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (pp. 318-327), ACM. DOI: 10.1145/3297280.3297314

CHESS State-Based Analysis (CHESS-SBA)

The CHESS "State-Based Analysis" plugin is part of PolarSys CHESS, an open source methodology and tool for the development of high-integrity embedded systems. The CHESS methodology was devised and implemented initially in the CHESS project, later extended in the CONCERTO project, and then further developed within other projects.

This plugin performs Quantitative Dependability Analysis using a variant of the Stochastic Petri Nets formalism, starting from models specified in the CHESS ML language. The plugin is able to automatically compute system-level dependability metrics, based on dependability properties of individual components, and a description of the system, software, and/or hardware architecture. 

The whole CHESS Framework is released as open source and it is now an Eclipse project under the PolarSys Working Group. CHESS-SBA is also available on GitHub, jointly with an extensive wiki as documentation.

For more information please contact  Paolo Lollini.

CHESS Statistical Trace Analyzer

This software is used to analyze the statistical properties of run-time execution traces of different variables. Its application domain is real-time safety-critical systems. Key variables like release jitter, execution time, etc. are analyzed to devise probabilistic upper bounds, which are guaranteed to be respected with a given given coverage. This tool is part of the CHESS Framework, and it has been developed within the CONCERTO project.

The tool is released as open source, and available on GitHub: https://github.com/montex/it.unifi.rcl.chess.traceanalysis

For more information please contact Paolo Lollini.