Κλιμακώσιμο Σύστημα για κατ' απαίτηση Συλλογή και Επεξεργασία Δεδομένων Δικτυακής Κίνησης - Σωτηρόπουλος, Ιωάννης Νεκτάριος (Εθνικό Μετσόβιο Πολυτεχνείο. Σχολή Ηλεκτρολόγων Μηχανικών και Μηχανικών Υπολογιστών. Τομέας Επικοινωνιών, Ηλεκτρονικής και Συστημάτων Πληροφορικής. Εργαστήριο Διαχείρισης και Βέλτιστου Σχεδιασμού Δικτύων, 2017)
The scope of this thesis is the development of a multilayer platform which is provided on demand to many users in order to store and monitor network traffic (sflow). Following the features of each layer, there was a focus on the adaptability and scalability of the whole platform. Therefore, the final service would be a personalized monitoring system provided on demand with data which corresponds to each user. Specifically, the main part of this platform is a distributed data collection system, where in this case the data is received from the switches of the network infrastructure. For this purpose, Apache Kafka was used for central logging, Logstash and Smonnet, a custom python application developed in-house for processing and data enrichment, Elasticsearch as an analytics engine which has a distributed storage system for monitoring and Kibana for the visualization of the correlations and metrics against time. This platform provides on demand the necessary tools in order to be used as the foundation for the creation of a unified framework for collecting –network– monitoring data, and automatically presenting them to the corresponding tenants of various (experimental) infrastructures as part of the functionality of the system.