Ανάλυση επίδοσης και μοντελοποίηση του αλγορίθμου συσταδοποίησης k-means σε κεντρικό και κατανεμημένο περιβάλλον - Αφεντουλίδης, Γρηγόριος (Εθνικό Μετσόβιο Πολυτεχνείο. Σχολή Ηλεκτρολόγων Μηχανικών και Μηχανικών Υπολογιστών. Τομέας Τεχνολογίας Πληροφορικής και Υπολογιστών, 2015)
In recent years, interest in execution of data analytics jobs in the science, technology and business fields as well, skyrockets. That led in the development of execution engines that are now offered as services in IaaS providers and take on the execution of such jobs. Meanwhile these execution engines introduce diverse characteristics and runtime architectures, so there exists the need to analyze not only the resources they require but also the execution time they manage to achieve. The complexity of the goal, raises significantly from the fact that this is also affected by the execution parameters of the underlying algorithm. The outcome of such analysis will provide us the means to understand the advantages and disadvantages of every execution engine under specific circumstances and also let us deploy user policies in cloud environments that relate to the cost and the time restraints of the executions. For this purpose we must conduct an experimental analysis on those engines through a profiling process, where we will measure the usage of the resources as well as the overall execution time with carefully selected execution samples. The results of this process will enable us to construct static predictive models that could simulate the performance of the engines for varying execution parameters. In this diploma thesis we are studying the k-means algorithm, which is used for clustering jobs, on the centralized environment of Weka and the distributed environment of Apache Spark. We suggest and deploy two profiling architectures that handle the collection of the metrics regarding the resources used and the time of every experimental execurion. We analyze, with the help of the results of the profiling procedure, the performance of those two engines as the execution parameters vary, and we mention the advantages and disadvantages we notice. We also use the collected data for the construction of predictive models for every metric and every engine, and we comment on the accuracy of those models as well as the usefullness they provide. We alter the size of the cluster in the distributed version in order to check the scalability of the algorithm and we notice up to 30% time improvement. Finally we attempt to contrast the two engines, which designates the supremacy of Spark even for very small datasets.