FedWKD: Federated learning weighted aggregation with knowledge distillation for IoT forecasting

Federated Learning (FL) has emerged as a promising solution for decentralized Machine Learning (ML) that does not have direct access to datasets in a centralized manner. However, the traditional FL methods are prone to overfitting and model drift at the client level and server divergence during classic aggregation in case of heterogeneous, non-independent and identically distributed (non-IID) time-series sensor data. In this paper, we propose a novel approach that integrates bidirectional Knowledge Distillation (KD) by using distilled soft predictions of each client model, called logits, as well as server model distilled logits. Specifically, clients use KD regularization techniques using the received server logits during model training, while the server uses received client logits to build a score for weighted global aggregation each round. Thus, we avoid local training overhead for clients, while also improving global aggregation using weighting on the server-side for each training round for non-IID data. Experimental results highlight its ability to improve forecasting metrics compared to other methods such as CADIS and FEDGKD, using loss, error, and execution time metrics, hence bettering generalization and minimizing client drift and bias.