Performance and Communication Cost of Deep Neural Networks in Federated Learning Environments: An Empirical Study.

Basmah K. Alotaibi; Fakhri Alam Khan; Yousef Qawqzeh; Gwanggil Jeon; David Camacho

doi:10.9781/ijimai.2024.12.001

Authors

Basmah K. Alotaibi King Fahd University of Petroleum and Minerals
Fakhri Alam Khan King Fahd University of Petroleum and Minerals
Yousef Qawqzeh University of Fujairah
Gwanggil Jeon Incheon National University
David Camacho Universidad Politécnica de Madrid

DOI:

https://doi.org/10.9781/ijimai.2024.12.001

Keywords:

Communication Cost, Convolutional Neural Network (CNN), Deep Neural Networks, Distributive Learning, Federated Learning, Neural Network, Performance, Residual Neural Network (ResNet), Visual Geometry Group (VGG)

Supporting Agencies

1: The authors would like to acknowledge the support received from the Saudi Data and AI Authority (SDAIA) and King Fahd University of Petroleum and Minerals (KFUPM) under SDAIA-KFUPM Joint Research Center for Artificial Intelligence Grant no. JRC-AI-RFP-12. 2: This work has been partially supported by the project PCI2022- 134990-2 (MARTINI) of the CHISTERA IV Cofund 2021 program; by MCIN/AEI/10.13039/501100011033/ and European Union NextGenerationEU/PRTR for XAI-Disinfodemics (PLEC 2021-007681) grant, by European Comission under IBERIFIER Plus - Iberian Digital Media Observatory (DIGITAL-2023-DEPLOY- 04-EDMO-HUBS 101158511); and by EMIF managed by the Calouste Gulbenkian Foundation, in the project MuseAI.

Abstract

Federated learning, a distributive cooperative learning approach, allows clients to train the model locally using their data and share the trained model with a central server. When developing a federated learning environment, a deep/machine learning model needs to be chosen. The choice of the learning model can impact the model performance and the communication cost since federated learning requires the model exchange between clients and a central server in several rounds. In this work, we provide an empirical study to investigate the impact of using three different neural networks (CNN, VGG, and ResNet) models in image classification tasks using two different datasets (Cifar-10 and Cifar-100) in a federated learning environment. We investigate the impact of using these models on the global model performance and communication cost under different data distribution that are IID data and non-IID data distribution. The obtained results indicate that using CNN and ResNet models provide a faster convergence than VGG model. Additionally, these models require less communication costs. In contrast, the VGG model necessitates the sharing of numerous bits over several rounds to achieve higher accuracy under the IID data settings. However, its accuracy level is lower under non-IID data distributions than the other models. Furthermore, using a light model like CNN provides comparable results to the deeper neural network models with less communication cost, even though it may require more communication rounds to achieve the target accuracy in both datasets. CNN model requires fewer bits to be shared during communication than other models.

Downloads

Download data is not yet available.

References

Z. Yang, M. Chen, K.-K. Wong, H. V. Poor and S. Cui, “Federated learning for 6G: Applications, challenges, and opportunities,” Engineering, vol. 8, pp. 33-41, 2022.

B. McMahan, E. Moore, D. Ramage, S. Hampson and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 2017.

D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li and H. V. Poor, “Federated learning for internet of things: A comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 23, pp. 1622-1658, 2021.

X. Ma, J. Zhu, Z. Lin, S. Chen and Y. Qin, “A state-of-the-art survey on solving non-IID data in Federated Learning,” Future Generation Computer Systems, vol. 135, pp. 244-258, 2022.

C. Carrascosa, F. Enguix, M. Rebollo and J. Rincon, “Consensus-based learning for MAS: definition, implementation and integration in IVEs,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 8, pp. 21-32, 2023.

M. Aledhari, R. Razzak, R. M. Parizi and F. Saeed, “Federated learning: A survey on enabling technologies, protocols, and applications,” IEEE Access, vol. 8, pp. 140699-140725, 2020.

T. Li, A. K. Sahu, A. Talwalkar and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, vol. 37, pp. 50-60, 2020.

S. AbdulRahman, H. Tout, H. Ould-Slimane, A. Mourad, C. Talhi and M. Guizani, “A survey on federated learning: The journey from centralized to distributed on-site learning and beyond,” IEEE Internet of Things Journal, vol. 8, pp. 5476-5497, 2020.

C. Briggs, Z. Fan and P. Andras, “A review of privacy-preserving federated learning for the Internet-of-Things,” Federated Learning Systems: Towards Next-Generation AI, pp. 21-50, 2021.

A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie and R. Pedarsani,“Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,” in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Online, 2020.

A. Khan, M. ten Thij and A. Wilbik, “Communication-Efficient VerticalFederated Learning,” Algorithms, vol. 15, p. 273, 2022.

C. Zhang, Y. Xie, H. Bai, B. Yu, W. Li and Y. Gao, “A survey on federated learning,” Knowledge-Based Systems, vol. 216, p. 106775, 2021.

B. Yu, W. Mao, Y. Lv, C. Zhang and Y. Xie, “A survey on federated learning in data mining,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 12, p. e1443, 2022.

L. Li, Y. Fan, M. Tse and K.-Y. Lin, “A review of applications in federated learning,” Computers & Industrial Engineering, vol. 149, p. 106854, 2020.

Z. Lian, J. Cao, Y. Zuo, W. Liu and Z. Zhu, “AGQFL: Communication-efficient Federated Learning via Automatic Gradient Quantization in Edge Heterogeneous Systems,” in 2021 IEEE 39th International Conference on Computer Design (ICCD), Storrs, CT, USA, 2021.

J. Xu, W. Du, Y. Jin, W. He and R. Cheng, “Ternary Compression for Communication-Efficient Federated Learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, p. 1162–1176, 2022.

D. Rothchild, A. Panda, E. Ullah, N. Ivkin, I. Stoica, V. Braverman, J. Gonzalez and R. Arora, “Fetchsgd: Communication-efficient federated learning with sketching,” in Proceedings of the 37th International Conference on Machine Learning, Virtual, 2020.

Y. Zhou, Q. Ye and J. Lv, “Communication-efficient federated learning with compensated overlap-fedavg,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, pp. 192-205, 2021.

Z. Qu, S. Guo, H. Wang, B. Ye, Y. Wang, A. Y. Zomaya and B. Tang, “Partial Synchronization to Accelerate Federated Learning Over Relay-Assisted Edge Networks,” IEEE Transactions on Mobile Computing, vol. 21, pp. 4502-4516, 2021.

B. Alotaibi, F. A. Khan and S. Mahmood, “Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study,” Applied Sciences, vol. 14, p. 2720, 2024.

J. Zhong, Y. Wu, W. Ma, S. Deng and H. Zhou, “Optimizing Multi-Objective Federated Learning on Non-IID Data with Improved NSGA-III and Hierarchical Clustering,” Symmetry, vol. 14, p. 1070, 2022.

X. Wu, X. Yao and C.-L. Wang, “FedSCR: Structure-based communication reduction for federated learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, pp. 1565-1577, 2020.

L. Gao, H. Fu, L. Li, Y. Chen, M. Xu and C.-Z. Xu, “Feddc: Federated learning with non-iid data via local drift decoupling and correction,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, USA, 2022.

Z. Lian, W. Liu, J. Cao, Z. Zhu and X. Zhou, “FedNorm: An Efficient Federated Learning Framework with Dual Heterogeneity Coexistence on Edge Intelligence Systems,” in 2022 IEEE 40th International Conference on Computer Design (ICCD), Olympic Valley, CA, USA , 2022.

Y. Gong, Y. Li and N. M. Freris, “FedADMM: A robust federated deep learning framework with adaptivity to system heterogeneity,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE), Kuala Lumpur, Malaysia, 2022.

S. Zhou, Y. Huo, S. Bao, B. Landman and A. Gokhale, “FedACA: An Adaptive Communication-Efficient Asynchronous Framework for Federated Learning,” in 2022 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS), CA, USA , 2022.

X. Li, Z. Qu, B. Tang and Z. Lu, “Fedlga: Toward system-heterogeneity of federated learning via local gradient approximation,” IEEE Transactions on Cybernetics, vol. 54, pp. 401 - 414, 2023.

Z. Chai, A. Ali, S. Zawad, S. Truex, A. Anwar, N. Baracaldo, Y. Zhou, H. Ludwig, F. Yan and Y. Cheng, “Tifl: A tier-based federated learning system,” in Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, Stockholm, 2020.

Q. Zeng, Y. Du, K. Huang and K. K. Leung, “Energy-efficient radio resource allocation for federated edge learning,” in 2020 IEEE International Conference on Communications Workshops (ICC Workshops), Dublin, Ireland, 2020.

N. M. Jebreel, J. Domingo-Ferrer, D. Sanchez and A. Blanco-Justicia, “LFighter: Defending against the label-flipping attack in federated learning,” Neural Networks, vol. 170, pp. 111-126, 2024.

H. Zhang, J. Jia, J. Chen, L. Lin and D. Wu, “A3fl: Adversarially adaptive backdoor attacks to federated learning,” in Advances in Neural Information Processing Systems, New Orleans, LA, USA, 2024.

S. K. Lo, Q. Lu, C. Wang, H.-Y. Paik and L. Zhu, “A systematic literature review on federated machine learning: From a software engineering perspective,” ACM Computing Surveys (CSUR), vol. 54, pp. 1-39, 2021.

Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, pp. 2278-2324, 1998.

B. Ghimire and D. B. Rawat, “Recent advances on federated learning for cybersecurity and cybersecurity for federated learning for internet of things,” IEEE Internet of Things Journal, vol. 9, pp. 8229-8249, 2022.

Z. Lu, H. Pan, Y. Dai, X. Si and Y. Zhang, “Federated Learning With Non-IID Data: A Survey,” IEEE Internet of Things Journal, pp. 1-1, 2024.

C. Janiesch, P. Zschech and K. Heinrich, “Machine learning and deep learning,” Electronic Markets, vol. 31, pp. 685-695, 2021.

A. Mathew, P. Amudha and S. Sivakumari, “Deep Learning Techniques: An Overview,” Advanced Machine Learning Technologies and Applications: Proceedings of AMLTA 2020, pp. 599-608, 2021.

Z. Li, F. Liu, W. Yang, S. Peng and J. Zhou, “A survey of convolutional neural networks: analysis, applications, and prospects,” IEEE transactions on neural networks and learning systems, vol. 33, pp. 6999 - 7019, 2021.

Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436-444, 2015.

J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, L. Wang, G. Wang, J. Cai and T. Chen, “Recent advances in convolutional neural networks,” Pattern recognition, vol. 77, pp. 354-377, 2018.

N. K. Chauhan and K. Singh, “A review on conventional machine learning vs deep learning,” in 2018 International conference on computing, power and communication technologies (GUCON), Greater Noida, India, 2018.

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.

A. S. Rao, T. Nguyen, M. Palaniswami and T. Ngo, “Vision-based automated crack detection using convolutional neural networks for condition assessment of infrastructure,” Structural Health Monitoring, vol. 20, pp. 2124-2142, 2021.

W. Wang, Y. Yang, X. Wang, W. Wang and J. Li, “Development of convolutional neural network and its application in image classification: a survey,” Optical Engineering, vol. 58, pp. 040901-040901, 2019.

M. Pak and S. Kim, “A review of deep learning in image recognition,” in 2017 4th international conference on computer applications and information processing technology (CAIPT), Kuta Bali, Indonesia, 2017.

K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016.

M. Shafiq and Z. Gu, “Deep Residual Learning for Image Recognition: A Survey,” Applied Sciences, vol. 12, p. 8972, 2022.

A. Krizhevsky, G. Hinton and others, “Learning multiple layers of features from tiny images,” University of Tront, Toronto, ON, Canada, 2009.

K. Hsieh, A. Phanishayee, O. Mutlu and P. Gibbons, “The Non-IID Data Quagmire of Decentralized Machine Learning,” in Proceedings of the 37th International Conference on Machine Learning, Virtual, 2020.