z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learning
报告题目：z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learning
报告专家：张纵辉 （香港中文大学（深圳）理工学院）
报告时间：2022年11月25日16：30-17：30
报告地点：ZOOM ID：819 5278 4619 (221125)
报告摘要：Federated Learning (FL) is a promising privacy-preserving distributed learning paradigm but suffers from high communication cost when training large-scale machine learning models. Sign-based methods, such as SignSGD, have been proposed as a biased gradient compression technique for reducing the communication cost. However, sign-based algorithms could diverge under heterogeneous data, which thus motivated the development of advanced techniques, such as the error-feedback method and stochastic sign-based compression, to fix this issue. Nevertheless, these methods still suffer from slower convergence rates. Besides, none of them allows multiple local SGD updates like FedAvg. In this paper, we propose a novel noisy perturbation scheme with a general symmetric noise distribution for sign-based compression, which not only allows one to flexibly control the tradeoff between gradient bias and convergence performance, but also provides a unified viewpoint to existing stochastic sign-based methods. More importantly, we propose the very first sign-based FedAvg algorithm (z-SignFedAvg). Theoretically, we show that z-SignFedAvg achieves a faster convergence rate than existing sign-based methods and, under the uniformly distributed noise, can enjoy the same convergence rate as its uncompressed counterpart. Extensive experiments are conducted to demonstrate that the z-SignFedAvg can achieve competitive empirical performance on real datasets.
专家简介：张纵辉，台湾清华大学博士，国家高层次青年人才，现为香港中文大学（深圳）理工学院副教授、助理院长（主管教育）、广东省大数据计算基础理论与方法重点实验室副主任、深圳市大数据研究院研究员。目前与过去分别担任国际信号处理顶级期刊《IEEE Transactions on Signal Processing》资深编委，网络信号处理顶级期刊《IEEE Transactions on Signal and Information Processing over Networks》、《IEEE Open Journal of Signal Processing》编委，IEEE信号处理协会（SPS）通信网络信号处理技术委员会委员、感知通信一体化工作组发起人与首届主席和IEEE SPS 董事会亚太区独立主席。
张纵辉教授专注于面向无线通信、机器学习的关键信号处理和优化方法的基础研究，已发表IEEE 国际顶级期刊/会议论文130余篇，包括5篇ESI高被引论文，总计被引6000余次。 “以优化及信号处理技术对无线通信的贡献”获得2015年IEEE通信学会亚太区杰出青年学者奖；与合作者在鲁棒波束赋形优化方面的基础性工作于2018年获得国际信号处理领域最具影响力的IEEE信号处理协会最佳论文奖；2021年以高效分布式优化方法的开创性工作第二次获得IEEE信号处理协会最佳论文奖（全球获奖超过2次以上仅10人）;他也获得香港中文大学（深圳）理工学院首届卓越科研奖。近年来主持和参与包括国家自然科学基金重点项目、面上项目、广东省重点项目、深圳市杰出青年项目以及华为、中兴等企业的横向项目10余项。其中“分布式基带架构的新型信道估计算法”获得华为2022年技术成果转化二等奖。
邀请人：王治国
