Simple Private Noisy SGD is Minimax-Optimal under Population Strong Convexity
Valentin Roth, Marco Avella Medina
Abstract
Stochastic gradient descent (SGD) is at the center of modern machine learning and is known to be statistically minimax-optimal in many regression settings, e.g., for convex Generalized Linear Models (GLMs). Yet, it is unknown whether its simplest differentially private Noisy SGD counterpart fulfills similar favorable guarantees even on strongly convex losses. We resolve this question affirmatively. We upper bound the parameter estimation error for a single pass of Noisy SGD with a fixed stepsize schedule and constant batch sizes on smooth and strongly convex Lipschitz losses over a compact convex domain under Renyi DP. Specialized to strongly convex GLMs, our error bounds match minimax lower bounds we derive based on the Score Attack framework. In particular, this provides the first minimax-optimal rate for Noisy SGD in private logistic regression, avoiding poly-logarithmic terms present in existing results. Our analysis is versatile as it relies on standard convex analysis and establishes the same rates in expectation and with high probability.
Create a lesson
Related papers
Anytime-valid inference for time series via self-normalization
Tim Kutta, Xiaofeng Shao
Local Clustering Tests and Botnet Localization in Sparse Random Geometric Networks
Luca Di Persio, Gianmarco Bet, Youness Outaleb
Asymptotic efficiency comparison of explicit Dirichlet shape estimators
Ioannis Oikonomidis, Samis Trevezas
Sharp minimax rates and adaptive estimation of latent proportions in bivariate Poisson mixtures
Essoham Ali
Inference in High-Dimensional Linear Mediation Models
Samriddha Lahiry, Rajarshi Mukherjee, Pragya Sur
Gaussian Quasi-Likelihood Estimation for Nonsynchronously Observed Lévy processes
Teppei Ogihara, Yuma Uehara