--------------------------------------------------------------------------
Mini-workshop UT1-ENSEEIHT-ANITI on "Optimization and Statistics"
September 12th, 2020 - INP-ENSEEIHT
--------------------------------------------------------------------------
The program is as follows:
9:00-9:40 : M. Teboulle (Tel Aviv University, Invited professor UT1)
A fresh look on the analysis of Lagrangian methods for composite
minimization
9:50-10:30 : S. Sabach (The Technion, Invited professor UT1)
Proximal minimization algorithms beyond Lipschitz gradient continuity.
10:30-10:50 : Coffee break
10:50-11:30 : S. Gadat (UT1, IUF)
Smooth stochastic optimization with second order methods.
11:40-12:20 : J. Fadili (ENSICAEN, IUF)
Model consistency with low-complexity regularization in learning and
optimization.
**************************************
Abstracts by chronological order:
**************************************
M. Teboulle, A fresh look on the analysis of Lagrangian methods for
composite minimization,
Lagrangian based methods have re-emerged as starring in optimization
algorithms, being effective in a wide spectrum of modern applications
arsing in data science. The analysis of these methods often appears as
sophisticated, obscure and lengthy.
In this talk I will present a novel and simplified convergence analysis
framework.
Joint work with Shoham Sabach (Technion).
S. Sabach, Proximal minimization algorithms beyond Lipschitz gradient
continuity,
We focus on nonconvex and nonsmooth minimization problems with a
composite objective, where the differentiable part of the objective is
freed from the usual and restrictive global Lipschitz gradient
continuity assumption. We study and analyze a Bregman-based proximal
gradient method for this model, which is proven to globally converge to
a critical point under natural assumptions on the problem’s data, and,
in particular, for semi-algebraic problems. To illustrate the power and
potential of our general framework and results, we present numerical
experiments on challenging applications in data science.
S. Gadat, Smooth stochastic optimization with second order methods,
With the striking development of machine learning optimization problems,
recursive methods like stochastic gradient descent has become a rising
theme of interest.
In this talk, we will present some recent advances on second order
methods for stochastic optimization of a smooth multidimensional real
function, among them the heavy ball, the Polyak and the Nesterov methods.
We will consider in particular some almost sure convergence result in
the case of non-convex multi-well objectives. For convex and KL
functions, we will also explain how we can obtain some non-asymptotic
and asymptotic results.
We will provide some examples on concrete statistical problems.
J. Fadili, Model consistency with low-complexity regularization in
learning and optimization,
Low-complexity non-smooth convex regularizers are routinely used to
impose some structure on the coefficients for linear predictors in
inverse problems and machine learning. Model consistency amounts then to
provably select the correct structure (e.g., support or rank) by
regularized empirical risk minimization. In this talk, I will introduce
two classes of convex regularizers enjoying a strong geometric structure
for which exact or enlarged model consistency can be proved. For the
first class, coined partly smooth functions, exact model consistency is
established under an appropriate non-degeneracy condition. In situations
where non-degeneracy condition is not fulfilled (typically with highly
correlated designs), regularization methods tend to select larger
models. We will then provide the theoretical underpinning of this
behaviour by introducing a class of convex regularizers that we coin
“mirror stratifiable”. This class of functions is ubiquitous in data
processing, machine learning, and statistics. We show that this
“mirror-stratifiable” structure allows to establish sharp model
consistency guarantees which are applicable to optimal solutions of the
learning problem, and also to the iterates computed by a certain class
of stochastic proximal splitting algorithms.