Gallier J., Quaintance J. Fundamentals of Optimization Theory With Applications to Machine Learning

University of Pennsylvania (Dept. of Computer and Information Science), 2019. — 645 p.This second volume covers some elements of optimization theory and applications, especially to machine learning. This volume is divided in five parts:Preliminaries of Optimization Theory.
Linear Optimization.
Nonlinear Optimization.
Applications to Machine Learning.
An appendix on Hilbert Bases and the Riesz–Fischer Theorem.Part I is devoted to some preliminaries of optimization theory. Part II deals with the special case where the objective function is a linear form and the constraints are affine inequality and equality constraints. This subject is known as linear programming, and the next four chapters give an introduction to the subject. Part III is devoted to nonlinear optimization, which is the case where the objective function J is not linear and the constaints are inequality constraints. Since it is practically impossible to say anything interesting if the constraints are not convex, we quickly consider the convex case. The next three chapters constitute Part IV, which covers some applications of optimization theory (in particular Lagrangian duality) to machine learning. Except for a few exceptions we provide complete proofs. We did so to make this book self-contained, but also because we believe that no deep knowledge of this material can be acquired without working out some proofs. However, our advice is to skip some of the proofs upon first reading, especially if they are long and intricate.