OPER627 Nonlinear Optimization

Instructor:Yongjia Song
Time: Monday, Wednesday, 5:30-6:45 PM
Location: 2104 Harris Hall
Office hours: 4140 Harris Hall, Tuesday, Thursday 3:00-4:00 PM, and by appointment.

Description: This course will address basic theory and algorithms for nonlinear optimization (unconstrained and constrained).

The course will cover the following topics in nonlinear optimization:

Part one: Unconstrained nonliner optimization:

Optimality conditions
Fundamental solution framework for nonlinear optimization
Nonlinear optimization algorithms

Steepest descent
Newton's methods
Quasi-Newtion methods

Large-scale nonlinear optimization algorithms

Large-scale Newton's method
Conjugate gradient method
Stochastic gradient descent (if time permits)

Part two: Constrained nonlinear optimization:

Optimization over simple constraints, gradient projection
Optimality conditions, KKT, constraint qualification
Lagrangian-based nonlinear optimization algorithms

Augmented Lagrangian
Interior point methods

Course Syllabus: Link

Lecture 1 (08/26/13): Introduction. Slides
Related Readings: Appendix A, N-W book
Handouts:

There are many online Matlab tutorials, for example, here
M. Ferris' math review
S. Wright's math review

Lecture 2 (08/28/13): Optimality conditions. Slides
Related Readings: Chapter 2, N-W book
Lecture 3 (09/04/13): Optimality conditions in the convex case. Slides
Lecture 4 (09/09/13): Line search. Slides
Lecture 5 (09/11/13): Convergence Analysis. Slides
Related Readings: Chapter 3.1, 3.2, N-W book
Lecture 6 (09/16/13): Convergence of line search methods. Slides
Related Readings: Chapter 3.2, 3.3, N-W book
Lecture 7 (09/18/13): Modified Newton and Quasi-Newton. Slides
Related Readings: Chapter 3.4, Chapter 6, N-W book
Lecture 8 (09/23/13): BFGS and L-BFGS. Slides
Related Readings: Chapter 6.4, 7.2, N-W book
Lecture 9 (09/25/13): Trust-region methods. Slides
Related Readings: Chapter 4 intro, Chapter 4.3, N-W book
Lecture 10 (09/30/13): Trust-region methods based on the Cauchy point. Slides
Related Readings: Chapter 4.1, 4.2, N-W book
Lecture 11 (10/02/13): Conjugate gradient method. Slides
Related Readings: Chapter 5.1, N-W book
Lecture 12 (10/09/13): CG methods in large-scale optimization. Slides
Related Readings: Chapter 5.2, 7.1, N-W book
Lecture 13 (10/14/13): Regression and least-squares problems. Slides
Related Readings: Chapter 10.1, 10.2, 10.3, N-W book
Lecture 14 (10/16/13): Midterm review. Slides
Lecture 15 (10/28/13): Optimization over a simple set. Slides
Lecture 16 (10/30/13): Algorithms for optimization over a simple set. Slides
Lecture 17 (11/04/13): Theory of constrained optimization: from geometry to algebra. Slides
Lecture 18 (11/06/13): Theory of constrained optimization: general constraint set. Slides
Lecture 19 (11/11/13): Theory of constrained optimization: Lagrangian duality. Slides
Lecture 20 (11/13/13): Overview of algorithms for constrained optimization. Slides
Related Readings: Chapter 15, N-W book
Lecture 21 (11/18/13): Penalty functions for constrained optimization. Slides
Related Readings: Chapter 17, N-W book
Lecture 22 (11/20/13): Augmented Lagrangian method. Slides
Related Readings: Chapter 17, N-W book
Lecture 23 (11/25/13): Interior point method. Slides
Related Readings: Chapter 19, N-W book
Lecture 24 (12/02/13): Sequential quadratic programming. Slides
Related Readings: Chapter 18, N-W book
Course materials are also available at Blackboard.