Lecture 12: Optimization: Multiple variables, constraints (part II)

This lecture discusses optimization algorithms, focusing on approximating Hessians for efficient minimization. It compares Broyden's method, Davidon-Fletcher-Powell (DFP), and the state-of-the-art BFGS algorithm. BFGS, using inverse Hessian updates, offers better convergence than DFP due to its handling of inverse matrix changes. The lecture then introduces constrained optimization, KKT conditions, and their application to problems like implicit surface representation and linear programming. This segment details the derivation of the BFGS algorithm, a state-of-the-art optimization method. It contrasts it with the Davidon-Fletcher-Powell (DFP) algorithm, highlighting the BFGS algorithm's superior convergence properties and its reliance on the inverse of the Hessian matrix rather than the Hessian itself. The discussion includes the importance of minimizing the norm of the difference of inverses for better behavior. This segment delves into the properties of the Hessian matrix (second derivative), specifically its symmetry and positive definiteness. It discusses the challenges of ensuring these properties are maintained during optimization, particularly when using updates that don't inherently respect them, leading to the need for refined optimization techniques. This segment explains Braden's algorithm for calculating the derivative of a function from R n to R m , focusing on the Secant style update method which uses the slope of a secant line as a proxy for the derivative. It highlights the practical application of approximating derivatives using readily available data points. This segment introduces constrained optimization problems, where the goal is to minimize a function subject to equality and inequality constraints. It explains the inherent difficulty of simultaneously handling minimization, equality constraints, and inequality constraints, emphasizing that finding even a better feasible point is a valuable outcome. This segment delves into the derivation of Karush-Kuhn-Tucker (KKT) conditions, a crucial concept in optimization. It explains how these conditions handle inequality constraints by cleverly introducing a complementary slackness condition (μᵢhᵢ = 0), which elegantly combines the Lagrange multiplier approach with the consideration of active and inactive constraints. This segment clearly explains the difference between active and inactive constraints in optimization problems. It illustrates how removing an inactive constraint doesn't affect the optimal solution, while removing an active constraint can potentially improve it, providing valuable intuition for understanding constraint behavior. This segment defines feasible points and critical points in the context of constrained optimization. It introduces the challenge of inequality constraints, where derivatives cannot be directly used to find solutions, and sets the stage for the introduction of the Karush-Kuhn-Tucker (KKT) conditions, which provide necessary conditions for optimality in such problems.