Chapter 3. Gradients and Optimization Methods

  1. Aapo Hyvärinen,
  2. Juha Karhunen and
  3. Erkki Oja

Published Online: 15 MAY 2002

DOI: 10.1002/0471221317.ch3

Independent Component Analysis

Independent Component Analysis

How to Cite

Hyvärinen, A., Karhunen, J. and Oja, E. (2001) Gradients and Optimization Methods, in Independent Component Analysis, John Wiley & Sons, Inc., New York, USA. doi: 10.1002/0471221317.ch3

Author Information

  1. Neural Networks Research Center, Helsinki University of Technology, Finland

Publication History

  1. Published Online: 15 MAY 2002
  2. Published Print: 21 MAY 2001

ISBN Information

Print ISBN: 9780471405405

Online ISBN: 9780471221319



  • vector gradient;
  • matrix gradient;
  • unconstrained optimization;
  • constrained optimization


In this chapter, the authors discuss some typical iterative optimization algorithms and their properties. Mostly, the algorithms are based on the gradients of the cost functions. Therefore, vector and matrix gradients are reviewed first, followed by the most typical ways to solve unconstrained and constrained optimization problems with gradient-type learning algorithms.