Chapter 5. Information Theory

  1. Aapo Hyvärinen,
  2. Juha Karhunen and
  3. Erkki Oja

Published Online: 15 MAY 2002

DOI: 10.1002/0471221317.ch5

Independent Component Analysis

Independent Component Analysis

How to Cite

Hyvärinen, A., Karhunen, J. and Oja, E. (2001) Information Theory, in Independent Component Analysis, John Wiley & Sons, Inc., New York, USA. doi: 10.1002/0471221317.ch5

Author Information

  1. Neural Networks Research Center, Helsinki University of Technology, Finland

Publication History

  1. Published Online: 15 MAY 2002
  2. Published Print: 21 MAY 2001

ISBN Information

Print ISBN: 9780471405405

Online ISBN: 9780471221319



  • information theory;
  • entropy;
  • mutual information;
  • maximum entropy;
  • negentropy;
  • cumulants;
  • nonpolynomial functions


In this chapter, the basic concepts of information theory are introduced. The latter half of the chapter deals with a more specialized topic: approximation of entropy. These concepts are needed in the ICA methods of Part II.