🚀 go-pugleaf

Simon wrote:
> 
> To all
> 
> X is a discrete random variable that takes on one of the values x1, x2,
> ..... , xn, with respective probabilities p1, p2, .... ,pn.
> 
> The entropy of X is defined as:  H(X) = -Sum{pi*log(pi)} in which log means
> log with base 2 and Sum means summation over all i.
> 
> Please, could someone explain to me why H(X) is maximized when all of the pi
> are equal??
> (It is quite easy to prove when n=2)(But what to do when n>2???)
> 
You have the function

H=-Sum[p_i log p_i]

which you like to maximize. But there is also a constraint, namely

Sum p_i=1

To solve this problem we use the Lagrange-method and build

h=-Sum[p_i(lambda+log p_i)]

Now we maximise this function treating all p_i as independently. Thus we
have to differntiate with respect to p_j

\partial_{p_j} h=-(lambda+1+log p_j)=0

Thus we get:

p_j=exp[-(lambda+1)]=const.

We have to chose further lambda such that you fulfill the constraint,
i.e.,

p_j=1/N

if j runs from 1...N.

How to solve the same problem for continuous probability variables read
my script about statistical physics from my home page. There you find
the physical applications to information theory the maximum entropy
principle comes from (your H has to be read as an eta, which notation
was introduced by Boltzmann ;-))).



-- 
Hendrik van Hees		Phone:  ++49 6159 71-2755
c/o GSI-Darmstadt SB3 3.162	Fax:    ++49 6159 71-2990
Planckstr. 1			mailto:h.vanhees@gsi.de
D-64291 Darmstadt		http://www.gsi.de/~vanhees/index.html