🚀 go-pugleaf

In article 92Sep14212000@image.mit.edu, popat@image.mit.edu (Kris Popat) writes:
>
>Suppose you have a parametric model for the probability density
>function of a discrete random variable, and a set of observed values.
>Call the model pdf p(x), and call the observed values x_1,...x_N.  The
>goal is to find parameter values that make the model pdf p(x_i)
>approximate the "true" but unknown pdf q(x_i).
>
>One way to fit the model to the data would be to find parameter values
>that minimize
>
>                    -sum(log p(x_i))
>
>i.e., to minimize the total self-information of the observed points
>with respect to the model.  This "works" because for a given true pdf
>q(x_i) and for all valid model pdfs p(x_i),
>
>         E[-log(p(x_i))] = -sum( q(x_i) log p(x_i) )
>
>is minimized when p = q.
>
>I'd like a pointer to any papers that discuss this or similar
>approaches to parametric pdf fitting.
>
>Kris Popat
>MIT Rm E15-391  Cambridge, MA  02139


This is the best approach. You're actually minimizing the "sample" relative
entropy D(p||q) which is has many properties of a distance metric between pdf's.
D(p||q) is defined as sum(p(x) log[p(x)/q(x)]). See Cover's information theory text.

For example for a two-sided Laplacian density q(x;A) = 0.5Aexp(-A|x|),
you end up with A = 1/E[|x|], where E[|x|] is the sample mean of the absolute values.

Jerry Shapiro