nltk.HeldoutProbDist¶

class nltk.HeldoutProbDist(base_fdist, heldout_fdist, bins=None)[source]¶

The heldout estimate for the probability distribution of the experiment used to generate two frequency distributions. These two frequency distributions are called the “heldout frequency distribution” and the “base frequency distribution.” The “heldout estimate” uses uses the “heldout frequency distribution” to predict the probability of each sample, given its frequency in the “base frequency distribution”.

In particular, the heldout estimate approximates the probability for a sample that occurs r times in the base distribution as the average frequency in the heldout distribution of all samples that occur r times in the base distribution.

This average frequency is Tr[r]/(Nr[r].N), where:

• Tr[r] is the total count in the heldout distribution for all samples that occur r times in the base distribution.
• Nr[r] is the number of samples that occur r times in the base distribution.
• N is the number of outcomes recorded by the heldout frequency distribution.

In order to increase the efficiency of the prob member function, Tr[r]/(Nr[r].N) is precomputed for each value of r when the HeldoutProbDist is created.

Variables:

_estimate – A list mapping from r, the number of times that a sample occurs in the base distribution, to the probability estimate for that sample. _estimate[r] is calculated by finding the average frequency in the heldout distribution of all samples that occur r times in the base distribution. In particular, _estimate[r] = Tr[r]/(Nr[r].N). _max_r – The maximum number of times that any sample occurs in the base distribution. _max_r is used to decide how large _estimate must be.