CCM Course

Appearance

Bayesian Number Game ​

This is an interactive demonstration of Tenenbaum's (1999, 2001) Bayesian model of concept learning, using the "number game" as an example. The model illustrates how people generalize from examples to infer the extension of a concept.

Examples
Sampling
Try
Probability each number is in the concept00.511102030405060708090100Most probable hypothesespowers of 4.33powers of 2.16squares.10ending in 6.10multiples of 8.08multiples of 4.04even numbers.02just 16<.01[15, 16]<.01[16, 17]<.01[14, 16]<.01[15, 17]<.011: 0.10. Click to add as an example.2: 0.19. Click to add as an example.3: 0.01. Click to add as an example.4: 0.67. Click to add as an example.5: 0.03. Click to add as an example.6: 0.15. Click to add as an example.7: 0.04. Click to add as an example.8: 0.36. Click to add as an example.9: 0.16. Click to add as an example.10: 0.09. Click to add as an example.11: 0.08. Click to add as an example.12: 0.16. Click to add as an example.13: 0.11. Click to add as an example.14: 0.15. Click to add as an example.15: 0.15. Click to add as an example.16: 1.00. Click to add as an example.17: 0.15. Click to add as an example.18: 0.16. Click to add as an example.19: 0.12. Click to add as an example.20: 0.17. Click to add as an example.21: 0.10. Click to add as an example.22: 0.11. Click to add as an example.23: 0.08. Click to add as an example.24: 0.22. Click to add as an example.25: 0.17. Click to add as an example.26: 0.18. Click to add as an example.27: 0.06. Click to add as an example.28: 0.11. Click to add as an example.29: 0.05. Click to add as an example.30: 0.06. Click to add as an example.31: 0.04. Click to add as an example.32: 0.34. Click to add as an example.33: 0.03. Click to add as an example.34: 0.05. Click to add as an example.35: 0.02. Click to add as an example.36: 0.28. Click to add as an example.37: 0.02. Click to add as an example.38: 0.04. Click to add as an example.39: 0.02. Click to add as an example.40: 0.16. Click to add as an example.41: 0.01. Click to add as an example.42: 0.03. Click to add as an example.43: 0.01. Click to add as an example.44: 0.07. Click to add as an example.45: 0.01. Click to add as an example.46: 0.13. Click to add as an example.47: 0.01. Click to add as an example.48: 0.15. Click to add as an example.49: 0.11. Click to add as an example.50: 0.03. Click to add as an example.51: 0.00. Click to add as an example.52: 0.06. Click to add as an example.53: 0.00. Click to add as an example.54: 0.02. Click to add as an example.55: 0.00. Click to add as an example.56: 0.24. Click to add as an example.57: 0.00. Click to add as an example.58: 0.02. Click to add as an example.59: 0.00. Click to add as an example.60: 0.06. Click to add as an example.61: 0.00. Click to add as an example.62: 0.02. Click to add as an example.63: 0.00. Click to add as an example.64: 0.74. Click to add as an example.65: 0.00. Click to add as an example.66: 0.12. Click to add as an example.67: 0.00. Click to add as an example.68: 0.06. Click to add as an example.69: 0.00. Click to add as an example.70: 0.02. Click to add as an example.71: 0.00. Click to add as an example.72: 0.14. Click to add as an example.73: 0.00. Click to add as an example.74: 0.02. Click to add as an example.75: 0.00. Click to add as an example.76: 0.16. Click to add as an example.77: 0.00. Click to add as an example.78: 0.02. Click to add as an example.79: 0.00. Click to add as an example.80: 0.14. Click to add as an example.81: 0.10. Click to add as an example.82: 0.02. Click to add as an example.83: 0.00. Click to add as an example.84: 0.06. Click to add as an example.85: 0.00. Click to add as an example.86: 0.12. Click to add as an example.87: 0.00. Click to add as an example.88: 0.14. Click to add as an example.89: 0.00. Click to add as an example.90: 0.02. Click to add as an example.91: 0.00. Click to add as an example.92: 0.06. Click to add as an example.93: 0.00. Click to add as an example.94: 0.02. Click to add as an example.95: 0.00. Click to add as an example.96: 0.24. Click to add as an example.97: 0.00. Click to add as an example.98: 0.02. Click to add as an example.99: 0.00. Click to add as an example.100: 0.16. Click to add as an example.
Several hypotheses share the posterior, so the prediction is graded. Add an example and watch the smaller consistent hypotheses pull ahead.

The Model ​

Given a set of observed positive examples X = {x₁, x₂, ..., xₙ}, the model infers which hypothesis h best explains the data, and uses this to predict whether new numbers belong to the concept.

Hypotheses ​

The model considers many hypotheses about what rule generates the numbers:

  • Mathematical rules: squares, cubes, primes, and powers of 2 through 10
  • Multiples: multiples of 3, 4, 5, ..., 12
  • Even/odd: all even or all odd numbers
  • Ending patterns: numbers ending in 0, 1, 2, ..., 9
  • Intervals: every range [n, m] with 1 ≤ n ≤ m ≤ 100, such as [10, 20] or [15, 25]

This is the hypothesis space from Homework 3: 34 mathematical hypotheses and 5,050 intervals. The mathematical hypotheses share a prior probability of λ equally (the slider; 2/3 by default), and the intervals share 1 − λ, weighted to favor intervals about ten numbers wide.

The Size Principle ​

The key insight is the size principle for likelihood:

(1)P(X|h)=(1|h|)n if all x∈X are in h

This means:

  • Smaller hypotheses that are consistent with the data get higher likelihood
  • With more examples, this preference for smaller hypotheses grows exponentially

For example, if you see the number 16:

  • "Powers of 2" (size 6) has likelihood 1/6 ≈ 0.17
  • "Even numbers" (size 50) has likelihood 1/50 = 0.02

But if you see 16, 8, 2, and 64:

  • "Powers of 2" has likelihood (1/6)⁴ ≈ 0.00077
  • "Even numbers" has likelihood (1/50)⁴ = 0.00000016

The smaller hypothesis wins by a much larger margin with more data.

Posterior & Generalization ​

Using Bayes' rule:

(2)P(h|X)∝P(X|h)⋅P(h)

The marginal probability that a new number y is in the concept:

(3)P(y∈C|X)=∑hP(y∈h)⋅P(h|X)

Try These Examples ​

Example 1: The number 16 ​

Enter just "16". Notice that many hypotheses are consistent: powers of 2, powers of 4, multiples of 4, even numbers, etc. The histogram shows broad generalization.

Example 2: Powers of 2 ​

Enter "16, 8, 2, 64". Now "powers of 2" dominates, and the model predicts only 4 and 32 (the other powers of 2 up to 100) are likely to be in the concept.

Example 3: Powers of 4 ​

Enter "4, 16, 64". The model strongly favors "powers of 4" over "powers of 2" because it's a smaller consistent hypothesis.

Example 4: An interval ​

Enter "16, 23, 19, 20". No mathematical rule fits well, so interval hypotheses like [16, 23] dominate. The model generalizes to nearby numbers.

Example 5: Squares ​

Enter "81, 25, 4". The model infers "square numbers" and predicts 1, 9, 16, 36, 49, 64, and 100 are likely in the concept.

How to Use ​

  1. Enter numbers in the input field, separated by commas or spaces
  2. Click example buttons to try preset demonstrations
  3. Click bars in the histogram to add numbers as observations
  4. Click number chips to remove observations
  5. Switch between strong and weak sampling, or move the λ slider, and watch how the posterior distribution and generalization histogram update

Key Insights ​

  1. Suspicious coincidences: If all examples happen to be powers of 2, that's unlikely to be a coincidence—the concept is probably "powers of 2"

  2. Size matters: The model prefers the smallest hypothesis consistent with the data

  3. More data, sharper inference: With more examples, the posterior concentrates on fewer hypotheses

  4. Rational generalization: The model captures human-like generalization patterns, neither too narrow nor too broad

References ​

Tenenbaum, J. B. (1999). A Bayesian framework for concept learning. PhD Thesis, MIT.

Tenenbaum, J. B., & Griffiths, T. L. (2001). Generalization, similarity, and Bayesian inference. Behavioral and Brain Sciences, 24(4), 629-640.