Chapter 2 Overview of Topic Modeling

We first lay out a basic topic model without any accounting for time.

In this model, we have a number of people, M.

Each individual has a number of diagnoses, N_{i=1M}.We will simulate this as a poisson RV with mean of 10 diagnoses per patient.

There are a total number of $N=\sum _{i=1}^{M}N_{i}$ diagnoses over all individuals.

There are a total of $V$ disease codes entirely.

For each of the M individuals, there exists a distribution over K topics, $\theta_{i}$.

This topic distribution for a given individual, $\theta_i$ will be sparsely simulated so individuals are loaded on a minimal number of topics using a random dirichlet with $\alpha$ = [0.1,…K] .

For each of the K topics, there is a distribution on V diseases. There are a total of V diseases. This parameter $\phi_k$, over diseases, is simulated also according to a sparse dirichlet, this time with hyperparameter $\beta$ for example also = 0.1_RV

Here we allow that each topic has a special ‘topic specific’ disease enrichment, such that for a select number d of randomly sampled diseases, here d=10, is (V-d)/V with the remaining = 1/V.

2.1 Generative model

Now each disease arises from a topic (there can be overlap such that one disease can be featured in several topics) with latent indicator $z_{i=1..M,j=1:Ni}$ with an integer from 1:K.

$1:Ni$ here represents the fact that an individual can have up to Ni diagnoses, so that $Z$ is a giant vector of length N.

According to the assigned topic (1:K) for that diagnosis, the probability of the given diagnosis is drawn according to $\phi_{kd}$

2.2 Table of Variables

Variable	Type	Meaning
K	integer	number of topics (e.g. 50)
V	integer	number of diseases in the vocabulary (e.g. 50,000 or 1,000,000)
M	integer	number of persons
$N_{i_{1..M}}$	integer	number of diseases in person $i$
N	integer	total number of diseases in all persons; sum of all $N_d$ values, i.e. $N = \sum_{i=1}^{M} N_i$
$\alpha_{k_{1..K}}$	positive real	prior weight of topic $k$ in a person; usually the same for all topics; normally a number less than 1, e.g. 0.1, to prefer sparse topic distributions, i.e. few topics per person
$\alpha$	K-dimensional vector of positive reals	collection of all $\alpha_k$ values, viewed as a single vector
$\beta_{w_{1..V}}$	positive real	prior weight of disease $w$ in a topic; usually the same for all diseases; normally a number much less than 1, e.g. 0.001, to strongly prefer sparse disease distributions, i.e. few diseases per topic
$\beta$	V-dimensional vector of positive reals	collection of all $\beta_w$ values, viewed as a single vector
$\phi_{k_{1..K},w_{1..V}}$	probability (real number between 0 and 1)	probability of disease $w$ occurring in topic $k$
$\Phi_{k_{1..K}}$	V-dimensional vector of probabilities, which must sum to 1	distribution of diseases in topic $k$
$\theta_{d_{1..M},k_{1..K}}$	probability (real number between 0 and 1)	probability of topic $k$ occurring in person $d$
$\Theta_{d_{1..M}}$	K-dimensional vector of probabilities, which must sum to 1	distribution of topics in person $d$
$z_{d_{1..M},w_{1..N_d}}$	integer between 1 and K	identity of topic of disease $w$ in person $d$
Z	N-dimensional vector of integers between 1 and K	identity of topic of all diseases in all persons
$w_{d_{1..M},w_{1..N_d}}$	integer between 1 and V	identity of disease $w$ in person $d$
W	N-dimensional vector of integers between 1 and V	identity of all diseases in all persons