Transformer Attention Lab

Transformers process clinical notes token-by-token but use Attention to focus on the most diagnostically relevant terms regardless of where they appear.

Click any token below to see its full attention pattern across the clinical note.

Self-Attention Mechanism

Each token creates Query (Q), Key (K), Value (V) vectors. Attention score between token i and j:

Attention(Q,K,V) = softmax(QKᵀ/√d_k)·V score(i,j) = (Qᵢ · Kⱼ) / √d_k α(i,j) = softmax of scores output_i = Σⱼ α(i,j)·Vⱼ High α(i,j) → token i "attends to" j Low temperature → sharper focus

Multi-Head Attention

Multiple heads run in parallel, each learning different clinical relationships:

MultiHead(Q,K,V) = Concat(h₁…hₙ)·Wᵒ Head 1 → negation ("no fever", "denies") Head 2 → entity co-reference Head 3 → temporal patterns ("worsening") Head 4 → numerical associations

Adjust Temperature to see how focused vs diffuse attention patterns change diagnostic confidence.

⚖️ Ethical Reality

Clinical NLP models trained on US hospital notes fail on notes from other regions due to different abbreviations and documentation styles. "Non-compliant patient" carries different clinical and socioeconomic implications across contexts.

① Clinical Note Tokens

Click a token to see its attention pattern:

Attention Heads4

parallel attention heads

Temperature (1/sharpness)1.0

low = sharp focus · high = diffuse

② Attention Formula

                score(i,j) = Qᵢ·Kⱼ / √d_k

                α(i,j) = exp(score/T) / Σ

                output_i = Σⱼ α(i,j)·Vⱼ

                d_k = embed_dim / num_heads

Query token: —
Max attention to: —
Attn entropy: — bits

③ Diagnostic Output

Clinical Assessment

Sepsis Risk

—

Confidence

—

Active Heads

—

Attn Params

—

d_k / Head

④ Attention Heatmap — row = query token · column = key token · brightness = attention weight · highlighted row = selected token