Tabular foundation models

Closing the Context Gap

Activation Alignment for Tabular In-Context Learning

Tabular foundation models must carry every training example through the forward pass on every prediction - because at their foundation, they are Transformers performing in-context learning over the entire training set. Shrink the context to go faster and accuracy collapses.

Activation alignment to the rescue: it fits a linear projection - on a CPU, in seconds - so a small-context student approximates the internal representations of a full-context teacher.

2-4× Effective data gain An aligned model with 10% of the data has the performance of an unaligned model with 20-40% of the data
38 TabArena datasets Evaluated on diverse real-world classification benchmarks
81% Beat the baseline Aligned student wins across 81% of evaluations
0 GPUs needed Trained via MSE with gradient descent in seconds on a CPU

How it works

The frozen model M is evaluated in two regimes: a teacher that sees all training examples, and a lightweight student that sees only a fraction. A simple linear projection is trained to align the student's layer-k activations onto the teacher's activations, using easy to generate unlabeled synthetic queries drawn from the feature distribution.

Sample efficiency and distillation gains

Explore benchmark results across all 38 TabArena datasets. Use the context budget slider to vary what percentage of training data the student model is allowed to see.

Sample efficiency (left) measures the effective training data matched by the student (points above the diagonal mean the aligned model matches an unaligned student with more data, demonstrating the aligner's success).

Distillation efficiency (right) measures what percentage of the performance gap to the full-context teacher is closed.

    10% 20% 30% 40% 50%
    Sample efficiency
    Distillation efficiency