Tabular foundation models
Activation Alignment for Tabular In-Context Learning
Tabular foundation models must carry every training example through the forward pass on every prediction - because at their foundation, they are Transformers performing in-context learning over the entire training set. Shrink the context to go faster and accuracy collapses.
Activation alignment to the rescue: it fits a linear projection - on a CPU, in seconds - so a small-context student approximates the internal representations of a full-context teacher.
The frozen model M is evaluated in two regimes: a teacher that sees all training examples, and a lightweight student that sees only a fraction. A simple linear projection is trained to align the student's layer-k activations onto the teacher's activations, using easy to generate unlabeled synthetic queries drawn from the feature distribution.
Explore benchmark results across all 38 TabArena datasets. Use the context budget slider to vary what percentage of training data the student model is allowed to see.
Sample efficiency (left) measures the effective training data matched by the student (points above the diagonal mean the aligned model matches an unaligned student with more data, demonstrating the aligner's success).
Distillation efficiency (right) measures what percentage of the performance gap to the full-context teacher is closed.