Skip to main content

TabNet Tabular (GPU)

Attention-based neural network for tabular data

Overview​

  • Model ID: tabnet-tabular-gpu
  • Type: Deep learning (tabular)
  • Framework: pytorch-tabnet
  • Hardware: GPU (small footprint, ~512 MB VRAM)
  • Best for: High-cardinality categorical features
  • Key feature: Interpretable attention
  • Multi-output: Yes (predicts multiple target columns simultaneously)

When to Use​

✅ Perfect for:

  • High-cardinality categoricals (e.g., user IDs, product IDs)
  • Need feature importance per sample
  • Large tabular datasets (>50k rows)
  • Interpretability required
  • Multi-output regression (predict multiple targets at once)

❌ Not ideal for:

  • Small datasets (<10k)
  • Purely numeric data - use MLP
  • Need fastest training - use LightGBM

Quick Start​

Single-output​

from colabhive import ColabHive

client = ColabHive(api_key="...", account_id="...")

dataset = client.datasets.upload("user_data", "./train.csv")

job = client.training.create(
    model="tabnet-tabular-gpu",
    dataset_id=dataset.id,
    hyperparameters={
        "epochs": 50,
        "batch_size": 512,
        "learning_rate": 0.02,
        "n_d": 64,
        "n_a": 64,
        "target_column": "target"
    }
)

job.wait()

Multi-output​

job = client.training.create(
    model="tabnet-tabular-gpu",
    dataset_id=dataset.id,
    hyperparameters={
        "epochs": 50,
        "batch_size": 512,
        "learning_rate": 0.02,
        "n_d": 64,
        "n_a": 64,
        "target_columns": ["price", "demand", "rating"]
    }
)
Multi-output

When using target_columns (plural), TabNet trains a single model that predicts all targets simultaneously. This is faster and often more accurate than training separate models.

Hyperparameters​

ParameterDefaultDescription
epochs100Training epochs
batch_size256Large batches work well
learning_rate0.02Higher LR than typical DL
n_d8Decision dimension
n_a8Attention dimension
n_steps3Sequential attention steps
gamma1.3Feature reuse across steps (1.0-2.0)
target_column—Single target column
target_columns—Multiple target columns (multi-output)

Key Features​

1. Attention Mechanism​

  • Sequential attention: Focuses on different features at each step
  • Feature importance: Explains predictions per sample
  • Interpretable: See which features matter for each decision

2. Handles Categoricals​

  • Native encoding: No need for one-hot encoding
  • Embeddings: Learns representations for categorical features
  • High cardinality: Handles 1000s of unique values

3. Multi-output Support​

  • Single model: Predicts multiple targets in one forward pass
  • Shared representation: Learns common patterns across targets
  • Efficient: Faster than training N separate single-output models

Performance​

Dataset TypeTabNetMLPXGBoost
High cardinality⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Pure numeric⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Interpretability⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Training speed⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐
Multi-output⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐

Example: E-commerce​

# Dataset with high-cardinality features
# - user_id: 100k unique users
# - product_id: 50k unique products
# - category: 1k categories

hyperparameters={
    "n_d": 64,       # Larger for complex patterns (max 64)
    "n_a": 64,
    "n_steps": 5,    # More attention steps
    "batch_size": 1024
}

Technical Details​

  • Model ID: tabnet-tabular-gpu
  • Framework: pytorch-tabnet
  • Category: deep_learning
  • Hardware: GPU (~512 MB VRAM)
  • Dependencies: pytorch-tabnet, numpy<2.0
numpy compatibility

TabNet requires numpy<2.0 due to compatibility with pytorch-tabnet's internal dtype handling. This is managed automatically when training through ColabHive.

Tips​

  1. Large batches: TabNet benefits from batch_size=512-1024
  2. More steps: n_steps=3-5 for complex datasets
  3. Categorical encoding: Let TabNet handle it, don't one-hot
  4. Feature importance: Use attention masks for interpretability
  5. Multi-output: Use target_columns when predicting related targets
  6. First run may be slower: Initial container setup installs PyTorch dependencies (~2-3 min overhead)

Next Steps​