A self-supervised Spectral Vision Transformer that unmixes hyperspectral pixels into endmembers and abundances — with no ground-truth labels required.
Hyperspectral unmixing decomposes each pixel into constituent endmembers and their abundances. Deep learning does this well, but reliable per-pixel abundance ground truth is essentially impossible to get in real scenes — which pushes the field toward unsupervised and self-supervised strategies.
S3ViT tackles this with a transformer that treats each 1×1 pixel as a token, using self-attention over the full sequence — augmented with learnable positional embeddings — to capture spectral-spatial structure. Rather than supervision, it bootstraps pseudo-labels: Singular Value Decomposition estimates the number of endmembers from the singular-value spectrum, and k-means clustering supplies cluster-derived priors used to build a contextual token and initialize spectral prototypes. Two initialization tokens — one from Vertex Component Analysis (VCA), one from the k-means priors — get embedded alongside patch tokens to guide training, while ReLU/Softmax layers enforce the standard abundance non-negativity and sum-to-one constraints.
Evaluated on the Samson, Jasper Ridge, and Washington DC Mall benchmarks against state-of-the-art geometrical and deep-learning baselines, S3ViT reaches competitive or superior performance — up to 31% better SAD and 25% better RMSE. The result: a compact pixel-token ViT, steered by weak spectral priors and trained purely on reconstruction loss, can match or beat fully supervised approaches on standard unmixing benchmarks.
