SSOLE: Rethinking Orthogonal Low-rank Embedding for Self-Supervised Learning

Abstract

Self-supervised learning (SSL) aims to learn meaningful representations from unlabeled data. Orthogonal Low-rank Embedding (OLE) shows promise for SSL by enhancing intra-class similarity in a low-rank subspace and promoting inter-class dissimilarity in a high-rank subspace, making it particularly suitable for multi-view learning tasks. However, directly applying OLE to SSL poses significant challenges: (1) the virtually infinite number of “classes” in SSL makes achieving the OLE objective impractical, leading to representational collapse; and (2) low-rank constraints may fail to distinguish between positively and negatively correlated features, further undermining learning. To address these issues, we propose SSOLE (Self-Supervised Orthogonal Low-rank Embedding), a novel framework that integrates OLE principles into SSL by (1) decoupling the low-rank and high-rank enforcement to align with SSL objectives; and (2) applying low-rank constraints to feature deviations from their mean, ensuring better alignment of positive pairs by accounting for the signs of cosine similarities. Our theoretical analysis and empirical results demonstrate that these adaptations are crucial to SSOLE’s effectiveness. Moreover, SSOLE achieves competitive performance across SSL benchmarks without relying on large batch sizes, memory banks, or dual-encoder architectures, making it an efficient and scalable solution for self-supervised tasks.

Key Ideas

Decoupled rank objectives
Low-rank (within-view) and high-rank (across-sample) terms are enforced separately, with nuclear-norm bounds tailored to SSL.
Deviation matrices
Applying the low-rank constraint to deviations from the mean accounts for the sign of cosine similarity, so positive pairs truly align.
Simple and efficient
Single encoder, no memory bank, no large batches, and still competitive on standard SSL benchmarks.

Poster

ICLR 2025 poster.

BibTeX

@inproceedings{huang2025ssole,
  title     = {SSOLE: Rethinking Orthogonal Low-rank Embedding for Self-Supervised Learning},
  author    = {Huang, Lun and Qiu, Qiang and Sapiro, Guillermo},
  booktitle = {The Thirteenth International Conference on Learning Representations (ICLR)},
  year      = {2025}
}