This paper introduces a principled clustering objective based on maximizing Mutual Information (MI) between paired data samples under a bottleneck, equivalent to distilling their shared abstract content (co-clustering), that tends to avoid degenerate clustering solutions.