Puneesh Deora

Hi, I'm Puneesh! I am a second year PhD student at UBC, fortunate to be advised by Prof. Christos Thrampoulidis. I am broadly interested in understanding foundations of deep learning optimization and generalization. More recently, I've been working on empirics and theory of phenomena fundamental to LLMs.

I completed my Master's from UBC where I worked on optimization and generalization theory on transformer models. Before that, I received my Bachelor's from IIT Roorkee in ECE, where I had the good fortune to receive guidance from Prof. Saumik Bhattacharya and Prof. P.M. Pradhan . My undergrad research was mainly focused on GANs for compressed sensing MRI reconstruction.

Email | Google Scholar | CV | Github

selected papers

(* denotes equal contribution)

  1. How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
    Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thrampoulidis.
    Preprint.
  1. In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
    Puneesh Deora, Bhavya Vasudeva, Tina Behnia, and Christos Thrampoulidis.
    COLM 2025, MOSS at ICML 2025 (Oral).
  1. Implicit Bias and Convergence Rates for Self-attention
    Bhavya Vasudeva*, Puneesh Deora*, and Christos Thrampoulidis.
    TMLR 2025, and BGPT at ICLR 2024.
  1. On the Optimization and Generalization of Multi-head Attention
    Puneesh Deora*, Rouzbeh Ghaderi*, Hossein Taheri*, and Christos Thrampoulidis.
    TMLR 2024 (also presented at ICLR 2025), and HiLD at ICML 2023.
  1. Fast Test Error Rates for Gradient-based Algorithms on Separable Data
    Puneesh Deora*, Bhavya Vasudeva*, Vatsal Sharan, Christos Thrampoulidis.
    HiLD at ICML 2023, and ICASSP 2024.