Gustaf Ahdritz

Gustaf Ahdritz

I'm a third-year PhD student in Computer Science at Harvard University. I'm a member of the Machine Learning Foundations Group and am advised by Boaz Barak and Jonathan Frankle. I'm supported by a fellowship from Harvard's Kempner Institute.


I'm broadly interested in empirical investigations of the properties of realistic deep neural networks. At the moment, I'm thinking about uncertainty in large language models.


In the summer of 2024, I interned at Apple, where I worked with Parikshit Gopalan, Udi Wieder, Moises Goldszmidt, and Anatoly Adamov (et al.). I graduated from Columbia with a B.A. in Computer Science & History (2020) and an M.S. in Computer Science (2021). There, I worked with Mohammed AlQuraishi on the applied task of protein structure prediction and led the development of OpenFold.

CV GitHub Google Scholar Twitter

Papers

Preprints

Real-Time Interactive Conversations
Preprint
Modeling Real-Time Interactive Conversations as Timed Diarized Transcripts
*Garrett Tanzer, *Gustaf Ahdritz, Luke Melas-Kyriazi
2024
Carefully sampling from language models trained directly on timed, diarized transcripts (e.g. instant messenger logs) permits true real-time interactivity.
Paper Code Tweetorial BibTex

Conference Papers

Higher-Order Calibration
ICLR
Spotlight
Provable Uncertainty Decomposition via Higher-Order Calibration
Gustaf Ahdritz, Aravind Gollakota, Parikshit Gopalan, Charlotte Peale, Udi Wieder (α)
2024
We define higher-order calibration, which turns out to be a necessary and sufficient condition for accurate uncertainty decomposition.
Paper BibTeX Tweetorial
Distinguishing Knowable from Unknowable
ICML
Distinguishing the Knowable from the Unknowable with Language Models
*Gustaf Ahdritz, *Tian Qin, Nikhil Vyas, Boaz Barak, Benjamin L. Edelman
2024
Linear probes of language model activations can predict when the predictive entropy of much larger and more knowledgeable models is close to zero, and they even work out-of-distribution!
Paper Code Tweetorial BibTex
OpenProteinSet
NeurIPS
OpenProteinSet: Training data for structural biology at scale
Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Lukas Jarosch, Daniel Berenberg, Ian Fisk, et al.
2023
We present the largest open repository of precomputed multiple sequence alignments (MSAs) of proteins, representing millions of compute hours.
Paper Data BibTeX
Soft Prompting
ICML Workshop
Soft prompting might be a bug, not a feature
*Luke Bailey, *Gustaf Ahdritz, *Anat Kleiman, Siddharth Swaroop, Finale Doshi-Velez, Weiwei Pan
2023
Contrary to prior speculation, we find that soft prompts (created with "prompt-" or "prefix-tuning") differ from natural token embeddings in key ways, complicating attempts to decode them back into natural language.
Paper BibTeX

Journal papers

OpenFold
Nature Methods
OpenFold: Retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization
*Gustaf Ahdritz, *Nazim Bouatta, Christina Floristean, Sachin Kadyan, Qinghui Xia, William Gerecke, et al.
2024
We created the first trainable, open-source reproduction of AlphaFold2 and used it to study how the model learns to fold.
Paper Code Talk BibTex
Single-sequence protein structure prediction
Nature Biotechnology
Single-sequence protein structure prediction using a language model and deep learning
*Ratul Chowdhury, *Nazim Bouatta, *Surojit Biswas, *Christina Floristean, Anant Kharkar, Gustaf Ahdritz, et al.
2022
We present RGN2, an end-to-end "single-sequence" protein structure prediction model that relies on a small protein language model rather than multiple sequence alignments.
Paper Code BibTeX

Teaching

I've served as a teaching fellow/assistant for the following courses at Harvard and Columbia:

  • Spring 2023: Foundations of Deep Learning (Harvard COMPSCI 229br) with Boaz Barak
  • Spring 2019 - Spring 2021: Advanced Programming (Columbia COMS 3157) with Jae Woo Lee

Awards & Fellowships