I’m a Research Scientist in MIT CSAIL with the MIT-IBM Watson AI Lab. I did my PhD in Brain and Cognitive Sciences at MIT, as an NSF Fellow working with Josh Tenenbaum and Antonio Torralba. My work investigates representations underlying intelligence in artificial (and previously, biological) neural networks.
I am also broadly interested in creativity underlying the human relationship to the world: from the brain’s fundamentally constructive role in sensory perception to the explicit creation of experiential worlds in art. I see this as a frontier for understanding intelligence and building intelligent machines. As a grad student I designed and co-taught MIT’s first course on Vision in Art and Neuroscience, which I continue to teach every fall. You can learn more about it here.
Selected Recent Publications
FIND: A Function Description Benchmark for Evaluating Interpretability Methods. An interactive dataset of functions resembling subcomputations inside trained neural networks, for validating and comparing open-ended labeling tools, and a new method that uses Automated Interpretability Agents to interpret other systems. S. Schwettmann*, T. Rott Shaham*, J. Materzynska, N. Chowdhury, S. Li, J. Andreas, D. Bau, A. Torralba. NeurIPS 2023.
Multimodal Neurons in Pretrained Text-Only Transformers. We find multimodal neurons in a transformer pretrained only on language. When image representations are aligned to the language model, these neurons activate on specific image features and inject related text into the model’s next token prediction. S. Schwettmann*, N. Chowdhury*, A. Torralba. ICCV CVCL Workshop 2023 (Oral).
MILAN: Natural Language Descriptions of Deep Features. We introduce a procedure that automatically labels neurons in deep networks with open-ended, compositional, natural language descriptions of their function. E. Hernandez, S. Schwettmann, D. Bau, T. Bagashvili, A. Torralba, J. Andreas. Natural Language Descriptions of Deep Features. ICLR 2022 (Oral).
Recent News
AI agents help explain other AI systems, MIT News
Demystifying machine-learning systems, MIT News
3Q: The Interface Between Art and Neuroscience, an interview with MIT News
How Artificial Intelligence Sees Art History Metropolitan Museum of Art