Publications

Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues
Chinmay Talegaonkar, Nikhil Gandudi Suresh, Zachary Novack, Yash Belhe, Priyanka Nagasamudra, Nicholas Antipa Spotlight (top 3.2%)
Paper / Summary / BibTex

This work injects defocus blur cues at inference time into Marigold, a pre-trained diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE), which effectively turns Marigold into a metric depth predictor in a training-free manner.

@inproceedings{talegaonkar2025repurposing,
title={Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues},
author={Talegaonkar, Chinmay and Suresh, Nikhil Gandudi and Novack, Zachary
and Belhe, Yash and Nagasamudra, Priyanka and Antipa, Nicholas}, booktitle={NeurIPS},
year={2025}
}

Fast Text-to-Audio Generation with Adversarial Post-Training
Zachary Novack, Zach Evans, Zack Zukowski, Josiah Taylor, CJ Carr, Julian Parker, Adnan Al-Sinan, Gian Marco Iodice, Julian McAuley, Taylor Berg-Kirkpatrick, Jordi Pons
IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2025
Paper / Website / Summary / BibTex

This work presents Adversarial Relativistic-Contrastive (ARC) post-training, the first distillation-free adversarial acceleration algorithm for diffusion/flow, enabling the development of Stable Audio Open Small, a lightweight Text-to-Audio model capable of generation 12 seconds of stereo 44.1kHz audio in 75 milliseconds.

@inproceedings{Novack2025Fast,
title={Fast Text-to-Audio Generation with Adversarial Post-Training},
author={Zachary Novack and Zach Evans and Zack Zukowski
and Josiah Taylor and CJ Carr and Julian Parker
and Adnan Al-Sinan and Gian Marco Iodice
and Julian McAuley and Taylor Berg-Kirkpatrick and Jordi Pons},
year={2025},
booktitle={IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)},
}

Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks
Yongyi Zang, Sean O'Brien, Taylor Berg-Kirkpatrick, Julian McAuley, Zachary Novack
International Society of Music Information Retrieval (ISMIR), 2025 Best Paper Award Winner!
Paper / Website / Summary / BibTex

We present RUListening, a framework that enhances perceptual evaluation in Music-QA benchmarks for Large Audio Language Models.

@inproceedings{zang2025you,
title={Are you really listening? boosting perceptual awareness in music-qa benchmarks},
author={Zang, Yongyi and O'Brien, Sean and Berg-Kirkpatrick, Taylor and McAuley, Julian and Novack, Zachary},
booktitle={International Society of Music Information Retrieval (ISMIR)}},
year={2025},
}

Aligning Text-to-Music Evaluation with Human Preferences
Yichen Huang, Zachary Novack, Koichi Saito, Jiatong Shi, Shinji Watanabe, Yuki Mitsufuji, John Thickstun, Chris Donahue
International Society of Music Information Retrieval (ISMIR), 2025
Paper / Code / Website / Summary / BibTex

We propose a new metric, the MAUVE Audio Divergence (MAD), that effectively captures diverse musical desiderata and also correlates more strongly with human preference data.

@inproceedings{huang2025aligning,
title={Aligning Text-to-Music Evaluation with Human Preferences},
author={Huang, Yichen and Novack, Zachary and Saito, Koichi and Shi, Jiatong
and Watanabe, Shinji and Mitsufuji, Yuki and Thickstun, John and Donahue, Chris},
booktitle={International Society of Music Information Retrieval (ISMIR)}},
year={2025},
}

Deriving Representative Structure from Music Corpora
Ilana Shapiro, Ruanqianqian Huang, Zachary Novack, Cheng-i Wang, Hao-Wen Dong, Taylor Berg-Kirkpatrick, Shlomo Dubnov, Sorin Lerner
Special Track on AI, Arts, and Creativity, International Joint Conference on Artificial Intelligence (IJCAI), 2025
Paper / Summary / BibTex

This work proposes a unified, hierarchical meta-representation of symbolic musical structure called the structural temporal graph (STG).

@inproceedings{Shapiro2025Deriving,
title={Deriving Representative Structure from Music Corpora},
author={Shapiro, Ilana and Huang, Ruanqianqian and Novack, Zachary
and Wang, Cheng-i and Dong, Hao-Wen and Berg-Kirkpatrick, Taylor
and Dubnov, Shlomo and Lerner, Sorin},
year={2025},
booktitle={Special Track on AI, Arts, and Creativity, International Joint Conference on Artificial Intelligence (IJCAI)},
}

Presto! Distilling Steps and Layers for Accelerating Music Generation
Zachary Novack, Ge Zhu, Jonah Casebeer, Julian McAuley, Taylor Berg-Kirkpatrick, Nicholas J. Bryan
International Conference on Learning Representations (ICLR), 2025 Spotlight (top 5.1%)
Paper / Website / Summary / BibTex

This work presents Presto!, a new method for accelerating audio-domain TTM models, which works by distilling the model to drop both diffusion steps and interior layers of the model itself, achieving <0.5s for generating 32s of 44.1kHz stereo audio.

@inproceedings{Novack2025Presto,
title={Presto! Distilling steps and layers for accelerating music generation.},
author={Zachary Novack and Ge Zhu and Jonah Casebeer and
Julian McAuley and Taylor Berg-Kirkpatrick and Nicholas J. Bryan},
year={2025},
booktitle={International Conference on Learning Representations (ICLR)},
}

CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation
Junda Wu, Warren Li, Zachary Novack, Amit Namburi, Carol Chen, Julian McAuley
International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025
SoCal NLP Symposium, 2024
Paper / Summary / BibTex

This work presents CoLLAP, a long-form vesrion of CLAP that is able to perform long-form reasoning and retrieval over minutes of musical audio.

@inproceedings{wu2024collap,
title={CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation},
author={Junda Wu and Warren Li and Zachary Novack and Amit Namburi and Carol Chen and Julian McAuley},
year={2025},
booktitle={International Conference on Acoustics, Speech, and Signal Processing (ICASSP)}
}

FUTGA-MIR: Enhancing Fine-grained and Temporally-aware Music Understanding with Music Information Retrieval
Junda Wu, Zachary Novack, Amit Namburi, Jiaheng Dai, Hao-Wen Dong, Zhouhang Xie, Carol Chen, Julian McAuley
International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025
Website / Summary / BibTex

This work presents FUTGA-MIR, an extension of our long context dense audio captioner FUTGA for downstream MIR tasks.

@inproceedings{wu2024futga,
title={FUTGA-MIR: Enhancing Fine-grained and Temporally-aware Music Understanding with Music Information Retrieval},
author={Junda Wu and Zachary Novack and Amit Namburi and Jiaheng Dai and Hao-Wen Dong and Zhouhang Xie and Carol Chen and Julian McAuley},
year={2025},
booktilte={International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
}

PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processing
Phillip Long, Zachary Novack, Taylor Berg-Kirkpatrick, Julian McAuley
International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025
NeurIPS Workshop on Creativity & Generative AI, 2024
Paper / Code / Website / Summary / BibTex

This work presents PDMX, the largest open source dataset of public-domain symbolic music scores in existence.

@inproceedings{long2024pdmx,
title={{PDMX}: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music Processing},
author={Long, Phillip and Novack, Zachary and Berg-Kirkpatrick, Taylor and McAuley, Julian},
booktitle={International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
year={2025},
}

DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation
Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrick, Nicholas J. Bryan
International Society of Music Information Retrieval (ISMIR), 2024
Paper / Website / Summary / BibTex

This work presents DITTO-2, a novel method for accelerating inference-time control in diffusion-based music models, leveraging diffusion distillation to speed up the sampling process.

@inproceedings{Novack2024Ditto2,
title={{DITTO-2}: Distilled Diffusion Inference-Time T-Optimization for Music Generation},
author={Novack, Zachary and McAuley, Julian and Berg-Kirkpatrick, Taylor and Bryan, Nicholas J.},
year={2024},
booktitle={International Society of Music Information Retrieval (ISMIR)}}

DITTO: Diffusion Inference-Time T-Optimization for Music Generation
Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrick, Nicholas J. Bryan
International Conference on Machine Learning (ICML), 2024 Oral (top 1.5%)
Paper / Website / Summary / BibTex

This work presents DITTO, a novel method for controlling pretrained text-to-music diffusion models without any model finetuning, enabling a wide range of artistic editing and control tasks.

@inproceedings{Novack2024Ditto,
title={{DITTO}: Diffusion Inference-Time T-Optimization for Music Generation},
author={Novack, Zachary and McAuley, Julian and Berg-Kirkpatrick, Taylor and Bryan, Nicholas J.},
year={2024},
booktitle={International Conference on Machine Learning (ICML)}}

CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets
Zachary Novack, Julian McAuley, Zachary Lipton, Saurabh Garg
International Conference on Machine Learning (ICML), 2023
1st Workshop on Multimodal Representation Learning, ICLR, 2023
Paper / Code / Summary / BibTex

This work introduces CHiLS, a strategy for zero-shot classification to improve CLIP-like models that focuses on improving class names and utilizes implicit semantic hierarchies to enhance accuracy without requiring additional training.

@inproceedings{novack2023chils,
title={CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets},
author={Novack, Zachary and McAuley, Julian and Lipton, Zachary and Garg, Saurabh},
year={2023},
booktitle={International Conference on Machine Learning (ICML)}, }

Disentangling the Mechanisms Behind Implicit Regularization in SGD
Zachary Novack, Simran Kaur, Tanya Marwah, Saurabh Garg, Zachary Lipton
International Conference on Learning Representations (ICLR), 2023
NeurIPS Workshop on The Benefits of Higher-Order Optimization in Machine Learning, 2022 Spotlight & Best Poster
Paper / Code / Summary / BibTex

This work presents a comparative empirical study of different gradient norm-based regularizers for improving Large-Batch Stochastic Gradient Descent, and ties this behavior to the trajectory of the micro-batch gradient norm during training.

@inproceedings{novack2023disentangling,
title={Disentangling the Mechanisms Behind Implicit Regularization in SGD},
author={Novack, Zachary and Kaur, Simran and Marwah, Tanya and Garg, Saurabh and Lipton, Zachary},
booktitle={International Conference on Learning Representations (ICLR)},
year={2023} }

FUTGA: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation
Junda Wu, Zachary Novack, Amit Namburi, Jiaheng Dai, Hao-Wen Dong, Zhouhang Xie, Carol Chen, Julian McAuley
3rd Workshop on NLP for Music and Audio, 2024
Paper / Website / Summary / BibTex

This work presents FUTGA, an open source AudioLLM capable of performing fine-grained reasoning and captioning for audio domain music.

@inproceedings{wu2024futga,
title={Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation},
author={Junda Wu and Zachary Novack and Amit Namburi and Jiaheng Dai and Hao-Wen Dong and Zhouhang Xie and Carol Chen and Julian McAuley},
year={2024},
booktilte={3rd Workshop on NLP for Music and Audio},
}

Unsupervised Lead Sheet Generation via Semantic Compression
Zachary Novack, Nikita Srivatsan, Taylor Berg-Kirkpatrick, Julian McAuley
AES International Symposium on AI and the Musician, 2024
Paper / Code / Summary / BibTex

This work presents a novel method Lead-AE for conditional lead sheet generation, leveraging a discrete auto-encoder setup that allows for fully unsupervised training.

@inproceedings{novack2023unsupervised,
title={Unsupervised Lead Sheet Generation via Semantic Compression},
author={Novack, Zachary and Srivatsan, Nikita and Berg-Kirkpatrick, Taylor and McAuley, Julian},
year={2024},
booktitle={AES International Symposium on AI and the Musician}}

Personalized Sequential Recommendation for Adaptive Itemization in MOBA Games
Zachary Novack
Paper / Summary / BibTex

This paper adapts the state-of-the-art work HT4-Rec for personalized sequential item recommendation in the popular video game League of Legends, incorporating user skill information to directly capture personal player ability.

@misc{novack2022personalized,
title={Personalized Sequential Recommendation for Adaptive Itemization in MOBA Games},
author={Novack, Zachary},
year={2022}}

Towards Generalizable Deep Speech Anonymization
Aaron Broukhim, Zachary Novack
Paper / Summary / BibTex

This work presents a method for learning language-agnostic speech anonymization systems, utilizing a GAN backbone and modified loss to actively learn an anonymized speaker distribution.

@misc{broukhim2022towards,
title={Towards Generalizable Deep Speech Anonymization},
author={Broukhim, Aaron and Novack, Zachary},
year={2022}}

Down the Rabbit Hole: Modeling Twitter Dynamics through Bayesian Inference
Zachary Novack
Paper / Summary / BibTex

This work presents an analytical study into addiction effects on Twitter, and uncovers the presence of different posting types that obey fundamentally chaotic behavior when modeled using an autoregressive Bayesian framework.

@article{novack2022down,
title={Down the Rabbit Hole: Modeling Twitter Dynamics through Bayesian Inference},
author={Novack, Zachary},
doi={"10.1184/R1/20638989.v1"},
year={2022}}

Approximating Optimal Transport via GANs for Recourse Disparity Analysis
Zachary Novack, Qi Xuan Teo, Ryan Steed
Paper / Summary / BibTex

This work presents a method for analyzing the recourse disparity across protected groups in algorithmic decision making systems, using GANs to approximate the optimal transport mapping between recourse-aligned subgroups.

@misc{novack2022approximating,
title={Approximating Optimal Transport via GANs for Recourse Disparity Analysis},
author={Novack, Zachary and Teo, Qi Xuan and Steed, Ryan},
year={2022}}

Lunch at the EigenSalad Bar: Linear Approaches to Dimensionality Reduction for Image Processing
Zachary Novack
Paper / Summary / BibTex

This pseudo-satirical work investigated linear dimensionality reduction techniques for processing images of food, attempting to find explainable bases for discrimintative food categories.

@misc{novack2021salad,
title={Lunch at the EigenSalad Bar: Linear Approaches to Dimensionality Reduction for Image Processing},
author={Novack, Zachary},
year={2021}}