Artificial Intelligence for Music

Hao-Wen (Herman) Dong

University of Michigan · Performing Arts Technology

Generative AI for Music: Challenges and Opportunities

Generative AI has been transforming the way we interact with technology and consume content. The recent successes of LLM-based chatbots, AI assistants, text-to-image, text-to-video, and text-to-music systems have showcased how AI can augment human creativity and boost human productivity. In the next decade, generative AI technology will also reshape how we create music in the music, film, TV, podcast, and gaming industries across the entertainment, commercial, and education sectors. In the first half of this talk, I will introduce some of our recent work on the various applications of generative AI in music creation, including multitrack music generation, automatic instrumentation, and violin performance synthesis. In the second half, I will discuss the unique challenges of applying, scaling, and deploying generative AI music models in practice. Finally, I will discuss research opportunities towards controllable and interactable generative AI systems for music.

Hao-Wen (Herman) Dong is an Assistant Professor in the Performing Arts Technology Department at the University of Michigan. Herman's research aims to empower music and audio creation with machine learning. His long-term goal is to lower the barrier of entry for music composition and democratize audio content creation. He is broadly interested in music generation, audio synthesis, multimodal machine learning, and music information retrieval. Herman received his PhD in Computer Science from UC San Diego, where he worked with Julian McAuley and Taylor Berg-Kirkpatrick. His research has been recognized by the UCSD CSE Doctoral Award for Excellence in Research, KAUST Rising Stars in AI, UChicago and UCSD Rising Stars in Data Science, ICASSP Rising Stars in Signal Processing, and UCSD GPSA Interdisciplinary Research Award.

• Homepage ↗
• Slides ↗

Zhiyao Duan

University of Rochester · ECE / CS / Data Science

AI Powered Interactive Music Making

Artificial Intelligence (AI) is profoundly transforming human society. As one of the most fun activities with a rich history, music making is being impacted by AI as well. Powered by AI, we can diversify the forms of music interaction, improve the productivity of musicians, lower barriers to entry for music interaction, and enlarge the music-making population. At the same time, however, generative AI poses significant challenges to current copyright regulation and economic norms. In this talk, I share my thoughts about these opportunities and challenges and introduce our recent work on real-time human–AI collaborative improvisation, Chinese–Western music style fusion, music inpainting from hand-drawn curves, software framework development for deploying music AI models on the web and to DAWs, and singing-voice deepfake detection.

Zhiyao Duan is an associate professor in Electrical and Computer Engineering, Computer Science, and Data Science at the University of Rochester. He is also a co-founder of Violy, a company aiming to improve music education through AI. His research interest is in computer audition and its connections with computer vision, natural language processing, and augmented and virtual reality. He received a best paper award at the Sound and Music Computing (SMC) Conference in 2017, a best paper nomination at ISMIR in 2017, and a CAREER award from NSF. His work has been funded by NSF, NIH, NIJ, the New York State Center of Excellence in Data Science, and University of Rochester internal awards on AR/VR, health analytics, and data science. He is a senior area editor of IEEE Signal Processing Letters, an associate editor for IEEE Open Journal of Signal Processing, and a guest editor for TISMIR. He is the President of ISMIR.

• Homepage ↗

Miguel Willis

University of Pennsylvania Law School · Future of the Profession Initiative

Miguel Willis is the Innovator in Residence at the Law School's Future of the Profession Initiative (FPI), University of Pennsylvania. He concurrently serves as the Executive Director of Access to Justice Tech Fellows, a national nonprofit organization that develops summer fellowships for law students seeking to leverage technology to create equitable legal access for low-income and marginalized populations. Prior to joining FPI, Willis served as the Law School Admissions Council's (LSAC) inaugural Presidential Innovation Fellow. Willis currently serves on the advisory board of the University of Arizona James E. Rogers College of Law's Innovation for Justice (i4J) program and serves on The Legal Services Corporation's Emerging Leaders Council.

• Homepage ↗

Gus Xia

MBZUAI · Machine Learning

Gus Xia is an assistant professor of Machine Learning at the Mohamed bin Zayed University of Artificial Intelligence in Masdar City, Abu Dhabi. His research includes the design of interactive intelligent systems to extend human musical creation and expression. This research lies at the intersection of machine learning, human–computer interaction, robotics, and computer music. Some representative works include interactive composition via style transfer, human–computer interactive performances, autonomous dancing robots, large-scale content-based music retrieval, haptic guidance for flute tutoring, and bio-music computing using slime mold.

• Homepage ↗
• Slides ↗

Ziyu Wang

NYU / NYU Shanghai · MBZUAI (visiting)

From Imitation to Creation: When Music AI Truly Understands

Recent advances in generative AI have led to impressive achievements in music generation. Yet, a fundamental challenge remains: how can these black-box models move beyond imitating music data to truly understand human creative intent and collaborate meaningfully with humans? We argue that the missing piece is a deeper alignment between humans and AI — one that involves shared musical concepts, structured knowledge, and even the principles behind how we learn music. In this talk, I will explore various approaches to establish such alignment in generative modeling, which naturally enhances model interpretability and controllability in music generation. Ultimately, we may find that the heart of this alignment challenge lies in understanding content and style, a timeless question that resonates across art and life.

Ziyu Wang is a PhD candidate in Computer Science at the Courant Institute of Mathematical Sciences, New York University, and holds an affiliation with NYU Shanghai. Currently, he is also a visiting scholar in the Machine Learning Department at MBZUAI. His research is conducted under the supervision of Prof. Gus Xia in Music X Lab, where he explores the intersection of music and machine learning. In 2019, he earned his undergraduate degree in Mathematics from Fudan University. Beyond his academic pursuits, he is a passionate conductor, pianist, and Erhu (a traditional Chinese string instrument) player. He has previously served as the conductor of the NYU Shanghai Jazz Ensemble and as the director of the Fudan Musical Club.

• Homepage ↗
• Slides ↗