A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians

Abstract

We interviewed professional comedians who perform live shows in front of audiences and who use artificial intelligence in their artistic process. To that end, we recruited ten participants at the Edinburgh Festival Fringe in August 2023 and ten more participants online for 3-hour workshops on "AI x Comedy", aiming to ensure linguistic, cultural, gender, sexual, national and racial diversity among participants. The workshop consisted of a comedy writing session with large language models (LLMs), a human-computer interaction questionnaire to assess the Creativity Support Index of the AI writing tool, and a focus group. Guided discussions in the focus group touched about the comedians' motivations and artistic processes, the biases and potentials of the LLM tools and their limitations as a Creativity Support Tool, as well as the comedians' ethical concerns about bias, censorship and copyright. Participants noted that the moderation strategies used during instruction-tuning of the LLMs reinforced hegemonic viewpoints by erasing minority groups, and qualified this as a form of censorship. At the same time, the LLMs did not succeed as a good creativity support tool, by producing stereotypical and biased comedy tropes, akin to "cruise ship comedy material from the 1950ies, but a bit less racist". Our work extends scholarship about the subtle difference between "offensive" language as a practice of resistance---and, equivalently, in comedy, of "punching up" and satire---and harmful speech. We also interrogate the global value alignment behind such language models and discuss community-based value alignment and data ownership to build AI tools that better suit the artists' needs.

Authors

Piotr Mirowski, Juliette Love, Kory Mathewson, Shakir Mohamed

Venue

FAccT 2024