Skip to content
Main Site News Console

Nemotron Halloween

· Hugging Face Translated
教程模型卡

Building a Haunted House with NVIDIA Nemotron Open Models

Or perhaps rather…

Boo! Now Get Off My Lawn!

Image 7: Blog Thumbnail

TL;DR: MonsterHouse is an interactive projection project I built for Halloween. It listens to trick-or-treaters, replies in the raspy voice of a grumpy old pirate ship, and softens up whenever visitors mind their manners. Built on the NVIDIA Nemotron Voice Agent, it brings together NVIDIA Nemotron ASR, Nemotron 3.5 Bolt E4B 4GB, Magpie TTS Zeroshot, and Audio2Face-3D—all running on a single workstation.

I wanted the kids visiting my house on Halloween to have a fun and unique way to interact. The house itself would run the show and hand out candy, leaning more towards grumpy and cranky than genuinely terrifying. Before I knew it, my solution was to slap a giant mouth onto the garage door and teach it how to roast people.

That’s how MonsterHouse came to life: a quirky old structure packed with pirate tales, sporting a fiery grin, and harboring a total weakness for the word “please.” The NVIDIA Nemotron Voice Agent provided the foundational conversational framework. Nemotron Speech does the listening, Bolt E4B serves as the brain, NVIDIA Magpie TTS Zeroshot generates real-time speech, and NVIDIA Audio2Face-3D brings the wooden mouth to life with animated lip-syncing.

Video 1

How Monster House Works

  • Listening: NVIDIA Nemotron Speech ASR transcribes visitor speech into text. [Confirmed: nemotron-speech-streaming-en-0.6b.q8_0.gguf ]
  • Thinking: NVIDIA Nemotron 3.5 Bolt E4B 4GB—an open-source small language model designed for on-device conversational agents—crafts every reply. LoRA fine-tuning weights give it a distinct personality.
  • Speaking: NVIDIA Magpie TTS Zeroshot synthesizes matching speech based on a short reference audio sample of the character.
  • Animating: NVIDIA Audio2Face-3D transforms clean speech into mouth animations inside Unreal Engine 5.
  • Seeing: NVIDIA Nemotron 3 Nano Omni identifies and describes Halloween costumes, while pairing with YOLO and ByteTrack to count and track visitors.
  • Running it: The NVIDIA Nemotron Voice Agent orchestrates all components on a single workstation equipped with one NVIDIA RTX PRO 6000 Blackwell Max-Q and one NVIDIA RTX PRO 4000 Blackwell.

A Grin Full of Splinters

The mouth was step one. I envisioned the wooden panels of the garage door splintering into jagged lips, with infernal flames glowing from within to illuminate the driveway whenever it opened its cursed maw. It needed to look aged, expressive, and mildly offended by the mere presence of visitors.

Image 8: Fiery Mouth_A house of expressions, complete with…