Enabling Live Performances in Virtual Reality
You are a stand-up comedian who can no longer perform at the local comedy club because of social distancing. You cannot perform via Facebook, Instagram, YouTube, Zoom or Google Meet because you need to connect with the audience. Will you locate to a different state with loose social distancing measures? Joe Rogan said he might, but there might be another solution.
It’s 7:55 PM. You have to go live in 5 minutes. You put on a virtual reality headset and get teleported to the stage with a full house in front. You thought this would be lame, like the simulated crowds in football games on consoles and computers. But this feels different, more real. You’re processing that thought when you recognize a friend in the front row! Wow. And they look… bored? Well, the show has not begun yet, but how is this happening?
The people you are seeing in the audience are like characters in an electronic game, except that information captured from their webcams is used to reconstruct their facial expressions and features in real-time. These characters are generated, but the engine generating them receives input parameters that define the character. Hair features, facial features, etc. In addition to that, a machine learning algorithm recognizes emotional expressions such as smiles, laughs, blushes, disapproval, etc. Another machine learning algorithm cleans up the audio feed from each viewer, removing all but speech, applause, laughs and groans.
It’s time. You walk a few steps and you’re on stage. You can look around as you wish. You crack the first joke and you see and hear the audience roar into laughter. This is working.
For it to work though, each viewer needs to be in front of a camera, on a smartphone, laptop or desktop. It may be possible for them to be experiencing this in virtual reality as well, if someone can figure out a way to capture the same data without needing another camera device in front.
As far as my understanding goes, this is already possible and can facilitate live experiences beyond just comedy. Think about education! Hope someone does it.
