Enabling Live Performances in Virtual Reality

You are a stand-up comedian who can no longer perform at the local comedy club because of social distancing. You cannot perform via Facebook, Instagram, YouTube, Zoom or Google Meet because you need to connect with the audience. Will you locate to a different state with loose social distancing measures? Joe Rogan said he might, but there might be another solution.

It’s 7:55 PM. You have to go live in 5 minutes. You put on a virtual reality headset and get teleported to the stage with a full house in front. You thought this would be lame, like the simulated crowds in football games on consoles and computers. But this feels different, more real. You’re processing that thought when you recognize a friend in the front row! Wow. And they look… bored? Well, the show has not begun yet, but how is this happening?

The people you are seeing in the audience are like characters in an electronic game, except that information captured from their webcams is used to reconstruct their facial expressions and features in real-time. These characters are generated, but the engine generating them receives input parameters that define the character. Hair features, facial features, etc. In addition to that, a machine learning algorithm recognizes emotional expressions such as smiles, laughs, blushes, disapproval, etc. Another machine learning algorithm cleans up the audio feed from each viewer, removing all but speech, applause, laughs and groans.

It’s time. You walk a few steps and you’re on stage. You can look around as you wish. You crack the first joke and you see and hear the audience roar into laughter. This is working.

For it to work though, each viewer needs to be in front of a camera, on a smartphone, laptop or desktop. It may be possible for them to be experiencing this in virtual reality as well, if someone can figure out a way to capture the same data without needing another camera device in front.

As far as my understanding goes, this is already possible and can facilitate live experiences beyond just comedy. Think about education! Hope someone does it.

--

--

--

Technology entrepreneur based in New York. Interested in art, music, philosophy, psychology, machine learning, neuroscience, physics, and nature.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

New Tesla Model S 100D Option Features 335-Mile Range

STEM Box update #10 — Wheelson’s final form

​​What is Motion Capture Technology? How does it work?

POCO M4 Pro — The Budget Flagship?

S22 Ultra vs POCO M4 Pro

Lenovo IdeaPad Flex 5 Review: Things You Need to Know

Remote Control for Autonomous Vehicles — A far worse idea than the use of Public Shadow “Safety”…

We are the robots: How and why our technology is changing us

Uefa introduces 70% squad cost rule as part of new financial regulations

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shan Rizvi

Shan Rizvi

Technology entrepreneur based in New York. Interested in art, music, philosophy, psychology, machine learning, neuroscience, physics, and nature.

More from Medium

Understanding Virtual Reality

How Augmented Reality and Virtual reality are shaping the future of Model-Based Systems Engineering

The Augmented Soldier

Signatures: Augmented Reality at the world scale