I recently created a new video of George Hamilton Green’s Triplets, as performed by five Ben Charleses. It was an enjoyable project to try out some new audio and video editing tricks I’ve picked up. (It was also an excuse to use my new MacBook Pro…thanks Apple for giving creative people so much power!) And, being a bad magician, I figured I would reveal how this sort of video is created for the uninitiated. But first, check out the video!
To make a long story short (and state the obvious): what you see is not a completely accurate representation of what is going on.
Another great example of duplicating a person for a music video can be found in Jacob Collier’s Hideaway. Obviously Jacob didn’t clone himself to make a video—he’s using an editing technique called masking, the same as I did in my Triplets video.
Here’s how the process works:
- Record yourself performing, preferably on one side of the frame.
- Without moving the camera, record yourself again on the other side of the room.
- Place one video on top of the other (think as if you have two pieces of construction paper stacked on top of each other).
- Cut a hole in one video so that the other shows through (again, the construction paper analogy works well here).
- Repeat…you can create an army of yourself!
Here's a screenshot of the editing process in Final Cut Pro. The red lines indicate the mask I have applied around light blue shirt Ben Charles. Notice how closely I had to cut next to purple shirt Ben Charles.
Some tricky situations can pop; for example, in my video, there are two Ben Charleses on each marimba, usually in very close quarters. This means I have to go through, sometimes frame-by-frame, and make adjustments to the cutout. It also means that, at some point, one person is probably going to be blocked by another. If you look closely at the marimbas on my video, you’ll see this happen (fairly regularly, in fact). I could have gone to very extreme lengths to try to get all of this perfect, but it was a fun side project, not a professional commercial operation. You also don't really notice it much on the fly. The same thing happens on Jacob’s video—if you look at 2:30 on the recording, the Jacob all the way on the left has an arm that disappears a bit.
(Side note: because of this editing trickery, my video contains two marimbas…which are actually the same instrument, just duplicated the video!)
And now, one more big secret: what you see is not what you hear. I would go as far as to say that this is the case with 90% or more professionally produced music videos.
Getting good sound is difficult. It requires a studio environment (or close to it), with very specifically placed mics, probably some physical separation between multiple players, and so on. It doesn’t exactly lend itself to making a music video on the beach, in a train, skydiving, and so on. The good news is that if you record something in a studio, then sing at the same tempo while skydiving, you can swap out the skydiving audio with the studio audio, and everything will be in sync. Your brain wants to believe what it hears and sees, so it just assumes that the audio and video go together.
In my case, I recorded all of the audio for the marimba parts in one night. When I recorded, I had a metronome going at 110 beats per minute in my headphones. As I recorded each new part, I added them into my click track so I could match the style. If I didn’t like the way something lined up, I could use a technique called splicing to remove one part and substitute a replacement. If I had absolutely no skill as a percussionist, I could even go as far as to record each individual note and splice them together—obviously that would be very time-consuming.
When I recorded the video, I had a metronome going at 110 beats per minute, always just out of the frame so the viewer couldn’t see it. It actually didn’t even matter if I hit wrong notes; for the most part, the camera is so far away that you can’t tell. (Don’t worry, I tried to hit the right notes, even if it wasn’t necessary.)
I then imported all of the video footage and synced it up to the “studio” audio, then got to all of the masking described above. So the audio and video were actually recorded on separate days. In fact, if anyone else recorded this same piece at 110 beats per minute, I could swap in their audio recording. (But don’t worry, you are both hearing and seeing me.)
And, if you didn’t notice, I still hadn’t actually done the studio recording of the xylophone solo part! So that was the final step. Once that was recorded, I tweaked my studio mix a bit, removed all of the existing audio from the video, and swapped that in.
The whole process took probably around 25-30 hours of work, and I’m pleased with the result!
Now here’s a bonus, just to show you what I mean by all of this…as an inside joke with my friend Akira, here’s Triplets as played on chairs. Notice that the audio and video are completely unrelated?