Let's get right down to it: This SXSW performance by Japanese pop trio Perfume is the most amazing thing I've seen in a long time. It's lightyears beyond straightforward projection mapping. It's lightyears beyond any sort of audiovisual spectacle I can think of, really. This thing is an ecstatic vision beamed back from a future in which the physical and digital have converged to the point of being utterly indistinguishable. It's a fever dream of atoms and pixels and photons. It's unbelievable. I've watched it about a hundred times and I still have no idea what's going on.
Here's what I do know. Mikiko, of the dance group Elevenplay, was responsible for the the stage show, visuals, and and choreography. The performance was produced by Kaoru Sugano, the famed creative director of the Japanese ad company Dentsu. The technical bits were overseen by Daito Manabe of the bleeding edge design crew Rhizomatiks. The whole thing demanded a small army of designers, engineers, and artists. The clip above was filmed in real time. None of it was done in post-production.
The video involves several "tricks," as Manabe puts it. The illusion began with preproduction, including an elaborate 3-D scan of the Austin venue and another of Perfume running through their performance.
The real magic, however, happened on the fly. During the performance, a dynamic projection mapping system cast visuals onto the semi-translucent screens in front of the singers; motion capture allowed the position of the projections to be calibrated automatically moment by moment. The cameras filming the performance were also watched by a motion capture system, each outfitted with a marker allowing the system to track the camera's position and orientation in space. This, Manabe says, was key for morphing seamlessly between perspectives, an effect conjured by Rhizomatiks computer vision wizard Yuya Hanai. The final video moves seamlessly between the live footage and the 3D model captured ahead of time.
Do I understand precisely how all that results in what I'm seeing here? Not really! But I don't understand exactly how the Chartes Cathedral was built either, and that doesn't make it any less tremendous.
