{"id":24405,"date":"2022-12-19T13:28:04","date_gmt":"2022-12-19T13:28:04","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/12\/19\/a-i-is-here-and-its-making-movies-is-hollywood-ready\/"},"modified":"2022-12-19T13:28:04","modified_gmt":"2022-12-19T13:28:04","slug":"a-i-is-here-and-its-making-movies-is-hollywood-ready","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/12\/19\/a-i-is-here-and-its-making-movies-is-hollywood-ready\/","title":{"rendered":"A.I. is here, and it’s making movies. Is Hollywood ready?"},"content":{"rendered":"

<\/a>\n<\/p>\n

\n

Scott Mann had a problem: too many f-bombs.<\/p>\n

The writer-director had spent production on \u201cFall,\u201d his vertigo-inducing thriller about rock climbers stuck atop a remote TV tower, encouraging the two leads to have fun with their dialogue. That improv landed a whopping 35 \u201cf-cks\u201d in the film, placing it firmly in R-rated territory.<\/p>\n

But when Lionsgate signed on to distribute \u201cFall,\u201d the studio wanted a PG-13 edit. Sanitizing the film would mean scrubbing all but one<\/a><\/u> of the obscenities.<\/p>\n

\u201cHow do you solve that?\u201d Mann recalled from the glass-lined conference room of his Santa Monica office this October, two months after the film\u2019s debut. A prop vulture he\u2019d commandeered from set sat perched out in the lobby.<\/p>\n

Reshoots, after all, are expensive and time-consuming. Mann had filmed \u201cFall\u201d on a mountaintop, he explained, and struggled throughout with not just COVID but also hurricanes and lightning storms. A colony of fire ants had taken up residence inside the movie\u2019s main set, a hundred-foot-long metal tube, at one point; when the crew woke them up, the swarm enveloped the set \u201clike a cloud.\u201d<\/p>\n

\u201c\u2018Fall\u2019 was probably the hardest film I ever made,\u201d said Mann. Could he avoid a redux?<\/p>\n

The solution, he realized, just might be a project he\u2019d been developing in tandem with the film: artificially intelligent software that could edit footage of the actors\u2019 faces well after principal photography had wrapped, seamlessly altering their facial expressions and mouth movements to match newly recorded dialogue.<\/p>\n

It\u2019s a deceptively simple use for a technology that experts say is poised to transform nearly every dimension of Hollywood, from the labor dynamics and financial models to how audiences think about what\u2019s real or fake.<\/p>\n

Artificial intelligence will do to motion pictures what Photoshop did to still ones, said Robert Wahl, an associate computer science professor at Concordia University Wisconsin who\u2019s written about the ethics of CGI, in an email. \u201cWe can no longer fully trust what we see.\u201d<\/p>\n

A software solution for dubious dubs<\/b><\/p>\n

It took a particularly dispiriting collaboration with Robert De Niro to push Mann into the world of software.<\/p>\n

De Niro was featuring in Mann\u2019s 2015 crime thriller \u201cHeist,\u201d and the two had put a lot of time and thought into the acclaimed actor\u2019s performance. But when it came time to adapt the film for foreign releases, Mann said, he was left unsatisfied.<\/p>\n

When films get released overseas, the dialogue is often re-recorded in other languages. That process, called \u201cdubbing,\u201d makes the movie internationally accessible but can also lead to the jarring sight of an actor\u2019s mouth flapping out-of-sync with the words they\u2019re supposedly saying. One typical solution is to rewrite dialogue so it pairs up better with the pre-existing visuals \u2014 but, for the sake of legibility, those changes sacrifice the creative team\u2019s original vision.<\/p>\n

\u201cAll the things I\u2019d worked out in nuance with Robert De Niro were now changed,\u201d Mann said of the dubs. \u201cI was kind of devastated.\u201d<\/p>\n

A follow-up film he worked on, \u201cFinal Score,\u201d deepened those frustrations. Mann tried scanning his cast-members\u2019 heads so he could better sync up their speech, but the process proved prohibitively expensive and the final outcome looked weird.<\/p>\n

It wasn\u2019t until researching more novel solutions that the visual effects enthusiast found a 2018 academic paper<\/a> outlining a possible solution: neural networks, or computer programs mimicking the structure<\/a> of a brain, that sought to transpose one actor\u2019s facial expression onto another\u2019s face.<\/p>\n

Fascinated, Mann reached out to the paper\u2019s authors and began collaborating with some of them on a rudimentary \u201cvubbing\u201d tool \u2014 that is, visual, rather than audio, dubbing. The subsequent addition of Nick Lynes, a friend-of-a-friend with a background in online gaming, gave the team a foothold in the tech sector, too.<\/p>\n

Together, the envoys of three very different worlds \u2014 cinema, science and the software industry \u2014 built Flawless, an A.I. filmmaking venture with offices in both Santa Monica and London.<\/p>\n

In very broad terms, the company\u2019s tech can identify patterns in an actors\u2019 phonemes (or the sounds they make) and visemes (or how they look when they\u2019re making those sounds), and then \u2014 when presented with newly recorded phonemes \u2014 update the on-screen visemes to match. Last year, Time magazine deemed<\/a><\/u> the company\u2019s \u201cfix for film dubbing\u201d one of the best inventions of 2021.<\/p>\n

The scramble to scrub dozens of f-bombs from \u201cFall,\u201d however, presented a question with potentially much broader ramifications: rather than just change what language characters spoke, could Flawless alter the very content of what they said?<\/p>\n

\u201cWe went into a recording studio down in \u2026 Burbank with the actresses and said, \u2018All right, here\u2019s the new lines,\u2019\u201d said Mann, who lives in Los Angeles. Then they plugged the new audio into the vubbing software, which adjusted the stars\u2019 on-screen facial movements accordingly. <\/p>\n

\u201cWe put the shots in, MPAA re-reviewed it and gave it PG-13, and that was what got into the cinemas,\u201d he said.<\/p>\n

Sitting in his Santa Monica conference room several weeks after the film came out, surrounded by posters for \u201cBlade Runner\u201d and \u201c2001: A Space Odyssey,\u201d Mann showed off the results with a scene wherein one of \u201cFall\u2019s\u201d protagonists bemoans their predicament.<\/p>\n

\u201cNow we\u2019re stuck on this stupid freaking tower in the middle of freaking nowhere!\u201d Virginia Gardner exclaimed to Grace Caroline Currey as the two huddled atop a precariously lofty platform.<\/p>\n

\n
\"Virginia <\/source><\/picture>\n
\n

Virginia Gardner and Grace Caroline Currey in \u201cFall.\u201d<\/p>\n

(Lionsgate)<\/p>\n<\/p><\/div>\n<\/figure>\n<\/div>\n

A moment later Mann replayed the scene. But this time, Gardner\u2019s dialogue was noticeably harsher: \u201cNow we\u2019re stuck on this stupid f-cking tower in the middle of f-cking nowhere.\u201d<\/p>\n

The first version was what went out in August to over<\/a><\/u> 1,500 American theaters. But the latter \u2014 the one with dialogue fit for a sailor \u2014 was what Mann actually filmed back on that fire ant-infested mountaintop. If you didn\u2019t know a neural network had reconstructed the actors\u2019 faces, you\u2019d probably have no idea their cleaned-up dialogue was a late addition.<\/p>\n

\u201cYou can\u2019t tell what\u2019s real and what\u2019s not,\u201d Mann said, \u201cwhich is the whole thing.\u201d<\/p>\n

The ethics of synthetics<\/b><\/p>\n

When it comes to filmmaking, that realism has obvious benefits. No one wants to spend money on something that looks like it came out of MS Paint.<\/p>\n

But the rise of software that can seamlessly change what someone seems to have said has major implications for a media environment already awash in misinformation. Flawless\u2019 core product is, after all, essentially just a more legit version of \u201cdeep-fakes<\/a><\/u>,\u201d or CGI that mimics someone\u2019s face and voice. <\/p>\n

It\u2019s not hard to imagine a troll who, instead of using these tools to cut cuss words from a movie, makes a viral video of Joe Biden declaring war on Russia. Porn made with someone\u2019s digital likeness has also become an issue<\/a>.<\/p>\n

And Flawless isn\u2019t the only company working in this space. Papercup, a company that generates synthetic human voices for use in dubs and voice-overs, aims \u201cto make any video watchable in any language,\u201d chief executive Jesse Shemen told The Times. <\/p>\n

And visual effects mainstay Digital Domain uses machine learning to render actors in cases where they can\u2019t appear themselves, such as scenes requiring a stunt double, said chief technology officer Hanno Basse.<\/p>\n

As these and other firms increasingly automate the entertainment industry, ethical questions abound.<\/p>\n

Hollywood is already reckoning with its newfound ability to digitally re-create dead actors, as with Anthony Bourdain\u2019s voice<\/a><\/u> in the documentary \u201cRoadrunner\u201d or Peter Cushing<\/a><\/u> and Carrie Fisher<\/a><\/u> in recent \u201cStar Wars\u201d sequels. Holographic revivals<\/a><\/u> of late celebs are also now possible.<\/p>\n

Digitally altered dialogue \u201crisks compromising the consent of those originally involved,\u201d said Scott Stroud, the director of the University of Texas at Austin\u2019s program in media ethics. \u201cWhat actors thought they were agreeing to isn\u2019t literally what is created.\u201d<\/p>\n

And this technology could open the door to films being changed long after they come out, said Denver D\u2019Rozario, a Howard University marketing professor who has studied the software resurrection of dead actors.<\/p>\n

\u201cLet\u2019s say \u2026 in a movie a guy\u2019s drinking a can of Pepsi, and 20 years from now you get a sponsorship from Coke,\u201d said D\u2019Rozario. \u201cDo you change the can of Pepsi to Coke?\u201d \u201cAt what point can things be changed? At what point can things be bought?\u201d<\/p>\n

Mann said the advantages of his technology are many, from breaking down language barriers and fomenting cross-border empathy to sparing actors the headache of reshoots. In his view, scenarios like D\u2019Rozario\u2019s hypothetical Coke sponsorship represent new revenue streams.<\/p>\n

Flawless has been proactive, Mann added, about building a product that aids rather than supplants authentic human performance.<\/p>\n

\u201cThere is a way to utilize technologies in a similar way that the [visual effects] industry has already established, which is like: do it securely, do it right, do it legally, with consent from everyone involved,\u201d he said.<\/p>\n

And the company has already engaged \u201call the big unions\u201d on how to make and use this technology in a sensible way, the director continued.<\/p>\n

SAG-AFTRA representatives stressed that A.I. filmmaking tech can either help or harm actors, depending on how it\u2019s used.<\/p>\n

\u201cTechnologies that do little more than digitally enhance our members\u2019 work may just require the ability to provide informed consent and, possibly, additional compensation,\u201d Jeffrey Bennett, SAG-AFTRA\u2019s general counsel, said in an email. \u201cAt the other end of the spectrum are the technologies that might replace traditional performance or that take our members\u2019 performances and create wholly new ones; for these, we maintain that they are a mandatory subject of bargaining.\u201d<\/p>\n

It\u2019s a train that, for better or worse, has already left the station. <\/p>\n

\u201cFall\u201d is currently streaming, and Mann said other movies his company worked on are coming out this Christmas \u2014 although he can\u2019t yet name them publicly.<\/p>\n

If you see a movie over the holidays, an A.I. might have helped create it.<\/p>\n

Will you be able to tell? Would it matter?<\/p>\n<\/div>\n