Do AI video-generators dream of San Pedro? Madonna among early adopters of AI’s next wave

Mar 4, 2024, 8:00 AM | Updated: 12:38 pm

Whenever Madonna sings the 1980s hit “La Isla Bonita” on her concert tour, moving images of swirling, sunset-tinted clouds play on the giant arena screens behind her.

To get that ethereal look, the pop legend embraced a still-uncharted branch of generative artificial intelligence – the text-to-video tool. Type some words — say, “surreal cloud sunset” or “waterfall in the jungle at dawn” — and an instant video is made.

Following in the footsteps of AI chatbots and still image-generators, some AI video enthusiasts say the emerging technology could one day upend entertainment, enabling you to choose your own movie with customizable story lines and endings. But there’s a long way to go before they can do that, and plenty of ethical pitfalls on the way.

For early adopters like Madonna, who’s long pushed art’s boundaries, it was more of an experiment. She nixed an earlier version of “La Isla Bonita” concert visuals that used more conventional computer graphics to evoke a tropical mood.

“We tried CGI. It looked pretty bland and cheesy and she didn’t like it,” said Sasha Kasiuha, content director for Madonna’s Celebration Tour that continues through late April. “And then we decided to try AI.”

ChatGPT-maker OpenAI gave a glimpse of what sophisticated text-to-video technology might look like when the company recently showed off Sora, a new tool that’s not yet publicly available. Madonna’s team tried a different product from New York-based startup Runway, which helped pioneer the technology by releasing its first public text-to-video model last March. The company released a more advanced “Gen-2″ version in June.

Runway CEO Cristóbal Valenzuela said while some see these tools as a “magical device that you type a word and somehow it conjures exactly what you had in your head,” the most effective approaches are by creative professionals looking for an upgrade to the decades-old digital editing software they’re already using.

He said Runway can’t yet make a full-length documentary. But it could help fill in some background video, or b-roll — the supporting shots and scenes that help tell the story.

“That saves you perhaps like a week of work,” Valenzuela said. “The common thread of a lot of use cases is people use it as a way of augmenting or speeding up something they could have done before.”

Runway’s target customers are “large streaming companies, production companies, post-production companies, visual effects companies, marketing teams, advertising companies. A lot of folks that make content for a living,” Valenzuela said.

Dangers await. Without effective safeguards, AI video-generators could threaten democracies with convincing “deepfake” videos of things that never happened, or — as is already the case with AI image generators — flood the internet with fake pornographic scenes depicting what appear to be real people with recognizable faces. Under pressure from regulators, major tech companies have promised to watermark AI-generated outputs to help identify what’s real.

There also are copyright disputes brewing about the video and image collections the AI systems are being trained upon (neither Runway nor OpenAI discloses its data sources) and to what extent they are unfairly replicating trademarked works. And there are fears that, at some point, video-making machines could replace human jobs and artistry.

For now, the longest AI-generated video clips are still measured in seconds, and can feature jerky movements and telltale glitches such as distorted hands and fingers. Fixing that is “just a question of more data and more training,” and the computing power on which that training depends, said Alexander Waibel, a computer science professor at Carnegie Mellon University who’s been researching AI since the 1970s.

“Now I can say, ‘Make me a video of a rabbit dressed as Napoleon walking through New York City,’” Waibel said. “It knows what New York City looks like, what a rabbit looks like, what Napoleon looks like.”

Which is impressive, he said, but still far from crafting a compelling storyline.

Before it released its first-generation model last year, Runway’s claim to AI fame was as a co-developer of the image-generator Stable Diffusion. Another company, London-based Stability AI, has since taken over Stable Diffusion’s development.

The underlying “diffusion model” technology behind most leading AI generators of images and video works by mapping noise, or random data, onto images, effectively destroying an original image and then predicting what a new one should look like. It borrows an idea from physics that can be used to describe, for instance, how gas diffuses outward.

“What diffusion models do is they reverse that process,” said Phillip Isola, an associate professor of computer science at the Massachusetts Institute of Technology. “They kind of take the randomness and they congeal it back into the volume. That’s the way of going from randomness to content. And that’s how you can make random videos.”

Generating video is more complicated than still images because it needs to take into account temporal dynamics, or how elements within the video change over time and across sequences of frames, said Daniela Rus, another MIT professor who directs its Computer Science and Artificial Intelligence Laboratory.

Rus said the computing resources required are “significantly higher than for still image generation” because “it involves processing and generating multiple frames for each second of video.”

That’s not stopping some well-heeled tech companies from trying to keep outdoing each other in showing off higher-quality AI video generation at longer durations. Requiring written descriptions to make an image was just the start. Google recently demonstrated a new project called Genie that can be prompted to transform a photograph or even a sketch into “an endless variety” of explorable video game worlds.

In the near term, AI-generated videos will likely show up in marketing and educational content, providing a cheaper alternative to producing original footage or obtaining stock videos, said Aditi Singh, a researcher at Cleveland State University who has surveyed the text-to-video market.

When Madonna first talked to her team about AI, the “main intention wasn’t, ’Oh, look, it’s an AI video,’” said Kasiuha, the creative director.

“She asked me, ‘Can you just use one of those AI tools to make the picture more crisp, to make sure it looks current and looks high resolution?’” Kasiuha said. “She loves when you bring in new technology and new kinds of visual elements.”

Longer AI-generated movies are already being made. Runway hosts an annual AI film festival to showcase such works. But whether that’s what human audiences will choose to watch remains to be seen.

“I still believe in humans,” said Waibel, the CMU professor. ”I still believe that it will end up being a symbiosis where you get some AI proposing something and a human improves or guides it. Or the humans will do it and the AI will fix it up.”


Associated Press journalists Joseph B. Frederick and Rodrique Ngowi contributed to this report.

United States News

Republican presidential candidate former President Donald Trump talks to Sen. Marco Rubio during th...

Associated Press

2024 Election Latest: Trump to speak at RNC as convention enters fourth day, Biden has COVID-19

The Republican National Convention culminates Thursday with former President Donald Trump expected to accept the party’s presidential nomination, achieving a comeback four years in the making and anticipated even more in the past week in light of Saturday’s assassination attempt. He is expected to accept his third consecutive party nod in prime time before thousands […]

10 minutes ago

Wall Street Journal reporter Evan Gershkovich stands in a glass cage in a courtroom in Yekaterinbur...

Associated Press

U.S. reporter Evan Gershkovich appear in court in Russia for hearing on espionage charges

Wall Street Journal reporter Evan Gershkovich appeared in court in Russia on Thursday for the second hearing in his trial on espionage charges, the court said. Gershkovich, his employer and the U.S. government vehemently deny the charges against him. Gershkovich was accused by the Russian Prosecutor General’s office of “gathering secret information” on orders from […]

5 hours ago

FILE - A pedestrian traverses the rotunda inside the New Mexico Capitol, March 16, 2023, in Santa F...

Associated Press

New Mexico governor cites ‘dangerous intersection’ of crime and homelessness, wants lawmakers to act

SANTA FE, N.M. (AP) — Citing what she calls the “dangerous intersection” of crime and homelessness, New Mexico’s governor is calling on lawmakers to address stubbornly high crime rates as they convene Thursday for a special legislative session. In issuing her proclamation, Gov. Michelle Lujan Grisham talked about a vulnerable segment of society that falls […]

7 hours ago

Republican presidential candidate former President Donald Trump arrives on third day of the Republi...

Associated Press

FACT FOCUS: Trump, in Republican convention video, alludes to false claim 2020 election was stolen

DONALD TRUMP, alluding that the 2020 vote was stolen: “Whether you vote early, absentee, by mail or in person, we are going to protect the vote. That’s the most important thing we have to do is protect the vote. Keep your eyes open because these people want to cheat and they do cheat. And frankly, […]

7 hours ago

Supporters cheer as Republican presidential candidate former President Donald Trump arrives for a c...

Associated Press

‘One screen, two movies’: Conflicting conspiracy theories emerge from Trump shooting

WASHINGTON (AP) — A former president is shot, the gunman quickly neutralized, and all of it is caught on camera. But for those who don’t believe their eyes, that’s just the start of the story. For some supporters of former President Donald Trump, the failure of the Secret Service to prevent the attempted assassination point […]

7 hours ago

Follow @ktar923...

Sponsored Content by Sanderson Ford

3 storylines to get you revved up for the 2024 Arizona Cardinals

Arizona Cardinals training camp is just a couple weeks away starting on July 25, and Sanderson Ford is revved up and ready to go.

Sponsored Articles


Sanderson Ford

3 new rides for 3 new road trips in Arizona

It's time for the Sanderson Ford Memorial Day sale with the Mighty Fine 69 Anniversary, as Sanderson Ford turned 69 years old in May.


Day & Night Air Conditioning, Heating and Plumbing

Beat the heat, ensure your AC unit is summer-ready

With temperatures starting to rise across the Valley, now is a great time to be sure your AC unit is ready to withstand the sweltering summer heat.



Desert Institute for Spine Care is the place for weekend warriors to fix their back pain

Spring has sprung and nothing is better than March in Arizona. The temperatures are perfect and with the beautiful weather, Arizona has become a hotbed for hikers, runners, golfers, pickleball players and all types of weekend warriors.

Do AI video-generators dream of San Pedro? Madonna among early adopters of AI’s next wave