UNITED STATES NEWS

NYC’s AI chatbot was caught telling businesses to break the law. The city isn’t taking it down

Apr 3, 2024, 4:05 PM

NEW YORK (AP) — An artificial intelligence-powered chatbot created by New York City to help small business owners is under criticism for dispensing bizarre advice that misstates local policies and advises companies to violate the law.

But days after the issues were first reported last week by tech news outlet The Markup, the city has opted to leave the tool on its official government website. Mayor Eric Adams defended the decision this week even as he acknowledged the chatbot’s answers were “wrong in some areas.”

Launched in October as a “one-stop shop” for business owners, the chatbot offers users algorithmically generated text responses to questions about navigating the city’s bureaucratic maze.

It includes a disclaimer that it may “occasionally produce incorrect, harmful or biased” information and the caveat, since-strengthened, that its answers are not legal advice.

It continues to dole out false guidance, troubling experts who say the buggy system highlights the dangers of governments embracing AI-powered systems without sufficient guardrails.

“They’re rolling out software that is unproven without oversight,” said Julia Stoyanovich, a computer science professor and director of the Center for Responsible AI at New York University. “It’s clear they have no intention of doing what’s responsible.”

In responses to questions posed Wednesday, the chatbot falsely suggested it is legal for an employer to fire a worker who complains about sexual harassment, doesn’t disclose a pregnancy or refuses to cut their dreadlocks. Contradicting two of the city’s signature waste initiatives, it claimed that businesses can put their trash in black garbage bags and are not required to compost.

At times, the bot’s answers veered into the absurd. Asked if a restaurant could serve cheese nibbled on by a rodent, it responded: “Yes, you can still serve the cheese to customers if it has rat bites,” before adding that it was important to assess the “the extent of the damage caused by the rat” and to “inform customers about the situation.”

A spokesperson for Microsoft, which powers the bot through its Azure AI services, said the company was working with city employees “to improve the service and ensure the outputs are accurate and grounded on the city’s official documentation.”

At a press conference Tuesday, Adams, a Democrat, suggested that allowing users to find issues is just part of ironing out kinks in new technology.

“Anyone that knows technology knows this is how it’s done,” he said. “Only those who are fearful sit down and say, ‘Oh, it is not working the way we want, now we have to run away from it all together.’ I don’t live that way.”

Stoyanovich called that approach “reckless and irresponsible.”

Scientists have long voiced concerns about the drawbacks of these kinds of large language models, which are trained on troves of text pulled from the internet and prone to spitting out answers that are inaccurate and illogical.

But as the success of ChatGPT and other chatbots have captured the public attention, private companies have rolled out their own products, with mixed results. Earlier this month, a court ordered Air Canada to refund a customer after a company chatbot misstated the airline’s refund policy. Both TurboTax and H&R Block have faced recent criticism for deploying chatbots that give out bad tax-prep advice.

Jevin West, a professor at the University of Washington and co-founder of the Center for an Informed Public, said the stakes were especially high when the models are promoted by the public sector.

“There’s a different level of trust that’s given to government,” West said. “Public officials need to consider what kind of damage they can do if someone was to follow this advice and get themselves in trouble.”

Experts say other cities that use chatbots have typically confined them to a more limited set of inputs, cutting down on misinformation.

Ted Ross, the chief information officer in Los Angeles, said the city closely curated the content used by its chatbots, which do not rely on large language models.

The pitfalls of New York’s chatbot should serve as a cautionary tale for other cities, said Suresh Venkatasubramanian, the director of the Center for Technological Responsibility, Reimagination, and Redesign at Brown University.

“It should make cities think about why they want to use chatbots, and what problem they are trying to solve,” he wrote in an email. “If the chatbots are used to replace a person, then you lose accountability while not getting anything in return.”

United States News

Associated Press

Shooting kills 3 people including a young child in a car on an Alabama street

Three people including a young child were killed when their car was targeted with multiple gunshots outside a residence in Birmingham, Alabama, on Saturday evening. The Birmingham Police Department said in a social media post that a triple homicide investigation was underway in the 3400 Block of 27th Street North. The police said officers responded […]

2 hours ago

Republican presidential candidate former President Donald Trump is covered by U.S. Secret Service a...

Associated Press

The Secret Service is investigating how a gunman who shot and injured Trump was able to get so close

The Secret Service is investigating how a gunman was able to get close enough to shoot at former President Donald Trump at a rally Saturday.

3 hours ago

Carlton Smith, left, and Justin Nimmo pose in from of the console of the Barton Opus 234 theater or...

Associated Press

Reviving Hollywood glamor of the silent movie era, experts piece together a century-old pipe organ

DETROIT (AP) — A massive pipe organ that underscored the drama and comedy of silent movies with live music in Detroit’s ornate Hollywood Theatre nearly a century ago was dismantled into thousands of pieces and stashed away. The Barton Opus, built in 1927, spent four decades stored in a garage, attic and basement in suburban […]

4 hours ago

FILE - Georgia Gov. Brian Kemp, left, and Centers for Medicare and Medicaid Services Administrator ...

Associated Press

Can a Medicaid plan that requires work succeed? First year of Georgia experiment is not promising

ATLANTA (AP) — By now, Georgia officials expected their new Medicaid plan, the only one in the nation with a work requirement, to provide health insurance to 25,000 low-income residents and possibly tens of thousands more. But a year since its launch, Pathways to Coverage has roughly 4,300 members, much lower than what state officials […]

4 hours ago

Catherine Ward, owner of One Acre Farms, a functioning educational farm to help special needs kids,...

Associated Press

After Beryl, Houston-area farmers pull together to face unique challenges

PORTER, Texas (AP) — Hurricanes cause trouble for everyone, but farmers have a whole other list of problems. Beryl has been no exception: Over the past week, the storm damaged crops, soaked rabbits and chickens, terrified goats, put horses at risk of developing colic and left cows without fences to keep them penned in. And […]

5 hours ago

Debris from the Titan submersible, recovered from the ocean floor near the wreck of the Titanic, is...

Associated Press

The first Titanic voyage in 14 years is happening in the wake of submersible tragedy. Hopes are high

PORTLAND, Maine (AP) — The company that owns the salvage rights to the Titanic is undertaking its first expedition to the ship’s wreckage in years, and those involved in the mission said they have both heavy hearts and lofty goals for a trip happening a year after a submersible disaster involving another firm killed five […]

5 hours ago

Sponsored Articles

...

Day & Night Air Conditioning, Heating and Plumbing

Beat the heat, ensure your AC unit is summer-ready

With temperatures starting to rise across the Valley, now is a great time to be sure your AC unit is ready to withstand the sweltering summer heat.

...

Condor Airlines

Condor Airlines can get you smoothly from Phoenix to Frankfurt on new A330-900neo airplane

Adventure Awaits! And there's no better way to experience the vacation of your dreams than traveling with Condor Airlines.

...

Midwestern University

Midwestern University Clinics: transforming health care in the valley

Midwestern University, long a fixture of comprehensive health care education in the West Valley, is also a recognized leader in community health care.

NYC’s AI chatbot was caught telling businesses to break the law. The city isn’t taking it down