UNITED STATES NEWS

NYC’s AI chatbot was caught telling businesses to break the law. The city isn’t taking it down

Apr 3, 2024, 4:05 PM

NEW YORK (AP) — An artificial intelligence-powered chatbot created by New York City to help small business owners is under criticism for dispensing bizarre advice that misstates local policies and advises companies to violate the law.

But days after the issues were first reported last week by tech news outlet The Markup, the city has opted to leave the tool on its official government website. Mayor Eric Adams defended the decision this week even as he acknowledged the chatbot’s answers were “wrong in some areas.”

Launched in October as a “one-stop shop” for business owners, the chatbot offers users algorithmically generated text responses to questions about navigating the city’s bureaucratic maze.

It includes a disclaimer that it may “occasionally produce incorrect, harmful or biased” information and the caveat, since-strengthened, that its answers are not legal advice.

It continues to dole out false guidance, troubling experts who say the buggy system highlights the dangers of governments embracing AI-powered systems without sufficient guardrails.

“They’re rolling out software that is unproven without oversight,” said Julia Stoyanovich, a computer science professor and director of the Center for Responsible AI at New York University. “It’s clear they have no intention of doing what’s responsible.”

In responses to questions posed Wednesday, the chatbot falsely suggested it is legal for an employer to fire a worker who complains about sexual harassment, doesn’t disclose a pregnancy or refuses to cut their dreadlocks. Contradicting two of the city’s signature waste initiatives, it claimed that businesses can put their trash in black garbage bags and are not required to compost.

At times, the bot’s answers veered into the absurd. Asked if a restaurant could serve cheese nibbled on by a rodent, it responded: “Yes, you can still serve the cheese to customers if it has rat bites,” before adding that it was important to assess the “the extent of the damage caused by the rat” and to “inform customers about the situation.”

A spokesperson for Microsoft, which powers the bot through its Azure AI services, said the company was working with city employees “to improve the service and ensure the outputs are accurate and grounded on the city’s official documentation.”

At a press conference Tuesday, Adams, a Democrat, suggested that allowing users to find issues is just part of ironing out kinks in new technology.

“Anyone that knows technology knows this is how it’s done,” he said. “Only those who are fearful sit down and say, ‘Oh, it is not working the way we want, now we have to run away from it all together.’ I don’t live that way.”

Stoyanovich called that approach “reckless and irresponsible.”

Scientists have long voiced concerns about the drawbacks of these kinds of large language models, which are trained on troves of text pulled from the internet and prone to spitting out answers that are inaccurate and illogical.

But as the success of ChatGPT and other chatbots have captured the public attention, private companies have rolled out their own products, with mixed results. Earlier this month, a court ordered Air Canada to refund a customer after a company chatbot misstated the airline’s refund policy. Both TurboTax and H&R Block have faced recent criticism for deploying chatbots that give out bad tax-prep advice.

Jevin West, a professor at the University of Washington and co-founder of the Center for an Informed Public, said the stakes were especially high when the models are promoted by the public sector.

“There’s a different level of trust that’s given to government,” West said. “Public officials need to consider what kind of damage they can do if someone was to follow this advice and get themselves in trouble.”

Experts say other cities that use chatbots have typically confined them to a more limited set of inputs, cutting down on misinformation.

Ted Ross, the chief information officer in Los Angeles, said the city closely curated the content used by its chatbots, which do not rely on large language models.

The pitfalls of New York’s chatbot should serve as a cautionary tale for other cities, said Suresh Venkatasubramanian, the director of the Center for Technological Responsibility, Reimagination, and Redesign at Brown University.

“It should make cities think about why they want to use chatbots, and what problem they are trying to solve,” he wrote in an email. “If the chatbots are used to replace a person, then you lose accountability while not getting anything in return.”

United States News

Associated Press

Autoworkers union celebrates breakthrough win in Tennessee and takes aim at more plants in the South

DALLAS (AP) — The United Auto Workers’ overwhelming election victory at a Volkswagen plant in Tennessee is giving the union hope that it can make broader inroads in the South, the least unionized part of the country. The UAW won a stunning 73% of the vote at VW after losing elections in 2014 and 2019. […]

2 hours ago

Associated Press

Conditions improve for students shot in Maryland park on ‘senior skip day’

GREENBELT, Md. (AP) — Three of the five youths shot in a suburban Washington, D.C., park during a high school “senior skip day” gathering have been released from the hospital, while the condition of the most seriously injured has improved, authorities said Saturday. Police in Greenbelt, Maryland, are asking the public to share video footage […]

3 hours ago

Associated Press

New York lawmakers pass $237 billion budget addressing housing construction and migrants

ALBANY, N.Y. (AP) — New York lawmakers passed a $237 billion state budget Saturday that includes plans to spur housing construction and combat unlicensed marijuana stores. The package also includes a raft of other measures ranging from expediting the closure of some state prisons, addressing the recent influx of migrants, and continuing the pandemic-era policy […]

3 hours ago

SANTA FE, NEW MEXICO - DECEMBER 9, 2019: A copy of the record album, 'Johnny Cash at Folsom Prison'...

Associated Press

Record Store Day celebrates indie retail music sellers as they ride vinyl’s popularity wave

Record Store Day, a holiday invented at a gathering of independent record store owners and employees in 2007, will be observed Saturday.

7 hours ago

Associated Press

Former Arkansas governor, US Sen David Pryor dies at 89

LITTLE ROCK, Ark. (AP) — Former Arkansas governor and U.S. Sen. David Pryor, a Democrat who was one of the state’s most beloved political figures and remained active in public service in the state long after he left office, has died. He was 89. Pryor, who went undercover to investigate nursing homes while a congressman, […]

7 hours ago

Associated Press

A rabbi serving 30 years to life in his wife’s contract killing has died, prison officials say

TRENTON, N.J. (AP) — The New Jersey rabbi serving a decadeslong sentence in a 1994 murder-for-hire plot targeting his wife has died. Fred Neulander, 82, was pronounced dead shortly after 6 p.m. Wednesday at a hospital in Trenton after he was found unresponsive in his cell in the New Jersey State Prison infirmary, news outlets […]

7 hours ago

Sponsored Articles

...

DESERT INSTITUTE FOR SPINE CARE

Desert Institute for Spine Care is the place for weekend warriors to fix their back pain

Spring has sprung and nothing is better than March in Arizona. The temperatures are perfect and with the beautiful weather, Arizona has become a hotbed for hikers, runners, golfers, pickleball players and all types of weekend warriors.

...

Day & Night Air Conditioning, Heating and Plumbing

Day & Night is looking for the oldest AC in the Valley

Does your air conditioner make weird noises or a burning smell when it starts? If so, you may be due for an AC unit replacement.

(KTAR News Graphic)...

Boys & Girls Clubs

KTAR launches online holiday auction benefitting Boys & Girls Clubs of the Valley

KTAR is teaming up with The Boys & Girls Clubs of the Valley for a holiday auction benefitting thousands of Valley kids.

NYC’s AI chatbot was caught telling businesses to break the law. The city isn’t taking it down