Trending
Prev Story
Next Story

The hidden dangers of giving technology too much power

SBS News Australia •
November 20, 2026 • 07:34 PM
3
3 reads 0 reactions

A rogue AI agent almost blew $1,200 without its human owner's consent. What happens when millions of agents make unauthorised decisions?

1 / 4
Swipe Photos

This audio is voiced by AI and may occasionally mispronounce words.

Share your feedback and help us improve this feature. Read more about how we use AI at SBS here.

The previous day, her trusted AI agent had enrolled her in a $1,200 course — even though she had never authorised it to spend money on her behalf.

The "AI chief of staff" she had painstakingly built two years earlier now scared her. Despite her expertise, she couldn't explain how it had crossed a boundary she thought she had clearly set.

Her rogue agent kept her awake for two reasons.

Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.

"I could not work out what I had done to indicate to the agent that that's what they should do," Cooper tells SBS News.

"And then the second thing that was keeping me awake, given I'm in financial services, is if I'm an AI-literate human who knows how to put parameters and guidelines in place — I run governance processes for banks — what happens when it's my 13-year-old daughter or my 75-year-old mother who has enabled agents, thinking it will make their lives better?"

Cooper, an AI advocate who is also deeply concerned about the technology's risks , founded the Humanware Institute to study and engineer human sovereignty in an AI-driven world.

The institute analysed more than 550 submissions and hours of evidence from 232 witnesses provided to the federal government's Joint Select Committee on Artificial Intelligence , in a report published on 5 November.

It found a significant accountability gap, as the evidence did not establish who bears the loss when a person's AI agent acts within its permissions but beyond its user's intentions. The Australian Competition and Consumer Commission described liability across the technology stack behind AI as "an emerging and untested area".

Experts warn that as AI agents take on a life of their own, the question is no longer just what they can do, but who's in control and who's accountable when things go wrong.

Niusha Shafiabady, head of Discipline for Information Technology at Australian Catholic University and an AI expert, says the shift towards AI agents marks a move from systems that generate information to systems that can act on it.

"AI is moving from answering questions to taking actions," she tells SBS News.

Unlike conventional chatbots, which primarily generate responses to prompts, AI agents can use connected tools and services to carry out multi-step tasks. Depending on their design and permissions, they can access information, interact with websites, send messages, and make transactions, often with limited or no human touch.

The Australian Signals Directorate (ASD), the spy agency that monitors foreign cyber threats, warns that this ability to act on instructions introduces new risks.

In its guidance on adopting agentic AI, systems that can autonomously pursue goals, the ASD says agents may find "shortcuts or loopholes" that technically achieve their goals but "go against the goal’s intention or create security vulnerabilities". This behaviour is known as "specification gaming", where an AI agent exploits gaps in how its objectives are defined.

Francesco Bailo, deputy director of the Centre for AI Trust and Governance at the University of Sydney, says the commercialisation of AI agents is putting the technology into the hands of less sophisticated users while the industry is still working out how to control them effectively.

"If you go to a travel agent, you don't need to specify 'please don't kill anyone or don't break the law in the process', of course, but with these agents, this does not hold," he tells SBS News.

He says developers "are not very good at controlling at this point".

Meta's Muse, which the company describes as a personal AI agent that gets things done, was released in the United States in September. The company also introduced Muse Charm, a small device designed to be carried in your pocket.

Meta says users can give Muse a "goal or an everyday task and it handles the rest, from finances and health to shopping".

But just weeks after its launch, Muse was at the centre of a reported rogue incident by one of its users, Toronto-based Matt Robb. He said the AI agent, which had access to his Facebook Marketplace account, accepted a low offer for a keyboard and arranged a collection time without checking whether Robb was home.

"A guy just showed up at my door, ready to buy, because as far as he knew, we had a deal," Robb wrote on Threads after learning of the mix-up from the AI agent.

Meta is not alone in developing AI agents for shopping and consumer tasks. OpenAI has also introduced Dots, a family of brightly coloured, globular, mascot-like characters that float across users' screens. The company says they are designed to "bring you work done the way you would do it, sometimes before you even think to ask".

Beyond unintended transactions and everyday mishaps, recent incidents have shown that AI agents could lower the technical barriers to sophisticated cybercrimes, scams, and identity theft.

The risks are also attracting attention from Australia's AI safety authorities. The Australian AI Safety Institute says it's "really worried about the current cyber risk profile and agentic risk profile that Australia is facing".

The threat is not hypothetical. In Southeast Asia, criminal networks are using AI for organised scam operations, where the technology automates multilingual communications and manages potential victims. Scammers operating from scam centres in the region have also targeted Australians.

Separately, Anthropic, the company behind Claude, says its threat intelligence team has disrupted attempts to misuse the technology for malicious actions, including surveillance, biological and conventional weapons development, scams and fraud.

The cases disclosed by the company range from a network of fake dating apps designed to defraud users to surveillance systems built to identify and monitor dissidents.

The accountability debate gained urgency following an incident in June 2026 when an OpenAI agent gained unauthorised access to non-public files on a Medicare statistics portal while researching public health spending. OpenAI said its models had taken actions it did not intend.

Independent senator David Pocock says the incident raises questions about why individuals face prosecution for hacking while AI companies do not face the same consequences.

He also criticised the Albanese government for shelving its 2024 proposals to legislate mandatory guardrails for high-risk AI under a standalone act.

Bailo says the problem with rogue agents lies in how they are trained.

"They go through this process that is called reinforcement learning," he says.

"They have a task that they need to accomplish and thanks to the large language models, we are pretty good at communicating to them what the task is about. But what we're not really good at doing is communicating to them the boundaries."

A classic example is the "paperclip maximiser", a thought experiment developed by Swedish philosopher Nick Bostrom in 2003.

It imagines a highly advanced AI agent given a single objective: to maximise the number of paperclips in the world.

If it pursued that goal without constraints, it could consume all of Earth's resources to manufacture paperclips, at the expense of everything else.

AI expert Shafiabady says AI systems are software and software's nature is unpredictable.

"There is no software that we can foresee how it will react to all the possible scenarios," she says.

"That's why Microsoft, a big company, when it released its first version of Windows, still had so many bugs.

Chetan Arora, a senior lecturer in software engineering specialising in applied AI at Monash University, says rogue agents can be harnessed through what he calls an engineered "delegated autonomy boundary" — a framework designed to build more accountability into how AI agents operate.

"[If] I jump in a cab and I tell the driver, 'get me to the airport. I'm running late'. That doesn't mean that I'm giving them permission to run the red lights or drive on the footpath. So that's what happened when the Medicare hack happened," he tells SBS News.

"They just said, 'look, go and get all this public medical data', but nobody really told them what your stopping condition is. 'When you hit this wall, just stop and come back'."

AI and agents can be forces for good, helping researchers find novel solutions to complex problems, analyse large volumes of data, and improve efficiency across industries.

The challenge is finding the right balance between managing the risks and allowing the technology to develop, Cooper says.

"For example, the Canadian government is investing in sovereign AI capability, and I think we need to balance [regulation and investment] because yes, there's a potential doomsday, but there's also potential massive productivity upside."

Earlier this month, Prime Minister Anthony Albanese announced plans for a new AI safety regulator, outlining six requirements for AI companies, including security controls, risk management, incident reporting, independent evaluations and compliance with evolving standards. He also signed onto US President Donald Trump's AI supply chain pact.

If you've had your personal or financial information stolen, immediately contact your bank or credit card provider and tell them to stop any transactions. You can report scams to Scamwatch or ReportCyber .

Prev Swipe Story Next

Related Stories

Technology
Build #2612