AI Agents Are Coming for Your Computers
AI agents are no longer just passive tools—they're becoming active decision-makers on our desktops. From automating workflows to managing tasks, these digital companions are stepping into roles once reserved for human operators. These bots are growing in sophistication and could soon take greater control of our computers, reshaping how we interact with technology. It'll raise new questions about security, trust, and the balance between convenience and oversight.
Although not considered an agent by definition, Microsoft's Recall was one of the earliest examples of artificial intelligence taking more active control of a laptop. The feature was touted as the equivalent of an on-device search engine, periodically snapping snapshots of your screen to help you find the content you need. Massive outcry over the exposed security flaws led to Microsoft not only making the feature off by default but also postponing the release multiple times.
However, such criticism hasn't stopped others from toying with using AI agents to assume operational control of computers. In October, Anthropic released an upgraded Claude 3.5 Sonnet model capable of understanding and interacting with any desktop application. It can "imitate keystrokes, button clicks, and mouse gestures, essentially emulating a person sitting at a PC." At least one security expert warns that Anthropic's Computer Use functionality could be used to launch cyberattacks. I reached out to Anthropic to learn how it ensured Computer Use wouldn't face the same blowback as Microsoft's Recall feature but the company has yet to respond.
Motorola is another company that is experimenting with similar AI. Last month, it showcased AI capable of taking action on your behalf, opening the Uber app and requesting a ride. However, that example is a concept; Motorola states it won't ship soon.
Google is also working on a project codenamed Jarvis that has somewhat similar functionality. Google offered a preview of what's to come when it accidentally launched it to the Chrome store this week—it has since been removed. It would take over a web browser to accomplish tasks on a user's behalf. Google's Gemini AI would power it, and Project Jarvis could gather research, purchase a product, or even book a flight. The idea, like with Microsoft and Anthropic's AI, is to automate away our everyday assignments.
After all, companies constantly remind us that AI agents aim to handle mundane, repetitive tasks, freeing us to focus on more creative pursuits.
Large Action Models (LAMs) form a crucial part of these features' foundation. These AI systems interpret queries and respond by taking action. The most recognizable usage today is with the Rabbit R1 device. The hardware features a trainable AI agent that executes user requests. LAMs can be helpful across various applications, from the personal to the enterprise. We may see LAM-powered agents help us troubleshoot when our computers are broken, place an appointment on our calendar based on a recent email, or—like with what Google is doing—help book a flight.
From a business perspective, LAM agents can be used for customer service, marketing and sales, process automation, user interface and experience testing, and more. Salesforce is leveraging LAMs for its Agentforce platform. Shelby Heinecke, the company's Senior AI Research Manager, tells me LAMs are what lie ahead for companies pursuing AI innovation:
"Large Language Models are basically Jack of all Trades. They're great at everything. But as we go into this agent era, we need models that are still great at generating text, even different modalities. But we specifically need models that are experts at generating the next action. Absolutely, I think [LAMs] are the next phase."
While we will see more AI agents capable of controlling our devices, we must not embrace such innovation blindly. They bring with them a unique set of benefits and risks. We must examine the potential vulnerabilities such agents might have if run on our computer, smartphone, or Internet of Things device.
Another point to consider is user readiness and the varying acceptance levels for AI agents in personal and business contexts. While some welcome these agents to offload repetitive tasks, others remain wary of relinquishing control. People want to be aware of an agent's decision-making process. This highlights a crucial gap companies must address: making AI interactions transparent and allowing users to intervene when needed.
This Week’s AI News
🏭 AI Trends and Industry Impact
🤖 AI Models and Technologies
✏️ Generative AI and Content Creation
💰 Funding and Investments
☁️ Enterprise AI Solutions
⚙️ Hardware, Robotics, and Autonomous Systems
🔬 Science and Breakthroughs
💼 Business, Marketing, Media, and Consumer Applications
⚖️ Legal, Regulatory, and Ethical Issues
🔎 Opinions, Analysis, and Editorials
🎧 Podcasts
End Output
Thanks for reading. Be sure to subscribe so you don’t miss any future issues of this newsletter.
Did you miss any AI articles this week? Fret not; I’m curating the big stories in my Flipboard Magazine, “The AI Economy.”
Connect with me on LinkedIn and check out my blog to read more insights and thoughts on business and technology.
Do you have a story you think would be a great fit for “The AI Economy”? Awesome! Shoot me a message – I’m all ears!
Until next time, stay curious!