AI that clicks for you: Microsoft’s research points to the future of GUI automation

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More A comprehensive new survey from Microsoft researchers and academic partners reveals that artificial intelligence agents powered by large language models (LLMs) are becoming increasingly capable of controlling graphical user interfaces (GUIs), potentially changing how humans interact with software. The technology essentially gives AI systems the ability to see and manipulate computer interfaces just like humans do — clicking buttons, filling out forms, and navigating between applications.

Rather than requiring users to learn complex software commands, these “GUI agents” can interpret natural language requests and automatically execute the necessary actions. “These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands,” the researchers write . “Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software.



” Think of it as having a highly skilled executive assistant who can operate any software program on your behalf. You simply tell the assistant what you want to accomplish, and they handle all the technical details of making it happen. The rise of enterprise AI assistants changes everything Major tech companies are already racing to incorporate these capabilities into their products.

Microsoft’s Power Automate uses LLMs to help users create automated workflows across applications. The company’s Copilot AI assistant can directly control software based on text commands. Anthropic’s Computer Use functionality for Claude enables the AI to interact with web interfaces and perform complex tasks.

Google is reportedly developing Project Jarvis , an AI system that would use Chrome browser to carry out web-based tasks like research, shopping, and travel booking, though this capability is still in development and hasn’t been publicly released. “The advent of Large Language Models, particularly multimodal models, has ushered in a new era of GUI automation,” the paper notes. “They have demonstrated exceptional capabilities in natural language understanding, code generation, task generalization, and visual processing.

” This represents a potential $68.9 billion market opportunity by 2028, according to analysts at BCC Research, as enterprises look to automate repetitive tasks and make their software more accessible to non-technical users. The market is projected to grow from $8.

3 billion in 2022 to this figure, at a compound annual growth rate (CAGR) of 43.9% during the forecast period. The enterprise impact: Challenges and opportunities in AI automation However, significant hurdles remain before the technology sees widespread enterprise adoption.

The researchers identify several key limitations, including privacy concerns when agents handle sensitive data, computational performance constraints, and the need for better safety and reliability guarantees. “While they are effective for predefined workflows, these methods lacked the flexibility and adaptability required for dynamic, real-world applications,” the paper states regarding earlier automation approaches. The research team provides a detailed roadmap for addressing these challenges, emphasizing the importance of developing more efficient models that can run locally on devices , implementing robust security measures, and creating standardized evaluation frameworks.

“By incorporating safeguards and customizable actions, these agents ensure efficiency and security when handling intricate commands,” the researchers note, highlighting recent progress in making the technology enterprise-ready. For enterprise technology leaders, the emergence of LLM-powered GUI agents represents both an opportunity and a strategic consideration. While the technology promises significant productivity gains through automation, organizations will need to carefully evaluate the security implications and infrastructure requirements of deploying these AI systems.

“The field of GUI agents is moving towards multi-agent architectures, multimodal capabilities, diverse action sets, and novel decision-making strategies,” the paper explains. “These innovations mark significant steps toward creating intelligent, adaptable agents capable of high performance across varied and dynamic environments.” Industry experts predict that by 2025, at least 60% of large enterprises will be piloting some form of GUI automation agents, potentially leading to massive efficiency gains but also raising important questions about data privacy and job displacement.

The comprehensive survey suggests we’re at an inflection point where conversational AI interfaces could fundamentally change how humans interact with software — though realizing this potential will require continued advances in both the underlying technology and enterprise deployment practices. “These developments are laying the groundwork for more versatile and powerful agents capable of handling complex, dynamic environments,” the researchers conclude, pointing to a future where AI assistants become an integral part of how we work with computers. Stay in the know! Get the latest news in your inbox daily By subscribing, you agree to VentureBeat's Terms of Service.

Thanks for subscribing. Check out more VB newsletters here . An error occured.

.