Gemini app for Mac adds computer use to control desktop apps
Key Points:
- Google is enhancing the Gemini app on Mac with new computer use capabilities that enable it to interact with local desktop applications and websites by clicking and typing on the user's behalf to complete multi-step tasks.
- Unlike the previous Task functionality, which accessed local files via Connected Apps like Gmail and Google Calendar, the new feature operates in an isolated background window with a live preview, allowing users to continue using their screen and cursor freely.
- Users can control which apps Gemini interacts with, stop the process at any time, and must confirm sensitive actions such as logins and payments, ensuring safety and privacy.
- Practical applications include speeding up creative workflows, automating manual form entries, compiling live data into presentations, and batch-listing items for sale by integrating data from various local and web sources.
- The computer use feature is available starting today in US English for Google AI Plus, Pro, and Ultra subscribers, with new desktop app integrations like Blender and Autodesk Fusion added alongside existing ones such as Git and Obsidian.