8 Wild Demos of Google’s AI Agents

Introduction

Google has introduced new features at Google I/O 2024, leveraging advanced AI to streamline tasks and improve productivity. These include organizing receipts, returning online orders, simplifying email threads, and planning personalized trips. Google’s AI Agent integration into Google Search enhances research capabilities, while Gemini and Chrome enable secure browsing. Google Workspace introduces an AI teammate named Chip, helping teams track projects, organize information, and provide context in collaborative environments. These innovations demonstrate Google’s commitment to building intelligent systems that anticipate user needs and plan effectively.

1. An Email Agent to Continuously Organize All Receipts in Your Inbox into a Spreadsheet

The new feature introduced by Google email is an email agent designed to streamline the organization of receipts directly within your inbox. Here’s how Google’s AI Agent works:

Imagine you receive an email with a receipt attached, such as a hotel bill in PDF format. Upon opening the email, a suggestion appears in the side panel prompting you to organize and track your receipts. Clicking on this suggestion reveals a set of steps:

• Create a Drive Folder: The agent creates a dedicated Google Drive folder and automatically moves the receipt from the email, along with other similar receipts it finds in your inbox, into this folder.
• Extract Information to Spreadsheet: Next, it extracts relevant details from these receipts (like date, amount, and merchant) and compiles them into a new spreadsheet.

Upon confirming these actions, Google’s AI Agent, dubbed Gemini—executes the tasks seamlessly. But here’s where it gets even more powerful: Gemini offers the option to automate this entire workflow for all future emails containing receipts. This means that every time you receive a receipt, whether it’s from a hotel, a store, or any other service provider, Gemini will automatically categorize it into the designated Drive folder and update the spreadsheet accordingly.

This feature not only saves time by automating the tedious task of organizing receipts but also ensures it systematically manages your financial records in one place, making expense tracking and reporting more efficient.

2. An Email Agent to Return an Order

An email agent to return an order is Google’s latest feature. Let’s explore another use case in the shopping department. Imagine you’ve recently bought a pair of shoes online, but unfortunately, they don’t fit. Here’s how Google’s AI Agent, Gemini, can assist you in returning them:

• Search and Retrieve Receipt: Gemini searches your inbox for the email receipt related to your shoe purchase.
• Locate Order Number: It identifies and extracts the order number from the email, which is usually required for processing returns.
• Fill Out Return Form: Agent automatically fills out the return form with necessary details like order number and reason for return.
• Schedule Pickup (if applicable): Gemini can also help schedule a pickup for the return, ensuring that the process is as convenient as possible for you.

By automating these steps, Google’s AI Agent simplifies the often cumbersome process of returning online purchases. Instead of manually searching through emails, finding order details, and filling out forms, Gemini handles these tasks efficiently and effectively. This saves time and reduces hassle with online returns, making the experience smoother and more convenient.

3. An Email Agent to Sort Through Multiple Conversations, Files to Answer Questions

An email agent to sort through multiple conversations and files to answer questions is the latest feature introduced by Gmail. Here’s how Google’s AI Agent works:

• Overview: Imagine you have an email about sorting out a roof repair, but the thread is long and you haven’t read all the emails. With this new feature, you can simply tap the summarize option to get a concise overview.
• Mobile Card: A helpful mobile card overlays the email, providing a summary of all the key information you need to know. For example, it might show you quotes from different contractors like Jeff at Green Roofing.
• Q&A Feature: You can now ask questions directly within the mobile card. For instance, you can ask to compare roof repair bids by price and availability, and Gemini will fetch and compare the relevant details for you.
• Ease of Use: This feature makes it effortless to get quick answers on anything in your inbox without having to search through multiple emails. Whether you’re checking on your shoe delivery or the time doors open for a Knicks game, you can quickly get the information you need.
• Organization and Comparison: Gemini organizes details from different email threads into a neatly structured summary. This makes it easy to compare bids and make informed decisions, like choosing Green Roofing for your roof repair.

This feature saves time by eliminating manual search and comparison. It also simplifies complex tasks, enhancing Gmail mobile as a powerful inbox management tool.

4. A Search Agent with Multi-Step Trip Planning and Reasoning

A search agent with multi-step trip planning and reasoning is the latest feature introduced by Gemini. Here’s how Google’s AI Agent works:

• Prompt-Based Start: The process begins with a prompt where you specify your trip details—destination (Miami), interests (art for son, seafood for husband), and existing flight and hotel details in Gmail. This sets the stage for Gemini to begin its comprehensive planning.
• Data Gathering and Analysis: Gemini uses search and Maps extensions to create a dynamic travel graph, considering diverse interests and constraints, to create a personalized vacation plan.
• Dynamic UI Presentation: The personalized itinerary is then presented in Gemini’s new dynamic user interface. This interface is interactive and intuitive, making it easy to visualize and adjust your travel plans according to your preferences.
• Spatial Decision Making: Utilizing spatial data, Gemini makes informed decisions about your itinerary. For instance, it schedules activities close to your hotel upon arrival and selects highly rated seafood restaurants in proximity to your accommodations.
• Interactive Adjustments: You have the flexibility to interact with the itinerary. For example, you can adjust the start time for activities to better suit your family’s schedule. Gemini responds by updating the plan accordingly, optimizing your trip experience.
• Efficiency and Time Savings: This feature significantly reduces the time and effort required for trip planning. What would have taken hours of researching and coordinating logistics across multiple platforms is streamlined into a seamless process within Gmail.

This new trip planning experience is rolling out to Gemini Advanced, enhancing its capabilities. It makes it easier to plan and enjoy personalized trips directly from your inbox.

5. A Search Agent with Multi-Step Researching and Reasoning to Find a Pilates Class

Google Search has introduced a new capability: a search agent that performs multi-step researching and reasoning to find a pilates class. Here’s how Google’s AI Agent works:

• Introduction: Google is introducing multi-step reasoning in search, allowing users to conduct extensive research, such as finding the right studio for yoga and pilates, based on numerous factors.
• Search Example: Soon you’ll be able to ask Search to find the best yoga or pilates studios in Boston and show you details on their intro offers and the walking time from Beacon Hill. Google gets to work for you, finding the most relevant information and bringing it together into your AI overview.
• AI Overview: You get some studios with great ratings and their introductory offers. You can see the distance for each, such as one that’s just a 10-minute walk away.
• Visual Layout: Right below, you see where they’re located, laid out visually. All this information is obtained from just a single search.
• Gemini Model: Under the hood, Google’s custom Gemini model acts as your AI agent, using what they call multi-step reasoning. It breaks your bigger question down into all its parts and figures out which problems it needs to solve and in what order.
• Real-Time Information: Google utilizes its real-time information index, which includes over 250 million places, to provide the highest quality information, including ratings, reviews, and business hours.
• Efficiency: Research that might have taken you minutes or even hours, Google can now do on your behalf. This feature saves you time and effort by providing comprehensive information and recommendations in one place.

This new capability in Google Search utilizes advanced AI technologies to simplify the process of finding pilates classes or any other service. It automates research and presents comprehensive results directly to you.

6. A search agent with multi-step researching, reasoning & shopping to prepare a meal plan

A search agent with multi-step researching, reasoning & shopping to prepare a meal plan is a new capability introduced by Google Search. Here’s how it works:

• Introduction: Now you can ask Search to create a three-day meal plan for a group that’s easy to prepare. This feature integrates multi-step researching, reasoning, and shopping into a seamless experience.
• Meal Plan Creation: You receive a comprehensive meal plan with a wide range of recipes sourced from across the web. For example, you might find a recipe for overnight oats that looks particularly interesting. You can easily access the website to learn how to prepare them.
• Customization: If you want to add more vegetables, you can simply ask Search to swap in a vegetarian dish. Search customizes your meal plan according to your preferences.
• Export and Shopping: You have the option to export your meal plan or get the ingredients as a list by tapping a button. Looking ahead, you can even ask Google to add everything to your preferred shopping cart.
• Integration with Shopping: This integration allows you to seamlessly transition from planning to shopping, making it easier to prepare meals at home.
• Efficiency: This feature saves time and effort by automating the meal planning process and providing easy access to recipes and ingredients in one place.

This new capability in Google Search improves meal preparation by combining advanced AI with real-time information and shopping integration. It simplifies meal planning, customization, and shopping, providing a convenient solution for users looking to prepare meals at home.

7. A Browser Agent to Work Across Multiple External Websites to Do Tasks Like Updating Addresses Across Dozens of Websites

Gemini and Chrome have introduced a new capability: a browser agent that works across multiple external websites to perform tasks like updating addresses. Here’s how it works:

• Introduction: Imagine you’ve just moved to Chicago. Gemini and Chrome can work together to assist you in a variety of tasks to help you settle in. These tasks include organizing, reasoning, and synthesizing information on your behalf.
• Task Automation: One of the tasks you’ll need to do is update your new address across dozens of websites. Gemini can automate this process, ensuring that your address is updated consistently across all platforms.
• Multi-Website Navigation: Gemini can also help you explore the city and find local services, such as dry cleaners or dog walkers, by navigating across multiple external websites.
• Prompt-Based Interaction: Gemini will prompt you for more information when needed, ensuring that you always control the tasks being performed.
• Privacy and Security: It’s essential that these interactions are private, secure, and user-controlled. Gemini and Chrome are designed to prototype these experiences with privacy as a top priority, ensuring they work for everyone.
• Intelligent Systems: These use cases demonstrate the type of problems Gemini aims to solve by building intelligent systems that can think ahead, reason, and plan on your behalf.

Gemini and Chrome now use advanced AI and browsing technologies to streamline tasks across multiple websites. This enhances efficiency and convenience while maintaining privacy and security.

8. An AI Teammate That Lives Inside Google Workspace to Do Collaborative Tasks

Google has introduced a new capability: an AI teammate living inside Google Workspace to perform collaborative tasks. Here’s how it works:

• Introduction: AI teammate Chip, with its own account, can be assigned specific roles such as monitoring projects, organizing information, and providing context to the team.
• Configuration: Once configured, Chip is ready to assist the team. For instance, in Google Chat, Chip is in relevant chat rooms, providing quick updates or information upon request.
• Project Monitoring: When planning for events like I-O, Chip tracks project progress. For example, if you ask about the approval status of I-O storyboards, Chip searches and provides the latest information.
• Collective Memory: As Chip is added to more group chats, files, and email threads, it builds a collective memory of the team’s work. This allows Chip to provide informed responses and summaries based on the history of interactions.
• Example with Project Sapphire: In discussions about Project Sapphire, Chip can quickly search and synthesize information to answer questions like, “Are we on track for launch?” Chip not only provides a clear timeline and summary but also flags potential issues that the team should be aware of.
• Efficiency: Chip can complete tasks that could take hours for a human, such as creating a document to address an issue, in just a few minutes. This efficiency saves time and increases productivity for the entire team.
• Collaborative Capability: Chip operates in a group space where everyone can follow along and interact. Team members can ask Chip to perform tasks or provide information, enhancing collaboration and workflow.

This new capability in Google Workspace enhances team collaboration by leveraging AI to automate tasks and monitor projects. It provides valuable insights, thereby improving productivity and efficiency across the organization.

Conclusion

Google I/O 2024 showcased a range of innovative AI features designed to automate tasks such as receipt organizing and trip planning, enhancing user efficiency. Google’s AI Agents integrated into Google Search improve research capabilities and personalized recommendations. Gemini and Chrome enable seamless multi-website tasks and intelligent browsing. The introduction of Chip in Google Workspace exemplifies Google’s commitment to AI-driven collaboration, advancing technologies to streamline workflows and anticipate user needs.

You can refer this thread for more details.

Ayushi Trivedi

My name is Ayushi Trivedi. I am a B. Tech graduate. I have 3 years of experience working as an educator and content editor. I have worked with various python libraries, like numpy, pandas, seaborn, matplotlib, scikit, imblearn, linear regression and many more. I am also an author. My first book named #turning25 has been published and is available on amazon and flipkart. Here, I am technical content editor at Analytics Vidhya. I feel proud and happy to be AVian. I have a great team to work with. I love building the bridge between the technology and the learner.