CrewAI Multi-Agent System for Writing Article from YouTube Videos

Introduction

In the era of an evolving digital world, automation redefines how humans approach everyday tasks, including research, content creation and much more. A new and exciting tool emerging in this field is CrewAI, which enables multiple AI agents to work together, solving complex problems with enhanced creativity and decision-making by prioritising collaborative intelligence.

In this article, we will dive deep into understanding the complete CrewAI multi-agent system and how this system operates while developing a project to write articles from YouTube videos. This hands-on project will help you gain valuable insights and leverage AI to generate and streamline content creation while maintaining a human touch. Let’s get started.

Project Objective

Picture this: You are part of Analytics Vidhya, a platform known for its extensive library of articles on Data Science, Machine Learning, AI, and more. While you’re likely familiar with these resources, Analytics Vidhya also produces a range of YouTube content: informative videos, podcasts with AI leaders, news updates, and more. Isn’t it a great idea to have articles summarising the content from these videos as well? However, the process of manually writing such articles is resource-intensive and complex. It requires a domain expert first to watch and validate the videos, followed by a content writer to structure and draft the articles. Imagine scaling this process for thousands of videos: communication breakdowns, delays, and potential errors become inevitable.

To overcome this, AI agents can streamline the entire workflow. These agents can efficiently analyze video content, validate it, and generate articles—fast, reliable, and with minimal human intervention. Let’s explore how this can be done using AI!

Also read: Build an AI Research Assistant Using CrewAI and Composio

What is an AI Agent?

An agent is a software system that understands its environment, processes data and takes actions to achieve its objectives. It doesn’t always need to be intelligent; for instance, a motion sensor light acts as an agent by automatically turning on when it detects movement.

On the other hand, an AI Agent is a more advanced form of an agent that incorporates Artificial intelligence specifically designed to automate tasks, enabling it to operate without direct human intervention. The AI Agent also functions by observing its environment, processing the acquired information, and executing actions or making decisions based on predefined objectives. Still, it also learns from its environment and improves its decision-making over time. AI Agents are engineered to replicate human-like decision-making and handle more complex tasks by learning from data, identifying patterns, and deriving insights or making predictions to accomplish assigned tasks efficiently.

You can perceive it as an intelligent assistant on which you don’t need to have constant supervision. For example, an AI agent can process large amounts of data, handle customer queries, do research tasks, write blogs, and create content. The most interesting thing about these agents is that they can work continuously without any breaks, analyze larger amounts of data than humans, and easily adapt to changing environments and situations based on the defined rules or ML algorithms they are designed with. By combining multiple AI agents, as we will do in this article, the system’s collective intelligence increases significantly, allowing it to handle even more complex tasks smoothly and effectively.

If you want to know about the Agentic AI design pattern then explore this: 4 Agentic AI design pattern

What is CrewAI?

CrewAI is a new, emerging, and innovative platform designed to enable the collaboration of multiple AI agents efficiently and seamlessly (CrewAI Multi-Agent System). With its framework, it enables users to use the power of collective intelligence, allowing AI agents to work together on complex tasks, which could be challenging and expensive for a single agent to handle. With CrewAI you can create your system with specialized agents, each with its own expertise, to perform various tasks. This collaborative approach enhances decision-making and creativity and also streamlines workflows, making it easier to achieve desired outcomes.

Also read: Building an Agentic Workflow with CrewAI and Groq

How are we using Agent for this Project?

The core concept of this project involves selecting a YouTube video, extracting all relevant content from it, processing that information, and writing a structured article based on the perceived content.

To create a multiple AI-Agent system, we generally divide it into three components: Agents, Tasks, and Tools.

• Agents: Intelligent entities designed to perform specific roles within the system. Each agent possesses unique capabilities, allowing for efficient handling of various aspects of the project.
• Tasks: Clearly defined objectives assigned to agents. These tasks ensure that each agent understands its responsibilities, optimizing workflow and productivity.
• Tools: Resources and technologies that support agents in completing their tasks. These may include software applications, data sources, APIs, and custom-designed solutions aimed at enhancing the agents’ efficiency, effectiveness, and workflow.

In this system, we will implement two agents: one will serve as a domain expert agent, tasked with researching the YouTube video and collecting content using an assigned tool, while the other agent will function as an article content writer, responsible for creating a structured article based on the information provided by the domain expert agent. This data flow from one agent to another is referred to as a sequential process. While we will focus on this approach, other processes, such as hierarchical processes, will be discussed in a later article.

Flow of the Project

Below is the complete flow for the project:

Also read: How to Build an AI Pair Programmer with CrewAI?

Let’s Start Writing Article from YouTube Videos

To build CrewAI Multi-Agent System, install Python on your system. It’s recommended that you use Anaconda or Miniconda to manage environments. 

I am developing this on a Linux system, using Visual Studio Code (VS Code) as my IDE. VS Code is lightweight and versatile, supports multiple programming languages, and its integrated terminal simplifies development. It’s also a preferred choice for many developers—and honestly, I think it just looks pretty cool. 

Now that you have all the necessary tools let’s dive into the article and get started!

Step 01: Setup the Environment and Requirements

In this step, we’ll start by creating the requirements.txt file to specify all the libraries necessary for the project. After that, we’ll set up the .env file, which will hold environment variables such as API keys and other confidential information. Finally, we will create a virtual environment. This is a best practice in development because it locks the versions of the libraries and technologies used, preventing future errors due to version mismatches.

1. Create a Folder
Create a folder anywhere on your system, for example, name it Multi_AI_Agent_Article_Project.

2. Open in VS Code
Open VS Code, then open the folder you just created in the VS Code Explorer.

3. Create requirements.txt
Inside the folder, create a file named requirements.txt and paste the following text inside it, then save the file:

crewai
crewai_tools
load_dotenv
langchain-huggingface

4. Generate OpenAI API Key
For this project, we’ll use an LLM from OpenAI, so you need to generate an OpenAI API Key. You can follow the link: How to get your own OpenAI API Key to get instructions. The cost of executing this project is around $0.04, making it quite inexpensive. If you already have an API key, feel free to proceed to the next step.

5. Create .env File
Create another file named .env and save your OpenAI API key in this format, then save the file:

OPENAI_API_KEY= paste_your_api_key_here

6. Create Virtual Environment
Open the terminal in VS Code, ensuring it is opened inside the Multi_AI_Agent_Article_Project folder. Then type the following command to create a virtual environment:

conda create -p venv python=3.10

This command will create a Conda virtual environment in the venv directory within the project folder instead of the default Conda directory. The -p flag specifies the path where the environment will be created (in this case, the venv folder) and the Python version is set to 3.10.

7. Activate Virtual Environment
Use this command to activate the virtual environment:

conda activate venv/

8. Install Required Libraries
To install all the necessary libraries from the requirements.txt file, run the following command:

pip install -r requirements.txt

Once all the steps above are complete, you’re ready to proceed with building the actual Multi AI Agent system!

Step 02: Create Tools

Create a file named tools.py and paste the code below, then save the file:

from crewai_tools import YoutubeChannelSearchTool

Code Explanation:

Here, we’re importing the YoutubeChannelSearchTool from CrewAI. This tool allows an AI agent to search and gather video data, titles, descriptions, and metadata from a specified YouTube channel, enabling efficient content research and analysis.

#Initialize the tool with a specific YouTube channel handle to target your search
yt_channel_tool = YoutubeChannelSearchTool(youtube_channel_handle='@SystemDesignSchool')

Code Explanation:

Next, we initialize an object for the tool and provide the YouTube handle for the channel SystemDesignSchool. We are using this channel because it has only 6 videos, which helps save time and resources when the system processes the videos.

You can learn more about this tool from the following link.

Step 03: Create Agents

Now, create another file named agents.py and paste the following code into it, then save the file:

from crewai import Agent, LLM
from tools import yt_channel_tool
import os

from dotenv import load_dotenv
load_dotenv() # Load environment variables from the .env file

#Setting up environment variables for OpenAI API access
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
os.environ["OPENAI_MODEL_NAME"] = "gpt-4-0125-preview"

Code Explanation:

Here, we import the necessary modules to create AI agents and configure environment variables. We use dotenv to load the OPENAI_API_KEY from the .env file we created earlier. We also set the model name to gpt-0125-preview.

# Creating the Domain Expert Agent, specialized in extracting relevant content from YouTube videos
domain_expert = Agent(
	role='Domain Expert content researcher from Youtube Videos',
	goal='To get all the relevant Content from the Youtube Video for the topic: {topic}',
	verbose=True,
	memory=True,
	backstory=(
    	'Domain Expert in understanding and grasping videos from Youtube which are about Artificial Intelligence,'
    	'Data Science, Machine Learning, Deep Learning, and Generative AI and providing content.'
	),
	tool = [yt_channel_tool],
	allow_delegation=True
)

# Creating the Content Writer Agent, responsible for writing a technical article based on YouTube video content
content_writer = Agent(
	role='Article Writer',
	goal='Write a structured and informative technical article about the Youtube Video topic: {topic}',
	verbose=True,
	memory=True,
	backstory=(
    	'Skilled at breaking down complex technical content into clear, engaging stories that educate readers, while presenting discoveries in an accessible and easy-to-understand format.'
	),
	tool = [yt_channel_tool],
	allow_delegation=False
)

Code Explanation:

Here, we create two agents. The Domain Expert agent is designed to extract relevant content from YouTube videos on topics like AI and Machine Learning using the yt_channel_tool. The Content Writer agent focuses on writing structured, technical articles based on the provided content.

Both agents have memory=True, allowing them to retain information and recall it when needed. verbose=True enables detailed logging to help understand the agent’s actions. allow_delegation=True for the Domain Expert agent allows task delegation, while it is set to False for the Content Writer agent, as the writer does not need to delegate tasks. Defining their roles, goals, and backstory ensures that the agents focus on their specific tasks efficiently.

You can learn more about agents from the following link.

Step 04: Create Tasks

Create another file named tasks.py and paste the following code into it, then save the file:

from crewai import Task
from tools import yt_channel_tool
from agents import domain_expert, content_writer

Code Explanation:

This code imports the Task class from CrewAI for creating tasks, the yt_channel_tool from the tools.py file, and the domain_expert and content_writer agents from the agents.py file, which will perform specific roles in the tasks.

# Domain Expert - ReseaAIrch Task
research_task = Task(
	description=(
    	"Get and identify the Youtube Video {topic}."
    	"Get a fully detailed, expert level content/information about the video from the channel."
	),
	expected_output='A detailed 6 paragraphs long structured report based on the {topic} of video content.',
	tools = [yt_channel_tool],
	agent=domain_expert
)


# Content Writer - Article Writing Task
article_writing_task = Task(
	description=(
    	"Get content/information from the Youtube Channel's Video on the topic: {topic}."
	),
	expected_output='Summarize the content form the Youtube Channel and write a detailed long structured Article from the Youtube Chanel Video based on the {topic}',
	tools = [yt_channel_tool],
	agent=content_writer,
	async_execution=False,
	output_file='new-article.md'
)

Code Explanation:

This section defines two tasks: the domain expert research task and the content writer’s article writing task. The research task focuses on identifying a YouTube video on the specified topic from the given channel and gathering detailed information to create a structured report. The article writing task summarizes the video’s content and creates a structured article based on that information. Both agents are assigned the yt_channel_tool to access YouTube data. In the article_writing_task, async_execution=False ensures that the task is completed fully before moving to the next step, preventing parallel execution of tasks, and the output will be saved in the “new-article.md” file.

You can learn more about tasks from the following link.

Step 05: Execution of the Code

Now, as the final step, create another file named crew.py and paste the following code into it, then save the file:

from crewai import Crew,Process
from agents import domain_expert, content_writer
from tasks import research_task, article_writing_task

Code Explanation:

This code imports the Crew and Process classes from crewai to manage multiple agents along with the defined agents and their tasks created in previous steps, enabling them to work together on the project.

# Creating a Crew instance with specified agents, tasks, and configuration
crew = Crew(
	agents=[domain_expert, content_writer],
	tasks=[research_task, article_writing_task],
	process=Process.sequential,
	memory=True,
	cache=True,
	max_rpm=100,
	share_crew=True
)

# Kicking Off the Task
result = crew.kickoff(inputs={'topic': 'File Storage VS Object Storage'})
print(result)

Code Explanation:

This code creates a Crew instance that combines the two agents, the Domain Expert and the Content Writer, with their specific tasks. We have defined the process as sequential, meaning the tasks will run one after the other. The memory and cache settings are both set to True to remember past information and improve efficiency. The max_rpm parameter sets the maximum number of requests the crew can handle per minute, ensuring efficient processing without overwhelming the system. The share_crew option enables the crew to be shared for collaboration or reuse, allowing other processes or users to access and utilize the same crew configuration for different tasks.

The crew is then initiated with the specified topic, which the agents will search for on the YouTube channel, perform their tasks, and print the result.

You can learn more from the following link.

Final Output

Screenshot of the output file “new-article.md”:

To learn more about What agents are, visit the following Article: AI Agent.

Conclusion

Using CrewAI Multi-Agent system significantly improves the efficiency and quality of content generation from YouTube videos. We optimize the workflow of the project by breaking it into agents, tools and tasks which significantly reduces the time and effort required to create high-quality articles. The Domain Expert agent gathers key information from the YouTube videos, while the Content Writer processes this information and writes well-organized articles.

This collaborative approach reduces the need for manual effort while ensuring consistent results that maintain the core message of the original content. As automation technologies progress, embracing these tools allows content creators to focus more on strategic and creative aspects of their work.

To stay ahead in this evolving field of Agentic AI, enroll in our Agentic AI Pioneer Program today!

Key Takeaways

• The Multi-Agent system divides tasks among specialized agents, streamlining content generation and reducing the time required to produce articles.
• The Domain Expert agent swiftly gathers valuable insights from YouTube videos, allowing for quick analysis and understanding of content.
• The Content Writer agent transforms extracted information into well-structured articles, ensuring clarity and coherence in the final output.
• Leveraging automation technologies like CrewAI enables content creators to focus more on strategic and creative aspects of their work, reducing repetitive tasks.

Frequently Asked Questions