Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add crewai integration docs #1484

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 209 additions & 0 deletions sources/platform/integrations/ai/crewai.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
---
title: 🤖🚀 CrewAI integration
sidebar_label: CrewAI
description: Learn how to build AI Agents with Apify and CrewAI 🤖🚀.
sidebar_position: 1
slug: /integrations/crewai
---

**Learn how to build AI Agents with Apify and CrewAI.**

---

## What is CrewAI

[CrewAI](https://www.crewai.com/) is an open-source Python framework designed to orchestrate autonomous, role-playing AI agents that collaborate as a "crew" to tackle complex tasks. It enables developers to define agents with specific roles, assign tasks, and integrate tools—like Apify Actors—for real-world data retrieval and automation.

:::note Explore CrewAI

For more in-depth details on CrewAI, check out its [official documentation](https://docs.crewai.com/).

:::

## How to use Apify with CrewAI

This guide demonstrates how to integrate Apify Actors with CrewAI by building a crew of agents that uses the [RAG Web Browser](https://apify.com/apify/rag-web-browser) Actor to search Google for TikTok profiles and the [TikTok Data Extractor](https://apify.com/clockworks/free-tiktok-scraper) Actor to extract and analyze data from the TikTok profiles.

### Prerequisites

- **Apify API token**: To use Apify Actors in CrewAI, you need an Apify API token. Learn how to obtain it in the [Apify documentation](https://docs.apify.com/platform/integrations/api).

- **OpenAI API key**: To power the agents in CrewAI, you need an OpenAI API key. Get one from the [OpenAI platform](https://platform.openai.com/account/api-keys).

- **Python packages**: Install the following Python packages:

```bash
pip install 'crewai[tools]' langchain-apify langchain-openai
```

### Building the TikTok profile search and analysis crew

First, import all required packages:

```python
import os
from crewai import Agent, Task, Crew
from crewai_tools import ApifyActorsTool
from langchain_openai import ChatOpenAI
```

Next, set the environment variables for the Apify API token and OpenAI API key:

```python
os.environ["OPENAI_API_KEY"] = "Your OpenAI API key"
os.environ["APIFY_API_TOKEN"] = "Your Apify API token"
```

Instantiate the LLM and Apify Actors tools:

```python
llm = ChatOpenAI(model="gpt-4o-mini")

browser_tool = ApifyActorsTool(actor_name="apify/rag-web-browser")
tiktok_tool = ApifyActorsTool(actor_name="clockworks/free-tiktok-scraper")
```

Define the agents with roles, goals, and tools:

```python
search_agent = Agent(
role="Web Search Specialist",
goal="Find the TikTok profile URL on the web",
backstory="Expert in web searching and data retrieval",
tools=[browser_tool],
llm=llm,
verbose=True
)

analysis_agent = Agent(
role="TikTok Profile Analyst",
goal="Extract and analyze data from the TikTok profile",
backstory="Skilled in social media data extraction and analysis",
tools=[tiktok_tool],
llm=llm,
verbose=True
)
```

Define the tasks for the agents:

```python
search_task = Task(
description="Search the web for the OpenAI TikTok profile URL.",
agent=search_agent,
expected_output="A URL linking to the OpenAI TikTok profile."
)

analysis_task = Task(
description="Extract data from the OpenAI TikTok profile URL and provide a profile summary and details about the latest post.",
agent=analysis_agent,
context=[search_task],
expected_output="A summary of the OpenAI TikTok profile including followers and likes, plus details about their most recent post."
)
```

Create and run the crew:

```python
crew = Crew(
agents=[search_agent, analysis_agent],
tasks=[search_task, analysis_task],
process="sequential"
)

result = crew.kickoff()
print(result)
```

:::note Search and analysis may take some time

The agent tasks may take some time as they search the web for the OpenAI TikTok profile and extract data from it.

:::

You will see the crew’s output in the console, showing the results of the search and analysis.

```text
Profile Summary:
- Username: OpenAI
- Profile URL: [OpenAI TikTok Profile](https://www.tiktok.com/@openai)
- Followers: 605,000
- Likes: 3,400,000
- Number of Videos: 152
- Verified: Yes
- Signature: low key research previews
- Bio Link: [OpenAI Website](https://openai.com/)

Latest Post Details:
- Post ID: 7474019216346287406
- Post Text: "@Adeline Mai is a photographer..."
- Creation Time: February 21, 2025
- Number of Likes: 863
- Number of Shares: 26
- Number of Comments: 33
- Number of Plays: 20,400
- Number of Collects: 88
- Music Used: Original Sound by OpenAI
- Web Video URL: [Watch Here](https://www.tiktok.com/@openai/video/7474019216346287406)
```

If you want to test the whole example, create a new file, `crewai_integration.py`, and copy the full code into it:

```python
import os
from crewai import Agent, Task, Crew
from crewai_tools import ApifyActorsTool
from langchain_openai import ChatOpenAI

os.environ["OPENAI_API_KEY"] = "Your OpenAI API key"
os.environ["APIFY_API_TOKEN"] = "Your Apify API token"

llm = ChatOpenAI(model="gpt-4o-mini")

browser_tool = ApifyActorsTool(actor_name="apify/rag-web-browser")
tiktok_tool = ApifyActorsTool(actor_name="clockworks/free-tiktok-scraper")

search_agent = Agent(
role="Web Search Specialist",
goal="Find the TikTok profile URL on the web",
backstory="Expert in web searching and data retrieval",
tools=[browser_tool],
llm=llm,
verbose=True
)

analysis_agent = Agent(
role="TikTok Profile Analyst",
goal="Extract and analyze data from the TikTok profile",
backstory="Skilled in social media data extraction and analysis",
tools=[tiktok_tool],
llm=llm,
verbose=True
)

search_task = Task(
description="Search the web for the OpenAI TikTok profile URL.",
agent=search_agent,
expected_output="A URL linking to the OpenAI TikTok profile."
)
analysis_task = Task(
description="Extract data from the OpenAI TikTok profile URL and provide a profile summary and details about the latest post.",
agent=analysis_agent,
context=[search_task],
expected_output="A summary of the OpenAI TikTok profile including followers and likes, plus details about their most recent post."
)

crew = Crew(
agents=[search_agent, analysis_agent],
tasks=[search_task, analysis_task],
process="sequential"
)

result = crew.kickoff()
print(result)
```

## Resources

- [Apify Actors](https://docs.apify.com/platform/actors)
- [CrewAI Documentation](https://docs.crewai.com/)
Comment on lines +206 to +207
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us add other resources, what is an AI Agent, How to build an AI Agent etc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added What are AI agents? and How to build an AI agent, looks much better with more links 👍