devdocs_to_llm_firecrawl.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "authorship_tag": "ABX9TyMK76muNMFRt0JTXy2fMPCy",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alexfazio/devdocs-to-llm/blob/main/devdocs_to_llm_firecrawl.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# DevDocs to LLM: turn any documentation into a GPT\n",
        "\n",
        "By Alex Fazio (https://twitter.com/alxfazio)\n",
        "\n",
        "Github repo: https://github.com/alexfazio/devdocs-to-llm\n",
        "\n",
        "This Jupyter notebook demonstrates how to use the Firecrawl API to crawl developer documentation, extract content, and process the information to be ready to be used in an assistant like ChatGPT.\n",
        "\n",
        "By the end of this notebook, you'll be able to:\n",
        "\n",
        "1. Set up the Firecrawl environment\n",
        "Crawl a website and generate a sitemap\n",
        "2. Extract content from crawled pages in Markdown\n",
        "3. Export the processed content to various platforms and platforms, including Rentry.co and Google Docs!\n",
        "\n",
        "This cookbook is designed for developers and data scientists who want to efficiently gather and analyze developer documentation at scale."
      ],
      "metadata": {
        "id": "yEkq-MhCH5bw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Requirements\n",
        "Before proceeding, ensure you have the following:\n",
        "\n",
        "- **Firecrawl API key**: Essential for accessing the Firecrawl service\n",
        "- Google Docs API credentials: (Optional) A JSON file named `client_secret_<...>.apps.googleusercontent.com.json` for Google Docs integration.\n",
        "\n",
        "Note: The Google Docs API credential is only required if you plan to export content to Google Docs. All other functionalities can be used without this optional component."
      ],
      "metadata": {
        "id": "puKp3vCs8Oex"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tested Documentation Sources\n",
        "\n",
        "| Status | Documentation Source | URL |\n",
        "|--------|----------------------|-----|\n",
        "| ✅ | CrewAI | https://docs.crewai.com/ |\n",
        "| ✅ | Brave Search API | https://api.search.brave.com/app/documentation/ |\n",
        "| ✅ | OpenAI | https://platform.openai.com/docs |\n",
        "| ✅ | FireCrawl | https://docs.firecrawl.dev/ |\n",
        "| ✅ | Anthropic | https://docs.anthropic.com/en/docs/ |\n",
        "| ✅ | LangChain | https://python.langchain.com/v0.2/docs |\n",
        "\n",
        "Note: A checkmark (✅) indicates successful testing with the corresponding documentation source."
      ],
      "metadata": {
        "id": "8othA_x4AvGK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "First, let's set up our environment with the necessary imports and initializations:\n",
        "\n",
        "This notebook requires the following libraries:\n",
        "\n",
        "- `firecrawl-py`: For web crawling and content extraction\n",
        "- `requests`: For making HTTP requests\n",
        "- `beautifulsoup4`: For parsing HTML content"
      ],
      "metadata": {
        "id": "Uag3A-M2Hs1L"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vAt71wSHVqgz"
      },
      "outputs": [],
      "source": [
        "%pip install firecrawl-py requests beautifulsoup4 --quiet\n",
        "print(\"---\")\n",
        "%pip show requests beautifulsoup4"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from firecrawl import FirecrawlApp"
      ],
      "metadata": {
        "id": "_lKdJlD2iFAI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, insert you Firecrawl API key `fc-...`"
      ],
      "metadata": {
        "id": "etLyo0ot3_ct"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from getpass import getpass\n",
        "fc_api_key = getpass(\"Enter your Firecrawl API key: \")\n",
        "assert fc_api_key != \"\", \"Error: fc_api_key should not be an empty string\""
      ],
      "metadata": {
        "id": "xnKnooAbTbHt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Crawling\n",
        "\n",
        "Now let's crawl some DevDocs pages to use in our examples.\n",
        "\n",
        "Enter the documentation URL in your browser's address bar to access the main documentation overview page, **rather than a specific section or page within the documentation**.\n",
        "\n",
        "This allows you to start the crawl of the main documentation page and navigate to specific topics as needed."
      ],
      "metadata": {
        "id": "dd0NwYwIAKbm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sub_url = \"https://docs.cursor.com/\" # @param {type:\"string\"}\n",
        "assert sub_url != \"\", \"Error: sub_url should not be an empty string\""
      ],
      "metadata": {
        "id": "kn8ar3iZgd-Y",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Attempt Site Map Display and Page Count Preview (Optional)\n",
        "\n",
        "The following cell will display the site map and preview the page count, providing only a general idea of the structure and number of pages. This process **will not** use the Firecrawl API or consume any usage tokens. Please be aware that this preview is not as accurate as crawling the entire website with Firecrawl."
      ],
      "metadata": {
        "id": "JTMP1UbTSQXs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "from urllib.parse import urljoin\n",
        "\n",
        "def get_site_map(url, base_url=None, depth=0, max_depth=3, visited=None, sitemap=None):\n",
        "    if visited is None:\n",
        "        visited = set()\n",
        "    if sitemap is None:\n",
        "        sitemap = []\n",
        "    if base_url is None:\n",
        "        base_url = url\n",
        "    if depth > max_depth or url in visited:\n",
        "        return\n",
        "\n",
        "    visited.add(url)\n",
        "    sitemap.append(url)\n",
        "\n",
        "    try:\n",
        "        response = requests.get(url)\n",
        "        soup = BeautifulSoup(response.text, 'html.parser')\n",
        "        print('| ' * depth + '+-- ' + url)\n",
        "\n",
        "        for link in soup.find_all('a', href=True):\n",
        "            href = link['href']\n",
        "            full_url = urljoin(url, href)\n",
        "            # Check if the full_url starts with the base_url\n",
        "            if full_url.startswith(base_url):\n",
        "                get_site_map(full_url, base_url, depth + 1, max_depth, visited, sitemap)\n",
        "    except Exception as e:\n",
        "        print(f\"Error processing {url}: {str(e)}\")\n",
        "\n",
        "    return sitemap\n",
        "\n",
        "def crawl_sub_url(sub_url, max_depth=3):\n",
        "    # Ensure the sub_url ends with a '/'\n",
        "    if not sub_url.endswith('/'):\n",
        "        sub_url += '/'\n",
        "\n",
        "    base_url = sub_url\n",
        "    print(f\"[REQUESTS CRAWL] Sitemap for {base_url}:\")\n",
        "    sitemap = get_site_map(base_url, base_url=base_url, max_depth=max_depth)\n",
        "    print(f\"\\n[REQUESTS CRAWL] Total pages crawled: {len(sitemap)}\")\n",
        "    return sitemap\n",
        "\n",
        "# Example usage\n",
        "preview_sitemap = crawl_sub_url(sub_url)\n",
        "print(\"\\nSite map:\")\n",
        "print(preview_sitemap)\n",
        "\n",
        "# Store the preview_sitemap in a global variable\n",
        "sitemap = preview_sitemap"
      ],
      "metadata": {
        "id": "qyNFRv_WSAa9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's initialize `FirecrawlApp()` a Python object that allows you to interact with the Firecrawl API. It essentially sets up a connection so you can use its methods (like crawl_url or scrape_url) to send requests to Firecrawl and get website data back."
      ],
      "metadata": {
        "id": "U-BkvAvMLUzL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "  app = FirecrawlApp(api_key=fc_api_key)"
      ],
      "metadata": {
        "id": "0FyyXaUPT6HB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To manage costs and control crawl scope, specify a maximum number of pages to crawl using the limit parameter below."
      ],
      "metadata": {
        "id": "S-NuxKlLL0qJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown Limit the crawl to a maximum of `limit` pages\n",
        "limit = 55 #@param {type:\"number\"}"
      ],
      "metadata": {
        "id": "koUvHtRhVdwi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Crawl Launch"
      ],
      "metadata": {
        "id": "lTFC4NkOqxUd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "from urllib.parse import urljoin, urlparse\n",
        "import random\n",
        "from firecrawl import FirecrawlApp\n",
        "import json\n",
        "\n",
        "def merged_crawl(start_url, limit, fc_api_key=fc_api_key):\n",
        "    def standard_crawl(start_url):\n",
        "        visited = set()\n",
        "        collected_urls = []\n",
        "\n",
        "        def normalize_url(url):\n",
        "            parsed_url = urlparse(url)\n",
        "            normalized_url = parsed_url._replace(fragment=\"\", query=\"\").geturl()\n",
        "            return normalized_url\n",
        "\n",
        "        def is_valid_subpage(url):\n",
        "            return url.startswith(start_url) and not url.startswith(start_url + '#')\n",
        "\n",
        "        def crawl(url):\n",
        "            if len(collected_urls) >= limit:\n",
        "                return\n",
        "\n",
        "            normalized_url = normalize_url(url)\n",
        "            if (normalized_url in visited) or (not is_valid_subpage(normalized_url)):\n",
        "                return\n",
        "            visited.add(normalized_url)\n",
        "            try:\n",
        "                response = requests.get(normalized_url)\n",
        "                if response.status_code != 200:\n",
        "                    return\n",
        "            except requests.exceptions.RequestException:\n",
        "                return\n",
        "\n",
        "            collected_urls.append(normalized_url)\n",
        "            print(f\"[REQUESTS CRAWLER] {len(collected_urls)}/{limit}: {normalized_url}\")\n",
        "\n",
        "            if len(collected_urls) >= limit:\n",
        "                return\n",
        "\n",
        "            soup = BeautifulSoup(response.content, \"html.parser\")\n",
        "            for link in soup.find_all(\"a\", href=True):\n",
        "                if len(collected_urls) >= limit:\n",
        "                    return\n",
        "                href = link['href']\n",
        "                full_url = urljoin(url, href)\n",
        "                normalized_full_url = normalize_url(full_url)\n",
        "                if is_valid_subpage(normalized_full_url):\n",
        "                    crawl(normalized_full_url)\n",
        "\n",
        "        crawl(start_url)\n",
        "        return collected_urls\n",
        "\n",
        "    def firecrawl_method(start_url, limit):\n",
        "        app = FirecrawlApp(api_key=fc_api_key)\n",
        "        crawl_result = app.crawl_url(\n",
        "            start_url,\n",
        "            {\n",
        "                'crawlerOptions': {\n",
        "                    'includePaths': ['/docs/', '/documentation/'],\n",
        "                    'limit': limit,\n",
        "                }\n",
        "            }\n",
        "        )\n",
        "\n",
        "        urls_to_scrape = []\n",
        "        for page_data in crawl_result:\n",
        "            metadata = page_data.get('metadata', {})\n",
        "            source_url = metadata.get('sourceURL')\n",
        "            if source_url:\n",
        "                urls_to_scrape.append(source_url)\n",
        "            links_on_page = metadata.get('linksOnPage')\n",
        "            if links_on_page:\n",
        "                urls_to_scrape.extend(links_on_page)\n",
        "\n",
        "        return urls_to_scrape\n",
        "\n",
        "    try:\n",
        "        print(\"[CRAWLER] Attempting standard crawl...\")\n",
        "        result = standard_crawl(start_url)\n",
        "        if not result:\n",
        "            raise Exception(\"[CRAWLER] Standard crawl returned no results\")\n",
        "        return result\n",
        "    except Exception as e:\n",
        "        print(f\"[CRAWLER] Standard crawl failed: {str(e)}\")\n",
        "        print(\"[CRAWLER] Falling back to firecrawl method...\")\n",
        "        return firecrawl_method(start_url, limit)\n",
        "\n",
        "# Usage\n",
        "sitemap = merged_crawl(sub_url, limit)\n",
        "print(\"---\")\n",
        "print(f\"[CRAWLER] Crawled URLs (sitemap): {sitemap}\")\n",
        "print(f\"[CRAWLER] Number of pages crawled: {len(sitemap)}\")"
      ],
      "metadata": {
        "id": "QtM_CYOM1q2V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Scraping\n",
        "\n",
        "With our sitemap in hand, we can now proceed to extract content from each page. Firecrawl's content extraction capabilities allow us to efficiently parse web pages and retrieve the main content as markdown, filtering out navigation menus, advertisements, and other non-essential elements."
      ],
      "metadata": {
        "id": "tygeo3dr0xi0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before we begin the extraction process, let's set some parameters:\n",
        "\n",
        "- `scrape_option`: Choose whether to scrape all pages or a specific number of pages.\n",
        "- `num_pages`: If scraping a specific number of pages, set the desired number here.\n",
        "\n",
        "Please set these parameters in the cell below."
      ],
      "metadata": {
        "id": "ce2N8HasNylT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Scraping Options"
      ],
      "metadata": {
        "id": "HZau_9Hsq1Jg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a dropdown for scrape options\n",
        "scrape_option = \"Specific number of pages\"  # @param [\"All pages\", \"Specific number of pages\"]\n",
        "\n",
        "# Create a numerical input for the specific number of pages\n",
        "num_pages = 55  # @param {type:\"number\"}\n",
        "\n",
        "# Initialize the num_pages variable depending on the scrape_option\n",
        "if scrape_option == \"Specific number of pages\":\n",
        "    # Check if the number of pages exceeds the length of the sitemap\n",
        "    num_pages = min(num_pages, len(sitemap))  # Set num_pages to the smaller of the two values\n",
        "else:\n",
        "    # If \"All pages\" is selected, set num_pages to the total length of the sitemap\n",
        "    num_pages = len(sitemap)\n",
        "\n",
        "# Now you can proceed with your scraping logic using num_pages\n",
        "print(f\"[SCRAPER] Number of pages to scrape: {num_pages}\")"
      ],
      "metadata": {
        "id": "CdIFd1fuA00_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, let's execute the content extraction process. Our script will:\n",
        "\n",
        "1. Iterate through the URLs in our sitemap\n",
        "2. Use Firecrawl's API to extract the main content from each page\n",
        "3. Store the extracted content in both XML and Markdown formats\n",
        "\n",
        "XML helps in structuring large documents before feeding them to an LLM for RAG or direct query."
      ],
      "metadata": {
        "id": "M7M9lAYzN9-k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is the edited table with only the scraping feature:\n",
        "\n",
        "| **Plan**     | **Max Pages Scraped/Minute** |\n",
        "|--------------|------------------------------|\n",
        "| Free         | 5                             |\n",
        "| Hobby        | 10                            |\n",
        "| Standard     | 50                            |\n",
        "| Growth       | 500                           |\n",
        "\n",
        "Set the times below, ensuring that you stay within the appropriate rate limits for your FireCrawl usage tier and avoid triggering any scraping restrictions.\n",
        "\n",
        "Check [FireCrawl docs](https://docs.firecrawl.dev/rate-limits) for further guidance on rate limits."
      ],
      "metadata": {
        "id": "4bvLnXVgfSQN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Rate Limiting and Retry Parameters\n"
      ],
      "metadata": {
        "id": "2fqRWptDfhPc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @markdown Set the rate limiting and retry parameters for the web scraper:\n",
        "\n",
        "pages_per_minute = 9 # @param {type:\"integer\"}\n",
        "# @markdown Number of pages that can be scraped per minute\n",
        "\n",
        "wait_time_between_chunks = 33 # @param {type:\"integer\"}\n",
        "# @markdown Waiting time (in seconds) between max chunks\n",
        "\n",
        "retry_attempts = 3 # @param {type:\"integer\"}\n",
        "# @markdown Number of times to retry failed scrapes\n",
        "\n",
        "print(f\"Pages per minute: {pages_per_minute}\")\n",
        "print(f\"Wait time between chunks: {wait_time_between_chunks} seconds\")\n",
        "print(f\"Number of retry attempts: {retry_attempts}\")\n",
        "\n",
        "# You can use these variables in your main scraping code"
      ],
      "metadata": {
        "id": "5pxAjeDma8c-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Check the status of the `sitemap` variable for debugging\n",
        "\n"
      ],
      "metadata": {
        "id": "hKKezwOKpQLq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(sitemap)"
      ],
      "metadata": {
        "id": "-0f_2R1DpPsV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Firecrawl Scrape Launch"
      ],
      "metadata": {
        "id": "2z23mapxqitK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "import math\n",
        "import requests\n",
        "\n",
        "# Initialize a file to store the XML content\n",
        "output_file = 'scraped_content.xml'\n",
        "\n",
        "# Initialize strings to store all the XML and markdown content\n",
        "all_xml = \"<document>\\n\"\n",
        "all_markdown = \"\"\n",
        "\n",
        "# Initialize a list to store failed scrape URLs\n",
        "failed_scrapes = []\n",
        "\n",
        "# Determine the number of pages to scrape\n",
        "pages_to_scrape = len(sitemap) if scrape_option == \"All pages\" else min(num_pages, len(sitemap))\n",
        "\n",
        "# Calculate the chunk size and total number of chunks\n",
        "chunk_size = pages_per_minute\n",
        "total_chunks = math.ceil(pages_to_scrape / chunk_size)\n",
        "\n",
        "def scrape_url(url, attempt=1):\n",
        "    print(f\"[FIRECRAWL SCRAPER] Attempting to scrape URL: {url} (Attempt {attempt})\")\n",
        "    start_time = time.time()\n",
        "    try:\n",
        "        response = app.scrape_url(url=url, params={\n",
        "            'onlyMainContent': True,\n",
        "            'waitFor': 5000,\n",
        "        })\n",
        "        markdown_content = response.get('markdown', '')\n",
        "        end_time = time.time()\n",
        "        scrape_time = end_time - start_time\n",
        "        print(f\"[FIRECRAWL SCRAPER] Successfully scraped URL: {url}\")\n",
        "        print(f\"[FIRECRAWL SCRAPER] Scrape time: {scrape_time:.2f} seconds\")\n",
        "        print(f\"[FIRECRAWL SCRAPER] Content length: {len(markdown_content)} characters\")\n",
        "        return markdown_content\n",
        "    except requests.exceptions.HTTPError as e:\n",
        "        end_time = time.time()\n",
        "        scrape_time = end_time - start_time\n",
        "        print(f\"[FIRECRAWL SCRAPER] Error scraping {url}: {str(e)}\")\n",
        "        print(f\"[FIRECRAWL SCRAPER] Scrape time (failed): {scrape_time:.2f} seconds\")\n",
        "        return None\n",
        "\n",
        "def process_scraped_content(url, markdown_content):\n",
        "    global all_xml, all_markdown\n",
        "    # Create XML structure for this page without indentation\n",
        "    page_xml = f\"<page>\\n<content>\\n{markdown_content}\\n</content>\\n</page>\\n\"\n",
        "\n",
        "    # Add the markdown content with a proper separator\n",
        "    if all_markdown:  # If it's not the first entry, add a separator\n",
        "        all_markdown += \"\\n\\n***\\n\\n\"\n",
        "    all_markdown += f\"# {url}\\n\\n{markdown_content}\"\n",
        "\n",
        "    # Append the page XML to the XML string\n",
        "    all_xml += page_xml\n",
        "    print(f\"[FIRECRAWL SCRAPER] Processed content for URL: {url}\")\n",
        "    print(f\"[FIRECRAWL SCRAPER] XML content length: {len(page_xml)} characters\")\n",
        "    return page_xml\n",
        "\n",
        "# Open the file in write mode\n",
        "with open(output_file, 'w') as file:\n",
        "    # Write the opening XML tag\n",
        "    file.write(\"<document>\\n\")\n",
        "\n",
        "    for chunk in range(total_chunks):\n",
        "        chunk_start = chunk * chunk_size\n",
        "        chunk_end = min((chunk + 1) * chunk_size, pages_to_scrape)\n",
        "        print(f\"[FIRECRAWL SCRAPER] Processing chunk {chunk+1}/{total_chunks} (URLs {chunk_start+1}-{chunk_end})\")\n",
        "\n",
        "        for i in range(chunk_start, chunk_end):\n",
        "            url = sitemap[i]\n",
        "            print(f\"[FIRECRAWL SCRAPER] Processing URL {i+1}/{pages_to_scrape}: {url}\")\n",
        "            markdown_content = scrape_url(url)\n",
        "\n",
        "            if markdown_content is not None:\n",
        "                page_xml = process_scraped_content(url, markdown_content)\n",
        "                # Write the page XML to the file\n",
        "                file.write(page_xml)\n",
        "                print(f\"[FIRECRAWL SCRAPER] Successfully wrote content for URL: {url}\")\n",
        "            else:\n",
        "                failed_scrapes.append(url)\n",
        "                print(f\"[FIRECRAWL SCRAPER] Failed to scrape URL: {url}\")\n",
        "\n",
        "        # Wait after each chunk, except for the last one\n",
        "        if chunk < total_chunks - 1:\n",
        "            print(f\"[FIRECRAWL SCRAPER] Chunk {chunk+1} completed. Waiting for {wait_time_between_chunks} seconds before the next chunk...\")\n",
        "            time.sleep(wait_time_between_chunks)\n",
        "\n",
        "    # Retry failed scrapes\n",
        "    for attempt in range(retry_attempts):\n",
        "        if not failed_scrapes:\n",
        "            break\n",
        "        print(f\"[FIRECRAWL SCRAPER] Retry attempt {attempt + 1} of {retry_attempts} for {len(failed_scrapes)} failed scrapes...\")\n",
        "        retry_urls = failed_scrapes.copy()\n",
        "        failed_scrapes = []\n",
        "        for url in retry_urls:\n",
        "            print(f\"[FIRECRAWL SCRAPER] Retrying URL: {url}\")\n",
        "            markdown_content = scrape_url(url, attempt=attempt+2)\n",
        "            if markdown_content is not None:\n",
        "                page_xml = process_scraped_content(url, markdown_content)\n",
        "                # Write the page XML to the file\n",
        "                file.write(page_xml)\n",
        "                print(f\"[FIRECRAWL SCRAPER] Successfully scraped and wrote content for retried URL: {url}\")\n",
        "            else:\n",
        "                failed_scrapes.append(url)\n",
        "                print(f\"[FIRECRAWL SCRAPER] Failed to scrape URL on retry: {url}\")\n",
        "\n",
        "        if failed_scrapes:\n",
        "            print(f\"[FIRECRAWL SCRAPER] Retry attempt {attempt + 1} completed. Waiting for {wait_time_between_chunks} seconds before the next retry attempt...\")\n",
        "            time.sleep(wait_time_between_chunks)\n",
        "\n",
        "    # Write the closing XML tag\n",
        "    file.write(\"</document>\")\n",
        "    print(\"[FIRECRAWL SCRAPER] Finished writing to XML file\")\n",
        "\n",
        "# Add the closing tag to the XML string variable\n",
        "all_xml += \"</document>\"\n",
        "\n",
        "# Now you can use the 'all_xml' and 'all_markdown' string variables as needed\n",
        "print(f\"[FIRECRAWL SCRAPER] Total characters in all_xml: {len(all_xml)}\")\n",
        "print(f\"[FIRECRAWL SCRAPER] Total characters in all_markdown: {len(all_markdown)}\")\n",
        "print(f\"[FIRECRAWL SCRAPER] Number of pages scraped: {pages_to_scrape}\")\n",
        "print(f\"[FIRECRAWL SCRAPER] Number of pages that failed to scrape after all retries: {len(failed_scrapes)}\")\n",
        "if failed_scrapes:\n",
        "    print(\"[FIRECRAWL SCRAPER] Failed URLs:\")\n",
        "    for url in failed_scrapes:\n",
        "        print(url)"
      ],
      "metadata": {
        "id": "5cGuX2YpMgMu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exporting Extracted Content\n",
        "\n",
        "After extracting the content, we have several options for exporting and storing the data. In this notebook, we'll demonstrate two export methods:\n",
        "\n",
        "1. Exporting to Rentry.co, a simple pastebin-like service\n",
        "2. Exporting to Google Docs"
      ],
      "metadata": {
        "id": "3vaXC_QG-689"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Export to Rentry.com\n",
        "\n",
        "# Import necessary libraries\n",
        "import os\n",
        "import requests\n",
        "import re\n",
        "\n",
        "# Function to strip HTML tags\n",
        "def strip_html_tags(text):\n",
        "    return re.sub('<[^<]+?>', '', text)\n",
        "\n",
        "# Function to create a new Rentry post\n",
        "def new_rentry(url, edit_code, text):\n",
        "    base_url = os.getenv('BASE_URL', 'https://rentry.co')\n",
        "    api_url = f\"{base_url}/api/new\"\n",
        "\n",
        "    # Get CSRF token\n",
        "    session = requests.Session()\n",
        "    response = session.get(base_url)\n",
        "    csrf_token = session.cookies.get('csrftoken')\n",
        "\n",
        "    # Prepare payload\n",
        "    payload = {\n",
        "        'csrfmiddlewaretoken': csrf_token,\n",
        "        'url': url,\n",
        "        'edit_code': edit_code,\n",
        "        'text': text\n",
        "    }\n",
        "\n",
        "    headers = {\n",
        "        \"Referer\": base_url,\n",
        "        \"X-CSRFToken\": csrf_token\n",
        "    }\n",
        "\n",
        "    # Make POST request\n",
        "    response = session.post(api_url, data=payload, headers=headers)\n",
        "    return response.json()\n",
        "\n",
        "# Function to export content to Rentry\n",
        "def export_to_rentry(content):\n",
        "    cleaned_content = strip_html_tags(content)\n",
        "\n",
        "    # Check if the content exceeds 200,000 characters\n",
        "    if len(cleaned_content) > 200000:\n",
        "        print(\"The content exceeds 200,000 characters. Please try using Google Docs instead due to the character limit.\")\n",
        "        return None, None\n",
        "\n",
        "    url = ''  # Leave empty for random URL\n",
        "    edit_code = ''  # Leave empty for random edit code\n",
        "    response = new_rentry(url, edit_code, cleaned_content)\n",
        "    if response['status'] == '200':\n",
        "        return response['url'], response['edit_code']\n",
        "    else:\n",
        "        return None, None\n",
        "\n",
        "# Main execution\n",
        "rentry_url, rentry_edit_code = export_to_rentry(all_xml)\n",
        "\n",
        "if rentry_url and rentry_edit_code:\n",
        "    print(f\"Rentry document created successfully!\")\n",
        "    print(f\"URL: {rentry_url}\")\n",
        "    print(f\"Edit code: {rentry_edit_code}\")\n",
        "elif len(strip_html_tags(all_xml)) > 200000:\n",
        "    # This message is already printed in the export_to_rentry function, but we'll keep it here for clarity\n",
        "    print(\"The content exceeds 200,000 characters. Please try using Google Docs instead due to the character limit.\")\n",
        "else:\n",
        "    print(\"Failed to create Rentry document.\")"
      ],
      "metadata": {
        "id": "QM2L5kAe2Q1r",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Export to Google Docs\n",
        "\n",
        "from google.colab import files\n",
        "from google.oauth2.credentials import Credentials\n",
        "from google_auth_oauthlib.flow import Flow\n",
        "from googleapiclient.discovery import build\n",
        "import json\n",
        "import io\n",
        "import getpass\n",
        "\n",
        "# Function to securely get input\n",
        "def secure_input(prompt):\n",
        "    return getpass.getpass(prompt)\n",
        "\n",
        "# Upload button for JSON credentials file\n",
        "print(\"Please upload your client secret JSON file.\")\n",
        "uploaded = files.upload()\n",
        "\n",
        "# Get the filename of the uploaded file\n",
        "filename = list(uploaded.keys())[0]\n",
        "\n",
        "# Read the contents of the uploaded file\n",
        "client_secret_json = io.StringIO(uploaded[filename].decode('utf-8')).read()\n",
        "\n",
        "# Parse the JSON content\n",
        "client_secret_data = json.loads(client_secret_json)\n",
        "\n",
        "# Create a Flow instance\n",
        "flow = Flow.from_client_config(\n",
        "    client_secret_data,\n",
        "    scopes=['https://www.googleapis.com/auth/documents'],\n",
        "    redirect_uri='urn:ietf:wg:oauth:2.0:oob')\n",
        "\n",
        "# Tell the user to go to the authorization URL.\n",
        "auth_url, _ = flow.authorization_url(prompt='consent')\n",
        "\n",
        "print(\"Please go to this URL to authorize the application:\")\n",
        "print(auth_url)\n",
        "\n",
        "# The user will get an authorization code. This line will wait for the user to input it securely.\n",
        "code = secure_input(\"Enter the authorization code: \")\n",
        "\n",
        "# Exchange the authorization code for credentials.\n",
        "flow.fetch_token(code=code)\n",
        "\n",
        "# Get the credentials\n",
        "creds = flow.credentials\n",
        "\n",
        "# Create a Docs API service object\n",
        "service = build('docs', 'v1', credentials=creds)\n",
        "\n",
        "# Create a new document\n",
        "document = service.documents().create(body={'title': 'My New Document'}).execute()\n",
        "print(f\"Created document with title: {document.get('title')}\")\n",
        "\n",
        "# Get the document ID\n",
        "document_id = document.get('documentId')\n",
        "\n",
        "# Prepare the content to be inserted\n",
        "requests = [\n",
        "    {\n",
        "        'insertText': {\n",
        "            'location': {\n",
        "                'index': 1,\n",
        "            },\n",
        "            'text': all_markdown\n",
        "        }\n",
        "    }\n",
        "]\n",
        "\n",
        "# Execute the request to insert the content\n",
        "result = service.documents().batchUpdate(documentId=document_id, body={'requests': requests}).execute()\n",
        "\n",
        "print(f\"Document content updated. You can find it at: https://docs.google.com/document/d/{document_id}/\")\n",
        "\n",
        "# Clear sensitive variables\n",
        "del client_secret_json, client_secret_data, code, creds"
      ],
      "metadata": {
        "id": "iEOCy5XymyhK",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}