project-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. TipTap Editor: https://www.youtube.com/watch?v=vphOHH5j5LQ&t=621s&ab_channel=AndreMadarang   \n",
    "2. Notion: https://www.youtube.com/watch?v=0OaDyjB9Ib8&t=25797s&ab_channel=CodeWithAntonio   \n",
    "3. Vector Search: https://www.youtube.com/watch?v=cminnoNmZWY&ab_channel=WebDevCody\n",
    "4. File Storage: https://www.youtube.com/watch?v=27hMNWcsa-Y&t=6951s&ab_channel=WebDevCody   \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Links"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. https://www.electronjs.org/\n",
    "2. https://nextjs.org/docs\n",
    "3. https://ui.shadcn.com/docs\n",
    "4. https://docs.convex.dev/home\n",
    "5. https://excalidraw.com/\n",
    "6. https://clerk.com/docs/\n",
    "7. https://docs.trunk.io/\n",
    "8. https://undraw.co/\n",
    "\n",
    "ShadCN Drag-n-Drop Kanban Interface \n",
    "a. https://github.com/Georgegriff/react-dnd-kit-tailwind-shadcn-ui?tab=readme-ov-file\n",
    "\n",
    "ShadCN Chat Interface\n",
    "b. https://shadcn-chatbot-kit.vercel.app/docs/components/chat\n",
    "\n",
    "React Force Graph NLP Mapping \n",
    "c. https://github.com/vasturiano/react-force-graph\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Fetch latest changes from remote**\n",
    "git fetch origin\n",
    "\n",
    "**Switch to main branch**\n",
    "git switch main\n",
    "\n",
    "**Update main branch with remote changes**\n",
    "git pull origin main\n",
    "\n",
    "**Merge feature branch into main**\n",
    "git merge feature/embedding-pipeline\n",
    "\n",
    "**(If conflicts arise, resolve them, then add and commit)**\n",
    "git add path/to/conflicted-file.js\n",
    "git commit\n",
    "\n",
    "**Push the merged main branch to remote**\n",
    "git push origin main\n",
    "\n",
    "**(Optional) Delete the feature branch locally and remotely**\n",
    "git branch -d feature/embedding-pipeline\n",
    "git push origin --delete feature/embedding-pipeline"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project Structure: solomon-electron\n",
    "\n",
    "## Dependency & Prerequisites:  \n",
    "npm   \n",
    "node   \n",
    "dependency versions/ etc.   \n",
    "\n",
    "## Overview\n",
    "This document outlines the directory structure for the `solomon-electron` project, a hybrid Electron/Next.js application. It is designed to be easily readable and expandable as the project evolves.\n",
    "\n",
    "## Directory Structure\n",
    "\n",
    "- **/root**\n",
    "  - **solomon-electron**\n",
    "    - **assets** - _Static assets like images, fonts, etc._\n",
    "    - **electron**\n",
    "      - `main.js` - _The main Electron process file._\n",
    "    - **next** - _Next.js specific files and directories._\n",
    "      - **.next** - _Generated by Next.js, contains compiled files._\n",
    "      - **convex**\n",
    "        - **_generated**\n",
    "          - `api.d.ts` - _TypeScript definitions for APIs._\n",
    "          - `api.js` - _Generated API logic._\n",
    "          - `dataModel.d.ts` - _Data model TypeScript definitions._\n",
    "          - `server.d.ts` - _Server TypeScript definitions._\n",
    "          - `server.js` - _Generated server logic._\n",
    "        - `tsconfig.json` - _TypeScript configuration for Convex directory._\n",
    "      - **node_modules** - _Project dependencies (auto-generated)._\n",
    "      - **public** - _Static files accessible by the public._\n",
    "        - `next.svg`\n",
    "        - `vercel.svg`\n",
    "      - **src** - _Source code for the Next.js frontend._\n",
    "        - **app**\n",
    "          - **dashboard**\n",
    "            - **_components** - _Dashboard-specific components._\n",
    "              - `Canvas.tsx`\n",
    "              - `Chat.tsx`\n",
    "              - `DashboardHeader.tsx`\n",
    "              - `Sidebar.tsx`\n",
    "                - `button.tsx` - _Button component used in Sidebar._\n",
    "            - `DashboardLayout.tsx`\n",
    "            - `page.tsx`\n",
    "          - `favicon.ico`\n",
    "          - `globals.css` - _Global CSS styles._\n",
    "          - `layout.tsx`\n",
    "          - `page.tsx`\n",
    "        - **components** - _Reusable UI components._\n",
    "          - **ui**\n",
    "            - `button.tsx`\n",
    "          - `header.tsx`\n",
    "        - **lib**\n",
    "          - `utils.ts` - _Utility functions._\n",
    "      - `.env.local` - _Local environment variables._\n",
    "      - `.eslintrc.json` - _ESLint configuration._\n",
    "      - `.gitignore`\n",
    "      - `components.json` - _Components metadata._\n",
    "      - `next-env.d.ts`\n",
    "      - `next.config.mjs`\n",
    "      - `package-lock.json`\n",
    "      - `package.json`\n",
    "      - `postcss.config.js`\n",
    "      - `tailwind.config.ts`\n",
    "      - `tsconfig.json`\n",
    "    - **node_modules**\n",
    "    - `.gitignore`\n",
    "    - `LICENSE`\n",
    "    - `package-lock.json`\n",
    "    - `package.json`\n",
    "    - `project-notebook.ipynb` - _Jupyter notebook for project notes._\n",
    "    - `README.md` - _Project overview and setup instructions._\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# App Setup and Initialization Instructions [1-11]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. run 'npm init -y' which will create a 'package.json' file with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "{\n",
    "  \"name\": \"solomon-electron\",\n",
    "  \"version\": \"1.0.0\",\n",
    "  \"description\": \"\",\n",
    "  \"main\": \"index.js\",\n",
    "  \"scripts\": {\n",
    "    \"test\": \"echo \\\"Error: no test specified\\\" && exit 1\"\n",
    "  },\n",
    "  \"keywords\": [],\n",
    "  \"author\": \"\",\n",
    "  \"license\": \"ISC\"\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. npm install electron next electron-builder\n",
    "<br>\n",
    "\n",
    "3. mkdir next && cd next\n",
    "<br>\n",
    "\n",
    "4. npx create-next-app@latest ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "Need to install the following packages:\n",
    "create-next-app@14.1.4\n",
    "Ok to proceed? (y) y\n",
    "✔ Would you like to use TypeScript? … Yes\n",
    "✔ Would you like to use ESLint? … Yes\n",
    "✔ Would you like to use Tailwind CSS? … Yes\n",
    "✔ Would you like to use `src/` directory? … Yes\n",
    "✔ Would you like to use App Router? (recommended) … Yes\n",
    "✔ Would you like to customize the default import alias (@)? … No\n",
    "\n",
    "Creating a new Next.js app in /Users/matthewsimon/Documents/GitHub/acdc.electron/solomon-electron/next.\n",
    "\n",
    "Using npm.\n",
    "\n",
    "Initializing project with template: app-tw \n",
    "\n",
    "\n",
    "Installing dependencies:\n",
    "- react\n",
    "- react-dom\n",
    "- next\n",
    "\n",
    "Installing devDependencies:\n",
    "- typescript\n",
    "- @types/node\n",
    "- @types/react\n",
    "- @types/react-dom\n",
    "- autoprefixer\n",
    "- postcss\n",
    "- tailwindcss\n",
    "- eslint\n",
    "- eslint-config-next\n",
    "\n",
    "\n",
    "added 369 packages, and audited 370 packages in 10s\n",
    "\n",
    "136 packages are looking for funding\n",
    "  run `npm fund` for details\n",
    "\n",
    "found 0 vulnerabilities\n",
    "Success! Created next at /Users/matthewsimon/Documents/GitHub/acdc.electron/solomon-electron/next"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. cd /Users/matthewsimon/Documents/GitHub/acdc.electron/solomon-electron\n",
    "<br>\n",
    "\n",
    "6. mkdir ../electron && cd ../electron\n",
    "<br> \n",
    "\n",
    "7. Inside the electron folder, create a file named main.js. This will be your Electron main process file. \n",
    "<br>\n",
    "\n",
    "8. Add the following scripts to 'package.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "\"scripts\": {\n",
    "\t\"dev\": \"next dev\",\n",
    "\t\"build\": \"next build && next export && electron-builder\",\n",
    "\t\"start\": \"next start\",\n",
    "\t\"electron-dev\": \"ELECTRON_START_URL=http://localhost:3000 electron .\",\n",
    "\t\"electron\": \"electron .\"\n",
    "  }"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9. cd next && npm run dev\n",
    "<br>\n",
    "\n",
    "10. npm run electron-dev\n",
    "<br>\n",
    "\n",
    "11. Packaging your application: When you're ready to package your application for distribution, use the following command in the project root:   \n",
    "'npm run build'    \n",
    "<br>\n",
    "\n",
    "This command builds your Next.js application, exports it, and packages it with Electron Builder based on the configurations you specify in your package.json.\n",
    "\n",
    "Remember, this is a basic setup to get you started. Depending on your project needs, you may want to customize the Electron configuration, Next.js settings, and the build process further."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Development Startup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: We've recently added Trunk (see helpful links for Docs). This way, we're getting additional Linting details, and also we can use Trunk to import/ clone our directory. \n",
    "<br>\n",
    "\n",
    "This way, we may want to git clone our directory first, ensure Trunk Extension is installed in our Editor, then save the directory on our Local Machine, and then Clone the directory from Local using Trunk which will then begin to scan and process our Project. I'm sure there's an easier way to do this, but I've only just started using Trunk so I'll have to update this as we go along."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. git clone https://github.com/acdc-digital/solomon-electron\n",
    "2. cd /Users/matthewsimon/Documents/GitHub/acdc.electron/solomon-electron   \n",
    "3. npm i \n",
    "4. cd /Users/matthewsimon/Documents/Github/solomon-desktop/solomon-Desktop/next\n",
    "5. npm i\n",
    "6. npm run dev   \n",
    "7. npx convex dev\n",
    "8. cd /Users/matthewsimon/Documents/GitHub/acdc.electron/solomon-electron   \n",
    "9. npm run electron-dev"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application Development: Chat"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Directory \n",
    "\n",
    "**Dashboard Component**\n",
    "- /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/app/dashboard/_components/Chat.tsx\n",
    "\n",
    "**Chat Sub-Components**\n",
    "- /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat   \n",
    "<br> \n",
    "\n",
    "\t-- Header: /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat/   Chatheader.tsx   \n",
    "\t-- Layout: /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat/Chatlayout.tsx   \n",
    "\t-- Footer: /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat/Chatfooter.tsx   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Scripts "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// Chat.tsx\n",
    "// /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/app/dashboard/_components/Chat.tsx\n",
    "\n",
    "import React from 'react';\n",
    "import ChatHeader from '@/components/chat/Chatheader';\n",
    "import ChatLayout from '@/components/chat/Chatlayout';\n",
    "import ChatFooter from '@/components/chat/Chatfooter';\n",
    "\n",
    "const Chat: React.FC = () => {\n",
    "    return (\n",
    "        <div className='flex flex-col w-[33%] h-screen border-l'>\n",
    "            <ChatHeader title=\"Chat\" />\n",
    "            <ChatLayout />\n",
    "            <ChatFooter />\n",
    "            {/* Chat content goes here */}\n",
    "        </div>\n",
    "    );\n",
    "};\n",
    "\n",
    "export default Chat;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// ChatHeader.tsx\n",
    "// /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat/Chatheader.tsx \n",
    "\n",
    "import React from 'react';\n",
    "\n",
    "import { Avatar, AvatarImage, AvatarFallback } from '@/components/ui/avatar';\n",
    "\n",
    "interface ChatHeaderProps {\n",
    "\t// You can add more props as needed\n",
    "    title: string; \n",
    "\tavatarUrl: string; // URL for the avatar image\n",
    "    fallbackText: string; // Fallback text, e.g., user initials\n",
    "}\n",
    "\n",
    "const ChatHeader: React.FC<ChatHeaderProps> = ({ title, avatarUrl, fallbackText }) => {\n",
    "    return (\n",
    "        <div className=\"px-4 py-4 border-b flex items-center\">\n",
    "\t\t\t<Avatar className='border border-gray-300'>\n",
    "                <AvatarImage src={avatarUrl} alt=\"User Avatar\" />\n",
    "                <AvatarFallback>{fallbackText}</AvatarFallback>\n",
    "            </Avatar>\n",
    "            <h3 className=\"text-lg font-semibold ml-3\">{title}</h3>\n",
    "            {/* You can add more header content here, like buttons or status indicators */}\n",
    "        </div>\n",
    "    );\n",
    "};\n",
    "\n",
    "export default ChatHeader;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// ChatLayout.tsx\n",
    "// /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat/Chatlayout.tsx\n",
    "\n",
    "import React from 'react';\n",
    "\n",
    "const ChatLayout: React.FC = () => {\n",
    "    return (\n",
    "        <div className=\"flex-grow overflow-hidden\">\n",
    "            {/* This is where the messages will be displayed */}\n",
    "            {/* For now, it's just a placeholder */}\n",
    "            <div className=\"m-4\">Ongoing conversation...</div>\n",
    "        </div>\n",
    "    );\n",
    "};\n",
    "\n",
    "export default ChatLayout;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// ChatFooter.tsx\n",
    "// /Users/matthewsimon/Documents/GitHub/acdc.solomon-electron/solomon-electron/next/src/components/chat/Chatfooter.tsx\n",
    "\n",
    "'use client'\n",
    "\n",
    "import React, { useRef, useState } from 'react';\n",
    "import {\n",
    "  FileImage,\n",
    "  Mic,\n",
    "  Paperclip,\n",
    "  PlusCircle,\n",
    "  SendHorizontal,\n",
    "  Smile,\n",
    "  ThumbsDown,\n",
    "  ThumbsUp,\n",
    "} from 'lucide-react';\n",
    "import Link from 'next/link';\n",
    "import { buttonVariants } from '../ui/button';\n",
    "import { cn } from '@/lib/utils';\n",
    "import { AnimatePresence, motion } from 'framer-motion';\n",
    "import { Message, loggedInUserData } from '@/app/data'; // Adjust this import based on your actual data handling\n",
    "import { Textarea } from '../ui/textarea';\n",
    "\n",
    "\n",
    "interface ChatFooterProps {\n",
    "  sendMessage: (newMessage: Message) => void;\n",
    "  isMobile: boolean;\n",
    "}\n",
    "\n",
    "export const FooterIcons = [{ icon: FileImage }, { icon: Paperclip }];\n",
    "\n",
    "const ChatFooter: React.FC<ChatFooterProps> = ({ sendMessage, isMobile }) => {\n",
    "  const [message, setMessage] = useState('');\n",
    "  const inputRef = useRef<HTMLTextAreaElement>(null);\n",
    "\n",
    "  const handleInputChange = (event: React.ChangeEvent<HTMLTextAreaElement>) => {\n",
    "    setMessage(event.target.value);\n",
    "  };\n",
    "\n",
    "  const handleThumbsUp = () => {\n",
    "    const newMessage: Message = {\n",
    "      id: Date.now(), // Consider using a more robust method for generating IDs\n",
    "      name: loggedInUserData.name,\n",
    "      avatar: loggedInUserData.avatar,\n",
    "      message: '👍',\n",
    "    };\n",
    "    sendMessage(newMessage);\n",
    "    setMessage('');\n",
    "  };\n",
    "\n",
    "  const handleSend = () => {\n",
    "    if (message.trim()) {\n",
    "      const newMessage: Message = {\n",
    "        id: Date.now(), // Consider using a more robust method for generating IDs\n",
    "        name: loggedInUserData.name,\n",
    "        avatar: loggedInUserData.avatar,\n",
    "        message: message.trim(),\n",
    "      };\n",
    "      sendMessage(newMessage);\n",
    "      setMessage('');\n",
    "\n",
    "      if (inputRef.current) {\n",
    "        inputRef.current.focus();\n",
    "      }\n",
    "    }\n",
    "  };\n",
    "\n",
    "  const handleKeyPress = (event: React.KeyboardEvent<HTMLTextAreaElement>) => {\n",
    "    if (event.key === 'Enter' && !event.shiftKey) {\n",
    "      event.preventDefault();\n",
    "      handleSend();\n",
    "    }\n",
    "\n",
    "    if (event.key === 'Enter' && event.shiftKey) {\n",
    "      event.preventDefault();\n",
    "      setMessage((prev) => `${prev}\\n`);\n",
    "    }\n",
    "  };\n",
    "\n",
    "  return (\n",
    "    <div className=\"p-3 flex justify-between w-full items-center gap-3\">\n",
    "      {/* Additional functionality and icons can be added here */}\n",
    "      {/* Input field and send button logic */}\n",
    "      <AnimatePresence initial={false}>\n",
    "        <motion.div\n",
    "          key=\"input\"\n",
    "          className=\"flex-grow relative flex items-center\"\n",
    "          initial={{ opacity: 0 }}\n",
    "          animate={{ opacity: 1 }}\n",
    "          exit={{ opacity: 0 }}\n",
    "          transition={{ duration: 0.2 }}\n",
    "        >\n",
    "          <Textarea\n",
    "            placeholder=\"Type a message...\"\n",
    "            value={message}\n",
    "            onChange={handleInputChange}\n",
    "            onKeyDown={handleKeyPress}\n",
    "            ref={inputRef}\n",
    "            style={{ height: '20px' }}\n",
    "          ></Textarea>\n",
    "          <div className=\"flex items-center\">\n",
    "            {/* Emoji picker and other buttons */}\n",
    "          </div>\n",
    "        </motion.div>\n",
    "      </AnimatePresence>\n",
    "      {/* Handling of the send button appearance based on message content */}\n",
    "      {message.trim() ? (\n",
    "        <button\n",
    "          onClick={handleSend}\n",
    "          className=\"...\">\n",
    "          <SendHorizontal size={24} />\n",
    "        </button>\n",
    "      ) : (\n",
    "        <button\n",
    "          onClick={handleThumbsUp}\n",
    "          className=\"...\">\n",
    "          <ThumbsUp \n",
    "\t\t  \tclassName='mb-3'\n",
    "\t\t  \tsize={24} />\n",
    "\t\t  <ThumbsDown size={24} />\n",
    "        </button>\n",
    "      )}\n",
    "    </div>\n",
    "  );\n",
    "};\n",
    "\n",
    "export default ChatFooter;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application Development: Canvas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Dynamic Component Rendering in Next.js**\n",
    "\n",
    "Implementing dynamic rendering of components based on user interactions is a common requirement in modern web applications. In Next.js projects, leveraging React's built-in state management features like useState and useEffect hooks can simplify this process. This guide outlines a straightforward approach to dynamically render components within a Next.js application without the need for complex state management solutions.\n",
    "\n",
    "#### **Step 1: Define Your Components**\n",
    "\n",
    "Ensure you have the components for Admin, Notes, and Files. These components will be displayed based on user interactions.\n",
    "\n",
    "jsx\n",
    "Copy code\n",
    "```\n",
    "// In your /components/canvas directory\n",
    "\n",
    "// Admin.jsx\n",
    "export const Admin = () => {\n",
    "  return <div>Admin Content</div>;\n",
    "};\n",
    "\n",
    "// Notes.jsx\n",
    "export const Notes = () => {\n",
    "  return <div>Notes Content</div>;\n",
    "};\n",
    "\n",
    "// Files.jsx\n",
    "export const Files = () => {\n",
    "  return <div>Files Content</div>;\n",
    "};\n",
    "```\n",
    "\n",
    "#### **Step 2: Update CanvasHeader for Navigation Callbacks**\n",
    "\n",
    "Modify CanvasHeader to accept and execute callback functions upon link interactions, facilitating dynamic content rendering without direct navigation.\n",
    "\n",
    "jsx\n",
    "Copy code\n",
    "```\n",
    "// CanvasHeader.tsx\n",
    "const CanvasHeader = ({ title, onAdminClick, onNotesClick, onFilesClick }) => {\n",
    "  // Button components updated to call the provided callbacks on click\n",
    "};\n",
    "```\n",
    "\n",
    "#### **Step 3: Manage State in Canvas Component**\n",
    "\n",
    "Use the useState hook within your Canvas component to track and update the component to render based on user selections.\n",
    "\n",
    "jsx\n",
    "Copy code\n",
    "```\n",
    "// Canvas.tsx\n",
    "import React, { useState } from 'react';\n",
    "import { Admin, Notes, Files } from './path-to-your-components'; // Adjust accordingly\n",
    "\n",
    "const Canvas = () => {\n",
    "  const [activeComponent, setActiveComponent] = useState('Admin');\n",
    "\n",
    "  const renderComponent = () => {\n",
    "    switch(activeComponent) {\n",
    "      case 'Admin': return <Admin />;\n",
    "      case 'Notes': return <Notes />;\n",
    "      case 'Files': return <Files />;\n",
    "      default: return <Admin />;\n",
    "    }\n",
    "  };\n",
    "\n",
    "  return (\n",
    "    <div style={{ flexGrow: 1, backgroundColor: '#fff' }}>\n",
    "      <CanvasHeader\n",
    "        title=\"Canvas\"\n",
    "        onAdminClick={() => setActiveComponent('Admin')}\n",
    "        onNotesClick={() => setActiveComponent('Notes')}\n",
    "        onFilesClick={() => setActiveComponent('Files')}\n",
    "      />\n",
    "      <h2 className='m-2'>Project Canvas</h2>\n",
    "      {renderComponent()}\n",
    "    </div>\n",
    "  );\n",
    "};\n",
    "\n",
    "export default Canvas;\n",
    "```\n",
    "\n",
    "#### **Step 4: Update Button Handlers in CanvasHeader**\n",
    "\n",
    "Adjust the Button components in CanvasHeader to invoke the relevant callback functions when clicked, thus enabling dynamic component rendering.\n",
    "\n",
    "jsx\n",
    "Copy code\n",
    "```\n",
    "<Button className='mr-2' variant=\"outline\" onClick={onAdminClick}>\n",
    "  Admin\n",
    "</Button>\n",
    "```\n",
    "<br> \n",
    "\n",
    "Repeat the process for Notes and Files, using their respective callbacks. This strategy allows for dynamic rendering within the Canvas area based on interactions with CanvasHeader. Adjust component paths and names as needed to fit your project's architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application Development: Editor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// TipTapEditor.tsx\n",
    "// /Users/matthewsimon/Documents/Github/solomon-electron/next/src/components/canvas/(Projects)/_components/TipTapEditor.tsx\n",
    "\n",
    "import React, { useState } from 'react';\n",
    "import {\n",
    "  BoldIcon,\n",
    "  CodeIcon,\n",
    "  ItalicIcon,\n",
    "  Strikethrough,\n",
    "  UnderlineIcon,\n",
    "  HighlighterIcon,\n",
    "  LinkIcon,\n",
    "  ImageIcon,\n",
    "  TableIcon,\n",
    "  UndoIcon,\n",
    "  RedoIcon,\n",
    "  ListIcon,\n",
    "  ListOrderedIcon,\n",
    "  AlignLeftIcon,\n",
    "  AlignCenterIcon,\n",
    "  AlignRightIcon,\n",
    "  Heading1Icon,\n",
    "  Heading2Icon,\n",
    "  Heading3Icon,\n",
    "  PlusCircle, // Alternative for Zoom In\n",
    "  MinusCircle, // Alternative for Zoom Out\n",
    "} from 'lucide-react';\n",
    "import { useEditor, EditorContent } from '@tiptap/react';\n",
    "\n",
    "import Gapcursor from '@tiptap/extension-gapcursor'\n",
    "import StarterKit from '@tiptap/starter-kit';\n",
    "import Underline from '@tiptap/extension-underline';\n",
    "import Highlight from '@tiptap/extension-highlight';\n",
    "import Link from '@tiptap/extension-link';\n",
    "import Image from '@tiptap/extension-image';\n",
    "import Table from '@tiptap/extension-table';\n",
    "import TableRow from '@tiptap/extension-table-row';\n",
    "import TableCell from '@tiptap/extension-table-cell';\n",
    "import TableHeader from '@tiptap/extension-table-header';\n",
    "import TextAlign from '@tiptap/extension-text-align';\n",
    "import BulletList from '@tiptap/extension-bullet-list';\n",
    "import OrderedList from '@tiptap/extension-ordered-list';\n",
    "import ListItem from '@tiptap/extension-list-item';\n",
    "import Heading from '@tiptap/extension-heading';\n",
    "import Placeholder from '@tiptap/extension-placeholder';\n",
    "\n",
    "import PageVisualization from './PageVisualization'; // Ensure correct import path\n",
    "\n",
    "interface TipTapEditorProps {\n",
    "  onChange: (content: string) => void;\n",
    "  initialContent?: string;\n",
    "}\n",
    "\n",
    "const TipTapEditor: React.FC<TipTapEditorProps> = ({\n",
    "  onChange,\n",
    "  initialContent,\n",
    "}) => {\n",
    "  const [zoom, setZoom] = useState(1); // Initialize zoom state\n",
    "\n",
    "  const editor = useEditor({\n",
    "    editorProps: {\n",
    "      attributes: {\n",
    "        class: 'prose focus:outline-none',\n",
    "      },\n",
    "    },\n",
    "    extensions: [\n",
    "      StarterKit.configure({\n",
    "        bulletList: false,\n",
    "        orderedList: false,\n",
    "        listItem: false,\n",
    "      }),\n",
    "      Gapcursor,\n",
    "      BulletList,\n",
    "      OrderedList,\n",
    "      ListItem,\n",
    "      Underline,\n",
    "      Highlight,\n",
    "      Link.configure({ openOnClick: false }),\n",
    "      Image,\n",
    "      Table.configure({ resizable: true }),\n",
    "      TableRow,\n",
    "      TableHeader,\n",
    "      TableCell,\n",
    "      TextAlign.configure({\n",
    "        types: ['heading', 'paragraph', 'listItem'],\n",
    "      }),\n",
    "      Heading.configure({\n",
    "        levels: [1, 2, 3],\n",
    "      }),\n",
    "      Placeholder.configure({\n",
    "        placeholder: 'Start typing here...',\n",
    "      }),\n",
    "    ],\n",
    "    content: initialContent,\n",
    "    onUpdate: ({ editor }) => {\n",
    "      const content = editor.getHTML();\n",
    "      onChange(content);\n",
    "    },\n",
    "  });\n",
    "\n",
    "  React.useEffect(() => {\n",
    "    return () => {\n",
    "      editor?.destroy();\n",
    "    };\n",
    "  }, [editor]);\n",
    "\n",
    "  if (!editor) {\n",
    "    return null;\n",
    "  }\n",
    "\n",
    "  const addImage = () => {\n",
    "    const url = window.prompt('Enter image URL');\n",
    "    if (url) {\n",
    "      editor.chain().focus().setImage({ src: url }).run();\n",
    "    }\n",
    "  };\n",
    "\n",
    "  const setLink = () => {\n",
    "    const previousUrl = editor.getAttributes('link').href;\n",
    "    const url = window.prompt('Enter URL', previousUrl);\n",
    "    if (url === null) return;\n",
    "    if (url === '') {\n",
    "      editor.chain().focus().unsetLink().run();\n",
    "    } else {\n",
    "      editor.chain().focus().setLink({ href: url }).run();\n",
    "    }\n",
    "  };\n",
    "\n",
    "  const addTable = () => {\n",
    "    editor\n",
    "      .chain()\n",
    "      .focus()\n",
    "      .insertTable({ rows: 3, cols: 3, withHeaderRow: true })\n",
    "      .run();\n",
    "  };\n",
    "\n",
    "  const buttonClass = (isActive: boolean) =>\n",
    "    `p-1 rounded hover:bg-gray-200 ${isActive ? 'bg-gray-300' : ''}`;\n",
    "\n",
    "  // Zoom Handlers\n",
    "  const handleZoomIn = () => {\n",
    "    setZoom((prev) => Math.min(prev + 0.1, 3)); // Max zoom 300%\n",
    "  };\n",
    "\n",
    "  const handleZoomOut = () => {\n",
    "    setZoom((prev) => Math.max(prev - 0.1, 0.5)); // Min zoom 50%\n",
    "  };\n",
    "\n",
    "  return (\n",
    "    <div className=\"h-[calc(87vh-100px)] overflow-hidden\">\n",
    "      {/* Editor Toolbar */}\n",
    "      <div className=\"flex flex-wrap items-center border-b bg-gray-50 p-2 gap-x-1\">\n",
    "        {/* Undo and Redo */}\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().undo().run()}\n",
    "          className={buttonClass(false)}\n",
    "          aria-label=\"Undo\"\n",
    "        >\n",
    "          <UndoIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().redo().run()}\n",
    "          className={buttonClass(false)}\n",
    "          aria-label=\"Redo\"\n",
    "        >\n",
    "          <RedoIcon size={18} />\n",
    "        </button>\n",
    "\n",
    "        <div className=\"w-px h-6 bg-gray-300 mx-2\" />\n",
    "\n",
    "        {/* Text Styles */}\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleBold().run()}\n",
    "          className={buttonClass(editor.isActive('bold'))}\n",
    "          aria-label=\"Toggle Bold\"\n",
    "        >\n",
    "          <BoldIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleItalic().run()}\n",
    "          className={buttonClass(editor.isActive('italic'))}\n",
    "          aria-label=\"Toggle Italic\"\n",
    "        >\n",
    "          <ItalicIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleUnderline().run()}\n",
    "          className={buttonClass(editor.isActive('underline'))}\n",
    "          aria-label=\"Toggle Underline\"\n",
    "        >\n",
    "          <UnderlineIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleStrike().run()}\n",
    "          className={buttonClass(editor.isActive('strike'))}\n",
    "          aria-label=\"Toggle Strikethrough\"\n",
    "        >\n",
    "          <Strikethrough size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleHighlight().run()}\n",
    "          className={buttonClass(editor.isActive('highlight'))}\n",
    "          aria-label=\"Toggle Highlight\"\n",
    "        >\n",
    "          <HighlighterIcon size={18} />\n",
    "        </button>\n",
    "\n",
    "        <div className=\"w-px h-6 bg-gray-300 mx-2\" />\n",
    "\n",
    "        {/* Lists */}\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleBulletList().run()}\n",
    "          className={buttonClass(editor.isActive('bulletList'))}\n",
    "          aria-label=\"Toggle Bullet List\"\n",
    "        >\n",
    "          <ListIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleOrderedList().run()}\n",
    "          className={buttonClass(editor.isActive('orderedList'))}\n",
    "          aria-label=\"Toggle Ordered List\"\n",
    "        >\n",
    "          <ListOrderedIcon size={18} />\n",
    "        </button>\n",
    "\n",
    "        <div className=\"w-px h-6 bg-gray-300 mx-2\" />\n",
    "\n",
    "        {/* Text Alignment */}\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().setTextAlign('left').run()}\n",
    "          className={buttonClass(\n",
    "            editor.isActive({ textAlign: 'left' }) as boolean\n",
    "          )}\n",
    "          aria-label=\"Align Left\"\n",
    "        >\n",
    "          <AlignLeftIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().setTextAlign('center').run()}\n",
    "          className={buttonClass(\n",
    "            editor.isActive({ textAlign: 'center' }) as boolean\n",
    "          )}\n",
    "          aria-label=\"Align Center\"\n",
    "        >\n",
    "          <AlignCenterIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().setTextAlign('right').run()}\n",
    "          className={buttonClass(\n",
    "            editor.isActive({ textAlign: 'right' }) as boolean\n",
    "          )}\n",
    "          aria-label=\"Align Right\"\n",
    "        >\n",
    "          <AlignRightIcon size={18} />\n",
    "        </button>\n",
    "\n",
    "        <div className=\"w-px h-6 bg-gray-300 mx-2\" />\n",
    "\n",
    "        {/* Insert Elements */}\n",
    "        <button\n",
    "          onClick={setLink}\n",
    "          className={buttonClass(editor.isActive('link'))}\n",
    "          aria-label=\"Insert Link\"\n",
    "        >\n",
    "          <LinkIcon size={18} />\n",
    "        </button>\n",
    "        <button onClick={addImage} className={buttonClass(false)} aria-label=\"Insert Image\">\n",
    "          <ImageIcon size={18} />\n",
    "        </button>\n",
    "        <button onClick={addTable} className={buttonClass(false)} aria-label=\"Insert Table\">\n",
    "          <TableIcon size={18} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleCodeBlock().run()}\n",
    "          className={buttonClass(editor.isActive('codeBlock'))}\n",
    "          aria-label=\"Toggle Code Block\"\n",
    "        >\n",
    "          <CodeIcon size={18} />\n",
    "        </button>\n",
    "\n",
    "        <div className=\"w-px h-6 bg-gray-300 mx-2\" />\n",
    "\n",
    "        {/* Headings */}\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleHeading({ level: 1 }).run()}\n",
    "          className={buttonClass(editor.isActive('heading', { level: 1 }))}\n",
    "          aria-label=\"Heading 1\"\n",
    "        >\n",
    "          <Heading1Icon size={20} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleHeading({ level: 2 }).run()}\n",
    "          className={buttonClass(editor.isActive('heading', { level: 2 }))}\n",
    "          aria-label=\"Heading 2\"\n",
    "        >\n",
    "          <Heading2Icon size={20} />\n",
    "        </button>\n",
    "        <button\n",
    "          onClick={() => editor.chain().focus().toggleHeading({ level: 3 }).run()}\n",
    "          className={buttonClass(editor.isActive('heading', { level: 3 }))}\n",
    "          aria-label=\"Heading 3\"\n",
    "        >\n",
    "          <Heading3Icon size={20} />\n",
    "        </button>\n",
    "\n",
    "        <div className=\"w-px h-6 bg-gray-300 mx-2\" />\n",
    "\n",
    "        {/* Zoom Controls */}\n",
    "        <button\n",
    "          onClick={handleZoomOut}\n",
    "          className=\"p-1 rounded hover:bg-gray-200\"\n",
    "          aria-label=\"Zoom Out\"\n",
    "        >\n",
    "          <MinusCircle size={18} /> {/* Use MinusCircle as Zoom Out */}\n",
    "        </button>\n",
    "        <span className=\"px-1\">{Math.round(zoom * 100)}%</span>\n",
    "        <button\n",
    "          onClick={handleZoomIn}\n",
    "          className=\"p-1 rounded hover:bg-gray-200\"\n",
    "          aria-label=\"Zoom In\"\n",
    "        >\n",
    "          <PlusCircle size={18} /> {/* Use PlusCircle as Zoom In */}\n",
    "        </button>\n",
    "      </div>\n",
    "\n",
    "      {/* Editor Content with Page Visualization */}\n",
    "      <div className=\"flex-grow bg-gray-200 p-2 h-full\">\n",
    "        <PageVisualization pageSize=\"A4\" zoom={zoom}>\n",
    "          <EditorContent\n",
    "            editor={editor}\n",
    "            className=\"h-full w-full\"\n",
    "            style={{ caretColor: 'black' }}\n",
    "          />\n",
    "        </PageVisualization>\n",
    "      </div>\n",
    "    </div>\n",
    "  );\n",
    "};\n",
    "\n",
    "export default TipTapEditor;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application Development: User Authentication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Identify routing from Homepage to Dashboard, and identify the Layout isAuth and isLoading notes from the Dashbaord layout to ensure we know what's allowed, and what's prevented using the User Authentication.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convex Schema "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "import { defineSchema, defineTable } from \"convex/server\";\n",
    "import { v } from \"convex/values\";\n",
    "\n",
    "export default defineSchema({\n",
    "  projects: defineTable({\n",
    "    title: v.string(),\n",
    "    userId: v.string(),\n",
    "    isArchived: v.boolean(),\n",
    "    parentProject: v.optional(v.id(\"projects\")),\n",
    "    content: v.optional(v.string()),\n",
    "    icon: v.optional(v.string()),\n",
    "    isPublished: v.boolean(),\n",
    "  })\n",
    "  .index(\"by_user\", [\"userId\"])\n",
    "  .index(\"by_user_parent\", [\"userId\", \"parentProject\"])\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [RAG] Document Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ***RAG Pipeline 001 - Conceptual***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "{/*\n",
    "// Generate Embeddings Function\n",
    "async function generateEmbeddings(chunks: string[], ctx: MutationCtx): Promise<number[][]> {\n",
    "\t// Initialize OpenAI client\n",
    "\tconst openai = new OpenAI();\n",
    "\n",
    "\ttry {\n",
    "\t  const response = await openai.embeddings.create({\n",
    "\t\tmodel: \"text-embedding-ada-002\",\n",
    "\t\tinput: chunks,\n",
    "\t  });\n",
    "\n",
    "\t  // Extract embeddings from the response\n",
    "\t  const embeddings = response.data.map((item) => item.embedding);\n",
    "\t  return embeddings;\n",
    "\t} catch (error) {\n",
    "\t  console.error(\"Error generating embeddings:\", error);\n",
    "\t  throw new Error(\"Failed to generate embeddings.\");\n",
    "\t}\n",
    "  }\n",
    "// Get Embedding Function\n",
    "async function getEmbedding(text: string, ctx: MutationCtx): Promise<number[]> {\n",
    "\t// Initialize OpenAI client\n",
    "\tconst openai = new OpenAI();\n",
    "\n",
    "\ttry {\n",
    "\t  const response = await openai.embeddings.create({\n",
    "\t\tmodel: \"text-embedding-ada-002\",\n",
    "\t\tinput: text,\n",
    "\t  });\n",
    "\n",
    "\t  // Extract the embedding from the response\n",
    "\t  const embedding = response.data[0].embedding;\n",
    "\t  return embedding;\n",
    "\t} catch (error) {\n",
    "\t  console.error(\"Error generating embedding:\", error);\n",
    "\t  throw new Error(\"Failed to generate embedding.\");\n",
    "\t}\n",
    "  }\n",
    "\n",
    "// Extract Text Function using pdf-parse\n",
    "async function extractTextFromBuffer(buffer: Buffer): Promise<string> {\n",
    "\ttry {\n",
    "\t  const data = await pdfParse(buffer);\n",
    "\t  return data.text;\n",
    "\t} catch (error) {\n",
    "\t  console.error('Error extracting text from buffer:', error);\n",
    "\t  throw new Error('Failed to extract text from the PDF file.');\n",
    "\t}\n",
    "  }\n",
    "\n",
    "// Chunk Text Function\n",
    "function chunkText(text: string, maxChunkSize = 1000): string[] {\n",
    "\tconst sentences = text.match(/[^\\.!\\?]+[\\.!\\?]+/g) || [];\n",
    "\tconst chunks: string[] = [];\n",
    "\tlet chunk = \"\";\n",
    "\n",
    "\tfor (const sentence of sentences) {\n",
    "\t  if (chunk.length + sentence.length > maxChunkSize) {\n",
    "\t\tchunks.push(chunk.trim());\n",
    "\t\tchunk = \"\";\n",
    "\t  }\n",
    "\t  chunk += sentence + \" \";\n",
    "\t}\n",
    "\tif (chunk) {\n",
    "\t  chunks.push(chunk.trim());\n",
    "\t}\n",
    "\treturn chunks;\n",
    "  }\n",
    "\n",
    "// Store Chunks and Embeddings Function\n",
    "async function storeChunksAndEmbeddings(\n",
    "\tctx: MutationCtx,\n",
    "\tdocumentId: Id<\"projects\">,\n",
    "\tchunks: string[],\n",
    "\tembeddings: number[][]\n",
    "  ): Promise<void> {\n",
    "\tawait ctx.db.patch(documentId, {\n",
    "\t  documentEmbeddings: embeddings,\n",
    "\t  documentChunks: chunks, // Include this if you added documentChunks to the schema\n",
    "\t});\n",
    "  }\n",
    "\n",
    "export const processDocument = mutation({\n",
    "\targs: {\n",
    "\t  documentId: v.id(\"projects\"),\n",
    "\t},\n",
    "\thandler: async (ctx, args) => {\n",
    "\t  const document = await ctx.db.get(args.documentId);\n",
    "\t  if (!document) {\n",
    "\t\tthrow new Error(\"Document not found.\");\n",
    "\t  }\n",
    "\n",
    "\t  // Generate a download URL for the file\n",
    "\t  const url = await ctx.storage.getUrl(document.fileId);\n",
    "\t  if (!url) {\n",
    "\t\tthrow new Error(\"Failed to get URL for the document's file.\");\n",
    "\t  }\n",
    "\n",
    "\t  // Fetch the file content\n",
    "\t  const response = await fetch(url);\n",
    "\t  const arrayBuffer = await response.arrayBuffer();\n",
    "\t  const buffer = Buffer.from(arrayBuffer);\n",
    "\n",
    "\t  // Extract text from the file (e.g., PDF)\n",
    "\t  const textContent = await extractTextFromBuffer(buffer);\n",
    "\n",
    "\t  // Proceed to chunking and embedding\n",
    "\t  const chunks = chunkText(textContent);\n",
    "\t  const embeddings = await generateEmbeddings(chunks, ctx);\n",
    "\n",
    "\t  // Store chunks and embeddings in Convex\n",
    "\t  await storeChunksAndEmbeddings(ctx, document._id, chunks, embeddings);\n",
    "\t},\n",
    "  });\n",
    "  */}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ***Original***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// /Users/matthewsimon/Documents/Github/solomon-electron/next/src/app/api/parse-pdf/route.ts\n",
    "// /src/app/api/parse-pdf/route.ts\n",
    "\n",
    "import { NextResponse } from 'next/server';\n",
    "import convex from '@/lib/convexClient';\n",
    "import { PDFLoader } from \"@langchain/community/document_loaders/fs/pdf\";\n",
    "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
    "import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
    "\n",
    "import fs from 'fs';\n",
    "import path from 'path';\n",
    "import fetch from 'node-fetch'; // Ensure node-fetch is installed\n",
    "\n",
    "export const runtime = \"nodejs\";\n",
    "\n",
    "export async function POST(request: Request) {\n",
    "  let tempFilePath = '';\n",
    "\n",
    "  try {\n",
    "    // **Step 1: Parse the Incoming JSON Body**\n",
    "    const { documentId, fileId } = await request.json();\n",
    "    console.log('Received POST request with:', { documentId, fileId });\n",
    "\n",
    "    // **Step 2: Validate Input**\n",
    "    if (!documentId || !fileId) {\n",
    "      console.error('Validation failed: Missing documentId or fileId');\n",
    "      return NextResponse.json(\n",
    "        { error: 'documentId and fileId are required' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // **Step 3: Validate fileId**\n",
    "    if (typeof fileId !== 'string' || fileId.trim() === '') {\n",
    "      console.error('Invalid fileId provided.');\n",
    "      return NextResponse.json(\n",
    "        { error: 'Invalid fileId provided.' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // **Step 4: Invoke the Convex Mutation to Get the PDF URL**\n",
    "    console.log('Invoking Convex mutation: projects:getFileUrl');\n",
    "    const response = await convex.mutation('projects:getFileUrl', { fileId });\n",
    "\n",
    "    if (!response || !response.url) {\n",
    "      console.error('No URL returned for PDF');\n",
    "      return NextResponse.json(\n",
    "        { error: 'No URL returned for PDF' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    console.log('PDF URL:', response.url);\n",
    "\n",
    "    // **Step 5: Fetch the PDF from the URL**\n",
    "    console.log('Fetching the PDF from the URL');\n",
    "    const pdfResponse = await fetch(response.url);\n",
    "\n",
    "    if (!pdfResponse.ok) {\n",
    "      console.error('Failed to fetch PDF:', pdfResponse.statusText);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Failed to fetch PDF' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 10, // Update progress after fetching\n",
    "    });\n",
    "\n",
    "    // **Step 6: Parse the PDF to Extract Text**\n",
    "    console.log('Parsing the PDF to extract text');\n",
    "\n",
    "    // Download and save the PDF to a temporary file\n",
    "    const tempDir = path.join(process.cwd(), 'temp');\n",
    "    if (!fs.existsSync(tempDir)) {\n",
    "      fs.mkdirSync(tempDir);\n",
    "    }\n",
    "    tempFilePath = path.join(tempDir, `${fileId}.pdf`);\n",
    "\n",
    "    const arrayBuffer = await pdfResponse.arrayBuffer();\n",
    "    const buffer = Buffer.from(arrayBuffer);\n",
    "    fs.writeFileSync(tempFilePath, buffer);\n",
    "    console.log('PDF saved to:', tempFilePath);\n",
    "\n",
    "    // Parse the PDF using PDFLoader\n",
    "    const loader = new PDFLoader(tempFilePath);\n",
    "    const docs = await loader.load();\n",
    "\n",
    "    if (!docs.length) {\n",
    "      throw new Error(\"No content extracted from the document.\");\n",
    "    }\n",
    "    console.log(\"Loaded docs:\", docs.length);\n",
    "\n",
    "    // Extract text from docs\n",
    "    const extractedText = docs.map(doc => doc.pageContent).join('\\n');\n",
    "    console.log('Extracted Text:', extractedText);\n",
    "\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 30, // Update progress after parsing\n",
    "    });\n",
    "\n",
    "    // **Step 7: Split the Document into Chunks**\n",
    "    console.log('Splitting the document into chunks');\n",
    "\n",
    "    const textSplitter = new RecursiveCharacterTextSplitter({\n",
    "      chunkSize: 1000,    // Adjust based on your needs\n",
    "      chunkOverlap: 200,  // Adjust based on your needs\n",
    "      separators: [\"\\n\\n\", \"\\n\", \" \", \"\"],\n",
    "    });\n",
    "\n",
    "    const splitDocs = await textSplitter.splitDocuments(docs);\n",
    "    let docChunks = splitDocs.map((doc) => doc.pageContent);\n",
    "\n",
    "    // Add numbering to each chunk\n",
    "    docChunks = docChunks.map((chunk, index) => `Chunk ${index + 1}:\\n${chunk}`);\n",
    "\n",
    "    console.log('Number of Chunks:', docChunks.length);\n",
    "    console.log('Preview of first chunk:', docChunks[0]);\n",
    "\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 50, // Update progress after chunking\n",
    "    });\n",
    "\n",
    "    // **Step 8: Retrieve parentProjectId from documentId**\n",
    "    console.log('Retrieving parentProjectId from documentId:', documentId);\n",
    "    const parentProjectId = await convex.query('projects:getParentProjectId', { documentId });\n",
    "\n",
    "    if (!parentProjectId) {\n",
    "      console.error(`No parentProjectId found for documentId: ${documentId}`);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Invalid documentId: parentProjectId not found.' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    console.log('Retrieved parentProjectId:', parentProjectId);\n",
    "\n",
    "    // **Step 9: Insert Chunks into the `chunks` Table**\n",
    "    console.log('Inserting chunks into the database');\n",
    "    try {\n",
    "      // Batch insert chunks for efficiency\n",
    "      const chunkDocs = docChunks.map((chunk, index) => ({\n",
    "        pageContent: chunk,\n",
    "        chunkNumber: index + 1,\n",
    "      }));\n",
    "\n",
    "      await convex.mutation('chunks:insertChunks', {\n",
    "        parentProjectId, // Updated: Use parentProjectId instead of projectId\n",
    "        chunks: chunkDocs,\n",
    "      });\n",
    "\n",
    "      console.log('All chunks inserted successfully.');\n",
    "\n",
    "      // Optionally, update the document's `isProcessed` and `processedAt` fields\n",
    "      await convex.mutation('projects:updateProcessingStatus', {\n",
    "        documentId,\n",
    "        isProcessing: false, // Set to 'false' if processing is complete\n",
    "        processedAt: new Date().toISOString(),\n",
    "        progress: 100, // Assuming processing is complete\n",
    "      });\n",
    "\n",
    "    } catch (insertError: any) {\n",
    "      console.error('Error inserting chunks:', insertError);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Error inserting chunks' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 70, // Update progress after chunk insertion\n",
    "    });\n",
    "\n",
    "    // **Step 10: Generate and Store Embeddings for Chunks**\n",
    "    console.log('Generating embeddings for the chunks');\n",
    "\n",
    "    // Initialize OpenAIEmbeddings with your API key\n",
    "    const openAIEmbeddings = new OpenAIEmbeddings({\n",
    "      openAIApiKey: process.env.OPENAI_API_KEY, // Ensure this is set in your environment variables\n",
    "    });\n",
    "\n",
    "    // Generate embeddings for the chunks\n",
    "    const chunkEmbeddings: number[][] = await openAIEmbeddings.embedDocuments(docChunks);\n",
    "    console.log('Generated Embeddings:', chunkEmbeddings.length);\n",
    "\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 90, // Update progress after generating embeddings\n",
    "    });\n",
    "\n",
    "    // **Step 11: Update the Embeddings for Each Chunk in the Database**\n",
    "    console.log('Updating chunk embeddings in the database');\n",
    "    try {\n",
    "      // Perform concurrent updates\n",
    "      await Promise.all(chunkEmbeddings.map((embedding, index) =>\n",
    "        convex.mutation('chunks:updateChunkEmbedding', {\n",
    "          parentProjectId, // Updated: Use parentProjectId instead of projectId\n",
    "          chunkNumber: index + 1,\n",
    "          embedding,\n",
    "        })\n",
    "      ));\n",
    "\n",
    "      console.log('All chunk embeddings updated successfully.');\n",
    "\n",
    "      await convex.mutation('projects:updateProcessingStatus', {\n",
    "        documentId,\n",
    "        isProcessing: false, // Mark as not processing\n",
    "        progress: 100, // Final progress update\n",
    "      });\n",
    "\n",
    "      return NextResponse.json(\n",
    "        {\n",
    "          pdfUrl: response.url,\n",
    "          text: extractedText,\n",
    "          chunks: docChunks,\n",
    "          embeddingsGenerated: chunkEmbeddings.length\n",
    "        },\n",
    "        { status: 200 }\n",
    "      );\n",
    "    } catch (embeddingsError: any) {\n",
    "      console.error('Error updating chunk embeddings:', embeddingsError);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Error updating chunk embeddings' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "  } catch (error: any) {\n",
    "    console.error('Error handling POST request:', error);\n",
    "    return NextResponse.json(\n",
    "      { error: error.message || 'Internal Server Error' },\n",
    "      { status: 500 }\n",
    "    );\n",
    "  } finally {\n",
    "    // Clean up: Delete the temporary file\n",
    "    if (tempFilePath && fs.existsSync(tempFilePath)) {\n",
    "      fs.unlinkSync(tempFilePath);\n",
    "      console.log('Temporary file deleted.');\n",
    "    }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ***RAG Pipeline 002 - Meta-Data***\n",
    "\n",
    "1.\tsplitPages: true\n",
    "In the PDFLoader constructor, we set splitPages: true. This loads each PDF page as a separate Document object so we can preserve and associate per-page metadata (like pageNumber).\n",
    "2.\tMetadata Merging\n",
    "We capture metadata from each page (like pageDoc.metadata?.loc?.pageNumber, pageDoc.metadata?.title, etc.). When chunking, we merge that data into our final chunk object.\n",
    "3.\textractHeadingsFromText()\n",
    "Demonstrates a very basic way to detect headings or important lines within the text. In a production scenario, you’d replace this with more advanced parsing logic or a library that can better extract structured elements (e.g., headings, footnotes, authors).\n",
    "4.\tmetadata\n",
    "Each chunk now includes a metadata object that references pageNumber, title, author, and headings. You can also add any additional fields, such as footnotes or other relevant context.\n",
    "5.\tSchema\n",
    "Your chunks table schema in schema.ts has a metadata field. We’re now populating it with actual data that can be used at retrieval time to provide more context (e.g., “Here’s chunk #7 from page 3, heading ‘Section 2: Introduction’ …”)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// /Users/matthewsimon/Documents/Github/solomon-electron/next/src/app/api/parse-pdf/route.ts\n",
    "// /src/app/api/parse-pdf/route.ts\n",
    "\n",
    "import { NextResponse } from 'next/server';\n",
    "import convex from '@/lib/convexClient';\n",
    "import { PDFLoader } from \"@langchain/community/document_loaders/fs/pdf\";\n",
    "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
    "import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
    "\n",
    "import fs from 'fs';\n",
    "import path from 'path';\n",
    "import fetch from 'node-fetch'; // Ensure node-fetch is installed\n",
    "\n",
    "export const runtime = \"nodejs\";\n",
    "\n",
    "/** Example function to detect headings in a chunk. */\n",
    "function extractHeadingsFromText(text: string): string[] {\n",
    "  const lines = text.split('\\n');\n",
    "  return lines.filter(line => {\n",
    "    const trimmed = line.trim();\n",
    "    // Example heuristic: lines in ALL CAPS or lines starting with \"Section\"\n",
    "    return /^[A-Z\\s]+$/.test(trimmed) || /^Section\\s+\\d+:/.test(trimmed);\n",
    "  });\n",
    "}\n",
    "\n",
    "export async function POST(request: Request) {\n",
    "  let tempFilePath = '';\n",
    "\n",
    "  try {\n",
    "    // -------------------------\n",
    "    // Step 1: Parse the Incoming JSON Body\n",
    "    // -------------------------\n",
    "    const { documentId, fileId } = await request.json();\n",
    "    console.log('Received POST request with:', { documentId, fileId });\n",
    "\n",
    "    // Step 2: Validate Input\n",
    "    if (!documentId || !fileId) {\n",
    "      console.error('Validation failed: Missing documentId or fileId');\n",
    "      return NextResponse.json(\n",
    "        { error: 'documentId and fileId are required' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // Step 3: Validate fileId\n",
    "    if (typeof fileId !== 'string' || fileId.trim() === '') {\n",
    "      console.error('Invalid fileId provided.');\n",
    "      return NextResponse.json(\n",
    "        { error: 'Invalid fileId provided.' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 4: Invoke Convex Mutation to Get the PDF URL\n",
    "    // -------------------------\n",
    "    console.log('Invoking Convex mutation: projects:getFileUrl');\n",
    "    const response = await convex.mutation('projects:getFileUrl', { fileId });\n",
    "    if (!response || !response.url) {\n",
    "      console.error('No URL returned for PDF');\n",
    "      return NextResponse.json(\n",
    "        { error: 'No URL returned for PDF' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    console.log('PDF URL:', response.url);\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 5: Fetch the PDF\n",
    "    // -------------------------\n",
    "    console.log('Fetching the PDF from the URL');\n",
    "    const pdfResponse = await fetch(response.url);\n",
    "    if (!pdfResponse.ok) {\n",
    "      console.error('Failed to fetch PDF:', pdfResponse.statusText);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Failed to fetch PDF' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // Update processing status (progress: 10)\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 10,\n",
    "      isProcessing: true, // Mark as processing if not yet set\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 6: Parse the PDF to Extract Text\n",
    "    // -------------------------\n",
    "    console.log('Parsing the PDF to extract text');\n",
    "\n",
    "    // Download and save the PDF to a temporary file\n",
    "    const tempDir = path.join(process.cwd(), 'temp');\n",
    "    if (!fs.existsSync(tempDir)) {\n",
    "      fs.mkdirSync(tempDir);\n",
    "    }\n",
    "    tempFilePath = path.join(tempDir, `${fileId}.pdf`);\n",
    "\n",
    "    const arrayBuffer = await pdfResponse.arrayBuffer();\n",
    "    const buffer = Buffer.from(arrayBuffer);\n",
    "    fs.writeFileSync(tempFilePath, buffer);\n",
    "    console.log('PDF saved to:', tempFilePath);\n",
    "\n",
    "    // Create a loader that keeps per-page metadata\n",
    "    const loader = new PDFLoader(tempFilePath, {\n",
    "      splitPages: true, // Each page is a separate document\n",
    "    });\n",
    "    const docs = await loader.load();\n",
    "    if (!docs.length) {\n",
    "      throw new Error(\"No content extracted from the document.\");\n",
    "    }\n",
    "    console.log(\"Total pages (docs) loaded:\", docs.length);\n",
    "\n",
    "    // Print sample metadata for debugging\n",
    "    console.log(\"Sample metadata:\", docs[0].metadata);\n",
    "\n",
    "    // Combine extracted text (for debugging or raw usage)\n",
    "    const extractedText = docs.map(doc => doc.pageContent).join('\\n');\n",
    "    console.log('Extracted Text:', extractedText.slice(0, 300) + '...');\n",
    "\n",
    "    // Update progress to 30\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 30,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 7: Split Each Page-Document into Sub-chunks\n",
    "    // -------------------------\n",
    "    console.log('Splitting the document into chunks');\n",
    "    const textSplitter = new RecursiveCharacterTextSplitter({\n",
    "      chunkSize: 1000,\n",
    "      chunkOverlap: 200,\n",
    "      separators: [\"\\n\\n\", \"\\n\", \" \", \"\"],\n",
    "    });\n",
    "\n",
    "    const allChunks = [];\n",
    "    for (let pageIndex = 0; pageIndex < docs.length; pageIndex++) {\n",
    "      const pageDoc = docs[pageIndex];\n",
    "\n",
    "      // If PDFLoader returns pageNumber in metadata\n",
    "      const pageNumber = pageDoc.metadata?.loc?.pageNumber ?? pageIndex + 1;\n",
    "      // Potential docTitle/author from PDF metadata\n",
    "      const docTitle = pageDoc.metadata?.title || \"Untitled\";\n",
    "      const docAuthor = pageDoc.metadata?.author || \"Unknown\";\n",
    "\n",
    "      // Do a chunk split for this page\n",
    "      const pageChunks = await textSplitter.splitDocuments([pageDoc]);\n",
    "\n",
    "      // Create chunk objects that include metadata\n",
    "      for (const chunkedDoc of pageChunks) {\n",
    "        const headings = extractHeadingsFromText(chunkedDoc.pageContent);\n",
    "\n",
    "        allChunks.push({\n",
    "          pageContent: chunkedDoc.pageContent,\n",
    "          metadata: {\n",
    "            pageNumber,\n",
    "            docTitle,\n",
    "            docAuthor,\n",
    "            headings,\n",
    "          }\n",
    "        });\n",
    "      }\n",
    "    }\n",
    "\n",
    "    // Number them globally\n",
    "    const docChunks = allChunks.map((item, index) => ({\n",
    "      ...item,\n",
    "      chunkNumber: index + 1,\n",
    "    }));\n",
    "\n",
    "    console.log('Total number of sub-chunks across all pages:', docChunks.length);\n",
    "\n",
    "    // Update progress to 50\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 50,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 8: Retrieve parentProjectId from documentId\n",
    "    // -------------------------\n",
    "    console.log('Retrieving parentProjectId from documentId:', documentId);\n",
    "    const parentProjectId = await convex.query('projects:getParentProjectId', { documentId });\n",
    "    if (!parentProjectId) {\n",
    "      console.error(`No parentProjectId found for documentId: ${documentId}`);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Invalid documentId: parentProjectId not found.' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "    console.log('Retrieved parentProjectId:', parentProjectId);\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 9: Insert Chunks into the chunks Table\n",
    "    // -------------------------\n",
    "    console.log('Inserting chunks into the database');\n",
    "    try {\n",
    "      // Prepare chunk docs for batch insert\n",
    "      const chunkDocs = docChunks.map((chunk) => ({\n",
    "        pageContent: chunk.pageContent,\n",
    "        chunkNumber: chunk.chunkNumber,\n",
    "        metadata: {\n",
    "          pageNumber: chunk.metadata.pageNumber,\n",
    "          docTitle: chunk.metadata.docTitle,\n",
    "          docAuthor: chunk.metadata.docAuthor,\n",
    "          headings: chunk.metadata.headings,\n",
    "        },\n",
    "      }));\n",
    "\n",
    "      await convex.mutation('chunks:insertChunks', {\n",
    "        parentProjectId,\n",
    "        chunks: chunkDocs,\n",
    "      });\n",
    "\n",
    "      console.log('All chunks inserted successfully.');\n",
    "\n",
    "      // (Optional) Mark doc as processed\n",
    "      await convex.mutation('projects:updateProcessingStatus', {\n",
    "        documentId,\n",
    "        isProcessing: false, // done inserting\n",
    "        isProcessed: true,   // if you store this at the project level\n",
    "        processedAt: new Date().toISOString(),\n",
    "      });\n",
    "    } catch (insertError: any) {\n",
    "      console.error('Error inserting chunks:', insertError);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Error inserting chunks' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // Update progress to 70\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 70,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 10: Generate Embeddings for Chunks\n",
    "    // -------------------------\n",
    "    console.log('Generating embeddings for the chunks');\n",
    "    const openAIEmbeddings = new OpenAIEmbeddings({\n",
    "      openAIApiKey: process.env.OPENAI_API_KEY,\n",
    "    });\n",
    "\n",
    "    // Get text from each chunk to embed\n",
    "    const textsForEmbedding = docChunks.map((c) => c.pageContent);\n",
    "    const chunkEmbeddings: number[][] = await openAIEmbeddings.embedDocuments(textsForEmbedding);\n",
    "    console.log('Generated Embeddings:', chunkEmbeddings.length);\n",
    "\n",
    "    // Update progress to 90\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 90,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 11: Update Each Chunk with Its Embedding\n",
    "    // -------------------------\n",
    "    console.log('Updating chunk embeddings in the database');\n",
    "    try {\n",
    "      await Promise.all(\n",
    "        chunkEmbeddings.map((embedding, index) =>\n",
    "          convex.mutation('chunks:updateChunkEmbedding', {\n",
    "            parentProjectId,\n",
    "            chunkNumber: docChunks[index].chunkNumber,\n",
    "            embedding,\n",
    "          })\n",
    "        )\n",
    "      );\n",
    "\n",
    "      console.log('All chunk embeddings updated successfully.');\n",
    "\n",
    "      // Mark final progress as 100\n",
    "      await convex.mutation('projects:updateProcessingStatus', {\n",
    "        documentId,\n",
    "        isProcessing: false,\n",
    "        progress: 100,\n",
    "      });\n",
    "\n",
    "      // Return final response\n",
    "      return NextResponse.json(\n",
    "        {\n",
    "          pdfUrl: response.url,\n",
    "          text: extractedText,\n",
    "          chunks: docChunks,\n",
    "          embeddingsGenerated: chunkEmbeddings.length\n",
    "        },\n",
    "        { status: 200 }\n",
    "      );\n",
    "    } catch (embeddingsError: any) {\n",
    "      console.error('Error updating chunk embeddings:', embeddingsError);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Error updating chunk embeddings' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "  } catch (error: any) {\n",
    "    console.error('Error handling POST request:', error);\n",
    "    return NextResponse.json(\n",
    "      { error: error.message || 'Internal Server Error' },\n",
    "      { status: 500 }\n",
    "    );\n",
    "  } finally {\n",
    "    // Cleanup\n",
    "    if (tempFilePath && fs.existsSync(tempFilePath)) {\n",
    "      fs.unlinkSync(tempFilePath);\n",
    "      console.log('Temporary file deleted.');\n",
    "    }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ***RAG Pipeline 002 - Hierarchical Chunking***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "javascript"
    }
   },
   "outputs": [],
   "source": [
    "// /Users/matthewsimon/Documents/Github/solomon-electron/next/src/app/api/parse-pdf/route.ts\n",
    "// /src/app/api/parse-pdf/route.ts\n",
    "\n",
    "import { NextResponse } from 'next/server';\n",
    "import convex from '@/lib/convexClient';\n",
    "import { PDFLoader } from \"@langchain/community/document_loaders/fs/pdf\";\n",
    "// We still import RecursiveCharacterTextSplitter for potential fallback or partial usage\n",
    "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
    "import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
    "\n",
    "import fs from 'fs';\n",
    "import path from 'path';\n",
    "import fetch from 'node-fetch'; // Ensure node-fetch is installed\n",
    "\n",
    "export const runtime = \"nodejs\";\n",
    "\n",
    "/** Example function to detect headings in a chunk. */\n",
    "function extractHeadingsFromText(text: string): string[] {\n",
    "  const lines = text.split('\\n');\n",
    "  return lines.filter(line => {\n",
    "    const trimmed = line.trim();\n",
    "    // Example heuristic: lines in ALL CAPS or lines starting with \"Section\"\n",
    "    return /^[A-Z\\s]+$/.test(trimmed) || /^Section\\s+\\d+:/.test(trimmed);\n",
    "  });\n",
    "}\n",
    "\n",
    "/** A naive check to identify headings in text for 'hierarchical' chunking. */\n",
    "function isLikelyHeading(line: string): boolean {\n",
    "  return /^[A-Z\\s]+$/.test(line.trim()) || /^Section\\s+\\d+:/.test(line.trim());\n",
    "}\n",
    "\n",
    "/**\n",
    " * Example function to split a string into sections based on headings.\n",
    " * Returns array of { heading, body } objects.\n",
    " */\n",
    "function splitByHeadings(text: string): { heading: string; body: string }[] {\n",
    "  const lines = text.split('\\n');\n",
    "  const sections: { heading: string; body: string }[] = [];\n",
    "  let currentHeading = 'UNTITLED SECTION';\n",
    "  let currentBuffer = '';\n",
    "\n",
    "  for (const line of lines) {\n",
    "    if (isLikelyHeading(line)) {\n",
    "      // Push the existing buffer to a section\n",
    "      if (currentBuffer.trim().length > 0) {\n",
    "        sections.push({ heading: currentHeading, body: currentBuffer });\n",
    "      }\n",
    "      currentHeading = line.trim();\n",
    "      currentBuffer = '';\n",
    "    } else {\n",
    "      currentBuffer += `${line}\\n`;\n",
    "    }\n",
    "  }\n",
    "\n",
    "  // Push the last buffer if it exists\n",
    "  if (currentBuffer.trim().length > 0) {\n",
    "    sections.push({ heading: currentHeading, body: currentBuffer });\n",
    "  }\n",
    "\n",
    "  return sections;\n",
    "}\n",
    "\n",
    "/** Split paragraphs within each heading section. Uses naive blank line detection. */\n",
    "function splitByParagraphs(text: string): string[] {\n",
    "  // Paragraphs delimited by one or more blank lines\n",
    "  return text.split(/\\n\\s*\\n/).map(p => p.trim()).filter(Boolean);\n",
    "}\n",
    "\n",
    "/**\n",
    " * Additional fallback: a simple character-based splitter with overlap.\n",
    " * This helps if a paragraph is still too large.\n",
    " */\n",
    "function charBasedSplit(\n",
    "  text: string,\n",
    "  chunkSize: number,\n",
    "  chunkOverlap: number\n",
    "): string[] {\n",
    "  const chunks: string[] = [];\n",
    "  let start = 0;\n",
    "  while (start < text.length) {\n",
    "    const end = Math.min(start + chunkSize, text.length);\n",
    "    chunks.push(text.slice(start, end));\n",
    "    start += (chunkSize - chunkOverlap);\n",
    "  }\n",
    "  return chunks;\n",
    "}\n",
    "\n",
    "/**\n",
    " * Semantic splitter that tries to split text by sentences.\n",
    " * For example, we split at each period, question mark, exclamation mark, etc.\n",
    " * Then enforce max length constraints with overlap.\n",
    " */\n",
    "function semanticSplit(\n",
    "  text: string,\n",
    "  maxTokens: number,\n",
    "  overlapTokens: number\n",
    "): string[] {\n",
    "  // Very naive sentence split (regex-based). In production, consider an NLP library.\n",
    "  const sentences = text.split(/(?<=[.?!])\\s+/);\n",
    "  const chunks: string[] = [];\n",
    "  let currentChunk = '';\n",
    "\n",
    "  for (const sentence of sentences) {\n",
    "    if ((currentChunk + sentence).length > maxTokens) {\n",
    "      // push the current chunk\n",
    "      chunks.push(currentChunk);\n",
    "      // add overlap from the end of currentChunk\n",
    "      const overlap = currentChunk.slice(\n",
    "        Math.max(0, currentChunk.length - overlapTokens)\n",
    "      );\n",
    "      currentChunk = overlap + ' ' + sentence;\n",
    "    } else {\n",
    "      currentChunk += (currentChunk ? ' ' : '') + sentence;\n",
    "    }\n",
    "  }\n",
    "  if (currentChunk) {\n",
    "    chunks.push(currentChunk);\n",
    "  }\n",
    "  return chunks;\n",
    "}\n",
    "\n",
    "/**\n",
    " * A combined hierarchical + semantic approach:\n",
    " * 1) Split by headings -> sections\n",
    " * 2) Within each section, split by paragraphs\n",
    " * 3) If paragraphs are too large, do a semantic or char-based split\n",
    " */\n",
    "function hierarchicalSemanticSplit(\n",
    "  text: string,\n",
    "  chunkSize: number,\n",
    "  chunkOverlap: number\n",
    "): string[] {\n",
    "  const sections = splitByHeadings(text);\n",
    "  const finalChunks: string[] = [];\n",
    "\n",
    "  for (const section of sections) {\n",
    "    // Break the section into paragraphs\n",
    "    const paragraphs = splitByParagraphs(section.body);\n",
    "\n",
    "    for (const paragraph of paragraphs) {\n",
    "      // If paragraph is bigger than chunkSize, do further splits\n",
    "      if (paragraph.length > chunkSize) {\n",
    "        // Option A: semantic split\n",
    "        const semChunks = semanticSplit(paragraph, chunkSize, chunkOverlap);\n",
    "\n",
    "        // If any sem-split chunk is STILL bigger than chunkSize,\n",
    "        // fallback to char-based\n",
    "        for (const semChunk of semChunks) {\n",
    "          if (semChunk.length > chunkSize) {\n",
    "            // fallback\n",
    "            const charChunks = charBasedSplit(semChunk, chunkSize, chunkOverlap);\n",
    "            // Prepend the heading for context\n",
    "            for (const cChunk of charChunks) {\n",
    "              finalChunks.push(`${section.heading}\\n${cChunk}`);\n",
    "            }\n",
    "          } else {\n",
    "            finalChunks.push(`${section.heading}\\n${semChunk}`);\n",
    "          }\n",
    "        }\n",
    "      } else {\n",
    "        // If paragraph is within chunk size, just store it with heading\n",
    "        finalChunks.push(`${section.heading}\\n${paragraph}`);\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "  return finalChunks;\n",
    "}\n",
    "\n",
    "/**\n",
    " * Demonstration of adaptive chunk sizing:\n",
    " * - Adjust chunkSize & chunkOverlap based on total number of characters in doc.\n",
    " */\n",
    "function getAdaptiveChunkParams(totalChars: number) {\n",
    "  if (totalChars < 5000) {\n",
    "    // smaller doc -> smaller chunkSize but bigger overlap\n",
    "    return { chunkSize: 500, chunkOverlap: 100 };\n",
    "  } else if (totalChars < 50000) {\n",
    "    // medium doc\n",
    "    return { chunkSize: 1000, chunkOverlap: 200 };\n",
    "  } else {\n",
    "    // large doc\n",
    "    return { chunkSize: 2000, chunkOverlap: 200 };\n",
    "  }\n",
    "}\n",
    "\n",
    "export async function POST(request: Request) {\n",
    "  let tempFilePath = '';\n",
    "\n",
    "  try {\n",
    "    // -------------------------\n",
    "    // Step 1: Parse the Incoming JSON Body\n",
    "    // -------------------------\n",
    "    const { documentId, fileId } = await request.json();\n",
    "    console.log('Received POST request with:', { documentId, fileId });\n",
    "\n",
    "    // Step 2: Validate Input\n",
    "    if (!documentId || !fileId) {\n",
    "      console.error('Validation failed: Missing documentId or fileId');\n",
    "      return NextResponse.json(\n",
    "        { error: 'documentId and fileId are required' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // Step 3: Validate fileId\n",
    "    if (typeof fileId !== 'string' || fileId.trim() === '') {\n",
    "      console.error('Invalid fileId provided.');\n",
    "      return NextResponse.json(\n",
    "        { error: 'Invalid fileId provided.' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 4: Invoke Convex Mutation to Get the PDF URL\n",
    "    // -------------------------\n",
    "    console.log('Invoking Convex mutation: projects:getFileUrl');\n",
    "    const response = await convex.mutation('projects:getFileUrl', { fileId });\n",
    "    if (!response || !response.url) {\n",
    "      console.error('No URL returned for PDF');\n",
    "      return NextResponse.json(\n",
    "        { error: 'No URL returned for PDF' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    console.log('PDF URL:', response.url);\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 5: Fetch the PDF\n",
    "    // -------------------------\n",
    "    console.log('Fetching the PDF from the URL');\n",
    "    const pdfResponse = await fetch(response.url);\n",
    "    if (!pdfResponse.ok) {\n",
    "      console.error('Failed to fetch PDF:', pdfResponse.statusText);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Failed to fetch PDF' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // Update processing status (progress: 10)\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 10,\n",
    "      isProcessing: true, // Mark as processing if not yet set\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 6: Parse the PDF to Extract Text\n",
    "    // -------------------------\n",
    "    console.log('Parsing the PDF to extract text');\n",
    "\n",
    "    // Download and save the PDF to a temporary file\n",
    "    const tempDir = path.join(process.cwd(), 'temp');\n",
    "    if (!fs.existsSync(tempDir)) {\n",
    "      fs.mkdirSync(tempDir);\n",
    "    }\n",
    "    tempFilePath = path.join(tempDir, `${fileId}.pdf`);\n",
    "\n",
    "    const arrayBuffer = await pdfResponse.arrayBuffer();\n",
    "    const buffer = Buffer.from(arrayBuffer);\n",
    "    fs.writeFileSync(tempFilePath, buffer);\n",
    "    console.log('PDF saved to:', tempFilePath);\n",
    "\n",
    "    // Create a loader that keeps per-page metadata\n",
    "    const loader = new PDFLoader(tempFilePath, {\n",
    "      splitPages: true, // Each page is a separate document\n",
    "    });\n",
    "    const docs = await loader.load();\n",
    "    if (!docs.length) {\n",
    "      throw new Error(\"No content extracted from the document.\");\n",
    "    }\n",
    "    console.log(\"Total pages (docs) loaded:\", docs.length);\n",
    "\n",
    "    // Print sample metadata for debugging\n",
    "    console.log(\"Sample metadata:\", docs[0].metadata);\n",
    "\n",
    "    // Combine extracted text (for debugging or raw usage)\n",
    "    const extractedText = docs.map(doc => doc.pageContent).join('\\n');\n",
    "    console.log('Extracted Text:', extractedText.slice(0, 300) + '...');\n",
    "\n",
    "    // Update progress to 30\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 30,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 7: Enhanced Split Each Page-Document into Sub-chunks\n",
    "    // -------------------------\n",
    "    console.log('Splitting the document into chunks');\n",
    "\n",
    "    // 7a) Determine total length to pick adaptive chunk size\n",
    "    const totalChars = docs.reduce((acc, doc) => acc + doc.pageContent.length, 0);\n",
    "    const { chunkSize, chunkOverlap } = getAdaptiveChunkParams(totalChars);\n",
    "    console.log(`Adaptive chunkSize=${chunkSize}, chunkOverlap=${chunkOverlap}`);\n",
    "\n",
    "    const allChunks = [];\n",
    "    for (let pageIndex = 0; pageIndex < docs.length; pageIndex++) {\n",
    "      const pageDoc = docs[pageIndex];\n",
    "      const pageNumber = pageDoc.metadata?.loc?.pageNumber ?? pageIndex + 1;\n",
    "      const docTitle = pageDoc.metadata?.title || \"Untitled\";\n",
    "      const docAuthor = pageDoc.metadata?.author || \"Unknown\";\n",
    "\n",
    "      // Instead of using RecursiveCharacterTextSplitter directly,\n",
    "      // we apply hierarchical + semantic chunking:\n",
    "      const pageChunks = hierarchicalSemanticSplit(\n",
    "        pageDoc.pageContent,\n",
    "        chunkSize,\n",
    "        chunkOverlap\n",
    "      );\n",
    "\n",
    "      // For each final chunk, gather the headings for that text\n",
    "      for (const chunkText of pageChunks) {\n",
    "        const headings = extractHeadingsFromText(chunkText);\n",
    "        allChunks.push({\n",
    "          pageContent: chunkText,\n",
    "          metadata: {\n",
    "            pageNumber,\n",
    "            docTitle,\n",
    "            docAuthor,\n",
    "            headings,\n",
    "          },\n",
    "        });\n",
    "      }\n",
    "    }\n",
    "\n",
    "    // Number them globally\n",
    "    const docChunks = allChunks.map((item, index) => ({\n",
    "      ...item,\n",
    "      chunkNumber: index + 1,\n",
    "    }));\n",
    "\n",
    "    console.log('Total number of sub-chunks across all pages:', docChunks.length);\n",
    "\n",
    "    // Update progress to 50\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 50,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 8: Retrieve parentProjectId from documentId\n",
    "    // -------------------------\n",
    "    console.log('Retrieving parentProjectId from documentId:', documentId);\n",
    "    const parentProjectId = await convex.query('projects:getParentProjectId', { documentId });\n",
    "    if (!parentProjectId) {\n",
    "      console.error(`No parentProjectId found for documentId: ${documentId}`);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Invalid documentId: parentProjectId not found.' },\n",
    "        { status: 400 }\n",
    "      );\n",
    "    }\n",
    "    console.log('Retrieved parentProjectId:', parentProjectId);\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 9: Insert Chunks into the chunks Table\n",
    "    // -------------------------\n",
    "    console.log('Inserting chunks into the database');\n",
    "    try {\n",
    "      // Prepare chunk docs for batch insert\n",
    "      const chunkDocs = docChunks.map((chunk) => ({\n",
    "        pageContent: chunk.pageContent,\n",
    "        chunkNumber: chunk.chunkNumber,\n",
    "        metadata: {\n",
    "          pageNumber: chunk.metadata.pageNumber,\n",
    "          docTitle: chunk.metadata.docTitle,\n",
    "          docAuthor: chunk.metadata.docAuthor,\n",
    "          headings: chunk.metadata.headings,\n",
    "        },\n",
    "      }));\n",
    "\n",
    "      await convex.mutation('chunks:insertChunks', {\n",
    "        parentProjectId,\n",
    "        chunks: chunkDocs,\n",
    "      });\n",
    "\n",
    "      console.log('All chunks inserted successfully.');\n",
    "\n",
    "      // (Optional) Mark doc as processed\n",
    "      await convex.mutation('projects:updateProcessingStatus', {\n",
    "        documentId,\n",
    "        isProcessing: false, // done inserting\n",
    "        isProcessed: true,   // if you store this at the project level\n",
    "        processedAt: new Date().toISOString(),\n",
    "      });\n",
    "    } catch (insertError: any) {\n",
    "      console.error('Error inserting chunks:', insertError);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Error inserting chunks' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "    // Update progress to 70\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 70,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 10: Generate Embeddings for Chunks\n",
    "    // -------------------------\n",
    "    console.log('Generating embeddings for the chunks');\n",
    "    const openAIEmbeddings = new OpenAIEmbeddings({\n",
    "      openAIApiKey: process.env.OPENAI_API_KEY,\n",
    "    });\n",
    "\n",
    "    // Get text from each chunk to embed\n",
    "    const textsForEmbedding = docChunks.map((c) => c.pageContent);\n",
    "    const chunkEmbeddings: number[][] = await openAIEmbeddings.embedDocuments(textsForEmbedding);\n",
    "    console.log('Generated Embeddings:', chunkEmbeddings.length);\n",
    "\n",
    "    // Update progress to 90\n",
    "    await convex.mutation('projects:updateProcessingStatus', {\n",
    "      documentId,\n",
    "      progress: 90,\n",
    "    });\n",
    "\n",
    "    // -------------------------\n",
    "    // Step 11: Update Each Chunk with Its Embedding\n",
    "    // -------------------------\n",
    "    console.log('Updating chunk embeddings in the database');\n",
    "    try {\n",
    "      await Promise.all(\n",
    "        chunkEmbeddings.map((embedding, index) =>\n",
    "          convex.mutation('chunks:updateChunkEmbedding', {\n",
    "            parentProjectId,\n",
    "            chunkNumber: docChunks[index].chunkNumber,\n",
    "            embedding,\n",
    "          })\n",
    "        )\n",
    "      );\n",
    "\n",
    "      console.log('All chunk embeddings updated successfully.');\n",
    "\n",
    "      // Mark final progress as 100\n",
    "      await convex.mutation('projects:updateProcessingStatus', {\n",
    "        documentId,\n",
    "        isProcessing: false,\n",
    "        progress: 100,\n",
    "      });\n",
    "\n",
    "      // Return final response\n",
    "      return NextResponse.json(\n",
    "        {\n",
    "          pdfUrl: response.url,\n",
    "          text: extractedText,\n",
    "          chunks: docChunks,\n",
    "          embeddingsGenerated: chunkEmbeddings.length\n",
    "        },\n",
    "        { status: 200 }\n",
    "      );\n",
    "    } catch (embeddingsError: any) {\n",
    "      console.error('Error updating chunk embeddings:', embeddingsError);\n",
    "      return NextResponse.json(\n",
    "        { error: 'Error updating chunk embeddings' },\n",
    "        { status: 500 }\n",
    "      );\n",
    "    }\n",
    "\n",
    "  } catch (error: any) {\n",
    "    console.error('Error handling POST request:', error);\n",
    "    return NextResponse.json(\n",
    "      { error: error.message || 'Internal Server Error' },\n",
    "      { status: 500 }\n",
    "    );\n",
    "  } finally {\n",
    "    // Cleanup\n",
    "    if (tempFilePath && fs.existsSync(tempFilePath)) {\n",
    "      fs.unlinkSync(tempFilePath);\n",
    "      console.log('Temporary file deleted.');\n",
    "    }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Llama Parser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that LlamaParse now gives the individual components of the page within the items array, in the shape of an object with a type, a value, and some bBox parameters. We are not using this at the moment, but it's something to keep in mind in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}