Text Summerisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df9ba7c9",
   "metadata": {},
   "source": [
    "# An Investigation Into Different Approaches For Summerisation Using NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1edb62be",
   "metadata": {},
   "source": [
    "## Abstract\n",
    "\n",
    "Text summarization in NLP is the process of summarising the large body of text into smaller chunk suitable which is more suitable for the readers comprehension. In this article, I will investigate different approaches to summerisation from traditional to advance methods such as generative AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bab445c",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "Summerisation of large body of text has been an essential need, during our daily lives when we are working or reading, we realise a lot of information we are reading can be summerised to a smaller more digestable format, we also have issue of needing to read a document but lacking the time as we are too busy with other matter. This could rnage from newspapers, articles, research papers, books etc. \n",
    "\n",
    "What I want to accomplish through this investigation is to discover the most cost-effective approach to summerisation and integrate this feature to our AI learning assistant we are developing. We will be researching and documenting various techniques used for summerisation and choose the most suitable approach after rigourously evaluating the method. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ec31a1c",
   "metadata": {},
   "source": [
    "## Types of Text Summarization\n",
    "After an extensive investigation finding numerous summerisation techniques we have categorised the two different types of approaches to text summerisation, these are the approaches, Extractive and Abstractive\n",
    "\n",
    "### Extractive Text Summarization\n",
    "It is the traditional method developed first. The main objective is to identify the significant sentences of the text and add them to the summary. You need to note that the summary obtained contains exact sentences from the original text.\n",
    "\n",
    "### Abstractive Text Summarization\n",
    "It is a more advanced method, with many advancements coming out frequently. The approach is to identify the important sections, interpret the context, and reproduce in a new way. This ensures that the core information is conveyed through the shortest text possible. Here, the sentences in the summary are generated, not just extracted from the original text.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5746738",
   "metadata": {},
   "source": [
    "## Extractive Summerisation Method: TextRank with Gensim\n",
    "TextRank is a graph-based ranking algorithm specifically adapted for text processing. It is similar to LexRank in that it is based on the concept of ranking sentences for importance within the text. The core idea is that sentences \"recommend\" each other, much like web pages do in Google's PageRank algorithm. Each sentence is treated as a node, and the connections between them are based on their similarity. A voting or recommendation process occurs where the importance of a sentence is determined by the importance of the sentences recommending it.\n",
    "\n",
    "As a result of this iterative voting or recommendation process, TextRank identifies sentences that are central to the text and thus should be included in the summary. The sentences chosen for the summary are those that are most highly ranked according to this algorithm.\n",
    "\n",
    "TextRank excels when applied to texts that have a dense web of semantic similarities between sentences, such as scientific articles, technical papers, and legal documents. It is particularly useful for documents where the use of domain-specific vocabulary leads to clear patterns of word use within the text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a27b8b2d",
   "metadata": {},
   "source": [
    "### Step 1: Add all neccessary installation and choose a File\n",
    "Add all neccessary imports and create function to prompt the user to select a PDF file from their system using a file dialog.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cddb7f58",
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install pdfminer.six nltk networkx gensim tk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ffd431a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tkinter as tk\n",
    "from tkinter import filedialog\n",
    "\n",
    "def choose_file():\n",
    "    # Initialize the Tkinter GUI\n",
    "    root = tk.Tk()\n",
    "    root.withdraw()  # Hide the root window\n",
    "    file_path = filedialog.askopenfilename(filetypes=[(\"PDF files\", \"*.pdf\")])  # Only allow PDFs\n",
    "    return file_path\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e4b4f9c",
   "metadata": {},
   "source": [
    "### Step 2: Text Extraction and Preprocessing\n",
    "\n",
    "The following functions are used to extract text from a PDF and preprocess it for summarization.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "460a0104",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pdfminer.high_level import extract_text\n",
    "\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    return extract_text(pdf_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "76ae85e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Error loading punkt: <urlopen error [Errno 11001]\n",
      "[nltk_data]     getaddrinfo failed>\n",
      "[nltk_data] Error loading stopwords: <urlopen error [Errno 11001]\n",
      "[nltk_data]     getaddrinfo failed>\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import PorterStemmer\n",
    "\n",
    "# Download required NLTK resources\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "\n",
    "def preprocess_text(text):\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    stemmer = PorterStemmer()\n",
    "    \n",
    "    # Split text into sentences\n",
    "    sentences = sent_tokenize(text)\n",
    "    \n",
    "    # Tokenize, stem, and remove stop words\n",
    "    preprocessed_text = []\n",
    "    for sentence in sentences:\n",
    "        tokens = [stemmer.stem(word) for word in word_tokenize(sentence.lower())\n",
    "                  if word not in stop_words and word.isalnum()]\n",
    "        if tokens:  # Only add non-empty lists\n",
    "            preprocessed_text.append(tokens)\n",
    "    \n",
    "    # Filter out sentences that correspond to empty preprocessed lists\n",
    "    sentences = [sentences[i] for i, tokens in enumerate(preprocessed_text) if tokens]\n",
    "    \n",
    "    return sentences, preprocessed_text\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cba2b42",
   "metadata": {},
   "source": [
    "### Step 3: Implement TextRank\n",
    "\n",
    "We will now define functions to build a similarity matrix using Word2Vec and apply the TextRank algorithm to rank sentences.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3fef8a28",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import networkx as nx\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "def build_similarity_matrix(sentences, preprocessed_text):\n",
    "    # Train a Word2Vec model\n",
    "    if preprocessed_text:\n",
    "        model = Word2Vec(preprocessed_text, vector_size=100, window=2, min_count=1, workers=2)\n",
    "        model_wv = model.wv\n",
    "    else:\n",
    "        raise ValueError(\"The preprocessed text is empty. Please check your preprocessing steps.\")\n",
    "    \n",
    "    # Create an empty similarity matrix\n",
    "    sim_matrix = np.zeros((len(sentences), len(sentences)))\n",
    "    \n",
    "    # Build the similarity matrix\n",
    "    for i in range(len(sentences)):\n",
    "        for j in range(len(sentences)):\n",
    "            if i != j:\n",
    "                # Ensure each sentence has at least one word after preprocessing\n",
    "                if preprocessed_text[i] and preprocessed_text[j]:\n",
    "                    vector_i = np.mean([model_wv[word] for word in preprocessed_text[i] if word in model_wv], axis=0)\n",
    "                    vector_j = np.mean([model_wv[word] for word in preprocessed_text[j] if word in model_wv], axis=0)\n",
    "                    \n",
    "                    # Check that vectors are valid (not NaN or infinite)\n",
    "                    if not np.all(np.isfinite(vector_i)) or not np.all(np.isfinite(vector_j)):\n",
    "                        continue\n",
    "                    \n",
    "                    sim_matrix[i][j] = cosine_similarity([vector_i], [vector_j])[0, 0]\n",
    "    \n",
    "    return sim_matrix\n",
    "\n",
    "\n",
    "def apply_text_rank(sim_matrix, sentences):\n",
    "    nx_graph = nx.from_numpy_array(sim_matrix)\n",
    "    scores = nx.pagerank(nx_graph, max_iter=10000, tol=1e-3)\n",
    "    ranked_sentences = sorted(((scores[i], s) for i, s in enumerate(sentences)), reverse=True)\n",
    "    return ranked_sentences\n",
    "\n",
    "def summarize(pdf_path, num_sentences=5):\n",
    "    text = extract_text_from_pdf(pdf_path)\n",
    "    sentences, preprocessed_text = preprocess_text(text)\n",
    "    sim_matrix = build_similarity_matrix(sentences, preprocessed_text)\n",
    "    ranked_sentences = apply_text_rank(sim_matrix, sentences)\n",
    "    summary = ' '.join([s[1] for s in ranked_sentences[:num_sentences]])\n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3e81d1b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5. But\n",
      "along with honor comes, or must come, the charismatic leader’s recognition of his\n",
      "‘Eigenverantwortung’  (‘Self-responsibility’)  (1992b:  180). For  him,  both  Stefan  George  and  Tolstoy  were  charismatic  leaders  who  were\n",
      "‘irrational’. For  Weber,  bureaucratic  authority  has  many  positive  features:  it  is  based\n",
      "upon reason, it is impartially implemented by paid trained ofﬁcials, and its future\n",
      "\n",
      " \n",
      "\f",
      "is stable. One of the differences between bureaucratic and traditional Herrschaft, if not the key one is that\n",
      "the former is based upon the concept of ‘competence’, which is lacking in the latter (see Weber,\n",
      "1988: 478, 482).\n"
     ]
    }
   ],
   "source": [
    "pdf_path = choose_file()\n",
    "summary = summarize(pdf_path)\n",
    "print(summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44248b6b",
   "metadata": {},
   "source": [
    "## Extractive Summerisation Method: LexRank\n",
    "\n",
    "LexRank is an unsupervised approach to text summarization based on graph theory. Sentences within a given text are represented as vertices in a graph. Edges between sentences are created based on the similarity between sentences, which can be computed using measures like cosine similarity with TF-IDF weighting. The LexRank algorithm then applies a method similar to Google's PageRank to this graph: a sentence is considered important if it is similar to many other sentences, and those sentences are themselves considered important.\n",
    "\n",
    "LexRank is an unsupervised approach to text summarization based on graph theory. Sentences within a given text are represented as vertices in a graph. Edges between sentences are created based on the similarity between sentences, which can be computed using measures like cosine similarity with TF-IDF weighting. The LexRank algorithm then applies a method similar to Google's PageRank to this graph: a sentence is considered important if it is similar to many other sentences, and those sentences are themselves considered important.\n",
    "\n",
    " LexRank is particularly effective on structured and well-written texts where the salient information is distributed throughout the document. It works well with news articles, research papers, and technical documents, where the recurrence of similar concepts can be used to gauge the importance of sentences.\n",
    "\n",
    "### Step 1: Import Necessary Libraries\n",
    "\n",
    "Before we begin, let's import all the necessary libraries. If you haven't installed these libraries, you can do so using `pip`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "805d62ad",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Run this cell to install the necessary libraries\n",
    "#!pip install pdfminer.six numpy scipy networkx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7ed5f9c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# Importing necessary libraries\n",
    "import pdfminer\n",
    "from pdfminer.high_level import extract_text\n",
    "import numpy as np\n",
    "from scipy.sparse.csgraph import connected_components\n",
    "from scipy.sparse import csr_matrix\n",
    "import networkx as nx\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import nltk\n",
    "nltk.download('punkt')  # Download the tokenizer model if not already downloaded\n",
    "nltk.download('stopwords')\n",
    "\n",
    "# Define function to extract text from PDF\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    return extract_text(pdf_path)\n",
    "\n",
    "# Define function to preprocess text\n",
    "def preprocess_text(text):\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    sentences = sent_tokenize(text)\n",
    "    preprocessed_sentences = []\n",
    "    for sentence in sentences:\n",
    "        words = word_tokenize(sentence)\n",
    "        filtered_words = [word for word in words if word.lower() not in stop_words and word.isalnum()]\n",
    "        preprocessed_sentences.append(' '.join(filtered_words))\n",
    "    return preprocessed_sentences\n",
    "\n",
    "# Define function to calculate sentence similarity\n",
    "def sentence_similarity(sent1, sent2):\n",
    "    vectorizer = TfidfVectorizer(min_df=1, stop_words='english')\n",
    "    \n",
    "    # Try-except block to handle the ValueError\n",
    "    try:\n",
    "        tfidf = vectorizer.fit_transform([sent1, sent2])\n",
    "        return ((tfidf * tfidf.T).A)[0, 1]\n",
    "    except ValueError:\n",
    "        # In case of an empty vocabulary, return 0 similarity\n",
    "        return 0.0\n",
    "\n",
    "# Define function to build the similarity graph\n",
    "def build_similarity_graph(sentences):\n",
    "    # Create an empty similarity matrix\n",
    "    S = np.zeros((len(sentences), len(sentences)))\n",
    "\n",
    "    # Populate the similarity matrix\n",
    "    for idx1, sentence1 in enumerate(sentences):\n",
    "        for idx2, sentence2 in enumerate(sentences):\n",
    "            if idx1 == idx2:\n",
    "                continue\n",
    "            S[idx1][idx2] = sentence_similarity(sentence1, sentence2)\n",
    "\n",
    "    # Convert the similarity matrix to a graph\n",
    "    graph = nx.from_numpy_array(S)\n",
    "    return graph\n",
    "\n",
    "# Define function to rank sentences using LexRank\n",
    "def lexrank_summarization(text, num_sentences=5):\n",
    "    # Preprocess the text\n",
    "    sentences = preprocess_text(text)\n",
    "\n",
    "    # Build the graph\n",
    "    graph = build_similarity_graph(sentences)\n",
    "\n",
    "    # Use the pagerank algorithm to rank sentences\n",
    "    scores = nx.pagerank(graph, max_iter=1000, tol=1e-06)\n",
    "    ranked_sentences = sorted(((scores[i], s) for i, s in enumerate(sentences)), reverse=True)\n",
    "\n",
    "    # Extract top N sentences as the summary\n",
    "    summary = \" \".join([s[1] for s in ranked_sentences[:num_sentences]])\n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0360e7ad",
   "metadata": {},
   "source": [
    "Now we will choose a PDF file and apply the summarization algorithm to extract the key points from the text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "380750a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      " 1993 Max Weber conceptualization charismatic authority inﬂuence organizational research Leadership Quarterly Vol Findings Weber writings charismatic authority continue instrumental shaping modern leadership theory charismatic form authority may particularly applicable effective today chaotic rapidly changing environments empowered organizational forms century may represent merely different incarnation Weber iron cage authority short propose Weber writings charismatic authority continue instrumental shaping modern leadership theory charismatic form authority may particularly applicable effective today chaotic rapidly changing environments empowered organizational forms century may simply represent different embodiment Weber iron cage authority Weber forwarded three basic types authority traditional charismatic Weber 1968 Indeed concepts along aspects Weber theory charismatic authority recently prompted lively debate among leadership scholars within pages Leadership Quarterly Bass 1999 Downloaded Queen Mary University London 02 January 2019 PT JMH 452 Beyer 1999 House 1999 Shamir 1999\n"
     ]
    }
   ],
   "source": [
    "# Prompt the user to select a PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# Extract text from the PDF\n",
    "text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Generate summary\n",
    "summary = lexrank_summarization(text)\n",
    "\n",
    "# Print the summary\n",
    "print(\"Summary:\\n\", summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dbdf959",
   "metadata": {},
   "source": [
    "## Extractive Summerisation Method: Latent Semantic Analysis (LSA)\n",
    "\n",
    "Next we will demostrate how to summarise a PDF document using the Latent Semantic Analysis (LSA) approach. LSA is based on the singular value decomposition (SVD) of a term-sentence matrix to reduce its dimensions and thus identify patterns that represent the underlying \"latent\" structure of the semantic relationships within the text. By mapping the high-dimensional space of terms into a lower-dimensional space, LSA can infer the importance of sentences based on the concepts they contain, even if they do not share specific keywords.\n",
    "\n",
    "The summarization effect of LSA is to identify and extract sentences that carry the essence of the topics within the text. These sentences may not necessarily be the most frequently occurring or the most interconnected but rather those that best capture the main themes and variations in topic within the document.\n",
    "\n",
    "LSA is effective for complex texts with sophisticated structures, such as academic literature, research papers, and technical documents, where simple word frequency is insufficient to understand the importance of sentences. It is particularly useful for summarizing texts that require the identification of thematic importance and where synonymy and polysemy (words that have multiple meanings) are prevalent.\n",
    "\n",
    "### Step 1: Import Necessary Libraries\n",
    "\n",
    "Let's begin by importing all necessary libraries. Install them using pip if they are not already installed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b8b36f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pdfminer.six numpy scipy scikit-learn nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02767713",
   "metadata": {},
   "source": [
    "### Step 2: Extract and Preprocess Text\n",
    "\n",
    "We need to preprocess the text by tokenizing sentences, removing stop words, and filtering non-alphanumeric characters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "36f51253",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# Importing necessary libraries\n",
    "import pdfminer\n",
    "from pdfminer.high_level import extract_text\n",
    "import numpy as np\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer\n",
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "nltk.download('punkt') # Download the tokenizer model if not already downloaded\n",
    "nltk.download('stopwords')\n",
    "\n",
    "# Define function to extract text from PDF\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    return extract_text(pdf_path)\n",
    "\n",
    "# Define function to preprocess text\n",
    "def preprocess_text(text):\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    sentences = sent_tokenize(text)\n",
    "    preprocessed_sentences = []\n",
    "    for sentence in sentences:\n",
    "        words = word_tokenize(sentence)\n",
    "        filtered_words = [word for word in words if word.lower() not in stop_words and word.isalnum()]\n",
    "        preprocessed_sentences.append(' '.join(filtered_words))\n",
    "    return preprocessed_sentences\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cac90d14",
   "metadata": {},
   "source": [
    "### Step 3: Implement LSA for Summarization\n",
    "\n",
    "LSA is used for summarization by applying dimensionality reduction to the term-sentence matrix and then extracting the sentences that contribute most to the resulting components.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2427b254",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define function to perform LSA summarization\n",
    "def lsa_summarization(sentences, num_topics=1, num_sentences=5):\n",
    "    # Create a TfidfVectorizer for vectorization of the sentences\n",
    "    vectorizer = TfidfVectorizer()\n",
    "    X = vectorizer.fit_transform(sentences)\n",
    "\n",
    "    # Perform SVD to reduce dimensionality\n",
    "    svd_model = TruncatedSVD(n_components=num_topics)\n",
    "    lsa = make_pipeline(svd_model, Normalizer(copy=False))\n",
    "    X_lsa = lsa.fit_transform(X)\n",
    "\n",
    "    # Rank sentences based on the weight in the first topic\n",
    "    ranked_sentences = [(sentence, X_lsa[index]) for index, sentence in enumerate(sentences)]\n",
    "    ranked_sentences = sorted(ranked_sentences, key=lambda x: x[1], reverse=True)\n",
    "\n",
    "    # Extract top N sentences as the summary\n",
    "    summary = \" \".join([sentence for sentence, weight in ranked_sentences[:num_sentences]])\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82b3e16d",
   "metadata": {},
   "source": [
    "### Step 4: Display the Summary\n",
    "\n",
    "Finally, we display the summarized text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "78304ab0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LSA Summary:\n",
      " current issue full text archive journal available Max Weber notion authority still hold century Jeffery Houghton College Business Economics West Virginia University Morgantown West Virginia USA Max Weber notion authority 449 Abstract Purpose purpose brief commentary provide brief overview Max Weber life work contributions management thought addressing question whether notion authority still holds century commentary begins brief biographical sketch followed examination Weber conceptualization authority inﬂuence ﬁeld management relevancy century Findings Weber writings charismatic authority continue instrumental shaping modern leadership theory charismatic form authority may particularly applicable effective today chaotic rapidly changing environments empowered organizational forms century may represent merely different incarnation Weber iron cage authority commentary makes important contribution management history literature examining important aspect Weber inﬂuence management thought theory practice\n"
     ]
    }
   ],
   "source": [
    "# Prompt the user to select a PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Preprocess the text to be summarized\n",
    "preprocessed_text = preprocess_text(text)\n",
    "\n",
    "# Generate summary\n",
    "summary = lsa_summarization(preprocessed_text)\n",
    "\n",
    "# Generate summary\n",
    "print(\"LSA Summary:\\n\", summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad8f668b",
   "metadata": {},
   "source": [
    "## Extractive Summerisation Method: Luhn Algorithm\n",
    "Luhn Algorithm algorithm is based on the frequency of words within the text; it assumes that words occurring more frequently are more significant. Luhn's insight was that there is a middle ground of word frequency that captures keywords: very common words are uninformative, and very rare words may be irrelevant. Furthermore, the Luhn algorithm pays attention to the proximity of these significant words to each other within a sentence, proposing that clusters of significant words are likely to convey more information.\n",
    "\n",
    "The summarization effect is that sentences that contain a higher density of these mid-frequency significant words, especially where they occur in close proximity, are selected for the summary. The resulting summary is therefore a collection of sentences that are rich in content-bearing words, which Luhn suggested represent the main points of the text.\n",
    "\n",
    " The Luhn algorithm is most effective for texts where the significant content can be identified through keyword frequency and distribution, such as news reports and business and technical papers. It works particularly well with texts that have a good signal-to-noise ratio in terms of keyword frequencies: significant words should stand out from the less important text while still being frequently enough used to indicate central themes.\n",
    "\n",
    "### Step 1: Import Necessary Libraries\n",
    "\n",
    "Before we begin, let's import all the necessary libraries. If you haven't installed these libraries, you can do so using pip.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c0a59c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pdfminer.six nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d2038a8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pdfminer\n",
    "from pdfminer.high_level import extract_text\n",
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from nltk.probability import FreqDist\n",
    "from nltk.corpus import stopwords\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6f9a775",
   "metadata": {},
   "source": [
    "### Step 2: Extract Text from PDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "41fdd1ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text_from_pdf(pdf_path):\n",
    "    return extract_text(pdf_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "087801d2",
   "metadata": {},
   "source": [
    "### Step 3: Preprocess the Text\n",
    "We define a function to preprocess text by tokenizing sentences, removing stop words, and filtering non-alphanumeric characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1ab556fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_text(text):\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    words = word_tokenize(text)\n",
    "    filtered_words = [word for word in words if word.lower() not in stop_words and word.isalnum()]\n",
    "    \n",
    "    return ' '.join(filtered_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc5e7606",
   "metadata": {},
   "source": [
    "### Step 4: Implement the Luhn Algorithm for Summarization\n",
    "The Luhn Algorithm scores sentences based on the frequency of significant words. Sentences with the highest scores are included in the summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "23723dae",
   "metadata": {},
   "outputs": [],
   "source": [
    "def luhn_summarization(text, num_sentences=5):\n",
    "    sentences = sent_tokenize(text)\n",
    "    preprocessed_text = preprocess_text(text)\n",
    "    word_frequencies = FreqDist(word_tokenize(preprocessed_text))\n",
    "    \n",
    "    # Compute the higher frequency threshold using average frequency\n",
    "    avg_frequency = sum(word_frequencies.values()) / len(word_frequencies)\n",
    "    significant_words = {word for word in word_frequencies if word_frequencies[word] > avg_frequency}\n",
    "    \n",
    "    ranked_sentences = {}\n",
    "    for i, sentence in enumerate(sentences):\n",
    "        words = word_tokenize(sentence.lower())\n",
    "        word_count = len(words)\n",
    "        score = sum([word_frequencies[word] for word in words if word in significant_words]) / word_count\n",
    "        ranked_sentences[i] = score\n",
    "    \n",
    "    selected_sentences = sorted(ranked_sentences, key=ranked_sentences.get, reverse=True)[:num_sentences]\n",
    "    summary = ' '.join([sentences[i] for i in selected_sentences])\n",
    "    \n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b364687",
   "metadata": {},
   "source": [
    "### Step 5: Display the Summary\n",
    "Finally, we display the summarized text after prompting the user to select a PDF file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "14ca8086",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Luhn Summary:\n",
      " The ﬁrst is traditional authority. KEYWORDS authority,  bureaucratic  authority,  charisma,  dominance,  leadership,\n",
      "traditional authority, Weber\n",
      "\n",
      "Max Weber’s longstanding interest in the notion of authority is well documented. This authority is based upon strong traditional rules and has\n",
      "much  in  common  with  legal  authority. The charismatic leader should\n",
      "and often does have these traits. The charismatic leader is a d¨amonischer type who\n",
      "appears  only  in  chaotic  times.\n"
     ]
    }
   ],
   "source": [
    "# Prompt the user to select a PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# Extract text from the PDF\n",
    "text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Generate summary\n",
    "summary = luhn_summarization(text)\n",
    "\n",
    "# Print the summary\n",
    "print(\"Luhn Summary:\\n\", summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93df519a",
   "metadata": {},
   "source": [
    "## Extractive Summerisation Method: KL-Sum\n",
    "KL-Sum algorithm is based on minimizing the Kullback-Leibler (KL) divergence, which is a way of measuring the difference between two probability distributions. For text summarization, KL-Sum aims to select sentences that, when taken together, provide a probability distribution of words as close as possible to the distribution in the original document. It iteratively adds sentences to the summary that most decrease the KL divergence.\n",
    "\n",
    "The effect of the KL-Sum algorithm is to create a summary that maintains the original distribution of words, which presumably represents the topics and the information content of the entire document. This tends to produce summaries that are representative of the original text's thematic structure.\n",
    "\n",
    "KL-Sum is effective for texts with distinct keyword distributions that are indicative of the text's content, such as scientific articles and technical documents. It can be especially useful in domains where the goal is to capture the key information without imposing much interpretation or paraphrasing, thus maintaining the document's original terminology and meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c61cedcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install numpy scipy nltk pymupdf nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb4d3498",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install --upgrade pymupdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c5c7ee3c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\wasim\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "import numpy as np\n",
    "from scipy.special import kl_div\n",
    "from scipy.stats import entropy\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import nltk\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from collections import Counter\n",
    "import fitz  \n",
    "\n",
    "# Ensure that you have the necessary NLTK models downloaded\n",
    "nltk.download('punkt')\n",
    "\n",
    "# Function to extract text from a PDF file\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    # Open the PDF file\n",
    "    with fitz.open(pdf_path) as pdf:\n",
    "        text = \"\"\n",
    "        # Iterate over each page in the PDF\n",
    "        for page in pdf:\n",
    "            # Extract text from the page\n",
    "            text += page.get_text()\n",
    "        return text\n",
    "    \n",
    "# Function to extract sentences from a string of text\n",
    "def get_sentences(text):\n",
    "    return sent_tokenize(text)\n",
    "\n",
    "# Function to create the word frequency distribution\n",
    "def word_freq_dist(text):\n",
    "    words = word_tokenize(text.lower())\n",
    "    return Counter(words)\n",
    "\n",
    "# Function to calculate the KL divergence\n",
    "# Function to calculate the KL divergence\n",
    "def kl_divergence(summary_freq_dist, doc_freq_dist, vocab_size):\n",
    "    \"\"\"\n",
    "    Calculate the KL divergence, adding a small value (1/vocab_size) to zero-counts to avoid infinity.\n",
    "    \"\"\"\n",
    "    P = np.array([summary_freq_dist.get(word, 0) + 1.0/vocab_size for word in doc_freq_dist])\n",
    "    Q = np.array([doc_freq_dist.get(word, 0) + 1.0/vocab_size for word in doc_freq_dist])\n",
    "    \n",
    "    return entropy(P, Q)\n",
    "\n",
    "# Function to perform KL-Sum summarization\n",
    "# Function to perform KL-Sum summarization\n",
    "def kl_sum(text, summary_size=5):\n",
    "    sentences = get_sentences(text)\n",
    "    vectorizer = CountVectorizer()\n",
    "    vectorizer.fit(sentences)\n",
    "    vocab = vectorizer.get_feature_names_out()\n",
    "    \n",
    "    doc_freq_dist = word_freq_dist(text)\n",
    "    \n",
    "    summary = []\n",
    "    summary_freq_dist = Counter()\n",
    "    remaining_sentences = sentences.copy()\n",
    "    \n",
    "    vocab_size = len(vocab)  # V is the size of the vocabulary\n",
    "    \n",
    "    while len(summary) < summary_size and remaining_sentences:\n",
    "        kl_scores = []\n",
    "        for sentence in remaining_sentences:\n",
    "            temp_summary = ' '.join(summary + [sentence])\n",
    "            temp_summary_freq_dist = word_freq_dist(temp_summary)\n",
    "            kl_score = kl_divergence(temp_summary_freq_dist, doc_freq_dist, vocab_size)\n",
    "            kl_scores.append((kl_score, sentence))\n",
    "            \n",
    "        # Select the sentence that minimizes the KL divergence\n",
    "        min_kl_sentence = min(kl_scores, key=lambda x: x[0])[1]\n",
    "        summary.append(min_kl_sentence)\n",
    "        summary_freq_dist.update(word_freq_dist(min_kl_sentence))\n",
    "        remaining_sentences.remove(min_kl_sentence)\n",
    "    \n",
    "    return ' '.join(summary)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e46a6718",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Luhn Summary:\n",
      " 408-37. (1993), “Max Weber’s conceptualization of charismatic authority: its inﬂuence\n",
      "on organizational research”, Leadership Quarterly, Vol. Findings – Weber’s writings on charismatic authority have been and continue to be instrumental\n",
      "in shaping modern leadership theory, that the charismatic form of authority may be particularly applicable\n",
      "and effective in today’s chaotic and rapidly changing environments, and that the empowered and\n",
      "self-managing organizational forms of the twenty-ﬁrst century may represent merely a different incarnation\n",
      "of Weber’s iron cage of legal/rational authority. Indeed, these concepts along with other\n",
      "aspects of Weber’s theory of charismatic authority recently prompted a lively debate\n",
      "among leadership scholars within the pages of Leadership Quarterly (Bass, 1999;\n",
      "Max Weber’s\n",
      "notion of\n",
      "authority\n",
      "451\n",
      "Downloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)\n",
      "Beyer, 1999; House, 1999; Shamir, 1999). Jeffery D. Houghton\n",
      "College of Business and Economics, West Virginia University,\n",
      "Morgantown, West Virginia, USA\n",
      "Abstract\n",
      "Purpose – The purpose of this brief commentary is to provide a brief overview of Max Weber’s life,\n",
      "work, and contributions to management thought before addressing the question of whether his notion\n",
      "of authority still holds in the twenty-ﬁrst century.\n"
     ]
    }
   ],
   "source": [
    "# Prompt the user to select a PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# Extract text from the PDF\n",
    "text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Generate summary\n",
    "summary = kl_sum(text)\n",
    "\n",
    "# Print the summary\n",
    "print(\"Luhn Summary:\\n\", summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f1c557e",
   "metadata": {},
   "source": [
    "## Extractive Summerisation Method: K-Mean Clustering\n",
    "This method breaks the document into smaller chunks, such as paragraphs, and then generates vector embeddings for each chunk. These embeddings capture the semantic meaning of the text in a multi-dimensional vector space, enabling the algorithm to measure the \"distance\" between chunks in terms of content and meaning. Paragraphs that are semantically similar will cluster together in this space. A clustering algorithm like K-means is then used to identify these clusters. The algorithm determines the central points of these clusters, which are the chunks that best represent the \"average meaning\" of the topics within each cluster.\n",
    "\n",
    "The approach essentially distills the document down to its key thematic elements. It identifies the main topics discussed throughout and selects the most representative sections for each topic. By combining the central chunks, a summary is created that is not only concise but also rich in context, reflecting the various key topics of the document.\n",
    "\n",
    "This method is versatile and can be effective across a range of text types. It is particularly useful for long documents such as reports, research papers, and lengthy articles where there are distinct sections or chapters. Since it relies on semantic understanding, it can handle complex materials where themes and ideas are more important than just individual keywords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd30ca09",
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install numpy scikit-learn gensim pdfplumber \"unstructured[pdf]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12046226",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#pip install \"unstructured[all-docs]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31049317",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pdfplumber\n",
    "\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    text = ''\n",
    "    with pdfplumber.open(pdf_path) as pdf:\n",
    "        for page in pdf.pages:\n",
    "            # Extract text\n",
    "            current_page_text = page.extract_text()\n",
    "            if current_page_text:  # Check if the text was extracted\n",
    "                text += current_page_text + \"\\n\\n\"  # Adding a double newline as a paragraph separator\n",
    "    return text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a62e3610",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Prompt the user to select a PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# After extracting text from the PDF\n",
    "text = extract_text_from_pdf(pdf_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75193509",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "169cb26c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from PyPDF2 import PdfReader\n",
    "\n",
    "# Function to extract text from a PDF\n",
    "def extract_text_from_pdf(pdf_path):\n",
    "    with open(pdf_path, 'rb') as file:\n",
    "        reader = PdfReader(file)\n",
    "        text = ''\n",
    "        for page in reader.pages:\n",
    "            text += page.extract_text()\n",
    "        return text\n",
    "\n",
    "# Summarization function\n",
    "def KMean_cluster_summerisation(document):\n",
    "    # Check the length of paragraphs after the split\n",
    "    paragraphs = document.split('\\n\\n')  # Adjust this line as needed based on the actual paragraph breaks\n",
    "    print(\"Number of paragraphs:\", len(paragraphs))\n",
    "\n",
    "    if len(paragraphs) < 2:\n",
    "        return \"Document is too short to summarize using clustering.\"\n",
    "\n",
    "    tagged_data = [TaggedDocument(words=par.split(), tags=[str(i)]) for i, par in enumerate(paragraphs)]\n",
    "    model = Doc2Vec(vector_size=50, min_count=2, epochs=40)\n",
    "    model.build_vocab(tagged_data)\n",
    "    model.train(tagged_data, total_examples=model.corpus_count, epochs=model.epochs)\n",
    "    embeddings = [model.infer_vector(par.split()) for par in paragraphs]\n",
    "    optimal_k = min(len(paragraphs), 10)\n",
    "    clustering_model = KMeans(n_clusters=optimal_k)\n",
    "    clustering_model.fit(embeddings)\n",
    "    cluster_assignment = clustering_model.labels_\n",
    "    centroids = clustering_model.cluster_centers_\n",
    "    representative_paragraphs = []\n",
    "\n",
    "    for i in range(optimal_k):\n",
    "        centroid = centroids[i]\n",
    "        distances = np.linalg.norm(embeddings - centroid, axis=1)\n",
    "        cluster_paragraphs = [par for par, cluster in zip(paragraphs, cluster_assignment) if cluster == i]\n",
    "        representative_paragraphs.append(cluster_paragraphs[np.argmin(distances)])\n",
    "    sorted_representative_paragraphs = sorted(representative_paragraphs, key=lambda par: paragraphs.index(par))\n",
    "    summary = '\\n\\n'.join(sorted_representative_paragraphs)\n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fec21e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate summary\n",
    "summary = KMean_cluster_summerisation(text)\n",
    "\n",
    "# Print the summary\n",
    "print(\"K-Mean Clustering Summary:\\n\", summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "822fa290",
   "metadata": {},
   "source": [
    "## Abstractive Summerisation\n",
    "Abstractive summarization is based on the principle of understanding and interpreting the content like a human would, and then expressing that understanding in a new, condensed form. This process involves paraphrasing and rephrasing the essence of the text rather than simply extracting sentences directly from it. Abstractive summarization systems must first comprehend the text on a deeper level than just identifying keywords or phrases. They use natural language processing (NLP) techniques to understand the context, nuances, and the relationships between concepts in the text.\n",
    "\n",
    "After understanding the text, the system generates new sentences that capture the core meanings and the most important information from the original content. This generation phase often employs advanced NLP techniques such as natural language generation (NLG) and machine learning models, especially sequence-to-sequence models.\n",
    "\n",
    "The generated sentences are then condensed to form a coherent summary. The summary should convey the main points of the original text but with fewer words and potentially different phrasing. The goal is to produce a shorter version of the text that retains the essential information and is coherent and fluent to read.\n",
    "\n",
    "The summarization process must ensure that the summary is not only concise but also logically structured and understandable, with transitions that make sense and maintain the flow of information.\n",
    "\n",
    "Abstractive summarization techniques are particularly useful when dealing with complex texts that require a high level of interpretation, such as news articles, stories, or even conversations where the context and the subtleties are crucial. This approach can also be effective for summarizing content where an extractive summary might be too fragmented or when a paraphrased, reworded summary would be more useful to the reader.\n",
    "\n",
    "Advanced abstractive summarization systems often leverage deep learning models like transformers, which have the ability to generate human-like text. These systems are trained on large datasets to learn patterns in language usage, enabling them to mimic the way humans summarize information. However, abstractive summarization is computationally intensive and can be prone to inaccuracies or loss of certain nuances, making it an ongoing area of research and development in the field of artificial intelligence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ddc74dd0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (4.34.1)\n",
      "Requirement already satisfied: filelock in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (3.12.4)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (0.17.3)\n",
      "Requirement already satisfied: numpy>=1.17 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (1.24.3)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (23.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (6.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (2023.8.8)\n",
      "Requirement already satisfied: requests in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (2.28.2)\n",
      "Requirement already satisfied: tokenizers<0.15,>=0.14 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (0.14.1)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (0.4.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from transformers) (4.66.1)\n",
      "Requirement already satisfied: fsspec in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from huggingface-hub<1.0,>=0.16.4->transformers) (2023.10.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from huggingface-hub<1.0,>=0.16.4->transformers) (4.5.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from tqdm>=4.27->transformers) (0.4.6)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (3.0.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests->transformers) (2022.12.7)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.2.1 -> 23.3.1\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "#!pip install transformers pymupdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a233c8b",
   "metadata": {},
   "source": [
    "## Abstractive Summerisation Method: Bart\n",
    "\n",
    "Bart uses a standard seq2seq/machine translation architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). The pretraining task involves randomly shuffling the order of the original sentences and a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE.\n",
    "### Importing Bart Model and Tokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "926398d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BartTokenizer, BartForConditionalGeneration\n",
    "import fitz  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "92174f48",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text_from_pdf(pdf_path):\n",
    "    # Open the PDF file\n",
    "    with fitz.open(pdf_path) as pdf:\n",
    "        text = \"\"\n",
    "        for page in pdf:\n",
    "            text += page.get_text()\n",
    "    return text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8500a17e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dde026f1e8549ec897f478d1c9f96c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/1.63G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ff118d730dd464db5a0431c3b9085f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)neration_config.json:   0%|          | 0.00/363 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')\n",
    "model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cf8be6b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def summarize_with_bart(text, max_length=1024, min_length=100):\n",
    "    inputs = tokenizer([text], max_length=max_length, return_tensors='pt', truncation=True)\n",
    "    \n",
    "    # Generate Summary\n",
    "    summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=max_length, min_length=min_length, early_stopping=True)\n",
    "    summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)\n",
    "    \n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2e675417",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      " Jeffery D. Houghton: Does Max Weber’s notion of authority still hold in the twenty-ﬁrst century? He says Weber's writings on charismatic authority have been and continue to be instrumental in shaping modern leadership theory. He argues that the empowered and self-managing organizational forms of the 20- ﬁRst century may represent merely a different incarnation of Weber's iron cage of legal/rational authority. The commentary makes an important contribution to the management history and management literature by examining an important aspect of his inﬂuence on management thought.\n"
     ]
    }
   ],
   "source": [
    "# Assume pdf_path is the path to your PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# Extract text from the PDF\n",
    "extracted_text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Summarize the extracted text\n",
    "summary = summarize_with_bart(extracted_text)\n",
    "\n",
    "# Print the summary\n",
    "print(\"Summary:\\n\", summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0de4491e",
   "metadata": {},
   "source": [
    "## Abstractive Summerisation Method: PEGASUS Model\n",
    "\n",
    "The next summerisation model we will investigate is the PEGASUS model from Hugging Face's Transformers library. PEGASUS is a state-of-the-art model for abstractive text summarization that can generate coherent and concise summaries.\n",
    "\n",
    "### Step 1: Install Required Libraries\n",
    "\n",
    "First, we need to install the transformers library and a library to read PDF documents, such as pdfplumber.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fffa9ce3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pdfplumber in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (0.10.3)\n",
      "Requirement already satisfied: pdfminer.six==20221105 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pdfplumber) (20221105)\n",
      "Requirement already satisfied: Pillow>=9.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pdfplumber) (9.4.0)\n",
      "Requirement already satisfied: pypdfium2>=4.18.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pdfplumber) (4.23.1)\n",
      "Requirement already satisfied: charset-normalizer>=2.0.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pdfminer.six==20221105->pdfplumber) (3.0.1)\n",
      "Requirement already satisfied: cryptography>=36.0.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pdfminer.six==20221105->pdfplumber) (41.0.5)\n",
      "Requirement already satisfied: cffi>=1.12 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from cryptography>=36.0.0->pdfminer.six==20221105->pdfplumber) (1.15.1)\n",
      "Requirement already satisfied: pycparser in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six==20221105->pdfplumber) (2.21)\n"
     ]
    }
   ],
   "source": [
    "#!pip install transformers tokenizers pdfplumber"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6362dbf1",
   "metadata": {},
   "source": [
    "### Step 2: Import Libraries\n",
    "\n",
    "Import the necessary libraries for the summarization process.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "145d2491",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pdfplumber\n",
    "from transformers import AutoTokenizer, PegasusForConditionalGeneration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d98575a",
   "metadata": {},
   "source": [
    "### Step 3: PDF Text Extraction\n",
    "\n",
    "Define a function to extract text from a PDF file using pdfplumber, which provides accurate text extraction and keeps the textual flow intact.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "709e6ccd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text_from_pdf(pdf_path):\n",
    "    with pdfplumber.open(pdf_path) as pdf:\n",
    "        text = ''\n",
    "        for page in pdf.pages:\n",
    "            text += page.extract_text()\n",
    "    return text\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c432e0f",
   "metadata": {},
   "source": [
    "### Step 4: Load PEGASUS Model and Tokenizer\n",
    "\n",
    "Load the PEGASUS model and its corresponding tokenizer. We will use the 'google/pegasus-xsum' pre-trained model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1dd947f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of PegasusForConditionalGeneration were not initialized from the model checkpoint at google/pegasus-xsum and are newly initialized: ['model.decoder.embed_positions.weight', 'model.encoder.embed_positions.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model_name = 'google/pegasus-xsum'\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = PegasusForConditionalGeneration.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b47f8fc",
   "metadata": {},
   "source": [
    "### Step 5: Summarize Text with PEGASUS\n",
    "\n",
    "Summarize the text using the PEGASUS model with the encoded text as input. Adjust the parameters like `num_beams` and `length_penalty` for different summarization quality.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "5d0968fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to summarize text\n",
    "def summarize_with_pegasus(text):\n",
    "    model = PegasusForConditionalGeneration.from_pretrained(model_name)\n",
    "    batch = tokenizer(text, truncation=True, padding='longest', return_tensors=\"pt\")\n",
    "    translated = model.generate(**batch)\n",
    "    return tokenizer.batch_decode(translated, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59cde8b2",
   "metadata": {},
   "source": [
    "### Step 6: Run the Summary\n",
    "\n",
    "Specify the path to your PDF document, extract the text, encode it, and then generate the summary using PEGASUS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "698a25f6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of PegasusForConditionalGeneration were not initialized from the model checkpoint at google/pegasus-cnn_dailymail and are newly initialized: ['model.decoder.embed_positions.weight', 'model.encoder.embed_positions.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      " Weber’s writings on charismatic authority have been and continue to be instrumental in shaping modern leadership.<n>The empowered and self-managing organizationalformsofthe twenty-first centurymayrepresentmerelyadifferentincarnation ofWeber’sironcageoflegal/rationalauthority.\n"
     ]
    }
   ],
   "source": [
    "# Assume pdf_path is the path to your PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# Extract text from the PDF\n",
    "extracted_text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Summarize the extracted text\n",
    "summary_texts = summarize_with_pegasus(extracted_text)\n",
    "\n",
    "# Print the summaries\n",
    "for summary in summary_texts:\n",
    "    print(\"Summary:\\n\", summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac072f8a",
   "metadata": {},
   "source": [
    "## Abstractive Summerisation Method: T5 Transformer \n",
    "\n",
    "Now we will test the summerisation capabilities of T5 model from Hugging Face's Transformers library. T5 stands for \"Text-to-Text Transfer Transformer\" and is a versatile model that frames all NLP tasks as a text-to-text problem. For summarization, it generates a concise version of a given input text.\n",
    "\n",
    "### Step 1: Install Required Libraries\n",
    "\n",
    "Begin by installing the `transformers` library, which provides the T5 model and utilities, and `pdfminer.six` for extracting text from PDFs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "118186a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install transformers pdfminer.six"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef3732a4",
   "metadata": {},
   "source": [
    "### Step 2: Import Libraries\n",
    "\n",
    "Now import the necessary libraries for the summarization process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "7184a2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import T5Tokenizer, T5ForConditionalGeneration\n",
    "from pdfminer.high_level import extract_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85c0d2c1",
   "metadata": {},
   "source": [
    "### Step 3: PDF Text Extraction\n",
    "\n",
    "We'll use `pdfminer.six` to extract text from PDF documents. This library allows us to convert PDF files to text, which can then be summarized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "c054a341",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text_from_pdf(pdf_path):\n",
    "    text = extract_text(pdf_path)\n",
    "    return text\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1aab8a8",
   "metadata": {},
   "source": [
    "### Step 4: Load T5 Model and Tokenizer\n",
    "\n",
    "Load the pre-trained T5 model and its corresponding tokenizer. We will use the 't5-small' model for this example, but larger models are available for better performance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "e3e4e285",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a3751b33e454048933dc001fe64541d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)okenizer_config.json:   0%|          | 0.00/2.32k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdf8f174f7244f4a872680d506aa6725",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)ve/main/spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69f481501bf540c39ca68a6bf8642842",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/1.39M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "022f91b813f94296a61d5e12b697628f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/1.21k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65c7024d90b04b75a4f54eb08ba87989",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading model.safetensors:   0%|          | 0.00/242M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be47f0ead91f4048b94946d7b84339ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)neration_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_name = 't5-small'\n",
    "tokenizer = T5Tokenizer.from_pretrained(model_name)\n",
    "model = T5ForConditionalGeneration.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10388c74",
   "metadata": {},
   "source": [
    "### Step 5: Prepare Text for T5\n",
    "\n",
    "T5 requires the task to be specified as part of the input text. For summarization, we prepend \"summarize: \" to the input text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "d47be1ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_text_for_t5(text, tokenizer, max_length=512):\n",
    "    preprocessed_text = \"summarize: \" + text\n",
    "    return tokenizer.encode(preprocessed_text, return_tensors=\"pt\", max_length=max_length, truncation=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8609e9e2",
   "metadata": {},
   "source": [
    "### Step 6: Summarize Text with T5\n",
    "\n",
    "We generate a summary by decoding the tokens produced by the T5 model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "dfbdfbc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def summarize_with_t5(encoded_text, model):\n",
    "    summary_ids = model.generate(encoded_text, min_length=30, max_length=200, length_penalty=2.0, num_beams=4, early_stopping=True)\n",
    "    summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffde72d3",
   "metadata": {},
   "source": [
    "### Step 7: Run the Summary\n",
    "\n",
    "Finally, we use the functions to extract text from a PDF, prepare it for T5, and generate the summary.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "1e49188e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "T5 Summary:\n",
      " the current issue and full text archive of this journal is available at www.emeraldinsight.com/1751-1348.htm Max Weber’s notion of authority 449 Abstract Purpose – The purpose of this brief commentary is to provide a brief overview of Weber’s life, work, and contributions to management thought.\n"
     ]
    }
   ],
   "source": [
    "# Example PDF file\n",
    "pdf_path = choose_file()\n",
    "\n",
    "# Extract text from the PDF\n",
    "text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Encode and prepare the text for T5\n",
    "encoded_text = prepare_text_for_t5(text, tokenizer)\n",
    "\n",
    "# Generate the summary\n",
    "summary = summarize_with_t5(encoded_text, model)\n",
    "\n",
    "# Print the summary\n",
    "print(\"T5 Summary:\\n\", summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "febf0a0b",
   "metadata": {},
   "source": [
    "## Abstractive Summerisation Method: OpenAI's GPT-3.5 and Langchain\n",
    "OpenAI's GPT (generative pre-trained transformer) models have been trained to understand natural language and code. GPTs provide text outputs in response to their inputs. The inputs to GPTs are also referred to as \"prompts\". Designing a prompt is essentially how you “program” a GPT model, usually by providing instructions or some examples of how to successfully complete a task.\n",
    "\n",
    "### Step 1: Setup and Authentication\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "45b22ed3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: PyPDF2 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (3.0.1)\n"
     ]
    }
   ],
   "source": [
    "#!pip install PyPDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "cb6149ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: openai in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (0.27.10)\n",
      "Requirement already satisfied: langchain in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (0.0.281)\n",
      "Requirement already satisfied: requests>=2.20 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from openai) (2.28.2)\n",
      "Requirement already satisfied: tqdm in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from openai) (4.66.1)\n",
      "Requirement already satisfied: aiohttp in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from openai) (3.8.5)\n",
      "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (6.0)\n",
      "Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (2.0.20)\n",
      "Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (0.5.14)\n",
      "Requirement already satisfied: langsmith<0.1.0,>=0.0.21 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (0.0.33)\n",
      "Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (2.8.5)\n",
      "Requirement already satisfied: numpy<2,>=1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (1.24.3)\n",
      "Requirement already satisfied: pydantic<3,>=1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (1.10.12)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchain) (8.2.3)\n",
      "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (22.2.0)\n",
      "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (3.0.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (6.0.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (4.0.3)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from aiohttp->openai) (1.3.1)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (3.20.1)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (0.9.0)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pydantic<3,>=1->langchain) (4.5.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests>=2.20->openai) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests>=2.20->openai) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests>=2.20->openai) (2022.12.7)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from SQLAlchemy<3,>=1.4->langchain) (2.0.1)\n",
      "Requirement already satisfied: colorama in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from tqdm->openai) (0.4.6)\n",
      "Requirement already satisfied: packaging>=17.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (23.0)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (1.0.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install openai langchain langchainhub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "9cbdc192",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "1684142b",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['OPENAI_API_KEY'] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "12d6b069",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_key = os.getenv('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "2c44db7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import ast\n",
    "import pandas as pd\n",
    "from langchain import LLMChain\n",
    "from langchain.chat_models import ChatAnthropic\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.chains.mapreduce import MapReduceChain\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.chains import (\n",
    "                StuffDocumentsChain,\n",
    "                LLMChain,\n",
    "                ReduceDocumentsChain,\n",
    "                MapReduceDocumentsChain,\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "8bd77a08",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    # Set a really small chunk size, just to show.\n",
    "    chunk_size = 1000,\n",
    "    chunk_overlap  = 200,\n",
    "    length_function = len,\n",
    "    is_separator_regex = False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "fc51aa8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "pdf_path = choose_file()\n",
    "loader = PyPDFLoader(pdf_path)\n",
    "pages = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "e7f69611",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = text_splitter.split_documents(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "b4e32820",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Does Max Weber’s notion\\nof authority still hold\\nin the twenty-ﬁrst century?\\nJeffery D. Houghton\\nCollege of Business and Economics, West Virginia University,\\nMorgantown, West Virginia, USA\\nAbstract\\nPurpose – The purpose of this brief commentary is to provide a brief overview of Max Weber’s life,\\nwork, and contributions to management thought before addressing the question of whether his notionof authority still holds in the twenty-ﬁrst century.\\nDesign/methodology/approach – The commentary begins with a brief biographical sketch followed\\nby an examination of Weber’s conceptualization of authority, its inﬂuence on the ﬁeld of management andits relevancy in the twenty-ﬁrst century.\\nFindings – Weber’s writings on charismatic authority have been and continue to be instrumental', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 0}),\n",
       " Document(page_content='Findings – Weber’s writings on charismatic authority have been and continue to be instrumental\\nin shaping modern leadership theory, that the charismatic form of authority may be particularly applicableand effective in today’s chaotic and rapidly changing environments, and that the empowered andself-managing organizational forms of the twenty-ﬁrst century may represent merely a different incarnation\\nof Weber’s iron cage of legal/rational authority.\\nOriginality/value – This commentary makes an important contribution to the management history\\nliterature by examining an important aspect of Weber’s inﬂuence on management thought, theory,\\nand practice.\\nKeywords Management theory, Authority, Charisma, Leadership\\nPaper type Conceptual paper\\nThe writings of Max Weber have had a profound and perhaps even unrivaled\\ninﬂuence on management thought and organizational theory over the past century\\n(Greenwood and Lawrence, 2005). Recently, however, some organizational theorists', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 0}),\n",
       " Document(page_content='inﬂuence on management thought and organizational theory over the past century\\n(Greenwood and Lawrence, 2005). Recently, however, some organizational theorists\\nhave questioned the current relevancy of Weber’s theories in today’s late-modernknowledge-based information age characterized by a very different set of economic,social, and technological realities from the time in which Weber’s ideas were born(Greenwood and Lawrence, 2005; Lounsbury and Carberry, 2005). A complete\\nexamination of the enduring inﬂuence of the entire breadth of Weber’s writings in the\\ncurrent context is well beyond the scope of this brief commentary. However, after a briefbiographical sketch and overview of Weber’s work, I will focus speciﬁcally on Weber’sconceptualization of authority and its relevancy in the twenty-ﬁrst century. In short, I will\\nsuggest that Weber’s writings on authority are still material in modern organizations and\\nare still helping to shape the thinking of today’s management scholars.', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 0}),\n",
       " Document(page_content='suggest that Weber’s writings on authority are still material in modern organizations and\\nare still helping to shape the thinking of today’s management scholars.\\nMax Weber was born in 1864 in Erfurt, Germany, the oldest of eight children. Weber\\nstudied law at the University of Heidelberg, but his educational experiences andsubsequent academic career would span a remarkably broad number of disciplines\\nincluding law, history, economics, philosophy, political science, and sociology. After\\ncompleting his doctoral dissertation and habilitation (the highest level of academicqualiﬁcation in certain European countries), Weber obtained his ﬁrst universityThe current issue and full text archive of this journal is available at\\nwww.emeraldinsight.com/1751-1348.htm\\nMax Weber’s\\nnotion of\\nauthority\\n449\\nJournal of Management History\\nVol. 16 No. 4, 2010\\npp. 449-453\\nqEmerald Group Publishing Limited\\n1751-1348\\nDOI 10.1108/17511341011073933', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 0}),\n",
       " Document(page_content='Max Weber’s\\nnotion of\\nauthority\\n449\\nJournal of Management History\\nVol. 16 No. 4, 2010\\npp. 449-453\\nqEmerald Group Publishing Limited\\n1751-1348\\nDOI 10.1108/17511341011073933\\nDownloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 0}),\n",
       " Document(page_content='appointment in 1892 at the University of Freiburg. He would also hold professorships at\\nthe University of Heidelberg, the University of Vienna, and the University of\\nMunich during an academic career marked by periods of intense writing productivitybut punctuated with bouts of neurosis resulting in periods of scholarly inactivity\\nand long leaves of absence from any teaching responsibilities. As Greenwood and\\nLawrence (2005) have noted, it is doubtful that Weber could have followed such a careerpath in today’s intense “publish or perish” academic culture. In 1904, after a ﬁve-year\\nperiod during which he published virtually nothing, Weber began publishing some of\\nhis most inﬂuential essays. These essays were later collected to comprise his mostinﬂuential book, The Protestant Ethic and the Spirit of Capitalism , which included\\nWeber’s famous metaphor of the iron cage (Weber, 1930). Subsequently, Weber\\npresented his fully developed ideas on bureaucracy, political leadership, domination,', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 1}),\n",
       " Document(page_content='Weber’s famous metaphor of the iron cage (Weber, 1930). Subsequently, Weber\\npresented his fully developed ideas on bureaucracy, political leadership, domination,\\nand authority in his three-volume masterpiece, Economy and Society (Weber, 1968),\\nwhich was ﬁrst published posthumously in 1922. Weber had died of pneumonia in 1920after contracting Spanish inﬂuenza. The inﬂuence of Weber’s writings on management\\nthought following his death was slowed somewhat by a lack of English translations of\\nhis work. Although, The Protestant Ethic was ﬁrst translated into English in 1930,\\nWeber’s essays on bureaucracy and authority were not widely available until the late\\n1940s. Yet despite the slow start, Weber’s inﬂuence on management and organizational\\ntheory in the middle decades of the twentieth century was tremendous (Greenwood andLawrence, 2005). Interested readers may refer to Ka ¨sler (1979) for a more detailed\\ndiscussion of Weber’s life and work.', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 1}),\n",
       " Document(page_content='discussion of Weber’s life and work.\\nWithout question, Max Weber’s ideas have had a broad and far-reaching inﬂuence on\\nthe development of the ﬁelds of management and organizational theory and his writings\\non authority certainly rank among his most inﬂuential. Weber forwarded three basictypes of authority: traditional, rational-legal, and charismatic (Weber, 1968). Traditional\\nauthority, such as that of tribal chiefs, feudal lords, and monarchs, is based on customs\\nand traditions that are passed down from one generation to the next. Rational-legalauthority, in contrast, is founded upon laws, rules, and the power stemming from a\\nlegitimate position or ofﬁce. Weber felt that the bureaucracy was a primary example of\\nrational-legal authority. Finally, charismatic authority results from extraordinarypersonal characteristics of a leader that have the capacity to inspire others.\\nWeber saw these three types of authority essentially as forces that would bring either', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 1}),\n",
       " Document(page_content='Weber saw these three types of authority essentially as forces that would bring either\\nstability and order (traditional/legal rational) or change and disorder (charismatic) to\\ninstitutions and society (Conger, 1993). Thus, for Weber, traditional authority was\\nviewed as stable, impersonal, and nonrational; rational/legal authority was seen asstable, impersonal, and rational; and charismatic authority was considered unstable,\\npersonal, and nonrational (Blau, 1963; Conger, 1993). Weber painted a distinct contrast\\nbetween the act of following a personal yet transitory charismatic leader as opposed tosubmitting to the more stable and impersonal traditional and rational/legal forms\\nof authority. He further saw both legal/rational and charismatic authority as forms of\\nrebellion against the stagnant status quo of traditional authority, the former throughprinciples and procedures based on consensus and rationality and the latter through an', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 1}),\n",
       " Document(page_content='rebellion against the stagnant status quo of traditional authority, the former throughprinciples and procedures based on consensus and rationality and the latter through an\\nemotional reaction to a heroic leader. Weber also suggested that charismatic authority is\\ninherently transitory and unstable, and is therefore most effective in times of crisis and\\nchange, serving primarily to facilitate the transition from one order to another (Conger,\\n1993). Having accomplished this transition, charismatic authority is either “routinized”JMH\\n16,4\\n450\\nDownloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 1}),\n",
       " Document(page_content='or simply fades away as charismatic leadership is replaced by the rules, tradition,\\nand institutionalized bureaucracy. Hence, the irony of charismatic authority is that it is\\noften replaced by the very forms of authority that it originally sought to overturn(Conger, 1993).\\nWeber developed his theory of authority in the early twentieth century and in that\\ncontext his ideas make a great deal of sense. The industrial age was creating larger andmore complex bureaucratic organizations and Weber observed that traditional authority\\nstructures were being replaced by legal/rational-based authority systems often aided by\\nlarger-than-life charismatic leaders whose entrepreneurial vision and energy were being\\ntransformed into the great corporations of the twentieth century. Weber, however, lived and\\nwrote in a simpler and arguably less dynamic context in which pace of change was slower\\nand less frenetic than today (Greenwood and Lawrence, 2005). Modern organizations are', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 2}),\n",
       " Document(page_content='wrote in a simpler and arguably less dynamic context in which pace of change was slower\\nand less frenetic than today (Greenwood and Lawrence, 2005). Modern organizations are\\nincreasingly characterized by new, decentralized network-based structures that are quite\\ndifferent from the large, complex bureaucracies of Weber’s day. Today, knowledge-basedwork and cutting-edge technologies are creating new organizational realities centered on\\nconcepts such as telecommuting, empowerment, and self-managing teams that may\\npotentially undermine the traditional and legal/rational forms of authority. Given these neworganizational forms and practices, does Weber’s notion of authority still hold in\\ntwenty-ﬁrst century? In the remainder of this commentary, I will suggest that Weber’s\\nideas on authority are just as relevant today as they were 100 years ago, but in differentways and for different reasons. In short, I will propose that Weber’s writings on charismatic', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 2}),\n",
       " Document(page_content='ideas on authority are just as relevant today as they were 100 years ago, but in differentways and for different reasons. In short, I will propose that Weber’s writings on charismatic\\nauthority have been and continue to be instrumental in shaping modern leadership theory,\\nthat the charismatic form of authority may be particularly applicable and effective in\\ntoday’s chaotic and rapidly changing environments, and that the empowered and\\nself-managing organizational forms of the twenty-ﬁrst century may simply represent adifferent embodiment of Weber’s iron cage of legal/rational authority.\\nWhen Weber redeﬁned the term “charisma” from its original ecclesiastical meaning\\nof “divinely bestowed power or talent” to mean “a special quality of an individualcapable of inspiring and inﬂuencing others,” he laid the foundation for the concept of\\ncharismatic leadership. As Conger (1993, p. 277) points out, Weber “is essentially the', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 2}),\n",
       " Document(page_content='charismatic leadership. As Conger (1993, p. 277) points out, Weber “is essentially the\\n“father of the ﬁeld” – responsible for the introduction of the concept as both a lay andscientiﬁc term”. Thus, beginning in the 1970s and continuing to the present, Weber’s\\nwritings on charismatic authority have served as the conceptual basis for the\\ndevelopment of theoretical models of charismatic leadership as well as for empirical\\nresearch on the subject (Conger, 1988, 1993). Moreover, Weber’s writings have done\\nmore than serve as the seminal works for one of the most popular concepts in modernleadership theory; they have also continued to move the ﬁeld forward as various nuances\\nof charismatic leadership have been identiﬁed and explored. For instance, nearly two\\ndecades ago following the initial development of the charismatic leadership theory,Conger (1993) called for researchers to examine two previously neglected aspects of', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 2}),\n",
       " Document(page_content='decades ago following the initial development of the charismatic leadership theory,Conger (1993) called for researchers to examine two previously neglected aspects of\\nWeber’s theory: the routinization of charismatic leadership and the role of context in\\ncharismatic leadership. In response, researchers have carefully examined the extent towhich the emergence and maintenance of charismatic leadership depends on the\\npresence of a dynamic context and/or crisis situation (Bligh et al. , 2004; de Hoogh et al. ,\\n2005; Pillai, 1996; Shamir and Howell, 1999). Indeed, these concepts along with other\\naspects of Weber’s theory of charismatic authority recently prompted a lively debate\\namong leadership scholars within the pages of Leadership Quarterly (Bass, 1999;Max Weber’s\\nnotion of\\nauthority\\n451\\nDownloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 2}),\n",
       " Document(page_content='Beyer, 1999; House, 1999; Shamir, 1999). The point is that Max Weber’s ideas are still\\ninforming and inspiring research, debate, and theory building in one of the most popularareas of organizational research today.\\nLeadership theorists have generally suggested that times of stress, turbulence, and\\nrapid change are more conducive to a charismatic leadership approach because thetransforming vision of a charismatic leader is more appealing in times of uncertainly(Bryman, 1993; Conger, 1999). Weber (1968) himself focused speciﬁcally on times of crisis\\nas a primary facilitating environment for charismatic authority. It seems reasonable then\\nto suggest that charismatic leadership may have even greater applicability and relevancein today’s turbulent and rapidly changing organizational environments than the conceptof charismatic authority did in Weber’s day.\\nFinally, as Barker (1993) suggested in his highly inﬂuential ethnographic study, new', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 3}),\n",
       " Document(page_content='Finally, as Barker (1993) suggested in his highly inﬂuential ethnographic study, new\\nparticipatory organizational forms and practices such as self-managing teams andemployee empowerment may not represent an escape from the iron cage of legal/rationalauthority and bureaucratic control. Quite to the contrary, these practices may simplyrepresent a shift in the locus of control from managers in traditional bureaucratic\\nstructures to the workers themselves (Barker, 1993). Workers become accountable totheir teammates and to themselves rather than to a relatively distant manager. Ironically\\nthen, empowerment and self-management practices may serve to tighten Weber’s iron\\ncage of rational control as organizational members effective police themselves and theirco-workers more closely than would be possible in the strictest bureaucracy. Once again,Weber’s notion of authority informs our modern understanding complex organizationalphenomena.', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 3}),\n",
       " Document(page_content='In conclusion, Hunt (1999, p. 129) has suggested that the development of charismatic\\nleadership was in part responsible for rejuvenating the study of leadership by creating“a paradigm shift that has attracted numerous new scholars and moved the ﬁeld as awhole out of its doldrums”. If this is true, then the ﬁeld of leadership was to a large extentrejuvenated by the inﬂuence of Max Weber as leadership scholars looked to the ideas of\\nthe past to create the leadership theory and practice of the present and future. Clearly,\\nMax Weber’s writings have been shaping the thinking of management scholars for morethan a century and his inﬂuence will likely continue into the foreseeable future.\\nReferences\\nBarker, J.R. (1993), “Tightening the iron cage: concertive control in self-managing teams”,\\nAdministrative Science Quarterly , Vol. 38, pp. 408-37.\\nBass, B.M. (1999), “On the taming of charisma: a reply to Janice Beyer”, Leadership Quarterly ,\\nVol. 10, pp. 541-53.', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 3}),\n",
       " Document(page_content='Administrative Science Quarterly , Vol. 38, pp. 408-37.\\nBass, B.M. (1999), “On the taming of charisma: a reply to Janice Beyer”, Leadership Quarterly ,\\nVol. 10, pp. 541-53.\\nBeyer, J.M. (1999), “Taming and promoting charisma to change organizations”, Leadership\\nQuarterly , Vol. 10, pp. 307-31.\\nBlau, P.M. (1963), “Critical remarks on Weber’s theory of authority”, American Political Science\\nReview , Vol. 57, pp. 308-9.\\nBligh, M.C., Kohles, J.C. and Meindl, J.R. (2004), “Charisma under crisis: presidential leadership\\nrhetoric, and media responses before and after the September 11th terrorist attacks”,\\nLeadership Quarterly , Vol. 15, pp. 211-40.\\nBryman, A. (1993), “Charismatic leadership in business organizations: some neglected issues”,\\nLeadership Quarterly , Vol. 4, pp. 289-304.JMH\\n16,4\\n452\\nDownloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 3}),\n",
       " Document(page_content='Conger, J.A. (1988), “Theoretical foundations of charismatic leadership”, in Conger, J.A. and\\nKanungo, R. (Eds), Charismatic Leadership: The Elusive Factor in Organizational\\nEffectiveness , Jossey-Bass, San Francisco, CA, pp. 12-39.\\nConger, J.A. (1993), “Max Weber’s conceptualization of charismatic authority: its inﬂuence\\non organizational research”, Leadership Quarterly , Vol. 4, pp. 277-88.\\nConger, J.A. (1999), “Charismatic and transformational leadership in organizations: an insider’s\\nperspective on these developing streams of research”, Leadership Quarterly , Vol. 10,\\npp. 145-70.\\nde Hoogh, A.H.B., den Hartog, D.N. and Koopman, P.L. (2005), “Linking the big ﬁve-factors\\nof personality to charismatic and transactional leadership: perceived dynamic workenvironment as a moderator”, Journal of Organizational Behavior , Vol. 26, pp. 839-65.\\nGreenwood, R. and Lawrence, T.B. (2005), “The iron cage in the information age: the legacy and', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 4}),\n",
       " Document(page_content='Greenwood, R. and Lawrence, T.B. (2005), “The iron cage in the information age: the legacy and\\nrelevance of Max Weber for organizational studies”, Organizational Studies , Vol. 26,\\npp. 493-9.\\nHouse, R.J. (1999), “Weber and the neo-charismatic leadership paradigm: a response to Beyer”,\\nLeadership Quarterly , Vol. 10, pp. 563-75.\\nHunt, J.G. (1999), “Transformational/charismatic leadership’s transformation of the ﬁeld:\\nan historical essay”, Leadership Quarterly , Vol. 10, pp. 129-45.\\nKa¨sler, D. (1979), Max Weber: An Introduction to His Life and Work , University of Chicago Press,\\nChicago, IL.\\nLounsbury, M. and Carberry, E.J. (2005), “From king to court jester? Weber’s fall from grace\\nin organizational theory”, Organizational Studies , Vol. 26, pp. 501-25.\\nPillai, R. (1996), “Crisis and the emergence of charismatic leadership in groups: an experimental\\ninvestigation”, Journal of Applied Social Psychology , Vol. 26, pp. 543-62.', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 4}),\n",
       " Document(page_content='Pillai, R. (1996), “Crisis and the emergence of charismatic leadership in groups: an experimental\\ninvestigation”, Journal of Applied Social Psychology , Vol. 26, pp. 543-62.\\nShamir, B. (1999), “Taming charisma for better understanding and greater usefulness: a response\\nto Beyer”, Leadership Quarterly , Vol. 10, pp. 555-63.\\nShamir, B. and Howell, J.M. (1999), “Organizational and contextual inﬂuences on the emergence\\nand effectiveness of charismatic leadership”, Leadership Quarterly , Vol. 10, pp. 257-84.\\nWeber, M. (1930), The Protestant Ethic and the Spirit of Capitalism , Unwin University Books,\\nLondon (translated Parsons, T.).\\nWeber, M. (1968) in Roth, G. and Wittich, C. (Eds), Economy and Society: An Outline of\\nInterpretive Sociology , Bedminstrer Press, New York, NY.\\nAbout the author\\nJeffery D. Houghton is an Associate Professor of Management and Director of the Master of', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 4}),\n",
       " Document(page_content='Interpretive Sociology , Bedminstrer Press, New York, NY.\\nAbout the author\\nJeffery D. Houghton is an Associate Professor of Management and Director of the Master of\\nScience in Human Resources and Industrial Relations program at West Virginia University.He has presented his research at various professional meetings and has published numerousarticles in a variety of respected journals. He holds a PhD in Organizational Studies from\\nVirginia Polytechnic Institute and State University. Jeffery D. Houghton can be contacted at:\\njeff.houghton@mail.wvu.eduMax Weber’s\\nnotion of\\nauthority\\n453\\nTo purchase reprints of this article please e-mail: reprints@emeraldinsight.com\\nOr visit our web site for further details: www.emeraldinsight.com/reprints\\nDownloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 4}),\n",
       " Document(page_content='This article has been cited by:\\n1.Kate Maclean. 2017. Disarming charisma? Mayoralty, gender and power in Medellín, Colombia. Political\\nGeography  59, 126-135. [ Crossref ]\\n2.Edna Rabenu. Positive Psychological Capital: From Strengths to Power 81-105. [ Crossref ]\\n3.Colleen Schwarz. 2015. A review of management history from 2010-2014 utilizing a thematic analysis\\napproach. Journal of Management History  21:4, 494-504. [ Abstract ] [Full Text ] [PDF]\\n4.Abdullah F. Alrebh. 2015. Covering the Building of a Kingdom: The Saudi Arabian Authority in The\\nLondon Times and The New York Times , 1901-1932. Digest of Middle East Studies  24:2, 187-212.\\n[Crossref ]\\n5.Sukumarakurup Krishnakumar, Jeffery D. Houghton, Christopher P . Neck, Christopher N. Ellison. 2015.\\nThe “good” and the “bad” of spiritual leadership. Journal of Management, Spirituality & Religion  12:1,\\n17-37. [ Crossref ]\\n6.Miguel Pereira Lopes. 2014. Rebuilding Lisbon in the aftermath of the 1755 earthquake. Journal of', metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 5}),\n",
       " Document(page_content=\"17-37. [ Crossref ]\\n6.Miguel Pereira Lopes. 2014. Rebuilding Lisbon in the aftermath of the 1755 earthquake. Journal of\\nManagement History  20:3, 278-291. [ Abstract ] [Full Text ] [PDF]\\n7.Ivana Milosevic, A. Erin Bass. 2014. Revisiting Weber's charismatic leadership: learning from the past\\nand looking to the future. Journal of Management History  20:2, 224-240. [ Abstract ] [Full Text ] [PDF]\\nDownloaded by Queen Mary University of London At 07:14 02 January 2019 (PT)\", metadata={'source': 'D:/Downloads/Houghton (2010) Does Max Webers Notion of Authority Still Hold in the Twenty-First Century.pdf', 'page': 5})]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "5b09f439",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reduce\n",
    "reduce_template = \"\"\"The following is set of summaries:\n",
    "{doc_summaries}\n",
    "Take these and distill it into a final, consolidated summary of the main themes. \n",
    "Helpful Answer:\"\"\"\n",
    "reduce_prompt = PromptTemplate.from_template(reduce_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "3581fb33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting langchainhub\n",
      "  Downloading langchainhub-0.1.13-py3-none-any.whl.metadata (478 bytes)\n",
      "Requirement already satisfied: requests<3,>=2 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchainhub) (2.28.2)\n",
      "Requirement already satisfied: types-requests<3.0.0.0,>=2.31.0.2 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from langchainhub) (2.31.0.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests<3,>=2->langchainhub) (3.0.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests<3,>=2->langchainhub) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests<3,>=2->langchainhub) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from requests<3,>=2->langchainhub) (2022.12.7)\n",
      "Requirement already satisfied: types-urllib3 in c:\\users\\wasim\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from types-requests<3.0.0.0,>=2.31.0.2->langchainhub) (1.26.25.14)\n",
      "Downloading langchainhub-0.1.13-py3-none-any.whl (3.4 kB)\n",
      "Installing collected packages: langchainhub\n",
      "Successfully installed langchainhub-0.1.13\n"
     ]
    }
   ],
   "source": [
    "!pip install langchainhub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "6a93ed85",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "# Note we can also get this from the prompt hub, as noted above\n",
    "reduce_prompt = hub.pull(\"rlm/map-prompt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "b089dc6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatPromptTemplate(input_variables=['docs'], output_parser=None, partial_variables={}, messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['docs'], output_parser=None, partial_variables={}, template='The following is a set of documents:\\n{docs}\\nBased on this list of docs, please identify the main themes \\nHelpful Answer:', template_format='f-string', validate_template=True), additional_kwargs={})])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduce_prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "8b9fe060",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run chain\n",
    "reduce_chain = LLMChain(llm=llm, prompt=reduce_prompt)\n",
    "\n",
    "# Takes a list of documents, combines them into a single string, and passes this to an LLMChain\n",
    "combine_documents_chain = StuffDocumentsChain(\n",
    "    llm_chain=reduce_chain, document_variable_name=\"docs\"\n",
    ")\n",
    "\n",
    "# Combines and iteravely reduces the mapped documents\n",
    "reduce_documents_chain = ReduceDocumentsChain(\n",
    "    # This is final chain that is called.\n",
    "    combine_documents_chain=combine_documents_chain,\n",
    "    # If documents exceed context for `StuffDocumentsChain`\n",
    "    collapse_documents_chain=combine_documents_chain,\n",
    "    # The maximum number of tokens to group documents into.\n",
    "    token_max=4000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "c43a9bdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combining documents by mapping a chain over them, then combining results\n",
    "map_reduce_chain = MapReduceDocumentsChain(\n",
    "    # Map chain\n",
    "    llm_chain=map_chain,\n",
    "    # Reduce chain\n",
    "    reduce_documents_chain=reduce_documents_chain,\n",
    "    # The variable name in the llm_chain to put the documents in\n",
    "    document_variable_name=\"docs\",\n",
    "    # Return the results of the map steps in the output\n",
    "    return_intermediate_steps=False,\n",
    ")\n",
    "\n",
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=1000, chunk_overlap=0\n",
    ")\n",
    "split_docs = text_splitter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "fc27cb3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the list of documents provided, the main themes that can be identified are:\n",
      "\n",
      "1. The historical context and impact of the 1755 Lisbon earthquake: The documents likely discuss the devastation caused by the earthquake and the subsequent efforts to rebuild and recover.\n",
      "\n",
      "2. Lessons from history for contemporary leadership: The documents likely explore the concept of charismatic leadership and its relevance in modern times, drawing on Weber's theories and applying them to current leadership practices.\n",
      "\n",
      "Overall, the main themes revolve around historical events and their implications for leadership and management.\n"
     ]
    }
   ],
   "source": [
    "print(map_reduce_chain.run(split_docs))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}