[Project_003]_Data_Science_Concepts.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "[Project 003] Data Science Concepts.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "3w5DoyKR0tXB",
        "gmOqXPgv08mb",
        "os5RIAop2NTg",
        "gGA1B0ja2kp4",
        "VKRTOatTpJ6N",
        "mxUoP4_zrKuH",
        "TSm_7BemtwqP",
        "ARBtOZdcx5z7",
        "6ElM77HxI7DM"
      ],
      "toc_visible": true,
      "authorship_tag": "ABX9TyM/RNthMj+Ol6P5shxmN3bi",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/id-shiv/knowledge_base/blob/master/%5BProject_003%5D_Data_Science_Concepts.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w5DoyKR0tXB",
        "colab_type": "text"
      },
      "source": [
        "# Data Science concepts"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gmOqXPgv08mb",
        "colab_type": "text"
      },
      "source": [
        "## Why?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHiiZEm51KCj",
        "colab_type": "text"
      },
      "source": [
        "- Huge amount of data generated.\n",
        "- Data could be structured or un-structured.\n",
        "\n",
        "Complex Algorithms (beyond traditional BI tools \\ algorithms) are required to process huge, unstructured, multi-source data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "os5RIAop2NTg",
        "colab_type": "text"
      },
      "source": [
        "## What?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nzei3-ps2PXI",
        "colab_type": "text"
      },
      "source": [
        "Process of extracting useful information from data using scientific methods.  \n",
        "Scientific methods : Programming + Statistics + Businesss.\n",
        "\n",
        "- Make better decisions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gGA1B0ja2kp4",
        "colab_type": "text"
      },
      "source": [
        "## Use Case"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vf24w7Nc2mcZ",
        "colab_type": "text"
      },
      "source": [
        "- Business : Retail (Walmart).\n",
        "- Problem : Find relationship between data.  \n",
        "e.g. : What products are liked by customers? (from social media data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VKRTOatTpJ6N",
        "colab_type": "text"
      },
      "source": [
        "## Roles"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W928Z_uNpLa7",
        "colab_type": "text"
      },
      "source": [
        "- Data Scientist : \n",
        "> Skills : Business + Mathematics (Statistics) + Technology (Programming).\n",
        "- Data Analyst :\n",
        "> Responsible for visualization, processing.  \n",
        "> Optimization of models.\n",
        "- Data Architect :\n",
        "> Data management blueprinting.\n",
        "- Data Engineers :\n",
        "> Update existing systems.\n",
        "- Staistician :\n",
        "> Creates methodologies, logic for problem solving.\n",
        "- DBA :\n",
        "> Administrator of Databases.\n",
        "- Business Analyst :\n",
        "> Focus on business goals.\n",
        "- Data & Analytics Manager :\n",
        "> Oversee data science operations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mxUoP4_zrKuH",
        "colab_type": "text"
      },
      "source": [
        "## Data Lifecycle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znQqMK3QrMr3",
        "colab_type": "text"
      },
      "source": [
        "- Business Requirement\n",
        "> Understand the problem.  \n",
        "> Parameters to be predicted.\n",
        "- Data acquisition\n",
        "> What data is needed?  \n",
        "> Where can i get it?  \n",
        "> Where do i store it?\n",
        "- Data processing\n",
        "> Transform.  \n",
        "> Clean.\n",
        "- Data exploration\n",
        "> Data analysis, brain-storming, find patterns.\n",
        "> Using visualizations.\n",
        "- Modelling\n",
        "> Model the prediction,  \n",
        "> Evaluate and  \n",
        "> Increase accurancy.\n",
        "- Deployment\n",
        "> Deploy to production environment. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-zqGbAtHsY0i",
        "colab_type": "text"
      },
      "source": [
        "# Statistics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TSm_7BemtwqP",
        "colab_type": "text"
      },
      "source": [
        "## Types of Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZxVn-3uksr97",
        "colab_type": "text"
      },
      "source": [
        "> Qualitative - cannot be measured\n",
        ">> Nominal : With no inherint order. e.g. : Gender.  \n",
        ">> Ordinal : Has Order. e.g. : Rating (Good, Bad, Average).  \n",
        "\n",
        "> Quantitative - can be measured\n",
        ">> Discrete : Categorical. e.g. : # of students.  \n",
        ">> Continous : Infinite. e.g. : Weight.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pIkdLksev1s0",
        "colab_type": "text"
      },
      "source": [
        "**Statistcs is an area of applied mathematics concerned with data collection, analysis, interpretation and presentation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DnMS7BeWwuTx",
        "colab_type": "text"
      },
      "source": [
        "**Population** : A collection whose properties are to be anlaysed.  \n",
        "**Sample** : Subset of **Population**.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ARBtOZdcx5z7",
        "colab_type": "text"
      },
      "source": [
        "## Sampling Techniques"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUDoUoNdyJI_",
        "colab_type": "text"
      },
      "source": [
        "Why Sampling? Method where a sample of population is analyzed to derive analysis of entire population, save time, cost with this."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xLeblxjIx8XM",
        "colab_type": "text"
      },
      "source": [
        "- Probability\n",
        "> Random : Each sample has equal chance of being selected.  \n",
        "> Systematic : Every nth record is selected.   \n",
        "> Stratified : Random sample of stratum (Stratum : samaple that shares at least one common property. e.g. : Gender).  \n",
        "- Non-probability\n",
        "> Snowball  \n",
        "> Quota  \n",
        "> Judgement    \n",
        "> Convenience   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pO94_hXE0eyg",
        "colab_type": "text"
      },
      "source": [
        "## Types of Statistics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWoNJO-P0htR",
        "colab_type": "text"
      },
      "source": [
        "- Descriptive : Provides description of population. e.g. Mean, Average, Minimum, etc.  \n",
        "- Inferential : Provides inferences based on samples of population. e.g. Large, Medium, Small."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L0nBwGHz2M1N",
        "colab_type": "text"
      },
      "source": [
        "### Descriptive Statistics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6JZ6x5v2Pt5",
        "colab_type": "text"
      },
      "source": [
        "- Measure of Centre :\n",
        "> Mean.  \n",
        "> Median.  \n",
        "> Mode : most reccurant.  \n",
        "- Measure of Spread \\ Dispersion : \n",
        "> Range.  \n",
        "> Inter Qurtile Range.  \n",
        "> Variance : How much differs from it expected value.  \n",
        "> Standard Deviation : Deviation between each element from the mean.   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ElM77HxI7DM",
        "colab_type": "text"
      },
      "source": [
        "#### Information Gain and Entropy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lMsNcQjXI-oj",
        "colab_type": "text"
      },
      "source": [
        "Used with Decision Trees or Random Forests.\n",
        "\n",
        "\n",
        "**Entropy** : Uncertinity in data.  \n",
        "**Information Gain** : How much a particular feature provides information at final outcome."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h4qhG50PMDlT",
        "colab_type": "text"
      },
      "source": [
        "#### Confusion Matrix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ZFxSBJ0MFaA",
        "colab_type": "text"
      },
      "source": [
        "Used for classification models. Finds accuracy of the model using the matrix - formula.  \n",
        "```\n",
        "Accuracy = (True +ve + True -ve) / (True +ve + True -ve + False +ve + False -ve)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t3IgeUToM1a9",
        "colab_type": "text"
      },
      "source": [
        "#### Probability"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tOLtrixM3l4",
        "colab_type": "text"
      },
      "source": [
        "Mathematical method used for statistical analysis.  \n",
        "Measure of likelihood of an event.\n",
        "```\n",
        "Probability = (desired outcomes) / (total outcomes)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yJkeS9RvO6AP",
        "colab_type": "text"
      },
      "source": [
        "##### Terminologies"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aBDHK5YqO7wm",
        "colab_type": "text"
      },
      "source": [
        "- Randon Experiment : An Experiment for which the outcome cannot be predicted with certainity.  \n",
        "- Sample Space : Entire possible set of possible outcomes.  \n",
        "- Event : One or more outcome (subset of sample space)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G9P0nI7GQK5F",
        "colab_type": "text"
      },
      "source": [
        "### Inferential Statistics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k6YIt3WZQOuK",
        "colab_type": "text"
      },
      "source": [
        "#### Point Estimation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-bOtQ7srQRzz",
        "colab_type": "text"
      },
      "source": [
        "Use of sample data - to serve unknown population paramter.  \n",
        "\n",
        "- Method of moments.  \n",
        "- Maximum of likelihood.\n",
        "- Bayes' Estimators.\n",
        "- Best Unbiased Estimators."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GPHHqnBOTbY1",
        "colab_type": "text"
      },
      "source": [
        "# Machine Learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_vT_TB1fUHzv",
        "colab_type": "text"
      },
      "source": [
        "https://github.com/id-shiv/knowledge_base/blob/master/%5BProject_001%5D_Machine_Learning_Concepts.ipynb"
      ]
    }
  ]
}