Initial commit

Author: devnich

.gitignore

@@ -0,0 +1,30 @@
+# Generated documentation and backups
+
+*.odt
+*.docx
+*.pdf
+*.html
+*.~lock.*
+*.bak
+tmp.md
+
+# Binaries
+*.xlsx
+
+# System files
+.DS_Store
+
+# R environmental variables
+.Rhistory
+.Rapp.history
+.RData
+.Ruserdata
+.Renviron
+/*.Rcheck/
+.Rproj.user/
+
+# Data folders
+data/*.sqlite
+
+# Archives
+*.zip

README.org

@@ -0,0 +1,342 @@
+#+STARTUP: fold indent
+#+OPTIONS: tex:t toc:2 H:6 ^:{}
+
+#+TITLE: Databases and SQL for Data Scientists
+#+AUTHOR: Derek Devnich
+#+BEGIN_SRC sql
+#+END_SRC
+#+BEGIN_SRC bash
+#+END_SRC
+* COMMENT SQL interaction
+1. Start SQLite inferior process
+   ~M-x sql-sqlite~
+
+2. Set SQL dialect for syntax highlighting
+   ~M-x sql-set-product~
+   ~sqlite~
+
+* Introducing databases and SQL: Why use a database?
+** Performance
+
+** Correctness
+There are two aspects of "correctness": Enforcing consistency and eliminating ambiguity. A database enforces consistency with a combination of data types, rules (e.g., foreign keys, triggers, etc.), and atomic transactions. It eliminates ambiguity by forbidding NULLs.
+
+1. You can represent simple data in a single table
+   [[file:images/animals.png]]
+
+2. The single table breaks down when your data is complex
+   [[file:images/animals_blob.png]]
+
+   If you use a nested representation, the individual table cells are no longer atomic. The tool for query, search, or perform analyses rely on the atomic structure of the table, and they break down when the cell contents are complex.
+
+3. Complex data with duplicate row
+   [[file:images/animals_dup.png]]
+
+   - Storing redundant information has storage costs
+   - Redundant rows violate the Don't Repeat Yourself [DRY] principle. Every copy is an opportunity to introduce errors or inconsistencies into the data.
+   - Storing multidimensional data in a single table increases the chance that your records will have NULL fields, which will complicate future queries (more on this later)
+
+4. Solution: Normalize the data by breaking it into multiple tables
+   [[file:images/animals_half.png]] [[file:images/sightings_half.png]]
+
+   - Every row of every table contains unique information
+   - Normalization is a continuum. We could normalize this data further, but there is a trade-off in terms of sane table management. Finding the correct trade-off is a matter of taste, judgment, and domain-specific knowledge.
+
+** Encode Domain Knowledge
+[[file:images/bank_account_schema.jpg]]
+
+- Encodes shape of domain
+- Embeds domain rules: e.g. cannot have a customer transaction without a customer account
+- Rules provide additional layer of correctness in the form of constraints
+- note that forbidding NULL seems much more reasonable in this context!
+
+** Extensions
+- Functions
+- Data types (GIS, JSON, date/time, searchable document, currency…)
+- Full-text search
+
+* Accessing data with queries
+** Basic queries
+1. Select everything from a table
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM surveys;
+   #+END_SRC
+
+2. Select a column
+   #+BEGIN_SRC sql
+   SELECT year
+   FROM surveys;
+   #+END_SRC
+
+3. Select multiple columns
+   #+BEGIN_SRC sql
+   SELECT year, month, day
+   FROM surveys;
+   #+END_SRC
+
+4. Limit results
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM surveys
+   LIMIT 10;
+   #+END_SRC
+
+5. Get unique values
+   #+BEGIN_SRC sql
+   SELECT DISTINCT species_id
+   FROM surveys;
+   #+END_SRC
+
+   #+BEGIN_SRC sql
+   -- Return distinct pairs
+   SELECT DISTINCT year, species_id
+   FROM surveys;
+   #+END_SRC
+
+6. Calculate values
+   #+BEGIN_SRC sql
+   -- Convert kg to g
+   SELECT plot_id, species_id, weight/1000
+   FROM surveys;
+   #+END_SRC
+
+7. SQL databases have functions
+   #+BEGIN_SRC sql
+   SELECT plot_id, species_id, ROUND(weight/1000, 2)
+   FROM surveys;
+   #+END_SRC
+
+** Filtering
+1. Filter by a criterion
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM surveys
+   WHERE species_id='DM';
+   #+END_SRC
+
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM surveys
+   WHERE year >= 2000;
+   #+END_SRC
+
+2. Combine criteria with booleans
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM surveys
+   WHERE (year >= 2000) AND (species_id = 'DM');
+   #+END_SRC
+
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM surveys
+   WHERE (species_id = 'DM') OR (species_id = 'DO') OR (species_id = 'DS');
+   #+END_SRC
+
+** *Challenge 1*: Large bois
+Get all of the individuals in Plot 1 that weighed more than 75 grams, telling us the date, species id code, and weight (in kg).
+
+** Building complex queries
+Use sets ("tuples") to condense criteria.
+#+BEGIN_SRC sql
+SELECT *
+FROM surveys
+WHERE (year >= 2000) AND (species_id IN ('DM', 'DO', 'DS'));
+#+END_SRC
+
+** Sorting
+1. Sort by a column value
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM species
+   ORDER BY taxa ASC;
+   #+END_SRC
+
+2. Descending sort
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM species
+   ORDER BY taxa DESC;
+   #+END_SRC
+
+3. Nested sort
+   #+BEGIN_SRC sql
+   SELECT *
+   FROM species
+   ORDER BY genus ASC, species ASC;
+   #+END_SRC
+
+** *Challenge 2*
+Write a query that returns year, species_id, and weight in kg from the surveys table, sorted with the largest weights at the top.
+
+** Order of execution
+Queries are pipelines
+[[file:images/written_vs_execution_order.png]]
+
+* Aggregating and grouping data (i.e. reporting)
+** COUNT and GROUP BY
+1. The COUNT function
+   #+BEGIN_SRC sql
+   SELECT COUNT(*)
+   FROM surveys;
+   #+END_SRC
+
+    #+BEGIN_SRC sql
+   -- SELECT only returns the non-NULL weights
+   SELECT COUNT(weight), AVG(weight)
+   FROM surveys;
+   #+END_SRC
+
+2.
+** Ordering aggregated results
+** Aliases
+** The HAVING keyword
+** Saving queries for future use
+** NULL
+Go to slides, rather than extensively demo (do demo "is null")
+
+* Combining data with joins
+
+* Data hygiene
+** TODO The problem with nulls
+Missing data and deceptive query results
+
+** Data integrity constraints: Keys, not null, etc
+
+** TODO Levels of Normalization
+
+* Creating and modifying data
+** Insert statements
+** Create tables
+** Table contraints
+sqlite check command
+https://stackoverflow.com/questions/29476818/how-to-avoid-inserting-the-wrong-data-type-in-sqlite-tables
+https://www.sqlitetutorial.net/sqlite-check-constraint/
+** Atomic commits
+By default, each INSERT statement is its own transaction. But if you surround multiple INSERT statements with BEGIN...COMMIT then all the inserts are grouped into a single transaction. The time needed to commit the transaction is amortized over all the enclosed insert statements and so the time per insert statement is greatly reduced.
+
+* (Optional) SQLite on the command line
+** Basic commands
+#+BEGIN_SRC bash
+sqlite3     # enter sqlite prompt
+.tables     # show table names
+.schema     # show table schema
+.help       # view built-in commands
+.quit
+#+END_SRC
+
+** Getting output
+1. Formatted output in the terminal
+   #+BEGIN_SRC sql
+   .headers on
+   .help mode
+   .mode column
+   #+END_SRC
+
+   #+BEGIN_SRC sql
+   select * from species where taxa == 'Rodent';
+   #+END_SRC
+
+2. Output to .csv file
+   #+BEGIN_SRC bash
+   .mode csv
+   .output test.csv
+   #+END_SRC
+
+   #+BEGIN_SRC sql
+   select * from species where taxa == 'Rodent';
+   #+END_SRC
+
+   #+BEGIN_SRC bash
+   .output stdout
+   #+END_SRC
+
+* TODO (Optional) Database access via programming languages
+** R language bindings
+** Python language bindings
+
+* (Optional) What kind of data storage system do I need?
+** Non-atomic write; sequential read
+1. Files
+
+** Single atomic write (database-level lock); query-driven read
+1. SQLite
+2. Microsoft Access
+
+** Multiple atomic writes (row-level lock); query-driven read
+1. PostgreSQL: https://www.postgresql.org
+2. MySQL/MariaDB
+   - https://mariadb.org
+   - https://www.mysql.com
+3. Oracle
+4. Microsoft SQL Server
+5. ...etc.
+
+* (Optional) Performance tweaks and limitations
+** Getting the most out of your database
+1. Use recommended settings, not default settings
+2. Make judicious use of indexes
+3. Use the query planner (this will provide feedback for item 2)
+4. Cautiously de-normalize your schema
+
+** Where relational databases break down
+1. Very large data (hardware, bandwidth, and data integration problems)
+2. Distributed data (uncertainty about correctness)
+
+** Why are distributed systems hard?
+1. CAP theorem
+   - In theory, pick any two: Consistent, Available, Partition-Tolerant
+   - In practice, Consistent or Available in the presence of a Partition
+
+2. Levels of data consistency
+   - https://jepsen.io/consistency
+   - https://github.com/aphyr/distsys-class
+
+3. Fallacies of distributed computing
+   1. The network is reliable
+   2. Latency is zero
+   3. Bandwidth is infinite
+   4. The network is secure
+   5. Topology doesn't change
+   6. There is one administrator
+   7. Transport cost is zero
+   8. The network is homogeneous
+
+* *Endnotes*
+* Credits
+- Data management with SQL for ecologists: https://datacarpentry.org/sql-ecology-lesson/
+- Databases and SQL: http://swcarpentry.github.io/sql-novice-survey/ (data hygiene, creating and modifying data)
+- Simplified bank account schema: https://soft-builder.com/bank-management-system-database-model/
+- Botanical Information and Ecology Network schema: https://bien.nceas.ucsb.edu/bien/biendata/bien-3/bien-3-schema/
+
+* References
+- C. J. Date, /SQL and Relational Theory/: https://learning.oreilly.com/library/view/sql-and-relational/9781491941164/
+- Common database mistakes: https://stackoverflow.com/a/621891
+- Fallacies of distributed computing: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
+
+* Data Sources
+- Portal Project Teaching Database: https://figshare.com/articles/dataset/Portal_Project_Teaching_Database/1314459
+  Specifically, portal_mammals.sqlite: https://figshare.com/ndownloader/files/11188550
+
+* COMMENT Export to Markdown using Pandoc
+  Do this if you want code syntax highlighting and a table of contents on Github.
+** Generate generic Markdown file
+#+BEGIN_SRC bash
+pandoc README.org -o tmp.md --wrap=preserve
+#+END_SRC
+
+** Edit generic Markdown file to remove illegal front matter
+1. Org directives
+2. Anything that isn't part of the document structure (e.g. TODO items)
+
+** Generate Github Markdown with table of contents
+#+BEGIN_SRC bash
+pandoc -f markdown --toc --toc-depth=2 --wrap=preserve -s tmp.md -o README.md
+#+END_SRC
+
+** Find and replace code block markers in final document (if applicable)
+#+BEGIN_EXAMPLE
+M-x qrr " {.python}" "python"
+M-x qrr " {.bash}" "bash"
+#+END_EXAMPLE

_config.yml

@@ -0,0 +1,5 @@
+title: Databases and SQL for Data Scientists
+description: Class notes and code examples
+
+remote_theme: jekyll/minima
+#theme: jekyll-theme-slate