Merge branch 'master' into materialize-master

Author: benesch

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "sqlparser"
 description = "Extensible SQL Lexer and Parser with support for ANSI SQL:2011"
-version = "0.3.2-alpha.0"
+version = "0.4.1-alpha.0"
 authors = ["Andy Grove <[email protected]>"]
 homepage = "https://github.com/andygrove/sqlparser-rs"
 documentation = "https://docs.rs/sqlparser/"

README.md

@@ -16,16 +16,124 @@ At some point, if the parsers diverge enough, it may be worth jettisoning
 compatibility with upstream so that we can perform large-scale refactors, but we
 should make such a decision deliberately, not accidentally.
 
-## Design
+## Upstream overview
+
+The goal of this project is to build a SQL lexer and parser capable of parsing
+SQL that conforms with the [ANSI/ISO SQL standard][sql-standard] while also
+making it easy to support custom dialects so that this crate can be used as a
+foundation for vendor-specific parsers.
+
+This parser is currently being used by the [DataFusion] query engine and
+[LocustDB].
+
+## Example
+
+To parse a simple `SELECT` statement:
+
+```rust
+use sqlparser::dialect::GenericDialect;
+use sqlparser::parser::Parser;
+
+let sql = "SELECT a, b, 123, myfunc(b) \
+           FROM table_1 \
+           WHERE a > b AND b < 100 \
+           ORDER BY a DESC, b";
+
+let dialect = GenericDialect {}; // or AnsiDialect, or your own dialect ...
+
+let ast = Parser::parse_sql(&dialect, sql.to_string()).unwrap();
+
+println!("AST: {:?}", ast);
+```
+
+This outputs
+
+```rust
+AST: [Query(Query { ctes: [], body: Select(Select { distinct: false, projection: [UnnamedExpr(Identifier("a")), UnnamedExpr(Identifier("b")), UnnamedExpr(Value(Long(123))), UnnamedExpr(Function(Function { name: ObjectName(["myfunc"]), args: [Identifier("b")], over: None, distinct: false }))], from: [TableWithJoins { relation: Table { name: ObjectName(["table_1"]), alias: None, args: [], with_hints: [] }, joins: [] }], selection: Some(BinaryOp { left: BinaryOp { left: Identifier("a"), op: Gt, right: Identifier("b") }, op: And, right: BinaryOp { left: Identifier("b"), op: Lt, right: Value(Long(100)) } }), group_by: [], having: None }), order_by: [OrderByExpr { expr: Identifier("a"), asc: Some(false) }, OrderByExpr { expr: Identifier("b"), asc: None }], limit: None, offset: None, fetch: None })]
+```
+
+## SQL compliance
 
-*These design notes were copied from upstream.*
+SQL was first standardized in 1987, and revisions of the standard have been
+published regularly since. Most revisions have added significant new features to
+the language, and as a result no database claims to support the full breadth of
+features. This parser currently supports most of the SQL-92 syntax, plus some
+syntax from newer versions that have been explicitly requested, plus some MSSQL-
+and PostgreSQL-specific syntax. Whenever possible, the [online SQL:2011
+grammar][sql-2011-grammar] is used to guide what syntax to accept. (We will
+happily accept changes that conform to the SQL:2016 syntax as well, but that
+edition's grammar is not yet available online.)
 
-The parser is implemented using the [Pratt Parser](https://tdop.github.io/)
-design, which is a top-down operator-precedence parser.
+Unfortunately, stating anything more specific about compliance is difficult.
+There is no publicly available test suite that can assess compliance
+automatically, and doing so manually would strain the project's limited
+resources. Still, we are interested in eventually supporting the full SQL
+dialect, and we are slowly building out our own test suite.
 
-This approach has the following benefits over parser generators:
+If you are assessing whether this project will be suitable for your needs,
+you'll likely need to experimentally verify whether it supports the subset of
+SQL that you need. Please file issues about any unsupported queries that you
+discover. Doing so helps us prioritize support for the portions of the standard
+that are actually used. Note that if you urgently need support for a feature,
+you will likely need to write the implementation yourself. See the
+[Contributing](#Contributing) section for details.
+
+### Supporting custom SQL dialects
+
+This is a work in progress, but we have some notes on [writing a custom SQL
+parser](docs/custom_sql_parser.md).
+
+## Design
+
+The core expression parser uses the [Pratt Parser] design, which is a top-down
+operator-precedence (TDOP) parser, while the surrounding SQL statement parser is
+a traditional, hand-written recursive descent parser. Eli Bendersky has a good
+[tutorial on TDOP parsers][tdop-tutorial], if you are interested in learning
+more about the technique.
+
+We are a fan of this design pattern over parser generators for the following
+reasons:
 
 - Code is simple to write and can be concise and elegant
 - Performance is generally better than code generated by parser generators
 - Debugging is much easier with hand-written code
-- It is far easier to extend and make dialect-specific extensions compared to using a parser generator
+- It is far easier to extend and make dialect-specific extensions
+  compared to using a parser generator
+
+## Contributing
+
+Contributions are highly encouraged!
+
+Pull requests that add support for or fix a bug in a feature in the SQL
+standard, or a feature in a popular RDBMS, like Microsoft SQL Server or
+PostgreSQL, will almost certainly be accepted after a brief review. For
+particularly large or invasive changes, consider opening an issue first,
+especially if you are a first time contributor, so that you can coordinate with
+the maintainers. CI will ensure that your code passes `cargo test`,
+`cargo fmt`, and `cargo clippy`, so you will likely want to run all three
+commands locally before submitting your PR.
+
+If you are unable to submit a patch, feel free to file an issue instead. Please
+try to include:
+
+  * some representative examples of the syntax you wish to support or fix;
+  * the relevant bits of the [SQL grammar][sql-2011-grammar], if the syntax is
+    part of SQL:2011; and
+  * links to documentation for the feature for a few of the most popular
+    databases that support it.
+
+Please be aware that, while we strive to address bugs and review PRs quickly, we
+make no such guarantees for feature requests. If you need support for a feature,
+you will likely need to implement it yourself. Our goal as maintainers is to
+facilitate the integration of various features from various contributors, but
+not to provide the implementations ourselves, as we simply don't have the
+resources.
+
+[tdop-tutorial]: https://eli.thegreenplace.net/2010/01/02/top-down-operator-precedence-parsing
+[`cargo fmt`]: https://github.com/rust-lang/rustfmt#on-the-stable-toolchain
+[current issues]: https://github.com/andygrove/sqlparser-rs/issues
+[DataFusion]: https://github.com/apache/arrow/tree/master/rust/datafusion
+[LocustDB]: https://github.com/cswinter/LocustDB
+[Pratt Parser]: https://tdop.github.io/
+[sql-2011-grammar]: https://jakewheat.github.io/sql-overview/sql-2011-foundation-grammar.html
+[sql-standard]: https://en.wikipedia.org/wiki/ISO/IEC_9075

src/ast/data_type.rs

@@ -11,6 +11,7 @@
 // limitations under the License.
 
 use super::ObjectName;
+use std::fmt;
 
 /// SQL data types
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
@@ -65,47 +66,53 @@ pub enum DataType {
     Array(Box<DataType>),
 }
 
-impl ToString for DataType {
-    fn to_string(&self) -> String {
+impl fmt::Display for DataType {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match self {
-            DataType::Char(size) => format_type_with_optional_length("char", size),
-            DataType::Varchar(size) => format_type_with_optional_length("character varying", size),
-            DataType::Uuid => "uuid".to_string(),
-            DataType::Clob(size) => format!("clob({})", size),
-            DataType::Binary(size) => format!("binary({})", size),
-            DataType::Varbinary(size) => format!("varbinary({})", size),
-            DataType::Blob(size) => format!("blob({})", size),
+            DataType::Char(size) => format_type_with_optional_length(f, "char", size),
+            DataType::Varchar(size) => {
+                format_type_with_optional_length(f, "character varying", size)
+            }
+            DataType::Uuid => write!(f, "uuid"),
+            DataType::Clob(size) => write!(f, "clob({})", size),
+            DataType::Binary(size) => write!(f, "binary({})", size),
+            DataType::Varbinary(size) => write!(f, "varbinary({})", size),
+            DataType::Blob(size) => write!(f, "blob({})", size),
             DataType::Decimal(precision, scale) => {
                 if let Some(scale) = scale {
-                    format!("numeric({},{})", precision.unwrap(), scale)
+                    write!(f, "numeric({},{})", precision.unwrap(), scale)
                 } else {
-                    format_type_with_optional_length("numeric", precision)
+                    format_type_with_optional_length(f, "numeric", precision)
                 }
             }
-            DataType::Float(size) => format_type_with_optional_length("float", size),
-            DataType::SmallInt => "smallint".to_string(),
-            DataType::Int => "int".to_string(),
-            DataType::BigInt => "bigint".to_string(),
-            DataType::Real => "real".to_string(),
-            DataType::Double => "double".to_string(),
-            DataType::Boolean => "boolean".to_string(),
-            DataType::Date => "date".to_string(),
-            DataType::Time => "time".to_string(),
-            DataType::Timestamp => "timestamp".to_string(),
-            DataType::Interval => "interval".to_string(),
-            DataType::Regclass => "regclass".to_string(),
-            DataType::Text => "text".to_string(),
-            DataType::Bytea => "bytea".to_string(),
-            DataType::Array(ty) => format!("{}[]", ty.to_string()),
-            DataType::Custom(ty) => ty.to_string(),
+            DataType::Float(size) => format_type_with_optional_length(f, "float", size),
+            DataType::SmallInt => write!(f, "smallint"),
+            DataType::Int => write!(f, "int"),
+            DataType::BigInt => write!(f, "bigint"),
+            DataType::Real => write!(f, "real"),
+            DataType::Double => write!(f, "double"),
+            DataType::Boolean => write!(f, "boolean"),
+            DataType::Date => write!(f, "date"),
+            DataType::Time => write!(f, "time"),
+            DataType::Timestamp => write!(f, "timestamp"),
+            DataType::Interval => write!(f, "interval"),
+            DataType::Regclass => write!(f, "regclass"),
+            DataType::Text => write!(f, "text"),
+            DataType::Bytea => write!(f, "bytea"),
+            DataType::Array(ty) => write!(f, "{}[]", ty),
+            DataType::Custom(ty) => write!(f, "{}", ty),
         }
     }
 }
 
-fn format_type_with_optional_length(sql_type: &str, len: &Option<u64>) -> String {
-    let mut s = sql_type.to_string();
+fn format_type_with_optional_length(
+    f: &mut fmt::Formatter,
+    sql_type: &'static str,
+    len: &Option<u64>,
+) -> fmt::Result {
+    write!(f, "{}", sql_type)?;
     if let Some(len) = len {
-        s += &format!("({})", len);
+        write!(f, "({})", len)?;
     }
-    s
+    Ok(())
 }