GitBook: [#10] No subject

Author: thclark

.gitbook/assets/Screenshot 2022-12-15 at 11.36.11.png

@@ -12,13 +12,15 @@
 * [FAIR principles](terms/fair-principles.md)
 * [Metadata](terms/live-edit-and-locked-edits.md)
 * [Ontology](terms/ontology.md)
+* [Pipeline](terms/pipeline.md)
 * [Pragmatics](terms/pragmatics.md)
 * [Property](terms/property.md)
 * [Schema](terms/schema.md)
 * [Schema Language](terms/schema-language.md)
 * [Semantics](terms/collections.md)
 * [Syntax](terms/the-gitbook-editor.md)
 * [Taxonomy](terms/taxonomy.md)
+* [Transformation](terms/transformation.md)
 * [Uniform Resource Identifier](terms/uniform-resource-identifier.md)
 * [Vocabulary](terms/change-requests.md)
 

.gitbook/assets/Screenshot 2022-12-15 at 11.38.42.png

@@ -33,6 +33,5 @@ The descriptive information about data.
 Information about how the data is structured.
 
 {% hint style="info" %}
-**Example:** Format, size, file type, properties, reference to a schema
+**Example:** Format, size, file type, properties, reference to a [schema](schema.md)
 {% endhint %}
-

.gitbook/assets/database schema example.png

@@ -0,0 +1,3 @@
+# Pipeline
+
+\<placeholder>

.gitbook/assets/example pipeline with jsonschema.svg

@@ -1,7 +1,7 @@
 # Property
 
-An aspect, feature, characteristic or parameter associated with a class or an object of a given class.
+An aspect, feature, characteristic or parameter associated with a [class](class.md) or an object of a given class.
 
 {% hint style="warning" %}
-In some programming languages, there is a significant difference between an attribute and a property (an attribute in python being a formal class member, for example), but for the the majority of the Task 43 work, they are broadly synonymous, and which is used depends on the context (for example, when writing a JSON schema you'd use the "property" terminology). 
+In some programming languages, there is a significant difference between an attribute and a property (an attribute in python being a formal class member, for example), but for the the majority of the Task 43 work, they are broadly synonymous, and which is used depends on the context (for example, when writing a JSON schema you'd use the "property" terminology).
 {% endhint %}

.gitbook/assets/preprocessing-pipeline.svg

@@ -1,7 +1,85 @@
 # Schema
 
-## TODO - copy from google docs
+A schema is a "blueprint" of what data looks like. More formally, it's an expression of descriptive and structural [metadata](live-edit-and-locked-edits.md) with defined [semantics](collections.md). A schema is a powerful communication tool, as it provides a clear and well-encapsulated expression of what data you have (or need).
+
+By "defined semantics" we mean that it is expressed in a particular [schema language](schema-language.md), the choice of which can be highly nuanced depending on your application.
+
+{% hint style="success" %}
+**Purpose**
+
+Schema allow you to:
+
+* de-risk projects involving data,
+* build robust data [transformations](transformation.md) and [pipelines](pipeline.md), and
+* communicate with other stakeholders about the data you require (or provide).
+{% endhint %}
+
+{% hint style="info" %}
+**Tip**
+
+The **pure act of writing down what's in the data** is far more important than the selection of [schema language](schema-language.md).
+
+Schemas can (in general) be translated between languages relatively easily; **the real value lies** in getting the data structure and description written down and communicated in the first place.
+{% endhint %}
 
 {% hint style="warning" %}
-In philosophy, a schema is a representation of a plan or theory in the form of an outline or model: "a schema of scientific reasoning". This is of course a much broader definition than we use for our purposes here.
+**Disambiguation**
+
+In philosophy, a schema is a representation of a plan or theory in the form of an outline or model: "a schema of scientific reasoning". This is a much broader definition than we use for our purposes here.
 {% endhint %}
+
+## Example uses for schema
+
+{% tabs %}
+{% tab title="Project management (!!!)" %}
+_Yes, you read this tab label correctly. Schema are an incredibly powerful de-risking tool for project management!_
+
+In any data-driven project with several stakeholders, many months can be spent in communication on how teams are going to work together. Example chunks of data are sent back and forth in CSV or Excel files, and it's frequently unclear what expectations are and where boundaries lie between teams. The meanings of particular columns are queried by email, just when people go out on holiday. Things need fixing when it turns out the real data is a bit different. And so on, and so on, as **the critical path grows ever longer...**
+
+****
+
+<figure><img src="../.gitbook/assets/Screenshot 2022-12-15 at 11.38.42.png" alt=""><figcaption><p>Gantt charts have a habit of shifting ever-right in digitalisation projects. A perfect way of getting dependencies under control, and prevent teams blocking one another, is to workshop a set of schema at each of the boundaries at the beginning of the project.</p></figcaption></figure>
+
+At the beginning of such projects, defining a set of schema at the boundaries where data is exchanged between teams:
+
+* Clarifies initial expectations&#x20;
+* Encourages disciplined, effective communication between the teams as the project and its data evolve
+
+This works well even if you don't yet have a clear understanding of your data (when schema are little more than a wild-guess!), because it introduces a framework for communication at the start.
+{% endtab %}
+
+{% tab title="Data validation" %}
+For many kinds of [data transformation](transformation.md), it's imperative that the input data has some kind of characteristic. For example, if you run a wind resource analysis, your input data must contain at least some information about the wind at a site!
+
+> But what if it doesn't? Or it's not in the right form?
+
+This is where schema come in. If you have a schema then you can use it to:
+
+* Check that the incoming data is valid
+* Issue coherent and useful error messages if it isn't.
+
+This is in contrast to accepting any kind of data, then behaving in undefined ways depending on what's in the data. Hence, it allows you to build robust data transformations more rapidly.
+
+<figure><img src="../.gitbook/assets/preprocessing-pipeline.svg" alt=""><figcaption><p>An example architecture diagram, in which a schema is used to validate data and return errors to user before accepting data for processing, launching a <a href="pipeline.md">pipeline</a> for <a href="transformation.md">transformation</a> with dataflow then persisting the output to a database</p></figcaption></figure>
+{% endtab %}
+
+{% tab title="Database" %}
+A schema for a relational database (such as [PostgreSQL](https://www.postgresql.org/)) describes how data is stored, by specifying tables, columns, column types and relations.
+
+Database schema are generally required to be in a language specific to the type of database.
+
+<figure><img src="../.gitbook/assets/database schema example.png" alt=""><figcaption><p>Graphical representation of a simple SQL database schema, from <a href="https://www.codecademy.com/learn/how-do-i-make-and-populate-my-own-database/modules/designing-a-database-schema/cheatsheet">codeacademy</a>.</p></figcaption></figure>
+{% endtab %}
+{% endtabs %}
+
+
+
+## Standardisation and evolution
+
+Industrial standards frequently emerge specifying data contents, often for stable industrial systems whose parameters and data outputs are well known.
+
+
+
+
+
+\

SUMMARY.md

@@ -0,0 +1,45 @@
+# Transformation
+
+An operation in which data is transformed from one form to another. The resulting form is entirely general but may be (for example):
+
+* A re-expression of the same fundamental data (eg rewriting with a different data structure or file format)
+* A combination of data sources (eg add wind direction to wind speed, to form a combined time series)
+* A result of the application of an algorithm (eg getting mean wind speed by averaging a time series)
+* An expression or visualisation (eg an HTML file to display a chart of wind speed time series and a table of monthly averages)
+
+Data transformation as a term is often used with reference to [pipelines](pipeline.md), which can be used to execute an arbitrary number and sequence of transformations.
+
+{% hint style="warning" %}
+**Disambiguation**
+
+The term **"Transformation"** could be perceived as meaning **"Translation"** of data (eg a re-expression of the same fundamental data in a different way). Translation, however, represents only one of the many kinds of operation that could be a transformation [(see below)](transformation.md#kinds-of-transformation).
+
+
+{% endhint %}
+
+## Kinds of transformation
+
+In the discussion that led to this entry, a range of other terms were discussed surrounding the "transformation" terminology, with some viewing other terms as more helpful. "Data Analysis" and "Data Translation" in particular were thought of. 
+
+However, such terms **have a connotation regarding the purpose of the transformation**, which can result in confusion when attempting to think clearly about the data engineering required to implement a transformation, particularly since an output of a given translation might have multiple uses.
+
+In general though, it may be helpful to have a shorthand to discuss different specific kinds of translation.  Corresponding with the above examples:
+
+* A **translation** might constitute a re-expression of the same fundamental data in a different form
+* A **view** might constitute the creation of a bridge between two data tables, combining two data sources to appear as one
+* An **analysis** might constitute application of some scientific algorithm to extract results or insight (which themselves constitute the output data of the transformation)
+* A **render** might constitute the expression or visualisation of data in a way which may not be well suited for ongoing use in an automated pipeline, but is good for human consumption (like an automated report).
+
+{% hint style="danger" %}
+**Why use the word "might"?**
+
+The boundaries blur between these items, particularly because it can't be known _a priori_ how the output of any given transformation will be used.
+
+* For example, using a[ jupyter notebook](https://jupyter.org/) to average some data and plot the result would combine algorithm implementation with visualisation. This would be colloquially called a "data analysis", but when moving to a production scenario, averaging of the data and the visualisation of the result would likely be in entirely different domains.
+{% endhint %}
+
+{% hint style="info" %}
+**Tip**
+
+Thinking simply in terms of transformations (as opposed to the different kinds and their purposes) will help you think more objectively about the separation of concerns between, and the role of, each step in a pipeline.
+{% endhint %}