Skip to content

Infer data type from JSON schema #477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 31 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
a3b4d56
feat: support readonly arrays in JsonSchema
cyrilletuzi Oct 19, 2020
990103f
first try of infering
cyrilletuzi Oct 20, 2020
931a787
infer arrays
cyrilletuzi Oct 23, 2020
333400d
feat: inference from JSONSchema
cyrilletuzi Oct 23, 2020
b73a306
Merge branch 'v11' into inferfromjsonschema
cyrilletuzi Oct 23, 2020
b8311f6
fix test
cyrilletuzi Oct 23, 2020
06ef853
tests: update all overloads tests with and without as const
cyrilletuzi Oct 23, 2020
bd2bb93
tests
cyrilletuzi Oct 23, 2020
9dfad66
better inference
cyrilletuzi Oct 24, 2020
4880f3e
doc
cyrilletuzi Oct 24, 2020
40301b6
unlimited tuple !!!
cyrilletuzi Oct 24, 2020
f2bf459
fix type recursion
cyrilletuzi Oct 24, 2020
df06beb
doc
cyrilletuzi Oct 24, 2020
c8e6e66
draft
cyrilletuzi Oct 25, 2020
165840f
fix errors
cyrilletuzi Oct 25, 2020
1fbf3d4
revert unwanted change
cyrilletuzi Oct 25, 2020
1f8c19e
fix lint
cyrilletuzi Oct 25, 2020
7aa0aba
fix tests
cyrilletuzi Oct 25, 2020
00182f1
fix accidental change
cyrilletuzi Oct 25, 2020
865b978
fix: use extends to limit code duplication
cyrilletuzi Oct 25, 2020
89ca34f
remove all code duplication
cyrilletuzi Oct 25, 2020
dd30787
tests: do compilation and runtime tests at the same time
cyrilletuzi Oct 25, 2020
5d2c874
tests: refactor
cyrilletuzi Oct 25, 2020
3dcc0fb
refactor: reorganize overloads
cyrilletuzi Oct 25, 2020
1a25a5d
test: stress test against heavy schema
cyrilletuzi Oct 25, 2020
7b04912
doc
cyrilletuzi Oct 25, 2020
ee2ccf2
tests: JSONSchema
cyrilletuzi Oct 26, 2020
2ba8d24
Merge branch 'v11' into inferfromjsonschema
cyrilletuzi Oct 26, 2020
1f4af4a
fix
cyrilletuzi Oct 26, 2020
4d61dc5
fix
cyrilletuzi Oct 26, 2020
1a97735
revert
cyrilletuzi Oct 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,578 changes: 257 additions & 1,321 deletions projects/ngx-pwa/local-storage/src/lib/storages/storage-map.service.spec.ts

Large diffs are not rendered by default.

351 changes: 18 additions & 333 deletions projects/ngx-pwa/local-storage/src/lib/storages/storage-map.service.ts

Large diffs are not rendered by default.

11 changes: 6 additions & 5 deletions projects/ngx-pwa/local-storage/src/lib/testing/cleaning.ts
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
import { StorageMap } from '../storages/storage-map.service';
import { IndexedDBDatabase } from '../databases/indexeddb-database';
import { MemoryDatabase } from '../databases/memory-database';
import { SafeStorageMap } from '../storages/safe-storage-map.service';

/**
* Helper to clear all data in storage to avoid tests overlap
* @param done Jasmine helper to explicit when the operation has ended to avoid tests overlap
* @param storageService Service
*/
export function clearStorage(done: DoneFn, storageService: StorageMap): void {
export function clearStorage(done: DoneFn, storageService: StorageMap | SafeStorageMap): void {

if (storageService.backingEngine === 'indexedDB') {

// tslint:disable-next-line: no-string-literal
const indexedDBService = storageService['database'] as IndexedDBDatabase;
const indexedDBService = (storageService as SafeStorageMap)['database'] as IndexedDBDatabase;

try {

Expand Down Expand Up @@ -82,7 +83,7 @@ export function clearStorage(done: DoneFn, storageService: StorageMap): void {
} else if (storageService.backingEngine === 'memory') {

// tslint:disable-next-line: no-string-literal
(storageService['database'] as MemoryDatabase)['memoryStorage'].clear();
((storageService as SafeStorageMap)['database'] as MemoryDatabase)['memoryStorage'].clear();

done();

Expand All @@ -103,13 +104,13 @@ export function clearStorage(done: DoneFn, storageService: StorageMap): void {
* @param doneJasmine helper to explicit when the operation has ended to avoid tests overlap
* @param storageService Service
*/
export function closeAndDeleteDatabase(done: DoneFn, storageService: StorageMap): void {
export function closeAndDeleteDatabase(done: DoneFn, storageService: StorageMap | SafeStorageMap): void {

/* Only `indexedDB` is concerned */
if (storageService.backingEngine === 'indexedDB') {

// tslint:disable-next-line: no-string-literal
const indexedDBService = storageService['database'] as IndexedDBDatabase;
const indexedDBService = (storageService as SafeStorageMap)['database'] as IndexedDBDatabase;

// tslint:disable-next-line: no-string-literal
indexedDBService['database'].subscribe({
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import { JSONSchema } from './json-schema';

/**
* JSON schemas of primitive types
*/
type JSONSchemaPrimitive = { type: 'string' | 'integer' | 'number' | 'boolean' };

/**
* Infer data from a JSON schemas describing a primitive type.
*/
type InferFromJSONSchemaPrimitive<Schema extends JSONSchemaPrimitive> =
/* Infer `const` and `enum` first, as they are more specific */
Schema extends { const: infer ConstType } ? ConstType :
Schema extends { enum: readonly (infer EnumType)[] } ? EnumType :
/* Infer primitive types */
Schema extends { type: 'string' } ? string :
Schema extends { type: 'integer' } ? number :
Schema extends { type: 'number' } ? number :
Schema extends { type: 'boolean' } ? boolean :
/* Default value, but not supposed to happen given the `JSONSchema` interface */
unknown;

/**
* Infer the data type from a tuple JSON schema describing a tuple of primitive types.
*/
type InferFromJSONSchemaTupleOfPrimitive<Schemas extends readonly JSONSchemaPrimitive[]> = {
-readonly [Key in keyof Schemas]: Schemas[Key] extends JSONSchemaPrimitive ? InferFromJSONSchemaPrimitive<Schemas[Key]> : never
};

/**
* Infer the data type from a JSON schema describing a tuple:
* - with 1 or 2 values of any type (especially usefull for handling `Map`s),
* - with unlimited values but of primitive types only.
*/
type InferFromJSONSchemaTuple<Schemas extends readonly JSONSchema[]> =
/* Allows inference from any JSON schema up to 2 values */
Schemas extends readonly [JSONSchema] ? [InferFromJSONSchema<Schemas[0]>] :
Schemas extends readonly [JSONSchema, JSONSchema] ? [InferFromJSONSchema<Schemas[0]>, InferFromJSONSchema<Schemas[1]>] :
/* Beyond 2 values, infer from primitive types only to avoid too deep type inference */
Schemas extends readonly JSONSchemaPrimitive[] ? InferFromJSONSchemaTupleOfPrimitive<Schemas> :
unknown[]
;

/**
* Infer the data type from a JSON schema.
*/
export type InferFromJSONSchema<Schema extends JSONSchema> =
/* Infer primitive types */
Schema extends JSONSchemaPrimitive ? InferFromJSONSchemaPrimitive<Schema> :
/* Infer arrays */
Schema extends { type: 'array', items: infer ItemsType } ?
/* Classic array */
ItemsType extends JSONSchema ? InferFromJSONSchema<ItemsType>[] :
/* Tuples (ie. array with different value types) */
ItemsType extends readonly JSONSchema[] ? InferFromJSONSchemaTuple<ItemsType> :
/* Default value, but not supposed to happen given the `JSONSchema` interface */
unknown[] :
/* Infer objects */
Schema extends { type: 'object', properties: infer Properties, required?: readonly (infer RequiredProperties)[] } ?
{
-readonly [Key in keyof Pick<Properties, RequiredProperties extends keyof Properties ? RequiredProperties : never>]:
Properties[Key] extends JSONSchema ? InferFromJSONSchema<Properties[Key]> : never;
} & {
-readonly [Key in keyof Omit<Properties, RequiredProperties extends keyof Properties ? RequiredProperties : never>]?:
Properties[Key] extends JSONSchema ? InferFromJSONSchema<Properties[Key]> : never;
} :
/* Default type if inference failed */
unknown;
16 changes: 15 additions & 1 deletion projects/ngx-pwa/local-storage/src/lib/validation/json-schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,19 @@ export interface JSONSchemaObject {

}

// TODO: documentation
/**
* JSON schema to describe a custom value (useful for `Blob`).
*/
export interface JSONSchemaUnknown {

/**
* Type for an unknown value.
*/
type: 'unknown';

}

/**
* Subset of the JSON Schema standard.
* Types are enforced to validate everything: each value **must** have a `type`.
Expand Down Expand Up @@ -284,5 +297,6 @@ export interface JSONSchemaObject {
* },
* required: ['firstName'],
* };
*
*/
export type JSONSchema = JSONSchemaString | JSONSchemaNumber | JSONSchemaInteger | JSONSchemaBoolean | JSONSchemaArray | JSONSchemaObject;
export type JSONSchema = JSONSchemaString | JSONSchemaNumber | JSONSchemaInteger | JSONSchemaBoolean | JSONSchemaArray | JSONSchemaObject | JSONSchemaUnknown;
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ export class JSONValidator {
return this.validateArray(data, schema);
case 'object':
return this.validateObject(data, schema);
// TODO: check how we handle this case (do not allow to bypass validation, or add a test)
case 'unknown':
return true;

}

Expand Down
3 changes: 2 additions & 1 deletion projects/ngx-pwa/local-storage/src/public_api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@

export {
JSONSchema, JSONSchemaBoolean, JSONSchemaInteger, JSONSchemaNumber, JSONSchemaString,
JSONSchemaArray, JSONSchemaArrayOf, JSONSchemaObject
JSONSchemaArray, JSONSchemaArrayOf, JSONSchemaObject, JSONSchemaUnknown
} from './lib/validation/json-schema';
export { JSONValidator } from './lib/validation/json-validator';

export { LocalDatabase } from './lib/databases/local-database';
export { SERIALIZATION_ERROR, SerializationError } from './lib/databases/exceptions';

export { SafeStorageMap } from './lib/storages/safe-storage-map.service';
export { StorageMap } from './lib/storages/storage-map.service';
export { LocalStorage } from './lib/storages/local-storage.service';
export { ValidationError, VALIDATION_ERROR } from './lib/storages/exceptions';
Expand Down