Skip to content

Commit

Permalink
add images to readme
Browse files Browse the repository at this point in the history
  • Loading branch information
brianzinn committed Oct 16, 2020
1 parent 8d8209b commit 268f2e2
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 18 deletions.
39 changes: 21 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# snowflake-ingest-node
simple API wrapper for Snowpipe for https://www.snowflake.com/ . At time of writing only Python and Java were available SDKs

[![NPM version](http://img.shields.io/npm/v/snowflake-ingest-node.svg?style=flat-square)](https://www.npmjs.com/package/snowflake-ingest-node)
[![NPM downloads](http://img.shields.io/npm/dm/snowflake-ingest-node.svg?style=flat-square)](https://www.npmjs.com/package/snowflake-ingest-node)

snowpipe intro:
https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro.html

Expand All @@ -10,18 +13,12 @@ Note that there is also an auto-ingest feature built into snowflake.
import * as dotenv from 'dotenv';

import { getLatestSecret } from '../src/SecretsManager';
import { APIEndpointHistory, createSnowpipeAPI } from 'snowflake-ingest-node';
import { createSnowpipeAPI, SnowpipeAPI, APIEndpointHistory } from 'snowflake-ingest-node';

describe(' > Snowflake API harness', () => {

let snowflakeAPI: {
loadHistoryScan: (pipeName: string, startTimeInclusive: string, endTimeExclusive?: string) => Promise<string>;
insertReport: (pipeName: string, beginMark?: string) => Promise<string>;
insertFile: (filenames: string[], pipeName: string) => Promise<string>;
endpointHistory: () => APIEndpointHistory;
};

const getSnowflakeAPI = async () => {
let snowpipeAPI: SnowpipeAPI;
const getSnowpipeAPI = async (): Promise<SnowpipeAPI> => {
const {
snowflake_username: username,
snowflake_region_id: regionId,
Expand All @@ -34,7 +31,7 @@ describe(' > Snowflake API harness', () => {
recordHistory: true
});
return result;
}
};

beforeEach(async () => {
dotenv.config();
Expand All @@ -46,34 +43,40 @@ describe(' > Snowflake API harness', () => {

it.skip('insertFile', async () => {
try {
/* const response =*/ await snowflakeAPI.insertFile(['/transactions/1602017572486-0000-00-00:00:00.csv'], PIPE_NAME);
const response = await snowflakeAPI.insertFile(['/path/file-name.csv'], PIPE_NAME);
console.log(response);
} catch (e) {
console.error(e);
}
console.log(snowflakeAPI.endpointHistory().insertFile[0].response);
const endpointHistory: APIEndpointHistory = snowpipeAPI.endpointHistory;
console.log(endpointHistory.insertFile[0].response);
});

it.skip('insertReport', async () => {
try {
/* const response = */ await snowflakeAPI.insertReport(PIPE_NAME, '33161bae019054d7aa5839ffbd60a85c');
const response = await snowflakeAPI.insertReport(PIPE_NAME, '<unique-request-id-optional>');
console.log(response);
} catch (e) {
console.error('error');
console.error(e);
}
console.log(snowflakeAPI.endpointHistory().insertReport[0].response);
});

it('loadHistoryScan', async () => {
// start time is in ISO 8601 format zulu timezone. Probably use a library like moment.tz.
try {
/* const response =*/ await snowflakeAPI.loadHistoryScan(PIPE_NAME, '2020-09-21T02:00:00.000Z');
const response = await snowflakeAPI.loadHistoryScan(PIPE_NAME, '2020-10-14T02:00:00.000Z');
console.log(response);
} catch (e) {
console.error('error', e);
console.error(e);
}
console.log(snowflakeAPI.endpointHistory().loadHistoryScan[0].response);
})
});
```

![Strongly Typed](https://raw.githubusercontent.com/brianzinn/snowflake-ingest-node/master/images/strong-typed.png)
![Runtime Debug](https://raw.githubusercontent.com/brianzinn/snowflake-ingest-node/master/images/runtime-debug.png)

# secrets and environment
You'll want to ensure your private key is secure in a vault or secret management (I am storing the lookup key as an environment variable). The rest could come from environment or hard coding. Here is a sample `.env` as above for running locally and against your setup in the cloud likely serverless:
```bash
Expand All @@ -90,4 +93,4 @@ You can just copy the one file or add via npm:
yarn add snowflake-ingest-node
yarn add jwt-simple
```
There is a peer dependency on `jwt-simple`, so make sure it is added as well. There are no other dependencies except for built-in node (https and crypto) modules.
There is a peer dependency on `jwt-simple`, so make sure it is added as well. There are no other dependencies except for built-in node (https and crypto) modules.
Binary file added images/runtime-debug.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/strong-typed.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 268f2e2

Please sign in to comment.