Skip to content

Commit ce25954

Browse files
daern91daern91
authored
feat(custom-objects-importer): add new importer package (#651)
#### Summary PR adds package to import custom objects to CTP. #### Todo * Tests * [x] Unit * [x] Integration * [x] Documentation * [ ] Add package to be used with [sphere-node-cli](https://github.com/sphereio/sphere-node-cli). (After merge of this PR) <!-- Two persons should review a PR, don't forget to assign them. --> resolves #435 resolves #552
1 parent d69d4c2 commit ce25954

File tree

16 files changed

+1550
-140
lines changed

16 files changed

+1550
-140
lines changed

‎README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ https://commercetools.github.io/nodejs/
7070
| [`csv-parser-price`](/packages/csv-parser-price) | [![csv-parser-price Version][csv-parser-price-icon]][csv-parser-price-version] | [![csv-parser-price Dependencies Status][csv-parser-price-dependencies-icon]][csv-parser-price-dependencies] |
7171
| [`csv-parser-state`](/packages/csv-parser-state) | [![csv-parser-state Version][csv-parser-state-icon]][csv-parser-state-version] | [![csv-parser-state Dependencies Status][csv-parser-state-dependencies-icon]][csv-parser-state-dependencies] |
7272
| [`custom-objects-exporter`](/packages/custom-objects-exporter) | [![custom-objects-exporter Version][custom-objects-exporter-icon]][custom-objects-exporter-version] | [![custom-objects-exporter Dependencies Status][custom-objects-exporter-dependencies-icon]][custom-objects-exporter-dependencies] |
73+
| [`custom-objects-importer`](/packages/custom-objects-importer) | [![custom-objects-importer Version][custom-objects-importer-icon]][custom-objects-importer-version] | [![custom-objects-importer Dependencies Status][custom-objects-importer-dependencies-icon]][custom-objects-importer-dependencies] |
7374
| [`personal-data-erasure`](/packages/personal-data-erasure) | [![personal-data-erasure Version][personal-data-erasure-icon]][personal-data-erasure-version] | [![personal-data-erasure Dependencies Status][personal-data-erasure-dependencies-icon]][personal-data-erasure-dependencies] |
7475
| [`discount-code-exporter`](/packages/discount-code-exporter) | [![discount-code-exporter Version][discount-code-exporter-icon]][discount-code-exporter-version] | [![discount-code-exporter Dependencies Status][discount-code-exporter-dependencies-icon]][discount-code-exporter-dependencies] |
7576
| [`discount-code-generator`](/packages/discount-code-generator) | [![discount-code-generator Version][discount-code-generator-icon]][discount-code-generator-version] | [![discount-code-generator Dependencies Status][discount-code-generator-dependencies-icon]][discount-code-generator-dependencies] |
@@ -111,6 +112,10 @@ https://commercetools.github.io/nodejs/
111112
[custom-objects-exporter-icon]: https://img.shields.io/npm/v/@commercetools/custom-objects-exporter.svg?style=flat-square
112113
[custom-objects-exporter-dependencies]: https://david-dm.org/commercetools/nodejs?path=packages/custom-objects-exporter
113114
[custom-objects-exporter-dependencies-icon]: https://img.shields.io/david/commercetools/nodejs.svg?path=packages/custom-objects-exporter&style=flat-square
115+
[custom-objects-importer-version]: https://www.npmjs.com/package/@commercetools/custom-objects-importer
116+
[custom-objects-importer-icon]: https://img.shields.io/npm/v/@commercetools/custom-objects-importer.svg?style=flat-square
117+
[custom-objects-importer-dependencies]: https://david-dm.org/commercetools/nodejs?path=packages/custom-objects-importer
118+
[custom-objects-importer-dependencies-icon]: https://img.shields.io/david/commercetools/nodejs.svg?path=packages/custom-objects-importer&style=flat-square
114119
[discount-code-exporter-version]: https://www.npmjs.com/package/@commercetools/discount-code-exporter
115120
[discount-code-exporter-icon]: https://img.shields.io/npm/v/@commercetools/discount-code-exporter.svg?style=flat-square
116121
[discount-code-exporter-dependencies]: https://david-dm.org/commercetools/nodejs?path=packages/discount-code-exporter

‎docs/SUMMARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
* [CSV Order Parser](/cli/csv-parser-orders.md)
2929
* [CSV State Parser](/cli/csv-parser-state.md)
3030
* [Custom Objects Exporter](/cli/custom-objects-exporter.md)
31+
* [Custom Objects Importer](/cli/custom-objects-importer.md)
3132
* [Personal Data Erasure](/cli/personal-data-erasure.md)
3233
* [Discount Code Generator](/cli/discount-code-generator.md)
3334
* [Discount Code Exporter](/cli/discount-code-exporter.md)

‎docs/cli/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Our CLI tools perform various functions from importing and exporting to syncing
1111
* [CSV Order Parser](/cli/csv-parser-orders.md)
1212
* [CSV State Parser](/cli/csv-parser-state.md)
1313
* [Custom Objects Exporter](/cli/custom-objects-exporter.md)
14+
* [Custom Objects Importer](/cli/custom-objects-importer.md)
1415
* [Personal Data Erasure](/cli/personal-data-erasure.md)
1516
* [Discount Code Generator](/cli/discount-code-generator.md)
1617
* [Discount Code Exporter](/cli/discount-code-exporter.md)

‎docs/cli/custom-objects-importer.md

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Custom Objects Importer
2+
3+
This package helps with importing [commercetools custom objects](https://docs.commercetools.com/http-api-projects-custom-objects.html) in JSON format to the [commercetools platform](https://docs.commercetools.com/).
4+
The package is built to be used in conjunction with [sphere-node-cli](https://github.com/sphereio/sphere-node-cli)
5+
6+
## Configuration
7+
8+
The constructor accepts five arguments:
9+
10+
* A required object containing the following values:
11+
* `apiConfig` (Object): `AuthMiddleware` options for authentication on the commercetools platform. (Required. See [here](https://commercetools.github.io/nodejs/sdk/api/sdkMiddlewareAuth.html#named-arguments-options))
12+
* `accessToken` (String): Access token to be used to authenticate requests to API. Requires scope of [`view_products`, `view_orders`, `view_customers`, `manage_products`, `manage_orders`, `manage_customers`].
13+
* `batchSize` (Number): Amount of custom objects to process concurrently (Optional. Default: 50)
14+
* `continueOnProblems` (Boolean): Flag whether to continue processing if an error occurs (Optional. Default: false)
15+
* An optional logger object having four functions (`info`, `warn`, `error` and `debug`)
16+
17+
## Usage with `sphere-node-cli`
18+
19+
You can use this package from the [`sphere-node-cli`](https://github.com/sphereio/sphere-node-cli). In order for the cli to import custom objects, the file to import from must be a valid JSON and follow the following structure:
20+
21+
```json
22+
[{
23+
"container": "Ludus",
24+
"key": "copperKey",
25+
"value": {
26+
"paymentMethod": "Cash",
27+
"paymentID": "1",
28+
"whateverElse": {
29+
"number": 1000
30+
}
31+
}
32+
},
33+
{
34+
"container": "Frobozz",
35+
"key": "jadeKey",
36+
"value": {
37+
"paymentMethod": "cc",
38+
"paymentID": "2",
39+
"whateverElse": {
40+
"digits": [1, 2, 3]
41+
}
42+
}
43+
},
44+
{
45+
"container": "Syrinx",
46+
"key": "crystalKey",
47+
"value": {
48+
"paymentMethod": "new",
49+
"paymentID": "3",
50+
"whateverElse": {
51+
"true": true
52+
}
53+
}
54+
}
55+
...
56+
]
57+
```
58+
59+
Then you can import this file using the cli:
60+
61+
```bash
62+
sphere import -t customObject -p my-project-key --host 'https://api.sphere.io' --authHost 'https://auth.sphere.io' -f /path/to/file.json -c
63+
'{"continueOnProblems": true}'
64+
```
65+
66+
## Direct Usage
67+
68+
You can also use this module directly in your Javascript project. To do this, you need to install it:
69+
70+
```bash
71+
npm install @commercetools/custom-objects-importer
72+
```
73+
74+
Then you can use it to import custom objects:
75+
76+
```js
77+
import CustomObjectsImport from '@commercetools/custom-objects-importer'
78+
79+
const customObjectsToImport = [
80+
{
81+
container: 'Ludus',
82+
key: 'copperKey',
83+
value: {
84+
paymentMethod: 'Cash',
85+
paymentID: '1',
86+
whateverElse: {
87+
number: 1000
88+
}
89+
},
90+
},
91+
{
92+
container: 'Frobozz',
93+
key: 'jadeKey',
94+
value: {
95+
paymentMethod: 'cc',
96+
paymentID: '2',
97+
whateverElse: {
98+
digits: [1,2,3]
99+
}
100+
},
101+
},
102+
{
103+
container: 'Syrinx',
104+
key: 'crystalKey',
105+
value: {
106+
paymentMethod: 'new',
107+
paymentID: '3',
108+
whateverElse: {
109+
true: false
110+
}
111+
},
112+
},
113+
...
114+
]
115+
116+
const options = {
117+
apiConfig: {
118+
host: 'https://auth.commercetools.com'
119+
projectKey: <PROJECT_KEY>,
120+
credentials: {
121+
clientId: '*********',
122+
clientSecret: '*********'
123+
}
124+
},
125+
accessToken: '123456yuhgfdwegh675412wefb3rgb',
126+
continueOnProblems: false
127+
}
128+
}
129+
130+
const logger = {
131+
error: console.error,
132+
warn: console.warn,
133+
info: console.log,
134+
debug: console.debug,
135+
}
136+
137+
const customObjectsImport = new CustomObjectsImport(options, logger)
138+
139+
customObjectsImport.run(customObjectsToImport)
140+
.then(() => {
141+
customObjectsImport.summaryReport()
142+
// handle successful import
143+
})
144+
.catch((error) => {
145+
// handle error
146+
})
147+
```
148+
149+
On successful completion, a call to the `.summaryReport()` method returns a report in the following format:
150+
151+
```js
152+
{
153+
reportMessage: 'Summary: there were 4 successfully imported custom objects. 2 were newly created, 2 were updated and 0 were unchanged.)',
154+
detailedSummary: {
155+
createErrorCount: 0,
156+
created: 2,
157+
errors: [],
158+
unchanged: 0,
159+
updateErrorCount: 0,
160+
updated: 2
161+
}
162+
}
163+
```
164+
165+
**Note:** By default, if a custom object exists, the module tries to build an update action for it, and if no update action can be built, the custom object will be ignored

‎integration-tests/cli/__snapshots__/custom-objects-import.it.js.snap

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Jest Snapshot v1, https://goo.gl/fbAQLP
2+
3+
exports[`Custom Object tests Custom Objects Importer should continueOnProblems if \`continueOnProblems\` 1`] = `
4+
Object {
5+
"detailedSummary": Object {
6+
"createErrorCount": 2,
7+
"createdCount": 1,
8+
"errors": Array [
9+
"key must at least have one character",
10+
"container must at least have one character",
11+
],
12+
"unchangedCount": 0,
13+
"updateErrorCount": 0,
14+
"updatedCount": 0,
15+
},
16+
"reportMessage": "Summary: there were 1 successfully imported custom objects. 1 were newly created, 0 were updated and 0 were unchanged. 2 errors occurred (2 create errors and 0 update errors.)",
17+
}
18+
`;
19+
20+
exports[`Custom Object tests Custom Objects Importer should continueOnProblems if \`continueOnProblems\` 2`] = `
21+
Array [
22+
"key must at least have one character",
23+
"container must at least have one character",
24+
]
25+
`;
26+
27+
exports[`Custom Object tests Custom Objects Importer should import custom objects to CTP 1`] = `
28+
Object {
29+
"detailedSummary": Object {
30+
"createErrorCount": 0,
31+
"createdCount": 3,
32+
"errors": Array [],
33+
"unchangedCount": 0,
34+
"updateErrorCount": 0,
35+
"updatedCount": 0,
36+
},
37+
"reportMessage": "Summary: there were 3 successfully imported custom objects. 3 were newly created, 0 were updated and 0 were unchanged.",
38+
}
39+
`;
40+
41+
exports[`Custom Object tests Custom Objects Importer should stop import on first errors by default 1`] = `
42+
Object {
43+
"createErrorCount": 1,
44+
"createdCount": 1,
45+
"errors": Array [
46+
"key must at least have one character",
47+
],
48+
"unchangedCount": 0,
49+
"updateErrorCount": 0,
50+
"updatedCount": 0,
51+
}
52+
`;
53+
54+
exports[`Custom Object tests Custom Objects Importer should update custom objects on the CTP 1`] = `
55+
Object {
56+
"detailedSummary": Object {
57+
"createErrorCount": 0,
58+
"createdCount": 0,
59+
"errors": Array [],
60+
"unchangedCount": 0,
61+
"updateErrorCount": 0,
62+
"updatedCount": 3,
63+
},
64+
"reportMessage": "Summary: there were 3 successfully imported custom objects. 0 were newly created, 3 were updated and 0 were unchanged.",
65+
}
66+
`;

‎integration-tests/cli/custom-objects-import.it.js

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
import CustomObjectsImporter from '@commercetools//custom-objects-importer'
2+
import { getCredentials } from '@commercetools/get-credentials'
3+
import cloneDeep from 'lodash.clonedeep'
4+
5+
import allCustomObjects from './helpers/custom-objects-export.data'
6+
import { clearData } from './helpers/utils'
7+
8+
let projectKey
9+
if (process.env.CI === 'true') projectKey = 'custom-objects-import-int-test'
10+
else projectKey = process.env.npm_config_projectkey
11+
12+
describe('Custom Object tests', () => {
13+
let apiConfig
14+
let customObjects
15+
16+
const logger = {
17+
error: () => {},
18+
warn: () => {},
19+
info: () => {},
20+
debug: () => {},
21+
}
22+
23+
beforeAll(async () => {
24+
const credentials = await getCredentials(projectKey)
25+
apiConfig = {
26+
host: 'https://auth.sphere.io',
27+
apiUrl: 'https://api.sphere.io',
28+
projectKey,
29+
credentials,
30+
}
31+
}, 15000)
32+
33+
describe('Custom Objects Importer', () => {
34+
let objectImport
35+
36+
beforeEach(() => {
37+
objectImport = new CustomObjectsImporter({ apiConfig }, logger)
38+
customObjects = cloneDeep(allCustomObjects)
39+
})
40+
41+
afterEach(async () => {
42+
await clearData(apiConfig, 'customObjects')
43+
}, 15000)
44+
45+
it(
46+
'should import custom objects to CTP',
47+
async () => {
48+
await objectImport.run(customObjects)
49+
const summary = objectImport.summaryReport()
50+
expect(summary).toMatchSnapshot()
51+
},
52+
15000
53+
)
54+
55+
it(
56+
'should update custom objects on the CTP',
57+
async () => {
58+
const oldCustomObjectsToUpdate = customObjects.map(object => ({
59+
...object,
60+
value: { paymentMethod: 'gold' },
61+
}))
62+
await objectImport.run(oldCustomObjectsToUpdate)
63+
64+
// Reset the summary
65+
objectImport._initiateSummary()
66+
67+
await objectImport.run(customObjects)
68+
const summary = objectImport.summaryReport()
69+
expect(summary).toMatchSnapshot()
70+
},
71+
15000
72+
)
73+
74+
it(
75+
'should stop import on first errors by default',
76+
async () => {
77+
// Set batchSize to 1 so it executes serially
78+
objectImport = new CustomObjectsImporter(
79+
{ apiConfig, batchSize: 1 },
80+
logger
81+
)
82+
83+
// Make last two objects invalid
84+
customObjects[1].key = ''
85+
customObjects[2].container = ''
86+
87+
try {
88+
await objectImport.run(customObjects)
89+
} catch (e) {
90+
// should create first object
91+
expect(e.summary.createdCount).toBe(1)
92+
// should stop after first error
93+
expect(e.summary.errors).toHaveLength(1)
94+
expect(e.summary).toMatchSnapshot()
95+
}
96+
},
97+
15000
98+
)
99+
100+
it(
101+
'should continueOnProblems if `continueOnProblems`',
102+
async () => {
103+
// Set batchSize to 1 so it executes serially
104+
objectImport = new CustomObjectsImporter(
105+
{
106+
apiConfig,
107+
batchSize: 1,
108+
continueOnProblems: true,
109+
},
110+
logger
111+
)
112+
113+
// Make two objects invalid
114+
customObjects[0].key = ''
115+
customObjects[1].container = ''
116+
117+
await objectImport.run(customObjects)
118+
const summary = objectImport.summaryReport()
119+
expect(summary).toMatchSnapshot()
120+
const errors = summary.detailedSummary.errors
121+
expect(errors).toMatchSnapshot()
122+
},
123+
15000
124+
)
125+
})
126+
})

0 commit comments

Comments
 (0)