Skip to content

Commit 8ed4786

Browse files
authored
Add missing conventions & examples to Component Guidelines docs (#2505)
* Add missing conventions & examples to Component Guidelines docs Added docs for Common Files and Pagination are taken from [this PR](#1693) by jverce * Add "getters" to .wordlist.txt * docs: Add guidelines for action name (omit article) * Formatting - surround package.json in backticks
1 parent 51f5f29 commit 8ed4786

File tree

2 files changed

+80
-2
lines changed

2 files changed

+80
-2
lines changed

.wordlist.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,7 @@ getLikedTweets
315315
getName
316316
GetResponse
317317
getter
318+
getters
318319
gf
319320
GFJjGQ
320321
GGfX

docs/docs/components/guidelines/README.md

Lines changed: 79 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,8 @@ Registry components are organized by app in the `components` directory of the
287287
folder and the name of the `js` file equivalent to the slugified component
288288
name). For example, the path for the "Search Mentions" source for Twitter is
289289
`/components/twitter/sources/search-mentions/search-mentions.js`.
290+
- Aside from `app_slug`, words in folder and file names are separated by dashes
291+
(-) (i.e., in kebab case)
290292

291293
You can explore examples in the [components
292294
directory](https://github.com/pipedreamhq/pipedream/tree/master/components).
@@ -297,6 +299,10 @@ If the app has a well-supported [Node.js client
297299
library](../api/#using-npm-packages), that should be preferred to manually
298300
constructed API requests to reduce code and improve maintenance.
299301

302+
#### Include dependencies in `package.json`
303+
304+
Each app should have a `package.json` in its root folder to track changes in its dependencies. To create a `package.json` file, run `npm init` in the app's root folder and customize it using [this `package.json`](https://github.com/PipedreamHQ/pipedream/blob/55236b3aa993cbcb545e245803d8654c6358b0a2/components/stripe/package.json) as a template.
305+
300306
#### Error-handling and input validation
301307

302308
When you use the SDK of a popular API, the SDK might raise clear errors to the user. For example, if the user is asked to pass an email address, and that email address doesn't validate, the library might raise that in the error message.
@@ -307,7 +313,27 @@ In general, **imagine you are a user troubleshooting an issue. Is the error easy
307313

308314
#### Pagination
309315

310-
When making API requests, handle pagination to ensure all data / events are processed.
316+
When making API requests, handle pagination to ensure all data/events are
317+
processed. Moreover, if the underlying account experiences and/or generates too
318+
much data paginating through the entire collection of records, it might cause
319+
out-of-memory or timeout issues (or both!), so as a rule of thumb the pagination
320+
logic should:
321+
322+
- Be encapsulated as a [generator](https://mzl.la/37z6Sh6) so that the component
323+
can start processing records after the very first API call. As an example, you
324+
can check the [Microsoft OneDrive
325+
methods](https://github.com/PipedreamHQ/pipedream/tree/master/components/microsoft_onedrive/microsoft_onedrive.app.js)
326+
to list files.
327+
- Accept a "next token/page/ID" whenever possible, so that API calls do not
328+
retrieve the entire collection of records during every execution but rather
329+
from a recent point in time. The `scanDeltaItems` generator method in the
330+
example above follows this pattern.
331+
- Persist the last page number, token or record ID right after processing, so
332+
that following executions of the component process new records to minimize the
333+
amount of duplicate events, execution time and delayed events. Following the
334+
same Microsoft OneDrive example, check the `processEvent` method [in this
335+
component](https://github.com/PipedreamHQ/pipedream/tree/master/components/microsoft_onedrive/sources/new-file/new-file.js)
336+
for an example.
311337

312338
#### Capturing Sensitive Data
313339

@@ -416,6 +442,13 @@ Drive](https://github.com/PipedreamHQ/pipedream/tree/master/components/google_dr
416442
example of this pattern. When using this approach, prop definitions should still
417443
be maintained in the app file.
418444

445+
Please note that the name `common` is just a convention and depending on each
446+
case it might make sense to name any common module differently. For example, the
447+
[AWS
448+
sources](https://github.com/PipedreamHQ/pipedream/tree/master/components/aws)
449+
contains a `common` directory instead of a `common.js` file, and the directory
450+
contains several modules that are shared between different event sources.
451+
419452
### Props
420453

421454
As a general rule of thumb, we should strive to only incorporate the 3-4 most relevant options from a given API as props. This is not a hard limit, but the goal is to optimize for usability. We should aim to solve specific use cases as simply as possible.
@@ -436,6 +469,7 @@ user understand what they need to do. Use Markdown as appropriate
436469
to improve the clarity of the description or instructions. When using Markdown:
437470

438471
- Enclose sample input values in backticks (`` ` ``)
472+
- Refer to other props using **bold** by surrounding with double asterisks (*)
439473
- Use Markdown links with descriptive text rather than displaying a full URL.
440474
- If the description isn't self-explanatory, link to the API docs of the relevant method to further clarify how the prop works. When the value of the prop is complex (for example, an object with many properties), link to the section of the API docs that include details on this format. Users may pass values from previous steps using [expressions](/workflows/steps/params/#entering-expressions), so they'll need to know how to structure that data.
441475

@@ -483,7 +517,11 @@ when interacting with the source (e.g., “My Project”). The code referencing
483517
selection receives the numeric ID (12345).
484518

485519
Async options should also support [pagination](../api/#async-options-example)
486-
(so users can navigate across multiple pages of options for long lists).
520+
(so users can navigate across multiple pages of options for long lists). See
521+
[Hubspot](https://github.com/PipedreamHQ/pipedream/blob/a9b45d8be3b84504dc22bb2748d925f0d5c1541f/components/hubspot/hubspot.app.mjs#L136)
522+
for an example of offset-based pagination. See
523+
[Twitter](https://github.com/PipedreamHQ/pipedream/blob/d240752028e2a17f7cca1a512b40725566ea97bd/components/twitter/twitter.app.mjs#L200)
524+
for an example of cursor-based pagination.
487525

488526
#### Interface & Service Props
489527

@@ -497,6 +535,11 @@ In the interest of consistency, use the following naming patterns when defining
497535
| `$.interface.timer` | `timer` |
498536
| `$.service.db` | `db` |
499537

538+
Use getters and setters when dealing with `$.service.db` to avoid potential
539+
typos and leverage encapsulation (e.g., see the [Search
540+
Mentions](https://github.com/PipedreamHQ/pipedream/blob/master/components/twitter/sources/search-mentions/search-mentions.mjs#L83-L88)
541+
event source for Twitter).
542+
500543
### Source Guidelines
501544

502545
These guidelines are specific to [source](/event-sources/) development.
@@ -613,6 +656,14 @@ end user. Generate and use a GUID for the shared secret value, save it to a
613656

614657
### Action Guidelines
615658

659+
#### Action Name
660+
661+
Like [source name](#source-name), action name should be a singular, title-cased
662+
name, should not be slugified, and should not include the app name.
663+
664+
As a general pattern, articles are not included in the action name. For example,
665+
instead of "Create a Post", use "Create Post".
666+
616667
#### Use `@pipedream/platform` axios for all HTTP requests
617668

618669
By default, the standard `axios` package doesn't return useful debugging data to the user when it `throw`s errors on HTTP 4XX and 5XX status codes. This makes it hard for the user to troubleshoot the issue.
@@ -625,6 +676,32 @@ When you `return` data from an action, it's exposed as a [step export](/workflow
625676

626677
For example, some APIs return XML responses. If you return XML from the step, it's harder for users to parse and reference in future steps. Convert the XML to a JavaScript object, and return that, instead.
627678

679+
#### "List" actions
680+
681+
##### Return an array of objects
682+
683+
To simplify using results from "list"/"search" actions in future steps of a
684+
workflow, return an array of the items being listed rather than an object with a
685+
nested array. [See this example for
686+
Airtable](https://github.com/PipedreamHQ/pipedream/blob/cb4b830d93e1495d8622b0c7dbd80cd3664e4eb3/components/airtable/actions/common-list.js#L48-L63).
687+
688+
##### Handle pagination
689+
690+
For actions that return a list of items, the common use case is to retrieve all
691+
items. Handle pagination within the action to remove the complexity of needing
692+
to paginate from users. We may revisit this in the future and expose the
693+
pagination / offset params directly to the user.
694+
695+
In some cases, it may be appropriate to limit the number of API requests made or
696+
records returned in an action. For example, some Twitter actions optionally
697+
limit the number of API requests that are made per execution (using a
698+
[`maxRequests`
699+
prop](https://github.com/PipedreamHQ/pipedream/blob/cb4b830d93e1495d8622b0c7dbd80cd3664e4eb3/components/twitter/twitter.app.mjs#L52))
700+
to avoid exceeding Twitter's rate limits. [See the Airtable
701+
components](https://github.com/PipedreamHQ/pipedream/blob/e2bb7b7bea2fdf5869f18e84644f5dc61d9c22f0/components/airtable/airtable.app.js#L14)
702+
for an example of using a `maxRecords` prop to optionally limit the maximum
703+
number of records to return.
704+
628705
#### Use `$.summary` to summarize what happened
629706

630707
[Describe what happened](/components/api/#returning-data-from-steps) when an action succeeds by following these guidelines:

0 commit comments

Comments
 (0)