Package & CLI v2

README.md

@@ -10,10 +10,6 @@ Mockingbird can send data to any downstream HTTP endpoint through Destination pl
 
 If you simply want to use the Web UI, you use the hosted one here: [https://mockingbird.tinybird.co/](https://mockingbird.tinybird.co/)
 
-## Docs
-
-Find the docs at [https://mockingbird.tinybird.co/docs](https://mockingbird.tinybird.co/docs)
-
 ## Usage
 
 ### Web UI
@@ -22,7 +18,7 @@ The Web UI provides and easy to use, guided experience. It should be pretty easy
 
 If you need help with defining schemas, or configuring particular Destinations, you can find more complete documentation [here](https://mockingbird.tinybird.co/docs).
 
-### Passing params in the URL
+#### Passing params in the URL
 
 If you want to re-use configurations from a previous session, you can simply save the URL. All settings are saved as parameters in the URL, so you can re-use and share configs with your team. For example: [http://localhost:3000/?schema=z_sales&eps=1&host=gcp_europe_west3&datasource=sales_dg&token=p.eyJ1IjogIjg4Nzk5NGUxLWZmNmMtNGUyMi1iZTg5LTNlYzBmNmRmMzlkZCIsICJpZCI6ICIwN2RlZThhMS0wNGMzLTQ4OTQtYmQxNi05ZTlkMmM3ZWRhMTgifQ.p_N4EETK7dbxOgHtugAUue3BUWwyGHT461Ha8P-d3Go](http://localhost:3000/?schema=z_sales&eps=1&host=gcp_europe_west3&datasource=sales_dg&token=p.eyJ1IjogIjg4Nzk5NGUxLWZmNmMtNGUyMi1iZTg5LTNlYzBmNmRmMzlkZCIsICJpZCI6ICIwN2RlZThhMS0wNGMzLTQ4OTQtYmQxNi05ZTlkMmM3ZWRhMTgifQ.p_N4EETK7dbxOgHtugAUue3BUWwyGHT461Ha8P-d3Go)
 
@@ -50,17 +46,129 @@ Here is an example of sending data to the Tinybird Events API:
   --limit 200
 ```
 
-Or if you use [Tinybird Local](https://www.tinybird.co/docs/cli/local-container):
+## Destinations
+
+### Tinybird
+
+[Tinybird](https://www.tinybird.co/) is a real-time data platform.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  Yes  |  Yes  |
+
+| Name        |                             Description                             |        Example value |
+| :---------- | :-----------------------------------------------------------------: | -------------------: |
+| Data Source |        The name of the Tinybird Data Source to send data to         |       my_data_source |
+| Auth Token  | The Auth Token with WRITE permissions on the configured Data Source | p.ePjdfbsdjnfeunf... |
+| Region      |          The Tinybird Region of the configured Data Source          |                   EU |
+| EPS         |     (Events Per Second) How many events per second to generate      |                 1000 |
+| Limit       |            Max number of rows to send (-1 for unlimited)            |                   -1 |
+
+### Ably
+
+[Ably](https://ably.com/) reliably distributes realtime data to your users using the publish/subscribe messaging pattern over WebSocket connections.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  Yes  |  Yes  |
+
+| Name       |                        Description                         | Example value |
+| :--------- | :--------------------------------------------------------: | ------------: |
+| API Key    |           Ably API key for basic authentication            |     xVLyHw... |
+| Channel ID |      The ID of the channel to which the data is send       |  rest-example |
+| EPS        | (Events Per Second) How many events per second to generate |          1000 |
+| Limit      |       Max number of rows to send (-1 for unlimited)        |            -1 |
+
+### AWS Kinesis
+
+[AWS Kinesis](https://aws.amazon.com/kinesis/) is a fully managed, scalable, and real-time data streaming service. It can continuously capture gigabytes of data per second from hundreds of thousands of sources, making it ideal for real-time analytics, application logs, and IoT device data.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  No   |  Yes  |
+
+| Name              |                        Description                         |   Example value |
+| :---------------- | :--------------------------------------------------------: | --------------: |
+| Region            |                         AWS region                         |       us-east-1 |
+| Access Key ID     |                      AWS credentials                       |     accessKeyId |
+| Secret Access Key |                      AWS credentials                       | secretAccessKey |
+| streamName        |       The name of the stream you want to publish to        |  my-stream-name |
+| partitionKey      |   Partition key to use for the data. Random by default.    |              "" |
+| kinesisOptions    |       Any additional options passed to KinesisClient       |              {} |
+| EPS               | (Events Per Second) How many events per second to generate |            1000 |
+| Limit             |       Max number of rows to send (-1 for unlimited)        |              -1 |
+
+### AWS SNS
+
+[AWS SNS](https://aws.amazon.com/sns/) is a fully managed Pub/Sub service for A2A and A2P messaging.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  Yes  |  Yes  |
+
+| Name              |                                 Description                                  |             Example value |
+| :---------------- | :--------------------------------------------------------------------------: | ------------------------: |
+| Region            |                                  AWS region                                  |                 us-east-1 |
+| Access Key ID     |                               AWS credentials                                |               accessKeyId |
+| Secret Access Key |                               AWS credentials                                |           secretAccessKey |
+| Topic ARN         |                       The topic you want to publish to                       | arn:aws:sns:us-east-1:... |
+| Subject           | Used as the "Subject" line when the message is delivered to email endpoints. |                        "" |
+| SNS Options       |                  Any additional options passed to SNSClient                  |                        {} |
+| EPS               |          (Events Per Second) How many events per second to generate          |                      1000 |
+| Limit             |                Max number of rows to send (-1 for unlimited)                 |                        -1 |
+
+### Confluent Cloud Kafka
+
+[Confluent Cloud Kafka](https://www.confluent.io/) is a cloud-native, fully managed Kafka service.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  No   |  Yes  |
+
+| Name          |                        Description                         |                                          Example value |
+| :------------ | :--------------------------------------------------------: | -----------------------------------------------------: |
+| REST endpoint |                        Endpoint URL                        | https://pkc-zpjg0.eu-central-1.aws.confluent.cloud:443 |
+| Cluster ID    |                      Kafka cluster ID                      |                                             p0s-f23... |
+| Topic         |                        Kafka Topic                         |                                                topic_1 |
+| API Key       |                     Confluent API Key                      |                                              FXVLSF... |
+| API Secret    |                    Confluent API Secret                    |                                             GxvPswX... |
+| Headers       |                      Optional headers                      |                                                     {} |
+| Key           |           Optional key to use for partitioning.            |                                                     {} |
+| EPS           | (Events Per Second) How many events per second to generate |                                                   1000 |
+| Limit         |       Max number of rows to send (-1 for unlimited)        |                                                     -1 |
+
+
+# Google Spanner
+
+[Google Spanner](https://cloud.google.com/spanner) is a relational, key-value, graph, and vector database. It is a highly scalable database that combines unlimited scalability with relational semantics, such as secondary indexes, strong consistency, schemas, and SQL providing 99.999% availability in one easy solution.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  No   |  Yes  |
+
+| Name        |                Description                |    Example value |
+| :---------- | :---------------------------------------: | ---------------: |
+| projectId   |              GCP Project ID               |   myfirstproject |
+| instanceId  |            Spanner Instance ID            |       myinstance |
+| databaseId  |            Spanner Database ID            |       mydatabase |
+| table       |            Spanner Table Name             |          mytable |
+| keyFilename | Path to GCP Service Account Key JSON file | ../my.creds.json |
+
+# RabbitMQ
+
+RabbitMQ is a popular open source message broker.
+
+|  Web  |  CLI  |
+| :---: | :---: |
+|  No   |  Yes  |
+
+| Name               |               Description                |        Example value |
+| :----------------- | :--------------------------------------: | -------------------: |
+| Endpoint           |            RabbitMQ endpoint             |     amqp://localhost |
+| Queue              |              RabbitMQ queue              |             my-queue |
+| AssertQueueOptions | RabbitMQ assertQueue options JSON string |   {"expires": 60000} |
+| PublishOptions     |   RabbitMQ publish options JSON string   | {"persistent": true} |
 
-```sh
-> TB_ENDPOINT=http://localhost mockingbird-cli tinybird
-  --schema schema.json \
-  --datasource "my_data_source" \
-  --token "e.Pdjdbfsbhksd...." \
-  --endpoint "custom" \
-  --eps 50 \
-  --limit 200
-```
 
 ## Contributing
 
@@ -85,7 +193,7 @@ The generator is written in TypeScript & uses [Faker.js](https://fakerjs.dev/) u
 
 #### Adding new Data Types
 
-DataTypes are defined in [/packages/mockingbird/src/extendedFaker.ts](./packages/mockingbird/src/extendedFaker.ts).
+Data Types are defined in [/packages/mockingbird/src/extendedFaker.ts](./packages/mockingbird/src/extendedFaker.ts).
 
 To add a new Data Type, add a new item to the `mockingbirdModule` object.
 
@@ -96,16 +204,18 @@ The value of the item must be a function that returns the desired value. The fun
 For example, a custom Data Type that takes no input params:
 
 ```javascript
-latitudeString: () => faker.address.latitude().toString(),
+myCustomType: () => "I return this awesome string!",
 ```
 
 A custom Data Type that accepts incoming parameters:
 
 ```javascript
-pick: (params: { values: unknown[] }) =>
-    params.values[Math.floor(Math.random() * params.values.length)],
+pick: (params: number) =>
+    params.values[Math.floor(Math.random() * params)],
 ```
 
+The `params` are passed directly to the function as-is, so they can be any type you like.
+
 #### Adding new preset schemas
 
 Preset schemas are defined in [/packages/mockingbird/src/schemas/](./packages/mockingbird/src/schemas).
@@ -117,13 +227,12 @@ The contents of the file should look like this:
 ```javascript
 import { Schema } from "../types";
 
-const newSchema: Schema = {
+export const newSchema: Schema = {
   timestamp: {
     type: "mockingbird.timestampNow",
   },
 };
 
-export default newSchema;
 ```
 
 Ensure a new schema is exported in [./packages/mockingbird/src/schemas/index.ts](./packages/mockingbird/src/schemas/index.ts)
@@ -153,12 +262,4 @@ npm install
 npm run dev
 ```
 
-This will serve both the Web UI & the documentation site locally. By default, the UI is available on `http://localhost:3001` and the docs are on `http://localhost:3000`.
-
-### Docs
-
-The Docs are under `./apps/docs`
-
-The Docs are written in MDX using [Nextra](https://nextra.site/) as a static site generator.
-
-To run the docs locally, see the instructions for the running the [Web UI](#running-the-local-dev-server).
+This will serve both the Web UI & the documentation site locally. By default, the UI is available on `http://localhost:3001` and the docs are on `http://localhost:3000`.

apps/cli/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@tinybirdco/mockingbird-cli",
   "description": "CLI for Mockingibrd",
-  "version": "1.5.1",
+  "version": "2.0.0-rc.0",
   "main": "index.js",
   "bin": "index.js",
   "scripts": {
@@ -25,7 +25,8 @@
     "url": "https://github.com/tinybirdco/mockingbird/apps/cli"
   },
   "dependencies": {
-    "@tinybirdco/mockingbird": "^1.5.0",
+    "@tinybirdco/mockingbird": "file:../../packages/mockingbird",
+    "builtin-modules": "^4.0.0",
     "yargs": "^17.7.2"
   },
   "devDependencies": {

apps/cli/subcommands.js

@@ -3,8 +3,11 @@ import {
   AWSSNSGenerator,
   LogGenerator,
   TinybirdGenerator,
-  NodeGenerators,
-} from "@tinybirdco/mockingbird";
+  GoogleSpannerGenerator,
+  AWSKinesisGenerator,
+  RabbitMQGenerator,
+  ConfluentCloudKafkaGenerator,
+} from "@tinybirdco/mockingbird/server";
 
 export const subcommands = [
   {
@@ -61,7 +64,7 @@ export const subcommands = [
   },
   {
     name: "confluent-cloud-kafka",
-    generator: NodeGenerators.ConfluentCloudKafkaGenerator,
+    generator: ConfluentCloudKafkaGenerator,
     options: {
       restEndpoint: {
         describe: "Confluent Cloud Kafka REST endpoint",
@@ -101,7 +104,7 @@ export const subcommands = [
   },
   {
     name: "rabbitmq",
-    generator: NodeGenerators.RabbitMQGenerator,
+    generator: RabbitMQGenerator,
     options: {
       endpoint: {
         describe: "RabbitMQ endpoint",
@@ -171,7 +174,7 @@ export const subcommands = [
   },
   {
     name: "aws-kinesis",
-    generator: NodeGenerators.AWSKinesisGenerator,
+    generator: AWSKinesisGenerator,
     options: {
       region: {
         describe: "AWS Region",
@@ -206,7 +209,7 @@ export const subcommands = [
   },
   {
     name: "google-spanner",
-    generator: NodeGenerators.GoogleSpannerGenerator,
+    generator: GoogleSpannerGenerator,
     options: {
       projectId: {
         describe: "GCP Project ID",

apps/cli/utils.js

@@ -1,4 +1,4 @@
-import { presetSchemas } from "@tinybirdco/mockingbird";
+import { presetSchemas } from "@tinybirdco/mockingbird/server";
 import fs from "fs";
 
 export const presetSchemaNames = Object.keys(presetSchemas);