|
1 | 1 | # async-api-generator |
2 | 2 |
|
3 | | -This tool parses source code - of any type and looks for tags / annotations to extract. |
| 3 | +This tool parses source code - of any type and looks for tags / annotations to extract expressions and assign them to GenDoc statements. |
| 4 | + |
| 5 | +The goal of this utility is twofold: |
| 6 | + |
| 7 | +- produce an interim state from a single service/application and convert to [AsyncAPI](https://www.asyncapi.com) spec |
| 8 | + |
| 9 | +- produce a holistic view across all application/service emitted AsyncAPI documents to provide a wider view |
| 10 | + |
| 11 | +The driver is to remove manual creation/maintanence of AsyncAPI Specs and have them be generated from code - which would also ~force~ encourage documentation of code to live with code and hence making it easier to modify relevant code and accompanying documentation. |
4 | 12 |
|
5 | 13 | _Limitations by design:_ |
6 | 14 |
|
7 | | -- requires a closing tag as the source can be of any type and formatting is not guaranteed. Essentially using things like `{` or cursor column indicator, wouldn't be reliable enough |
| 15 | +- requires a closing tag as the source can be of any type and formatting is not guaranteed. Essentially using things like `{` or cursor column indicator, wouldn't be reliable enough as not everyone may have uniform formatting. |
| 16 | + |
| 17 | +_High level flow_ |
| 18 | + |
| 19 | + |
| 20 | +Living documentation [link](https://lucid.app/lucidspark/4d09749b-ad86-456b-a5ad-2f9e70a2b546/edit?viewport_loc=-1181%2C-734%2C4250%2C2266%2C0_0&invitationId=inv_bc6af9c8-3b41-4b00-b535-255d109f0afb) |
8 | 21 |
|
9 | 22 | ## Flow Overview |
10 | 23 |
|
11 | | -walk the directory and create a list of all source files - excluding specified such as bin, dist, vendor, node_modules, etc... |
| 24 | +Walk the directory and create a list of all source files - excluding specified such as bin, dist, vendor, node_modules, etc... |
12 | 25 |
|
13 | 26 | running each file as a source through a lexer i.e. performing a lexical analysis, where we identify tokens and build an AST (Abstract Syntax Tree) |
14 | 27 |
|
15 | | -> each file can be done in a separate go routine as there is no intrinsic relationship between them at this stage |
| 28 | +> each file can be done in a separate go routine as there is no intrinsic relationship between them at this stage. |
16 | 29 |
|
17 | | -### Lexer |
| 30 | +Here is a more [detailed internal overview](./docs/internals.md) |
18 | 31 |
|
19 | | -TOKENs are kept to the following types - `token.TokenType` - not listed as the list is likely to change/grow. |
| 32 | +### Usage |
20 | 33 |
|
21 | | -Special consideration will need to be given to files that are not able to contain comments or anything outside of their existing syntax - e.g. .json most commonly containing schemas. |
| 34 | +See [usage](./docs/usage.md) for more details. |
22 | 35 |
|
23 | | -> these cases a convention will need to be followed where by the name of the message that it is describing must be in the name of the file. |
| 36 | +## AsyncAPI standard |
24 | 37 |
|
25 | | -### Parser |
| 38 | +[AsyncAPI at Next](./docs/asyncapi.md) |
26 | 39 |
|
27 | | -Not using an existing parser generator like CGF, BNF, or EBNF is on purposes as the input source will only ever really be composed of parts we care about i.e. `gendoc` markers their beginning and end and what they enclose within them as text. |
| 40 | +## Local development |
28 | 41 |
|
29 | | -We'll use the overly simplified Pratt Parser (top down method) as we'll have no need to for expression parsing onyl statement node creation with the associated/helper attributes. |
| 42 | +To run the full flow locally you can use the local:// storage protocol. |
30 | 43 |
|
31 | | -## AsyncAPI standard |
| 44 | +See usage [local example](./docs/usage.md#LocalExample) for an example of how to work with and experiment locally with the outputs. |
32 | 45 |
|
33 | | -The current [AsyncAPI standard spec](https://www.asyncapi.com/docs/reference/specification/v2.6.0) is at version `2.6.0`. |
| 46 | +## Components |
34 | 47 |
|
35 | | -The tool will deal with all the relevant sections to be able to build an AsyncAPI spec file from within a single repo. |
| 48 | +For a full view of required components including publishing to the existing event catalog, the AsyncAPI document generator consists of the following auxilary components. |
36 | 49 |
|
37 | | -The asyncAPI is built from the `Application` - i.e. service down, each service will have a toplevel description - `info` key, which will in turn include |
| 50 | +### Schema Generation from DotNet |
38 | 51 |
|
39 | | -### Important Properties |
40 | | - |
41 | | -- `id` name of the service. Will default to parent folder name - unless overridden |
42 | | - - format: `urn:$business_domain:$bounded_context_domain:$service_name` => `urn:whds:packing:whds.packing.app` |
43 | | -- `application` |
44 | | - this is info about the service/application including descriptions and titles |
45 | | -- `channels` outlines the topics/subscriptions or queues the application produces or is subscribed to. |
46 | | - - `topic/queue/subscription` |
47 | | - will each contain a message summary description, schema, any traits - i.e. re-useable components - such as the envelop for common parameters |
| 52 | +Download SchemaGenerator for relevant architecture and run binary with required flags, see [here](./src/dotnet/Schema.Generator/README.md) for details. |
48 | 53 |
|
49 | 54 | ### EventCatalog binding |
50 | 55 |
|
51 | | -The translation of AsyncAPI into an EventCatalog set up. Whilst there are fairly standard mappings between the 2 processes - there are some nuances and requirements. |
| 56 | +The translation of AsyncAPI into an EventCatalog set up. Whilst there are fairly standard mappings between the 2 processes - there are some nuances and requirements, but this [plugin](./src/ts/eventcatalog-plugin-doc-generator-azblob/README.md) will be used to generate the eventcatalog compliant output. |
| 57 | + |
| 58 | +See on [NPM](TODO) and how to use it insid the existing eventcatalog app. |
| 59 | + |
| 60 | +### Infrastructure |
| 61 | + |
| 62 | +For remote publishing of generated interim and processed AsyncAPI documents a centralised bucket is required. The [definition](./terraform/examples/full/full.tf) and description of the flow can be found [here](./terraform/README.md). |
| 63 | + |
| 64 | +## Evolution |
| 65 | + |
| 66 | +Once satisfactory outcomes are achieved, the next step could be to generate clients directly. There are various options out there: |
| 67 | + |
| 68 | +- [Official AsyncAPI Generators](https://github.com/asyncapi/generator) |
| 69 | +- Write own? |
| 70 | +- There will be use cases when a single repo contains an entire definition needed for the doc generation to work, i.e. it will contain all the infrastructure/notes/models/schemas examples. IN this case a wrap around command can be exposed to combine the interim output generation with reading it and generating the full AsyncAPI. |
| 71 | + - E.g. the backstage.io required CRD can embed an entire AsyncAPI doc inside its spec payload. |
| 72 | + |
| 73 | +### Still needing attention |
| 74 | + |
| 75 | +- the merging of content within the same level could use a bit of work, namely not overwriting content defined in multiple places and perform a non-destructive merge where possible. |
| 76 | +- `servers` section inside the main `service` definition |
| 77 | +- `wip`: naming of operations and category types to allow for a more generic labelling types. e.g. categoryType of pubOperation is actually an eventcatalog subscriber. |
| 78 | + |
| 79 | +#### schemagenerator |
| 80 | + |
| 81 | +Dotnet CLI to inspect and generate schemas and JSON payload samples. |
| 82 | + |
| 83 | +TODO: |
| 84 | + - Generated Samples when includes datetime |
| 85 | + - ENUMs in the generated SCHEMAs can sometimes come up as empty `""` |
0 commit comments