Plugins Support into Chipmunk branch #2196
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
*** These commits has been squashed for rebase *** *** Changes while rebase included too *** ******************************************************** Change parse function return type Parse function returns how many bytes have been consumed instead the rust of the slice to eliminate the reference and the lifetime Parse returns vector of results (in progress) - Change parse signature to return a vector of values instead of one item at a time. - Basic adjustments for producer logic to accept vector of parse results. - Adjust the current parsers temporally by wrapping their single return value inside a vector - Adjust parsers unit tests assuming that each parser will still return a vector of one value on each call. Plugin: Add simple temporary benchmarking code Assert parsing ends on first error temporarlly - Producer logic assumes that parsing will stop after encountering the first error. This check should be done in unit tests but for now it's done in code for now Plugin Host and API (in progress) - Create library for plugins host - Create wit file for parser on directory where plugins API crate will be initialized Parse return `impl IntoIterator` instead of `vec` - This can give more flexibility for other implementations of pares to return other types than vector - Parsers with a single return value can use `iter::onc` or an array to enable more compiler optimization on the code Move return statement in producer loop - Returning the results should happen after the parse iterator is finished. It was positioned at the end of the first iteration of parse call. Plugin: Wit files definitions & Prototype Clients - Wit files: - Use a directory for each version. - Use a separate interface for shared types. - Create wit file for bytesource - Small changes on parse wit file - Initialize rust lib and created place holder plugins Plugin: Initialize WASM host lib Plugins: Initialize WASM Engine Plugins: Provide general configuration for plugins - Parser and Bytesource plugins should get general chipmunk configuration besides their user-define configurations as string (or path to a file) - Currently the configurations structs are just palce-holders for chipmunk actual configs Plugin: Implementing Parser Host (in progress) - Create Parse Message type for parser plugins supporting only strings - Create Plugin Parser State, but its implementation is still missing - Bindgen Macro Plugin: Implementing Parser Host (in progress) - Implement Initialization of Parser Plugin - Error definitions for plugins - Move shared types and functionality to shared module - Move bindings module to their own file - WasmPlugin trait with basic implementation - Create Bytesoruce Plugin place holder with implementation for input types only Plugin: Parser Host & Versioning (in progress) - Move version specific parts to their own module to have one module for each version for the parts that can be changed - Define shared types and functions among different versions in their separate module - Provide Mapping between bindings types and shared types. - Create semantic version structure with parsing - Implement reading component meta data by loading and perform simple validation and version parsing to initiate the corresponding host version - Use std::OnceLock instead of tokio::OnceCell for creating wasm host since we don't need the function to be async. - Change error type `unsupported` in wit file to provide a custom message too. Plugin: Clippy Fixes Plugin Parser Host: Implement Parser Trait - Implement both parse functions using add method or returning a collection of results - Provide the needed mapping between guest and host results Plugin Parser: Add Unrecoverable Error type - Add Unrecoverable error to parse results to cover panics inside the plugins - Currently we log the error and print it to stderr and stop the parsing Plugins: Parser Integrations (in progress) - Introduce new parser type for the plugins with it's settings data type - Add Error Kind to the native errors for the plugins and map plugin initialization error to it. - Implement plugins parser Integrations in sessions export and observing - Make configurations path optional for parser and source plugins - Set `duplication_if_necessary` to true on parser host bindgen Integrate Plugin parser in CLI interactive tool - Option `plugin` is added to CLI interactive sessions - Plugin path will be defined via environment variables temporally. Plugins: Use interfaces in wit files Interfaces in wit files contain the current version meta data as well which is used to detect the plugin version to select the matching host version. Plugins: Implement missing mapping for parser host Plugin: Fix producer loop logic - Return was called from within the inner loop Producer Stream: Change return type to vector of values - Producer stream will yield all the available items at once instead of caching them Force using plugin parsers via env vars temporally Plugins: Parser Client Defs & Macro (in progress) - Define `Parser` trait which must be implemented by plugins authors - Implementing `export_parser` macro to be used by users - global paths for crate types still missing - Unit tests still missing
- Define `Parser` trait which must be implemented by plugins authors - Implementing `export_parser` macro to be used by users - global paths for crate types still missing - Unit tests still missing
- Define which types from bindings will be exported to the plugins implementers - Export types needed in export macros as hidden - Modify generate! Macro attributes to make its export macro public and set the bindings types for its needed types - Set all types withing export macros with their full path.
Apply fixes and changes for the parser client after trying to use it with real external plugin. Changes include: - Separate parser and bytesource with features to prevent generate! Macro from being called multiple times. - Move add function on the host to it's own interface that will be imported separately to avoid compilation errors because of duplicate definitions of types that occur when using the crate with separate projects as plugins only + Make the needed adjustments on the host side. - Make generated bindings public but hide them in documentations to because they are needed with export Macro - Move Prototyping code to test module temporally.
- Insure Parser trait from Chipmunk is in scope inside export macro so we can call parse method successfully. - Remove prototyping suppressing warning + Corresponding fixes
- Use the crate `TryBuild` to insure that the wrong patterns won't compile - Add successful test case when Parser trait is implemented. - Add failing tests for not implementing Parser Macro or for completely wrong input with trait or expression. - Ignore the comparison of compilation error messages for the tests since we still not sure if we gonna include this comparison on the future, because the comparison is text-based and can be broken on different Rust editions or different environments - Write script to run tests since they are behind inactive features
- Insure that one feature form Plugin-API crate at most is enabled at a time to provide the users with meaningful error message instead of the encrypted linking error if they enable many features at once since doing so will lead to linking errors on release builds only with very weird error messages
- Add test for the case when parser implementation and export macros are in different modules
- Internal modules now start with underscore to make it more clear for the users that they shouldn't use them - A warning have been added for the users if they explored the source code - This idea is taken from the crate 'Log'
- Add logging definitions and functions to wit file - Creating module on the client side to provide clean logging public API while handling log level complexity internally to direct avoid log calls to the host when the log level isn't allowed - Integrating the log level in the macro is still missing
- Provide custom logger that implements `log::Log` trait and register it using boxed logger. - This enables the users from using the general log macros from `log` crate and the message will be validated on the client side then sent to the host. - Added log level to the custom configurations for the parser in the wit file and implement it on the client side only for now.
- Use static logger instead of the boxed one because since the current max log level can be retrieved from log crate itself.
- Add unit tests for plugins parser - Change attributes and module name for prototyping code to make sure this wouldn't be compiled on any use case beside activating it manually with commenting out the attribute
- Implement logging on parser host. - Log Level will be always the current leg level in chipmunk since we use the same log functionality to log plugins' messages
Reexport log crate from chipmunk client crate because it's used in export macro, and the users can use it directly without having import their own version of the same library.
- Use sending parse results to the host instead of using the host add function by default since it's more readable and until know there is no noticeable performance difference between the two approaches
- Add configurations, attributes to show all the features in crate documentation, showing a hint for items available on features only. - Provide a script file to show the documentation with all needed attributes and environment variable to replicate what will be shown on docs.rs once the crate is published - Provide documentations and comments the implemented part of the crate
- Fixes inside the macro - Adding missing use statements for the doc example
- Use add method on the host to add parse results after changing client implementation to avoid collecting memory for all the results before sending them to the host. - Rust warns about undefined cfgs ab version 1.80, therefore the prototyping code must go under test cfg for now
- We intended to have our own plugins within the same repo and it makes sense to preserve the directory name `Plugins` for the actual plugins
- Add logging definitions to wit file - Implement WIT bindings in API crate, reexporting the needed types on to the public API - Define Bytesource Trait for plugins developers to provide the bytesource functionality. - Implement export macro for bytesource using the ByteSource trait and providing initialization for the instance and logging - Documentation tests are provided but the integration tests are still missing
- Extending trait test should pass if it's implemented currently. The previous implementation failed because of syntax errors - Remove unused using statement
- Add tests for multiple cases where the code must and can't compiled and assert the compilation results
- Added a temporary check to insure at least one feature at least is activated when running tests. This check can be removed after including the tests in CI pipeline - Remove scripts to run tests since we can use `cargo test --all-features` command instead of it.
- Provide an example of a simple implementation of the bytesource plugin. Which opens a file and read its content, providing them to Chipmunk when it calls for them - The implementation uses the config path as file path temporary until an input source type is defined and used. - Documentation are still work in progress
- Move initialization errors to plugins shared module - Re-activate warning checks for parser and general parts and fix all warnings - Add missing renaming libraries to rs-bindings Cargo.lock - TODOs & Comments
- Move plugins types to their own modules in sessions lib - Basic types and methods for byte source - Fix typos in directory name
Seek trait bound isn't used in any case for byte source and removing it will spare us from forcing implementing it on the source plugins
Deliver all possible configuration items in string parser and print their values, to be used for debugging while implementing the templates of them in Chipmunk.
* Added configurations renderer for integer and float types. * Drop-down and path aren't implemented and won't be shown
* Change Config type `Number` to `Integer` for clarity. * Change Config type `Path` to `Paths` making it provide multiple file paths at once. * According Adjustments in Plugins, Examples, Core and UI. * Implement UI for paths providing file picker. * UI for directory is still missing.
* Use different controls for the paths of files and directories. * File paths provide the allowed files extensions. * Show the full paths for files and dirs with tooltips.
* Extends the types for the render options for parser plugins providing more infos like description and widths. * Extend proptets and tests in ts-bindings with the new types. * Implement render options for plugins.
Extend `string parser` showing how to return render options changing its return type to two columns one for the message and the other for its length.
* Extend file types with `Plugins` type to avoid changing the current logic. * Extend the code everywhere where the file types are used with suitable code for the plugins. * Add menu command to open folder with plugins.
* Rename file type `plugins` to `parserPlugin` since it's intended to work with the parser plugins only and not the bytesource or other plugin kinds. * Add missing dialogs to electron file and folder services. * Extend types and implementation in rust core and their corresponding proptests
Add plugins parser to all kinds of stream inputs.
* Fixes exporting the results of plugins into new tap. * Resolve various todos in code. * Remove deprecated communication types.
* Split WIT interface into three packages: (plugins, bytesource, shared) to generate the shared items once only and enable removing the features on the plugins-api crate since the bindings don't clash anymore. * This separation changed the whole structure of the WIT files and made it similar to the structure of wasi crate. * Generate unified bindings for the shared types and move all their code into the shared module in plugins-api crate. * Adjust the unit tests for the macros accordingly, and remove all the wild card imports. * Adjust example plugins and make then compile. * Plugin host doesn't compile currently. * Documentations in WIT and plagins-api still missing.
* Document all modules and types in the public API of the plugins_api crate * Small improvements for the wit files. * Fix warnings in bytesource tests.
* Make the need adjustments after separating the plugins to different packages unifying the implementation for the shared types. * Generating the shared types doesn't generate not referenced types with `bindgen!()` macro. The current workaround is to generate the types from parser package inside the shared module. * State `generate_used_types` explicitly to true in `generat!()` macro for plugins_api because this macro supports this option
* Provides functions to validate the retrieve the values of the configurations to reduce the boilerplates for the plugins developers. * A macro has been used to generate the functions with a reminder to get compiler errors on type changes. * Change the code in the example plugins to use the new functionality.
Introduce another macro to return owned value for configuration values where it doesn't make sense to return a reference to a boolean or for an integer.
Example shouldn't be tracked while building Chipmunk and they should be visible on the top level of the repo.
Not all logging functions can be run in the plugins environment and changes within the logging configurations can cause problem. Therefore we re export logging macros only and the other functions needed withing the export macros will be undocumented and start with underscore.
* Create an example of DLT parser plugin that replicates the dlt parser built in Chipmunk to check how powerful the current implementation is and for benchmarking. * Current solution is working without filter support and without timestamps and formatting options. * Solution still have TODOs in the code for improvements.
Conflicts:
application/apps/indexer/stypes/Cargo.toml
application/client/src/app/ui/tabs/module.ts
application/client/src/app/ui/tabs/observe/parsers/general/module.ts
application/holder/src/service/actions/index.ts
application/platform/ipc/request/actions/index.ts
* Fixes included adapting the UI components to current angular version.
* Define default value for plugins in wit file and plugins api * Matching adjustments on the example plugins. * Implementation in host is still missing (Host won't compile currently)
* Implement support for configurations default values in backend and front end of the host, changing the types and fixing there proptests as well. * All control on front-end provide a default value on initialization to avoid having some configurations not existed if users haven't changed them. * Move shared configurations functions to state struct to be reachable in multiple places in code.
Provide documentation for developing plugins, building the examples and integrating them within Chipmunk.
Rust: * Plugin api: Remove checks for building with all features since they are not conflicting anymore. * Remove all code for forcing parser plugin via environment variables since it's not needed anymore for parser plugins. * Resolve smaller todos in plugins host and api. * Documentation and comments about plugins in Rust code. Type Script: * Remove not needed methods for byte-source plugins. * Remove not needed angular imports. * Change some TODOs asking Dmitry to help with them since I don't have enough overview on the front-end aspect. * Resolve other TODOs in front-end
Conflicts:
application/client/src/app/service/actions.ts
application/client/src/app/service/actions/index.ts
application/holder/src/service/actions/index.ts
application/platform/ipc/request/actions/index.ts
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will merge the current implementation of the plugins system into a separate branch on the mainstream repo to enable working on the plugins system from multiple developers at the same time.
Related PR: #2056 with more infos.
Current implementation:
ParserandByte Sourceplugins are defined using WASM and the component model.plugins-apiprovides WIT type bindings, more clear definition via Rust type system, export macros and multiple helper functions for logging and extracting configurations. This crate should be publish to crates.io.dlt_parser,string_parserandfile_sourceare provided as well.