-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Description
Goal
Homogenize the three extractor crates (firehose, evm-rpc, solana) to have consistent structure, module names, function names, and type names. The end result should look as if all extractors were created from the same cookie-cutter template. Aligning them will improve maintainability, make it easier to onboard new extractors, and is a prerequisite for the dataset↔extractor decoupling needed for SQL DDL execution.
Current State
The three extractor crates have inconsistent structure:
| Aspect | firehose |
evm-rpc |
solana |
|---|---|---|---|
| Client module | client.rs → Client |
client.rs → JsonRpcClient |
rpc_client.rs + of1_client.rs + extractor.rs → SolanaExtractor |
| Error handling | Error enum in lib.rs |
error.rs → ProviderError |
error.rs → ExtractorError |
| Dataset kind | FirehoseDatasetKind |
EvmRpcDatasetKind |
SolanaDatasetKind |
| Metrics | metrics.rs (pub) |
metrics.rs (pub) |
metrics.rs (private) |
| Tables | evm/tables/ (nested) |
tables/ |
tables.rs + tables/ |
| Provider config | N/A (inline) | provider.rs (pub) |
re-exports from amp-providers-solana |
| Extra modules | proto/, evm/ |
— | — |
Module structure comparison
firehose/src/ evm-rpc/src/ solana/src/
├── client.rs ├── client.rs ├── rpc_client.rs
├── dataset_kind.rs ├── dataset_kind.rs ├── of1_client.rs
├── dataset.rs ├── dataset.rs ├── extractor.rs
├── evm/ ├── error.rs ├── dataset_kind.rs
│ ├── mod.rs ├── lib.rs ├── dataset.rs
│ ├── pb_to_rows.rs ├── metrics.rs ├── error.rs
│ └── tables/ ├── provider.rs ├── lib.rs
│ ├── mod.rs └── tables/ ├── metrics.rs
│ ├── calls.rs ├── mod.rs ├── tables.rs
│ └── transactions.rs └── transactions.rs └── tables/
├── lib.rs ├── block_headers.rs
├── metrics.rs ├── block_rewards.rs
└── proto/ ├── instructions.rs
├── messages.rs
└── transactions.rs
Naming inconsistencies
- Error types:
ErrorvsProviderErrorvsExtractorError - Client types:
ClientvsJsonRpcClientvsSolanaExtractor - Module visibility: mixed
pubvs private for equivalent modules
Desired Outcome
All extractor crates should follow a consistent template. The end result should look as if they were all generated from the same cookie-cutter:
- Same module names for equivalent functionality
- Same type naming conventions (e.g., error type name pattern, client type name pattern)
- Same module visibility for equivalent modules
- Same structure for
tables/organization - Consistent approach to provider config, error handling, and metrics
Notes
- One crate per PR
- The dataset↔extractor decoupling is essential for executing SQL DDL properly
- This work can be done in parallel with the config decoupling effort (Decouple internal crate config types from amp-config #1790)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels