Skip to content

feat(lazer): improve metadata in protobuf #2856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat(lazer): improve metadata in protobuf #2856

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
48521b5
feat(lazer): update metadata proto, rename and clarify docs
Riateche Jul 14, 2025
b1e808e
feat(lazer): add DynamicValue parsed type
Riateche Jul 16, 2025
1cb5d89
test(lazer): add DynamicValue tests
Riateche Jul 16, 2025
0129da1
feat(lazer): add FeedKind to protocol
Riateche Jul 17, 2025
f8c719b
feat(lazer): add feed kind to protocol
Riateche Jul 17, 2025
1054759
fix(lazer): expose dynamic_value protobuf as rust module, move type t…
Riateche Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lazer/publisher_sdk/proto/dynamic_value.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ syntax = "proto3";
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";

package pyth_lazer_transaction;
package pyth_lazer;

// A dynamically typed value similar to `google.protobuf.Value`
// but supporting more types.
Expand Down
109 changes: 75 additions & 34 deletions lazer/publisher_sdk/proto/governance_instruction.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,12 @@ import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";

import "dynamic_value.proto";
import "state.proto";

// If any field documented as `[required]` is not present in the instruction,
// the instruction will be rejected.

package pyth_lazer_transaction;
package pyth_lazer;

// Representation of a complete governance instruction. This value will be signed
// by a governance source.
Expand Down Expand Up @@ -130,9 +131,10 @@ message Permissions {
enum UpdateFeedAction {
// Required by protobuf. Instruction will be rejected if this value is encountered.
UPDATE_FEED_ACTION_UNSPECIFIED = 0;
UPDATE_FEED_METADATA = 101;
ACTIVATE_FEED = 102;
DEACTIVATE_FEED = 103;
UPDATE_FEED_PROPERTIES = 101;
UPDATE_FEED_METADATA = 102;
ENABLE_FEED_IN_SHARD = 103;
DISABLE_FEED_IN_SHARD = 104;
REMOVE_FEED = 199;
}

Expand Down Expand Up @@ -302,17 +304,31 @@ message SetPublisherActive {
optional bool is_active = 1;
}

// Feed is inactive when added, meaning that it will be available to publishers but not to consumers.
// Add a new feed. Refer to `Feed` message fields documentation.
message AddFeed {
// [required] ID of the feed. Must be unique (within the shard).
// [required]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this description removed? Is it no longer accurate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a feed ID needs to change as a result of moving to another shard which already has that ID (further, how do we prevent this from happening?), how would that impact subscribing to the feed? Is this ID separate from the customer/publisher facing feed ID?

optional uint32 feed_id = 1;
// [required] Feed metadata. Some properties are required (name, exponent, etc.).
// Known properties must have the expected type.
// Additional arbitrary properties are allowed.
// (TODO: document known metadata properties)
optional DynamicValue.Map metadata = 2;
// IDs of publishers enabled for this feed.
repeated uint32 permissioned_publishers = 3;
// [required]
optional DynamicValue.Map metadata = 3;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason we skip field ID 2 if this is not in use yet?

// [required]
optional string name = 101;
// [required]
optional sint32 exponent = 102;
// [required]
optional uint32 min_publishers = 103;
// [required]
optional google.protobuf.Duration min_rate = 104;
// [required]
optional google.protobuf.Duration expiry_time = 105;
// [required]
optional string market_schedule = 106;
// [required]
optional FeedState state = 107;
// [required]
optional bool is_enabled_in_shard = 201;

// TODO: IDs of publishers enabled for this feed.
// repeated uint32 permissioned_publishers = 3;
}

message UpdateFeed {
Expand All @@ -321,43 +337,68 @@ message UpdateFeed {
// [required]
// Note: when adding a new variant here, update `Permissions` as well.
oneof action {
UpdateFeedMetadata update_feed_metadata = 101;
ActivateFeed activate_feed = 102;
DeactivateFeed deactivate_feed = 103;
UpdateFeedProperties update_feed_properties = 101;
UpdateFeedMetadata update_feed_metadata = 102;
EnableFeedInShard enable_feed_in_shard = 103;
DisableFeedInShard disable_feed_in_shard = 104;
google.protobuf.Empty remove_feed = 199;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably don't need this, since we will make feeds inactive instead of deleting.

}
}

// Update a feed's properties. The feed will be updated with values present in each field.
// If a value is not supplied, the corresponding property will remain unchanged.
// Refer to `Feed` message fields documentation.
message UpdateFeedProperties {
// [optional]
optional DynamicValue.Map metadata = 3;
// [optional]
optional string name = 101;
// [optional]
optional sint32 exponent = 102;
// [optional]
optional uint32 min_publishers = 103;
// [optional]
optional google.protobuf.Duration min_rate = 104;
// [optional]
optional google.protobuf.Duration expiry_time = 105;
// [optional]
optional string market_schedule = 106;
// [optional]
optional FeedState state = 107;
// [optional]
optional bool is_enabled_in_shard = 201;
}

message UpdateFeedMetadata {
// [required] Property name.
optional string name = 1;
// [optional] Property value. If unset, the property will be removed.
optional DynamicValue value = 2;
}

// Set the feed as active or shedule an activation.
// If there was already a pending activation or deactivation, it will be cleared
// Set the feed as enabled in this shard or shedule it for a certain timestamp.
// If there was already a pending status change, it will be cleared
// when this governance instruction is processed.
// Warning: there must never be two feeds with the same name active at the same time
// Warning: there must never be two feeds with the same name enabled at the same time
// within a shard group. This cannot be enforced within a shard. When a feed needs to be
// moved between shards, use `activation_timestamp` and `deactivation_timestamp`
// to deactivate it in the old shard and activate it in the new shard at the same time.
message ActivateFeed {
// [optional] If provided, the feed will activate at the specified timestamp.
// If `activation_timestamp` is already passed or if it's unset,
// the feed will be activated immediately when this
// moved between shards, use `enable_in_shard_timestamp` and `disable_in_shard_timestamp`
// to disable it in the old shard and enable it in the new shard at the same time.
message EnableFeedInShard {
// [optional] If provided, the feed will be enabled at the specified timestamp.
// If `enable_in_shard_timestamp` is already passed or if it's unset,
// the feed will be enabled immediately when this
// governance instruction is processed.
optional google.protobuf.Timestamp activation_timestamp = 1;
optional google.protobuf.Timestamp enable_in_shard_timestamp = 1;
}

// Set the feed as inactive or shedule a deactivation.
// If there was already a pending activation or deactivation, it will be cleared
// Set the feed as disabled in this shard or shedule it for a certain timestamp.
// If there was already a pending status change, it will be cleared
// when this governance instruction is processed.
// See also: `ActivateFeed` docs.
message DeactivateFeed {
// [optional] If provided, the feed will deactivate at the specified timestamp.
// If `deactivation_timestamp` is already passed or if it's unset,
// the feed will be deactivated immediately when this
// See also: `EnableFeedInShard` docs.
message DisableFeedInShard {
// [optional] If provided, the feed will be disabled at the specified timestamp.
// If `disable_in_shard_timestamp` is already passed or if it's unset,
// the feed will be disabled immediately when this
// governance instruction is processed.
optional google.protobuf.Timestamp deactivation_timestamp = 1;
optional google.protobuf.Timestamp disable_in_shard_timestamp = 1;
}
4 changes: 2 additions & 2 deletions lazer/publisher_sdk/proto/publisher_update.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
syntax = "proto3";
package pyth_lazer_transaction;
package pyth_lazer;

import "google/protobuf/timestamp.proto";

Expand Down Expand Up @@ -28,7 +28,7 @@ message FeedUpdate {
// [required] timestamp when this data was first acquired or generated
optional google.protobuf.Timestamp source_timestamp = 2;

// [required] one type of update containing specific data
// [required] one type of update containing specific data
oneof update {
PriceUpdate price_update = 3;
FundingRateUpdate funding_rate_update = 4;
Expand Down
2 changes: 1 addition & 1 deletion lazer/publisher_sdk/proto/pyth_lazer_transaction.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
syntax = "proto3";
package pyth_lazer_transaction;
package pyth_lazer;

import "publisher_update.proto";
import "governance_instruction.proto";
Expand Down
103 changes: 56 additions & 47 deletions lazer/publisher_sdk/proto/state.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
syntax = "proto3";
package lazer;
package pyth_lazer;

import "google/protobuf/duration.proto";
import "google/protobuf/timestamp.proto";

import "dynamic_value.proto";

// All optional fields should always be set unless documented otherwise.

// State of a Pyth Lazer shard.
Expand Down Expand Up @@ -45,67 +47,74 @@ message Publisher {
}

enum FeedState {
COMING_SOON = 0; // Default value
// Default value. Feeds in this state are not available to consumers.
COMING_SOON = 0;
// A fully available feed.
STABLE = 1;
// Inactive feeds are not available to consumers or publishers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is technically available to publishers now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by "technically available"? Can you suggest a proper description of this state?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think he means publishers can publish to them regardless of the state of the feed? We should change this in the future though.

Perhaps you can specify they are feeds which can/can't be subscribed to?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Inactive feeds are not available to consumers or publishers.
// Inactive feeds are not available to consumers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only difference in INACTIVE and COMING_SOON is to inform the end user that inactive feed will never become active.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might let some people consume comin_soon ones too.

INACTIVE = 2;
}

// Static data for a feed.
message FeedMetadata {
// [required] ID of the price feed.
optional uint32 price_feed_id = 1;
// [required] Feed name.
optional string name = 2;
// [required] Feed symbol.
optional string symbol = 3;
// [required] Feed description.
optional string description = 4;
// [required] Feed asset type.
optional string asset_type = 5;
// Feed kind determines the set of data fields available in the feed.
// It also determines the kind of data accepted from publishers for this feed
// (`PriceUpdate` or `FundingRateUpdate`).
enum FeedKind {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intended to introduce this in some form. I will use the state enum to guard rail what updates a feed is able to accept.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should use this for guard railing updates

// Fields: price, best_bid_price, best_ask_price
PRICE = 0;
// Fields: price, rate.
FUNDING_RATE = 1;
}

// An item of the state describing a feed.
message Feed {
// [required] ID of the feed.
optional uint32 feed_id = 1;
// Additional state per publisher.
// If an eligible publisher is not listed here, the corresponding state should be considered empty.
repeated FeedPublisherState per_publisher = 2;
// [required] Additional metadata values. These values will be exposed in the APIs, but
// are not directly used in the aggregator.
optional DynamicValue.Map metadata = 3;

// [required] A readable feed name. It must be unique across all feeds in the shard.
// Used for logs, metrics, feed search API, TradingView API.
optional string name = 101;
// [required] Exponent applied to all price and rate values for this feed.
// Actual value is `mantissa * 10 ^ exponent`.
// Restricted to int16.
optional sint32 exponent = 6;
// [optional] CoinMarketCap ID. Can be absent if there is no CoinMarketCap ID for this symbol.
optional uint32 cmc_id = 7;
// [optional] Funding rate interval. Only present for funding rate feeds.
optional google.protobuf.Duration funding_rate_interval = 8;
optional sint32 exponent = 102;
// [required] Minimal number of publisher prices required to produce an aggregate.
optional uint32 min_publishers = 9;
optional uint32 min_publishers = 103;
// [required] Minimal rate of aggregation performed by the aggregator for this feed.
// Cannot be lower than the shard's top level `State.min_rate`.
optional google.protobuf.Duration min_rate = 10;
optional google.protobuf.Duration min_rate = 104;
// [required] Time after which the publisher update is discarded.
optional google.protobuf.Duration expiry_time = 11;
// [required] If true, the feed is visible to the consumers. This can be used to prepare and verify
// new feeds before releasing them. This can also be used to migrate a feed from
// one shard to another. If a feed is present in
optional google.protobuf.Duration expiry_time = 105;
// [required] Market schedule in Pythnet format.
optional string market_schedule = 106;
// [required] Feed state.
optional FeedState state = 107;
// [required] Feed kind.
optional FeedKind kind = 108;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add asset_type here too. Because we will be using asset_type to determine what kind of feed type is expected (e.g funding-rate vs regular price).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's call it feed type.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ali-behjati are you saying it's different from asset_type and we have both? Because I was thinking we could infer feed type based on asset_type. and multiple asset_types can have the same feed type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also prefer to have feed type (Price/FundingRate) because those options are what we care about in the implementation. (It's already like this in the transaction protobuf). Asset type has a lot of options and I think it's better to leave it in the dynamic metadata.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I called it FeedKind because type is annoying to use in Rust 😅


// [required] Feed status in the current shard. Disabled feeds will not be visible in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry i got a bit lost here with feedState. what cases does it cover that feed state is not :? is it like we add a same feed to all the shards that we need this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feed state is a consumer-facing and publisher-facing property that's also used to limit access to the feeds. is_enabled_in_shard is a purely internal thing for moving live feeds between shards.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it kind of feels complex but got it. but how do you envision it? on my head i was like we do it as a new feed creation but i think the price feed state is replicated everywhere.

// the consumer API for the current shard. This setting should only be used
// to migrate a feed from one shard to another.
//
// If a feed is present in
// multiple shards, it must only be active in one of them at each time.
// To enforce this, `pending_activation` and `pending_deactivation` fields
// To enforce this, `enable_in_shard_timestamp` and `disable_in_shard_timestamp` fields
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm my understanding, Aggregator pulls all shards. So, it is able to handle this switch in logic?

// can be used to deactivate a feed in one shard and activate it in another shard
// at the same instant.
optional bool is_activated = 12;
// [optional] ID of the corresponding price feed in Hermes (Pythnet).
optional string hermes_id = 13;
// [optional] Quote currency of the asset.
optional string quote_currency = 14;
// [optional] Market schedule in Pythnet format.
// If absent, the default schedule is used (market is always open).
optional string market_schedule = 15;
// [required] Feed state
optional FeedState state = 16;
}
optional bool is_enabled_in_shard = 201;
// [optional] If present, the aggregator will enable the feed in the current shard
// at the specified instant.
optional google.protobuf.Timestamp enable_in_shard_timestamp = 202;
// [optional] If present, the aggregator will disable the feed in the current shard
// at the specified instant.
optional google.protobuf.Timestamp disable_in_shard_timestamp = 203;

// An item of the state describing a feed.
message Feed {
optional FeedMetadata metadata = 1;
// [optional] If present, the aggregator will activate the feed at the specified instant.
optional google.protobuf.Timestamp pending_activation = 2;
// [optional] If present, the aggregator will deactivate the feed at the specified instant.
optional google.protobuf.Timestamp pending_deactivation = 3;
// Additional state per publisher.
// If an eligible publisher is not listed here, the corresponding state should be considered empty.
repeated FeedPublisherState per_publisher = 4;
// TODO: list of permissioned publisher IDs.
}

Expand Down
2 changes: 1 addition & 1 deletion lazer/publisher_sdk/proto/transaction_envelope.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
syntax = "proto3";
package pyth_lazer_transaction;
package pyth_lazer;

import "google/protobuf/timestamp.proto";
import "pyth_lazer_transaction.proto";
Expand Down
5 changes: 4 additions & 1 deletion lazer/publisher_sdk/rust/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,12 @@ repository = "https://github.com/pyth-network/pyth-crosschain"
pyth-lazer-protocol = { version = "0.9.0", path = "../../sdk/rust/protocol" }
anyhow = "1.0.98"
protobuf = "3.7.2"
serde-value = "0.7.0"
humantime = "2.2.0"
tracing = "0.1.41"
serde = { version = "1.0.219", features = ["derive"] }
serde_json = "1.0.140"
derive_more = { version = "2.0.1", features = ["from"] }
hex = "0.4.3"

[build-dependencies]
fs-err = "3.1.0"
Expand Down
Loading