uAgent almanac tracking

[bryanchriswhite]

Acceptance criteria

1. Almanac registration events are tracked by a respective entity.
2. Live almanac registrations must be distinct from historic ones and must exclude expired registrations.

[bryanchriswhite]

Proposal

To facilitate expiration, we can add some code to the block primitive handler which checks for "resolutions" which are expired/ing (and deletes them). "Resolutions" would reference "registrations", which are the entity tracking historical registration events.

Separating "resolutions" and "registrations" allows us to remove "resolution" records as they expire while still serving the basic indexer function of preserving historical state; i.e. historical registration event data is recoverable via "registrations". It is in fact prudent that we delete these records because we're then asking the block handler (each block) to do an index scan, minimum, of that table. This approach decouples the performance loss in the block handler from the cumulative number of registrations, mitigating an unnecessary slow-down that would otherwise be introduced to the block handler. There will still be a performance hit but it will instead be correlated to the number un-expired registrations ("resolutions").

Schema:

type AlmanacService @jsonField {
  protocols: [String!]
  endpoints: [String!]!
}

type AlmanacRecord @entity {
  id: ID!
  service: AlmanacService
  registration: AlmanacRegistration! @derivedFrom(field: "record")
}

type AlmanacResolution @entity {
  id: ID!
  address: String! @index
  record: AlmanacRecord!
}

type AlmanacRegistration @entity {
  id: ID!
  # TODO: account (?)
  address: String! @index
  expiryHeight: BigInt! @index
  record: AlmanacRecord!
  event: Event!
  transaction: Transaction!
  block: Block!
}

[ejfitzgerald]

Makes sense to me the only thing to consider is that with the new version of the contract API

type AlmanacService @jsonField {
  protocols: [String!]
  endpoints: [String!]!
}

becomes more like

type AlmanacEndpoint @jsonField {
  endpoint: String!
  weight: Int!
}

type AlmanacService @jsonField {
  protocols: [String!]
  endpoints: [AlmanacEndpoint!]!
}

The weights work in the similar way to DNS fields

[bryanchriswhite]

I ran into some unexpected complications regarding foreign key constraints when attempting to relate AlmanacRegistration to Event. My current hypothesis is that handleEvent may not have finished at the time which handleAlmanacRegistration is attempting to save the registration, resulting in a foreign key constraint violation as the related event hasn't been committed yet.

I also experienced an issue while trying to add the reverse relationship between AlmanacResolution and AlmanacRegistration (@derivedFrom(field: "record")):

ledger-subquery-subquery-node-1  | 2022-11-21T14:06:05.297Z <store> ERROR Having a probl
em when syncing schema SequelizeDatabaseError: syntax error at or near "'almanac_registr
ations_record_id_uindex'"   

[bryanchriswhite]

Waiting for the contract-almanac v0.2 changes.

[bryanchriswhite]

Sprint 12 > 13 update:

Registrations

• E2E tests
• Schema updates
• Handler implementation
• Migration (+ debugging & testing)
• Dependency permission issues
  • submodule access
  • private repo release asset
• Logging woes (see fix: set WARNING level on gql aiohttp transport logger #223)

Resolutions

• E2E tests
• Schema updates
• Handler implementation
  • Cache / expire control flow & basic logic
  • DB query optimization on expiration (each block)
• Migration (+ testing)