Skip to content

Latest commit

 

History

History
119 lines (97 loc) · 8.79 KB

architecture.md

File metadata and controls

119 lines (97 loc) · 8.79 KB

🏗️ Architecture

Simplified (TL;DR)

OPAL consists of two key components that work together:

  1. OPAL Server

    • Creates a Pub/Sub channel clients subscribe to
    • Tracks a git repository (via webhook / polling) for updates to policy (or static data)
      • Additional versioned repositories can be supported (e.g. S3, SVN)
    • Accepts data update notifications via Rest API
    • pushes updates to clients (as diffs)
    • scales with other server instances via a configurable backbone pub/sub (Currently supported: Postgres, Redis, Kafka; more options will be added in the future)

  2. OPAL Client

    • Deployed alongside a policy-agent, and keeping it up to date
    • Subscribes to Pub/Sub updates, based on topics for data and policy
    • Downloads data-source configurations from server
      • Fetches data from multiple sources (e.g. DBs, APIs, 3rd party services)
    • Downloads policy from server
    • Keeps policy agents up to date


simplified


In Depth Architecture (Components diagram)

main-numbered

  • Legend:

    • Blue line: Data flows
    • Purple line: Policy flows

  • Components numbered in the diagram

    1. OPAL-Server

      • The Server managing data and policy; exposing REST routes for clients to retrieve configurations and Pub/Sub channel for clients to subscribe to updates

    2. OPAL-Client

      • The client, running at edge, adjacent to a policy-agent. Subscribes to data and policy updates. Act's on data-updates to approach data sources and aggregate data from them.
      • Clients open an outgoing websocket connection to servers, thus overcoming most firewall / networking challenges, creating a bi-directional Pub/Sub channel.

    3. Open-Policy-Agent (OPA)

      • The policy engine OPAL augments - by default this is OPA

    4. Application services

      • The application OPAL powers the authorization layer for

    5. Data service

      • A service (or multiple services) as part of the application that connect to the OPAL-server to notify of changes to authorization data (guiding OPAL to aggregate data from the relevant data-sources)

    6. GIT repository

      • A versioned store for the authorization policy
      • Triggers a webhook to the OPAL server to notify of policy changes
      • Can potentially be other version controlled stores (e.g. SVN, Perforce, S3)

    7. Data sources

      • Various services (internal and external) that hold and serve data that needs to be used for authorization decisions (by the policy agents)
      • Examples:
        • App REST API service for user list
        • SQL DB storing user roles
        • Billing SaaS service (e.g. Stripe) to
      • OPAL-Clients can be extended with different FetchProviders to allow extraction of data from various sources.

    8. OPAL admins

      • Developers maintaining the application.
      • Can drive new policies in realtime by simply pushing to their version control (Can simply be the same repository used by CI/CD for the whole app)
      • Can configure and control OPAL clients and their associated policy-agents from a unified control plane
      • Can change the applications data and easily sync authorization to match it.

    9. End users

      • The users of the application, enjoying a seamless experience - working each within their authorization bounds (enforced by open-policy-agents) - e.g. user permissions, roles, tenants.
      • Through OPAL the application's authorization layer adjusts to their needs in realtime - a new user invited can access instantly; new permissions assigned take affect at once. No waiting, no redeployments.
      • Thanks to OPA all requests are processed for authorization in record-breaking speed.

  • Lightweight Pub/Sub and Backbone Pub/Sub

    • OPAL's architecture potentially uses two Pub/Sub channels-

      1. Client <> Server - lightweight websocket Pub/Sub
      2. Server <> Server - backbone Pub/Sub

      While the lightweight channel requires no additional infrastructure, and can suffice for the can we are running only a single OPAL-server. If we wish to scale-out OPAL-servers, we achieve this using a backbone Pub/Sub (such as Redis, Kafka, or Postgres Listen/Notify) to sync all the servers (So a client connecting to one server, receive notifications of updates that are triggered by another server) The backbone Pub/Sub is connected to the lightweight Pub/SUb through the Broadcaster module.

  • Communication flows

    The following text describes common data/policy events/scenarios in an OPAL system.

    • User flows:

      • Flows triggered by

      • Authorization queries:

        • Users -> App -> PolicyAgent
        • Users interact with the application triggering authorization queries that the application resolves with the policy agent (directly). The policy agent is constantly kept up-to date by OPAL so it can answer all the queries correctly.

      • Authorization data changes:

        • Users -> App data service -> OPAL-server -> OPAL-clients -> Policy Agents
        • Users (e.g. customers, operators, partners, developers) affect the applications authorization layer (e.g. create new users, assign new roles). Application service triggers an event to OPAL-server, which notifies the clients, which collect the needed data from the affected sources.

    • Policy flows

      • Admin updates policy:

        • Admin -> Git -> OPAL-server -> OPAL-clients-> Policy Agents
        • The application admin commits a new version of the application policy (or subset thereof), triggering a webhook to the OPAL-server,which analyzes the new version, creates a differential update and notifies the OPAL-clients of it via Pub/Sub. The OPAL-clients connect back to the OPAL server (via REST) to retrieve the update itself.

      • OPAL-server starts :

        • Git -> OPAL-server -> OPAL-clients -> Policy Agents
        • A new OPAL-server node starts, spinning out workers - one worker will be elected as the GitPolicySource. The GitPolicySource worker will retrieve the policy from the repository and share it with the other workers (via the FastAPI Pub/Sub channel) and with other server instances (via the backbone pub/sub [e.g. Kafka, Redis, ...]), and through those to all the currently connected clients (via FastAPI Pub/Sub)

      • OPAL-clients starts :

        • OPAL-server -> OPAL-clients -> Policy Agents
        • The OPAL-client starts and connects to the OPAL-server (via REST) to retrieve it's full policy configuration, caching it directly into the policy-agent.
        • OPAL-clients keep the policy up to date, by subscribing to policy updates (via FastAPI Pub/Sub)

    • Data flows

      • Application updates authorization data:

        • App Service -> OPAL-server -> OPAL-clients -> Data-Sources -> OPAL-clients -> Policy-Agents
        • Following an a state changing event (e.g. user interaction, API call) The application changes authorization data, and triggers an update event to the OPAL-server (via REST), including a configured data-topic. The OPAL-server will then propagate (via FastAPI Pub/Sub) the event to all OPAL-clients which subscribed to the data-topic. Each client (using it's configured Data FetchProviders) will then approach each relevant data-source directly, aggregate the data, and store it in the policy agent.

      • A third-party updates authorization data:

        • Third-party -> App Data Monitoring Service -> OPAL-server -> OPAL-clients -> Data-Sources -> OPAL-clients -> Policy-Agents
        • A third-party such as a SaaS service updates data relevant for authorization. The Application monitors such changes, and triggers an update event to the OPAL-server (via REST), including a configured data-topic. The OPAL-server will then propagate (via FastAPI Pub/Sub) the event to all OPAL-clients which subscribed to the data-topic. Each client (using it's configured Data FetchProviders) will then approach each relevant data-source directly, aggregate the data, and store it in the policy agent.

      • OPAL-client starts :

        • OPAL-server -> OPAL-clients -> Data-Sources -> OPAL-clients -> Policy-Agents
        • OPAL-client connects to OPAL-server (via REST) to download base data-source configuration (for its configured topics); and then using it's configured Data FetchProviders) will approach each relevant data-source directly, aggregate the data, and store it in the policy agent.