Skip to content

Add WAL content for 2.0 release #481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners
# Order is important; the last matching pattern takes the most precedence.

* @dutow @dAdAbird
/contrib/pg_tde/documentation/ @nastena1606 @Andriciuc
/.github/ @artemgavrilov
4 changes: 1 addition & 3 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,7 @@ on:
- contrib/pg_tde/documentation/**

env:
# Avoid failures on slow recovery
PGCTLTIMEOUT: 120
PG_TEST_TIMEOUT_DEFAULT: 300
PGCTLTIMEOUT: 120 # Avoid failures on slow recovery

jobs:
collect:
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/sanitizers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,7 @@ env:
LSAN_OPTIONS: log_path=${{ github.workspace }}/sanitize.log print_suppressions=0 suppressions=${{ github.workspace }}/ci_scripts/suppressions/lsan.supp
ASAN_SYMBOLIZER_PATH: /usr/bin/llvm-symbolizer-14
EXTRA_REGRESS_OPTS: "--temp-config=${{ github.workspace }}/test_postgresql.conf"
# Avoid failures on slow recovery
PGCTLTIMEOUT: 120
PG_TEST_TIMEOUT_DEFAULT: 300

jobs:
run:
Expand Down
8 changes: 0 additions & 8 deletions ci_scripts/suppressions/lsan.supp
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,3 @@ leak:bin/pg_waldump/pg_waldump.c
#pg_dump
leak:getSchemaData
leak:dumpDumpableObject

#pg_rewind
leak:decide_file_actions
leak:GenerateRecoveryConfig

# Returns strdup'd string which never gets freed in frontend tools. Trying to
# free it leads to comiler complains that it discards 'const' qualifier.
leak:get_progname
4 changes: 1 addition & 3 deletions contrib/pg_tde/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@ __pycache__
/configure~
/log
/results
/src/bin/pg_tde_archive_decrypt
/src/bin/pg_tde_change_key_provider
/src/bin/pg_tde_restore_encrypt
/src/pg_tde_change_key_provider
/t/results
/tmp_check

Expand Down
24 changes: 3 additions & 21 deletions contrib/pg_tde/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,10 @@ src/libkmip/libkmip/src/kmip_bio.o \
src/libkmip/libkmip/src/kmip_locate.o \
src/libkmip/libkmip/src/kmip_memset.o

SCRIPTS_built = src/bin/pg_tde_archive_decrypt \
src/bin/pg_tde_change_key_provider \
src/bin/pg_tde_restore_encrypt
SCRIPTS_built = src/pg_tde_change_key_provider

EXTRA_INSTALL += contrib/pg_buffercache contrib/test_decoding
EXTRA_CLEAN += src/bin/pg_tde_archive_decrypt.o \
src/bin/pg_tde_change_key_provider.o \
src/bin/pg_tde_restore_encrypt.o \
xlogreader.c \
xlogreader.o
EXTRA_CLEAN += src/pg_tde_change_key_provider.o

ifdef USE_PGXS
PG_CONFIG = pg_config
Expand All @@ -77,21 +71,9 @@ endif

override SHLIB_LINK += -lcurl -lcrypto -lssl

src/bin/pg_tde_change_key_provider: src/bin/pg_tde_change_key_provider.o $(top_srcdir)/src/fe_utils/simple_list.o $(top_builddir)/src/libtde/libtde.a
src/pg_tde_change_key_provider: src/pg_tde_change_key_provider.o $(top_srcdir)/src/fe_utils/simple_list.o $(top_builddir)/src/libtde/libtde.a
$(CC) -DFRONTEND $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)

src/bin/pg_tde_archive_decrypt: src/bin/pg_tde_archive_decrypt.o xlogreader.o $(top_srcdir)/src/fe_utils/simple_list.o $(top_builddir)/src/libtde/libtdexlog.a $(top_builddir)/src/libtde/libtde.a
$(CC) -DFRONTEND $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)

src/bin/pg_tde_restore_encrypt: src/bin/pg_tde_restore_encrypt.o xlogreader.o $(top_srcdir)/src/fe_utils/simple_list.o $(top_builddir)/src/libtde/libtdexlog.a $(top_builddir)/src/libtde/libtde.a
$(CC) -DFRONTEND $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)

xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .

xlogreader.o: xlogreader.c
$(CC) $(CPPFLAGS) -DFRONTEND -c $< -o $@

# Fetches typedefs list for PostgreSQL core and merges it with typedefs defined in this project.
# https://wiki.postgresql.org/wiki/Running_pgindent_on_non-core_code_or_development_code
update-typedefs:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Overview of technical capabilities
# Technical Reference

This section covers the internal components and tools that power `pg_tde`.

Use it to understand how encryption is implemented, fine-tune a configuration, leverage advanced CLI tools and functions for diagnostics and customization.

[Architecture](../architecture/architecture.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button}
[Architecture](../architecture/index.md){.md-button} [GUC Variables](../variables.md){.md-button} [Functions](../functions.md){.md-button}
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ The following sections break down the key architectural components of this desig
* Temporary tables
* Write Ahead Log (WAL), still in beta. **Do not enable this feature in production environments**.

**Extension** means that `pg_tde` should be implemented only as an extension, possibly compatible with any PostgreSQL distribution, including the open source community version. This requires changes in the PostgreSQL core to make it more extensible. Therefore, `pg_tde` currently works only with the [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/index.html) - a binary replacement of community PostgreSQL and included in Percona Distribution for PostgreSQL.
**Extension** means that `pg_tde` should be implemented only as an extension, possibly compatible with any PostgreSQL distribution, including the open source community version. This requires changes in the PostgreSQL core to make it more extensible. Therefore, `pg_tde` currently works only with the [Percona Server for PostgreSQL :octicons-link-external-16:](https://docs.percona.com/postgresql/17/index.html) - a binary replacement of community PostgreSQL and included in Percona Distribution for PostgreSQL.

## Main components

The main components of `pg_tde` are the following:

* **Core server changes** focus on making the server more extensible, allowing the main logic of `pg_tde` to remain separate, as an extension. Core changes also add encryption-awareness to some command line tools that have to work directly with encrypted tables or encrypted WAL files.

[Percona Server for PostgreSQL location](https://github.com/percona/postgres/tree/{{tdebranch}})
[Percona Server for PostgreSQL location :octicons-link-external-16:](https://github.com/percona/postgres/tree/{{tdebranch}})

* The **`pg_tde` extension itself** implements the encryption code by hooking into the extension points introduced in the core changes, and the already existing extension points in the PostgreSQL server.

Expand All @@ -50,7 +50,7 @@ In the future these could be extracted into separate shared libraries with an op

`pg_tde` uses one principal key per database. Every internal key for the given database is encrypted using this principal key.

Internal keys are used for specific database files: each file with a different [Object Identifier (OID)](https://www.postgresql.org/docs/current/datatype-oid.html) has a different internal key.
Internal keys are used for specific database files: each file with a different [Object Identifier (OID) :octicons-link-external-16:](https://www.postgresql.org/docs/current/datatype-oid.html) has a different internal key.

This means that, for example, a table with 4 indexes will have at least 5 internal keys - one for the table, and one for each index.

Expand Down Expand Up @@ -82,12 +82,9 @@ Later decisions are made using a slightly modified Storage Manager (SMGR) API: w

### WAL encryption

!!! note
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

WAL encryption is controlled globally via a global GUC variable, `pg_tde.wal_encrypt`, that requires a server restart.

WAL keys also contain the [LSN](https://www.postgresql.org/docs/17/wal-internals.html) of the first WAL write after key creation. This allows `pg_tde` to know which WAL ranges are encrypted or not and with which key.
WAL keys also contain the [LSN :octicons-link-external-16:](https://www.postgresql.org/docs/17/wal-internals.html) of the first WAL write after key creation. This allows `pg_tde` to know which WAL ranges are encrypted or not and with which key.

The setting only controls writes so that only WAL writes are encrypted when WAL encryption is enabled. This means that WAL files can contain both encrypted and unencrypted data, depending on what the status of this variable was when writing the data.

Expand Down Expand Up @@ -147,8 +144,8 @@ Principal keys are stored externally in a Key Management Services (KMS). In `pg_

The following key providers are supported:

* [HashiCorp Vault](https://developer.hashicorp.com/vault/docs/what-is-vault) KV2 secrets engine
* [OpenBao](https://openbao.org/) implementation of Vault
* [HashiCorp Vault :octicons-link-external-16:](https://developer.hashicorp.com/vault/docs/what-is-vault) KV2 secrets engine
* [OpenBao :octicons-link-external-16:](https://openbao.org/) implementation of Vault
* KMIP compatible servers
* A local file storage. This storage is intended only for development and testing and is not recommended for production use.

Expand All @@ -167,7 +164,7 @@ Key provider configuration or location may change. For example, a service is mov

In certain cases you can't use SQL functions to manage key providers. For example, if the key provider changed while the server wasn't running and is therefore unaware of these changes. The startup can fail if it needs to access the encryption keys.

For such situations, `pg_tde` also provides [command line tools](../command-line-tools/cli-tools.md) to recover the database.
For such situations, `pg_tde` also provides [command line tools](../command-line-tools/index.md) to recover the database.

### Sensitive key provider information

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Overview of pg_tde CLI tools
# pg_tde CLI Tools

The `pg_tde` extension introduces new command-line utilities and extends some existing PostgreSQL tools to support encrypted WAL and tables. These include:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@

[`pg_waldump` :octicons-link-external-16:](https://www.postgresql.org/docs/current/pgwaldump.html) is a tool to display a human-readable rendering of the Write-Ahead Log (WAL) of a PostgreSQL database cluster.

!!! warning
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

To read encrypted WAL records, `pg_waldump` supports the following additional arguments:

* `keyring_path` is the directory where the keyring configuration files for WAL are stored. The following files are included:
Expand Down
7 changes: 2 additions & 5 deletions contrib/pg_tde/documentation/docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,6 @@ The principal key is used to encrypt the internal keys. The principal key is sto

### WAL encryption

!!! note
WAL encryption is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

WAL encryption is done globally for the entire database cluster. All modifications to any database within a PostgreSQL cluster are written to the same WAL to maintain data consistency and integrity and ensure that PostgreSQL cluster can be restored to a consistent state. Therefore, WAL is encrypted globally.

When you turn on WAL encryption, `pg_tde` encrypts entire WAL files starting from the first WAL write after the server was started with the encryption turned on.
Expand Down Expand Up @@ -140,9 +137,9 @@ Since the `SET ACCESS METHOD` command drops hint bits and this may affect the pe
You must restart the database in the following cases to apply the changes:

* after you enabled the `pg_tde` extension
* when enabling WAL encryption, which is currently in beta. **Do not enable this feature in production environments**.
* when enabling WAL encryption

After that, no database restart is required. When you create or alter the table using the `tde_heap` access method, the files are marked as those that require encryption. The encryption happens at the storage manager level, before a transaction is written to disk. Read more about [how tde_heap works](index/table-access-method.md#how-tde_heap-works-with-pg_tde).
After that, no database restart is required. When you create or alter the table using the `tde_heap` access method, the files are marked as those that require encryption. The encryption happens at the storage manager level, before a transaction is written to disk. Read more about [how tde_heap works](index/table-access-method.md#how-tde_heap-works).

## What happens to my data if I lose a principal key?

Expand Down
10 changes: 5 additions & 5 deletions contrib/pg_tde/documentation/docs/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ The Percona Server for PostgreSQL provides an extended Storage Manager API that

The following features are available for the extension:

* [Table encryption](test.md#encrypt-data-in-a-new-table), including:
* Table encryption, including:
* Data tables
* Index data for encrypted tables
* TOAST tables
Expand All @@ -14,9 +14,9 @@ The following features are available for the extension:
!!! note
Metadata of those tables is not encrypted.

* Single-tenancy support via a [global keyring provider](global-key-provider-configuration/set-principal-key.md)
* [Multi-tenancy support](how-to/multi-tenant-setup.md)
* Single-tenancy support via a global keyring provider
* Multi-tenancy support
* Table-level granularity for encryption and access control
* Multiple [Key management options](global-key-provider-configuration/index.md)
* Multiple Key management options

[Learn more about TDE and pg_tde :material-arrow-right:](index/about-tde.md){.md-button} [Get started with installation :material-arrow-right:](install.md){.md-button}
[Overview](index/index.md){.md-button} [Get Started](install.md){.md-button}
5 changes: 0 additions & 5 deletions contrib/pg_tde/documentation/docs/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@

The `pg_tde` extension provides functions for managing different aspects of its operation:

!!! note
If no error is reported when running the commands below, the operation completed successfully.

## Key provider management

A key provider is a system or service responsible for managing encryption keys. `pg_tde` supports the following key providers:
Expand Down Expand Up @@ -283,8 +280,6 @@ SELECT pg_tde_set_server_key_using_global_key_provider(
);
```

!!! warning
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.
=======
The `ensure_new_key` parameter instructs the function how to handle a principal key during key rotation:

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Key management overview
# Configure Key Management (KMS)

In production environments, storing encryption keys locally on the PostgreSQL server can introduce security risks. To enhance security, `pg_tde` supports integration with external Key Management Systems (KMS) through a Global Key Provider interface.

Expand All @@ -13,6 +13,4 @@ To use an external KMS with `pg_tde`, follow these two steps:

Select your prefered configuration from the links below:

[KMIP Configuration :material-arrow-right:](kmip-server.md){.md-button}
[Vault Configuration :material-arrow-right:](vault.md){.md-button}
[Keyring File Configuration (not recommended) :material-arrow-right:](keyring.md){.md-button}
[KMIP Configuration :material-arrow-right:](kmip-server.md){.md-button} [Vault Configuration :material-arrow-right:](vault.md){.md-button} [Keyring File Configuration (not recommended) :material-arrow-right:](keyring.md){.md-button}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Keyring file configuration
# Keyring File Configuration

This setup is intended for development and stores the keys unencrypted in the specified data file.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Fortanix KMIP server configuration
# Fortanix KMIP Server Configuration

`pg_tde` is compatible with Fortanix Data Security Manager (DSM) via the KMIP protocol. For a full setup guide, see [the Fortanix KMIP documentation here](https://support.fortanix.com/docs/users-guide-account-client-configurations?highlight=KMIP#23-kmip-clients).

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Using OpenBao as a key provider
# Using OpenBao as a Key Provider

You can configure `pg_tde` to use OpenBao as a global key provider for managing encryption keys securely.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# KMIP configuration
# KMIP Configuration

To use a Key Management Interoperability Protocol (KMIP) server with `pg_tde`, you must configure it as a global key provider. This setup enables `pg_tde` to securely fetch and manage encryption keys from a centralized key management appliance.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Thales KMIP server configuration
# Thales KMIP Server Configuration

`pg_tde` is compatible with the Thales CipherTrust Manager via the KMIP protocol. For a full setup guide, see [the Thales documentation](https://thalesdocs.com/ctp/cm/2.19/reference/kmip-ref/index.html?).

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
# Global Principal Key configuration
# Global Principal Key Configuration

You can configure a default principal key using a global key provider. This key will be used by all databases that do not have their own encryption keys configured. The function **both** sets the principal key and rotates internal keys as needed.

## Create a default principal key

!!! note
The sample output below is for demonstration purposes only. Be sure to replace the key name and provider with your actual values.

To create a global principal key, run:

```sql
Expand All @@ -16,18 +13,6 @@ SELECT pg_tde_create_key_using_global_key_provider(
);
```

??? example "Sample output"
```sql
postgres=# SELECT pg_tde_create_key_using_global_key_provider(
'keytest1',
'file-keyring'
);
pg_tde_create_key_using_global_key_provider
---------------------------------------------

(1 row)
```

## Configure a default principal key

To configure a global principal key, run:
Expand All @@ -39,33 +24,34 @@ SELECT pg_tde_set_default_key_using_global_key_provider(
);
```

??? example "Sample output"
```sql
postgres=# SELECT pg_tde_set_default_key_using_global_key_provider(
'keytest1',
'file-keyring'
);
pg_tde_set_default_key_using_global_key_provider
--------------------------------------------------

(1 row)
```

## Parameter description

* `key-name` is the name under which the principal key is stored in the provider.
* `global_vault_provider` is the name of the global key provider you previously configured.

!!! note
If no error is reported, the action completed successfully.

## How key generation works

The key material (actual cryptographic key) is auto-generated by `pg_tde` and stored securely by the configured provider.

!!! note
This process sets the **default principal key for the entire server**. Any database without a key explicitly configured will fall back to this key.

## Example

This example is for testing purposes only. Replace the key name and provider name with your values:

```sql
SELECT pg_tde_create_key_using_global_key_provider(
'test-db-master-key',
'file-vault'
);

SELECT pg_tde_set_key_using_global_key_provider(
'test-db-master-key',
'file-vault'
);
```

## Next steps

[Validate Encryption with pg_tde :material-arrow-right:](../test.md){.md-button}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Vault configuration
# Vault Configuration

You can configure `pg_tde` to use HashiCorp Vault as a global key provider for managing encryption keys securely.

Expand Down
Loading
Loading