Skip to content

Releases: aleph-im/pyaleph

v0.5.0-rc2

13 Apr 16:50
@odesenfans odesenfans
v0.5.0-rc2
7a8b99e
Compare
Choose a tag to compare
v0.5.0-rc2 Pre-release
Pre-release

This release introduces multiple changes to the way node operates.

TL;DR

  • A new database: we switch from MongoDB to PostgreSQL.
  • A new implementation of the processing pipeline: the message pipeline is now split in two parts to fix race conditions and optimize the overall throughput.
  • Separate API processes: the REST API is now running in a separate Docker container and spawns several worker processes for improved response times.
  • Materialized aggregates: aggregates are now faster to query through the API.
  • New endpoints: we make it easier to post new Aleph messages and determine if your messages were processed or rejected.
  • Major dependency updates: CCNs now run on Python 3.11.

Switch to PostgreSQL

One of the main features of this release is the switch from MongoDB to PostgreSQL. This switch is motivated by the development of new features for which we feel a relational database is more appropriate.

Each type of message is now associated with one or more DB tables that store the actual objects mentioned in Aleph messages. API endpoints and internal operations can now directly access these object tables instead of having to search through messages.

Additionally, we now use a DB migration system that guarantees the consistency of the data across updates.

As we dropped MongoDB, files are now stored on the local file system in a dedicated volume.

New message pipeline

Fetcher and processor

The new message pipeline addresses two issues: determinism and observability. We now use two separate processes:

  • the fetcher performs network accesses for messages that require additional downloads. It ensures that all the data required to process a message is available on the node before any further processing. It uses asyncio tasks to fetch data for multiple messages in parallel.
  • the message processor is in charge of checking the integrity of messages and permissions. It processes messages atomically, guaranteeing the absence of race condition.

This new architecture allows to process messages as soon as they are fetched. As most messages are immediately ready for processing, this maximizes the throughput of the message pipeline.

Errors and error codes

The error checking mechanism of the message pipeline was completely rewritten. Each error is now specified as its own exception type and is made visible to the user as an error code. By using the new GET /api/v0/messages/{item_hash}, users can now determine if and why their message was rejected by a node.

Additionally, we now use exponential retry times to reduce the total amount of retries and the CPU/network load that comes with them. Messages are now retried up to 10 times within a span of around 20 minutes.

Materialized aggregates

Aggregates are now re-calculated as soon as a new aggregate message is processed. This improves the performance when querying large aggregates.

API updates

New endpoints

  • POST /api/v0/messages: allows users to post a new message and then track the progress of the message in the processing pipeline. This endpoint supports a synchronous mode where the response is only sent once the node processes the message or a timeout occurs.
  • GET /api/v0/messages/{item_hash}: allows users to track the status individual messages. The status field allows users to determine if their message is processed, rejected, pending or forgotten.
  • GET /api/v0/addresses/{address}/balance: returns the balance in Aleph of a wallet address.
  • GET /api/v0/addresses/{address}/files: returns the list of files stored by the user along the total number of files they store on Aleph and the total space used.
  • GET /api/v1/posts.json: a new implementation of the /posts/ endpoint. This new implementation removes message-specific files and focuses on the post content and metadata. /api/v0/posts.json is now deprecated.

New features

  • The messages websocket now allows history = 0. It was reimplemented to use a RabbitMQ queue to read new messages directly from the message pipeline.

Breaking changes

  • GET /api/v0/messages:
    • the endpoint only returns processed messages. Forgotten messages are now ignored.
    • The size, content_type and engine_info fields added by the node on STORE messages are not returned anymore. If you need this information, use the new GET /api/v0/addresses/{address}/files endpoint.
  • GET /api/v0/posts: a lot of fields were dropped as they were redundant
  • GET /api/v0/addresses/stats.json: removed the address field. It was redundant with the key of the dictionary.
  • Message specification:
    • The content field of aggregate messages is now required to be a dictionary.
    • The ref field of program volumes is now required to be a message hash.
    • Dropped support for the NaN float value and the \u0000 character in aggregates and posts.

Upgrade guide

Prerequisites

Make sure that your node is up-to-date with the latest release. Specifically, you must ensure that your private key is in the format introduced in the 0.4.x releases. You can find the full upgrade guide here. You can also skip this update and convert your private key file in the right format using openssl in your keys directory:

cd ./keys
openssl pkcs8 -topk8 -inform PEM -outform DER -in node-secret.key -out node-secret.pkcs8.der -nocrypt

Stop the node

This release requires a full re-sync of your node. While you wait for your node to resynchronize, use any of our official nodes to access data: official.aleph.cloud.

The full resync is the simplest option and will work for all node operators who do not require their node to be up at the time.

The following instructions assume that you use one of our official Docker Compose files.
First, switch off your node:

docker-compose down

Now, retire your old Docker Compose file and download the new one.

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.5.0-rc2/deployment/samples/docker-compose/docker-compose.yml"

The new Docker Compose file comes with a default password for PostgreSQL. Generate a new password and specify it in your docker-compose.yml and config.yml files:

Update docker-compose.yml:

services:
  postgres:
    env:
      POSTGRES_PASSWORD: "<new-password>"

Add to config.yml:

postgres:
  host: "postgres"
  password: "<new-password>"

redis:
  host: "redis"

Do not forget to keep other passwords from the previous Docker Compose file, like the one you generated for RabbitMQ.

You can now restart your node:

docker-compose up -d

The sync process takes around a full day.

If you are using a custom Docker Compose file, beware that there are multiple new services:

  • The API lives in its own container (pyaleph-api), using the same Docker image as pyaleph.
  • The node now uses a Redis cache.
  • MongoDB is replaced by PostgreSQL.

Check the official Docker Compose file to see how the services are configured.

Cleanup

Once you are confident that you will not need to roll back the release, you can delete the MongoDB volume:

docker volume rm <docker-compose-directory>_pyaleph-mongodb

Full Changelog: v0.4.7...v0.5.0-rc2
Changes from v0.5.0-rc1: v0.5.0-rc1...v0.5.0-rc2

Loading

v0.5.0-rc1

28 Mar 13:16
@odesenfans odesenfans
v0.5.0-rc1
b162fc9
Compare
Choose a tag to compare
v0.5.0-rc1 Pre-release
Pre-release

This release introduces multiple changes to the way node operates.

TL;DR

  • A new database: we switch from MongoDB to PostgreSQL.
  • A new implementation of the processing pipeline: the message pipeline is now split in two parts to fix race conditions and optimize the overall throughput.
  • Materialized aggregates: aggregates are now faster to query through the API.
  • New endpoints: we make it easier to post new Aleph messages and determine if your messages were processed or rejected.
  • Major dependency updates: CCNs now run on Python 3.11.

Switch to PostgreSQL

One of the main features of this release is the switch from MongoDB to PostgreSQL. This switch is motivated by the development of new features for which we feel a relational database is more appropriate.

Each type of message is now associated with one or more DB tables that store the actual objects mentioned in Aleph messages. API endpoints and internal operations can now directly access these object tables instead of having to search through messages.

Additionally, we now use a DB migration system that guarantees the consistency of the data across updates.

As we dropped MongoDB, files are now stored on the local file system in a dedicated volume.

New message pipeline

Fetcher and processor

The new message pipeline addresses two issues: determinism and observability. We now use two separate processes:

  • the fetcher performs network accesses for messages that require additional downloads. It ensures that all the data required to process a message is available on the node before any further processing. It uses asyncio tasks to fetch data for multiple messages in parallel.
  • the message processor is in charge of checking the integrity of messages and permissions. It processes messages atomically, guaranteeing the absence of race condition.

This new architecture allows to process messages as soon as they are fetched. As most messages are immediately ready for processing, this maximizes the throughput of the message pipeline.

Errors and error codes

The error checking mechanism of the message pipeline was completely rewritten. Each error is now specified as its own exception type and is made visible to the user as an error code. By using the new GET /api/v0/messages/{item_hash}, users can now determine if and why their message was rejected by a node.

Additionally, we now use exponential retry times to reduce the total amount of retries and the CPU/network load that comes with them. Messages are now retried up to 10 times within a span of around 20 minutes.

Materialized aggregates

Aggregates are now re-calculated as soon as a new aggregate message is processed. This improves the performance when querying large aggregates.

API updates

New endpoints

  • POST /api/v0/messages: allows users to post a new message and then track the progress of the message in the processing pipeline. This endpoint supports a synchronous mode where the response is only sent once the node processes the message or a timeout occurs.
  • GET /api/v0/messages/{item_hash}: allows users to track the status individual messages. The status field allows users to determine if their message is processed, rejected, pending or forgotten.
  • GET /api/v0/addresses/{address}/balance: returns the balance in Aleph of a wallet address.
  • GET /api/v0/addresses/{address}/files: returns the list of files stored by the user along the total number of files they store on Aleph and the total space used.
  • GET /api/v1/posts.json: a new implementation of the /posts/ endpoint. This new implementation removes message-specific files and focuses on the post content and metadata. /api/v0/posts.json is now deprecated.

New features

  • The messages websocket now allows history = 0. It was reimplemented to use a RabbitMQ queue to read new messages directly from the message pipeline.

Breaking changes

  • GET /api/v0/messages:
    • the endpoint only returns processed messages. Forgotten messages are now ignored.
    • The size, content_type and engine_info fields added by the node on STORE messages are not returned anymore. If you need this information, use the new GET /api/v0/addresses/{address}/files endpoint.
  • GET /api/v0/posts: a lot of fields were dropped as they were redundant
  • GET /api/v0/addresses/stats.json: removed the address field. It was redundant with the key of the dictionary.
  • Message specification:
    • The content field of aggregate messages is now required to be a dictionary.
    • The ref field of program volumes is now required to be a message hash.
    • Dropped support for the NaN float value and the \u0000 character in aggregates and posts.

Upgrade guide

Prerequisites

Make sure that your node is up-to-date with the latest release. Specifically, you must ensure that your private key is in the format introduced in the 0.4.x releases. You can find the full upgrade guide here. You can also skip this update and convert your private key file in the right format using openssl in your keys directory:

cd ./keys
openssl pkcs8 -topk8 -inform PEM -outform DER -in node-secret.key -out node-secret.pkcs8.der -nocrypt

Stop the node

This release requires a full re-sync of your node. While you wait for your node to resynchronize, use any of our official nodes to access data: official.aleph.cloud.

The full resync is the simplest option and will work for all node operators who do not require their node to be up at the time.

The following instructions assume that you use one of our official Docker Compose files.
First, switch off your node:

docker-compose down

Now, retire your old Docker Compose file and download the new one.

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.5.0-rc1/deployment/samples/docker-compose/docker-compose.yml"

The new Docker Compose file comes with a default password for PostgreSQL. Generate a new password and specify it in your docker-compose.yml and config.yml files:

Update docker-compose.yml:

services:
  postgres:
    env:
      POSTGRES_PASSWORD: "<new-password>"

Add to config.yml:

postgres:
  host: "postgres"
  password: "<new-password>"

Do not forget to keep other passwords from the previous Docker Compose file, like the one you generated for RabbitMQ.

You can now restart your node:

docker-compose up -d

The sync process takes around a full day.

Cleanup

Once you are confident that you will not need to roll back the release, you can delete the MongoDB volume:

docker volume rm <docker-compose-directory>_pyaleph-mongodb

Full Changelog: v0.4.7...v0.5.0-rc1

Loading

v0.4.7

21 Mar 11:31
@odesenfans odesenfans
v0.4.7
8c2fd3e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.7

Minor fix release for the Ethereum/BSC connectors.

What's Changed

Upgrade guide

From any of the v0.4.x releases, upgrading is as simple as bumping the Docker image version of alephim/pyaleph-node to v0.4.7 and restarting your node.

Full Changelog: v0.4.6...v0.4.7

Contributors

odesenfans
Loading

v0.4.6

20 Mar 17:19
@odesenfans odesenfans
v0.4.6
df6bff9
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.6

Minor fix over v0.4.5.

What's Changed

Upgrade guide

From any of the v0.4.x releases, upgrading is as simple as bumping the Docker image version of alephim/pyaleph-node to v0.4.6 and restarting your node.

Full Changelog: v0.4.5...v0.4.6

Contributors

odesenfans
Loading

v0.4.5

20 Mar 16:50
@odesenfans odesenfans
v0.4.5
ae8350b
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.5

This release connects the Core Channel Nodes to the Aleph message indexer for Ethereum and Binance Smart Chain.

What's Changed

  • Chore: Allow provider polkadot signature verification by @Rgascoin in #358
  • Fix: Default peers of p2p-service did not work by @hoh in #374
  • Feature: fetch message/sync events from the Aleph indexer (MongoDB) by @odesenfans in #379
  • Feature: support all message types in trusted messages by @odesenfans in #380
  • Fix: use tx hash in messages parsed from the indexer by @odesenfans in #382

New Contributors

Full Changelog: 0.4.4...v0.4.5

Contributors

hoh, odesenfans, and Rgascoin
Loading

v0.4.4

03 Feb 15:40
@hoh hoh
v0.4.4
0e7fe28
Compare
Choose a tag to compare
v0.4.4

This release adds support for syncing messages from the Tezos blockchain, as well as a few fixes.
https://medium.com/aleph-im/introducing-fully-decentralized-nft-smart-contracts-for-tezos-3219fc2e51ef

What's Changed

  • Fix: Messages sent to the pubsub endpoint were only forwarded to other nodes.
  • Fix: The ref field on STORE messages was not stored
  • Upgrade dependencies p2p-service and aleph-client
  • Support disabling IPFS in configuration

In details

Full Changelog: v0.4.3...v0.4.4

How to upgrade

From 0.4.3

Replace the version tag with v0.4.4 in your docker-compose.yml file.

Or download the new Docker Compose file

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.4.4/deployment/samples/docker-compose/docker-compose.yml"

Edit the docker-compose.yml file with your secrets (ex: RabbitMQ).

Restart the node

docker-compose -f docker-compose-old.yml down
docker-compose -f docker-compose.yml up -d

From older versions

Follow the guide here.

If you have an issue with the keys

Make sure that your node is up-to-date with the latest release. Specifically, you must ensure that your private key is in the format introduced in the 0.4.x releases. You can find the full upgrade guide here. You can also skip this update and convert your private key file in the right format using openssl in your keys directory:

cd ./keys
openssl pkcs8 -topk8 -inform PEM -outform DER -in node-secret.key -out node-secret.pkcs8.der -nocrypt

Contributors

hoh and odesenfans
Loading

v0.4.3

20 Oct 10:53
@odesenfans odesenfans
v0.4.3
b2ead3e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.3

This release fixes a minor issue related to message processing introduced with v0.4.0.

How to upgrade

From 0.4.x

Download the new Docker Compose file

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.4.3/deployment/samples/docker-compose/docker-compose.yml"

Restart the node

docker-compose -f docker-compose-old.yml down
docker-compose -f docker-compose.yml up -d

From older versions

Follow the guide here.

Loading

v0.4.2

18 Oct 10:17
@odesenfans odesenfans
v0.4.2
52856f3
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.2

This release fixes default configuration values to make the transition to v0.4.x simpler.

How to upgrade

0.4.0 introduces a new service for P2P communication. This implies a few modifications to the configuration of your node.

Node private key file

This release includes a change in the private key format. The P2P service requires the key to be in PKCS8 DER format. This change is automatically managed in a migration script that you must run with the upgrade command (see below).

P2P topics

This release introduces new configuration options for the P2P service. The default configuration should work out of the box, but if you customized your install we advise to set the following configuration options. Append these options to your config.yml file:

aleph:
  queue_topic: ALEPH-TEST

p2p:
  daemon_host: p2p-service
  http_port: 4024
  port: 4025
  control_port: 4030
  listen_port: 4031
  reconnect_delay: 60
  peers:
    - /dns/api1.aleph.im/tcp/4025/p2p/Qmaxufiqdyt5uVWcy1Xh2nh3Rs3382ArnSP2umjCiNG2Vs
    - /dns/api2.aleph.im/tcp/4025/p2p/QmZkurbY2G2hWay59yiTgQNaQxHSNzKZFt2jbnwJhQcKgV

Note that you are free to configure bootstrap peers as you wish. The two default servers listed above will fit most use cases.

RabbitMQ config

The Core Channel Node communicates with the new P2P service through an instance of RabbitMQ for P2P pubsub topics. We strongly advise to change the default username and password in the config.yml and docker-compose.yml files:

config.yml

rabbitmq:
    host: rabbitmq
    port: 5672
    username: <new-username>
    password: <new-password>

docker-compose.yml

services:
  rabbitmq:
    environment:
      RABBITMQ_DEFAULT_USER: <new-username>
      RABBITMQ_DEFAULT_PASS: <new-password>

Upgrade your node

From any version >= 0.2.0

Download the new Docker Compose file

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.4.2/deployment/samples/docker-compose/docker-compose.yml"

Stop, upgrade and restart the node

docker-compose -f docker-compose-old.yml down --remove-orphans
# Adapt if your config.yml and/or keys/ directory are in different places
docker run \
      --entrypoint /opt/pyaleph/migrations/config_updater.py \
      -v $(pwd)/keys:/opt/pyaleph/keys \
      -v $(pwd)/config.yml:/opt/pyaleph/config.yml \
      alephim/pyaleph-node:v0.4.2 \
      --key-dir /opt/pyaleph/keys \
      --key-file /opt/pyaleph/keys/node-secret.key \
      --config /opt/pyaleph/config.yml \
      upgrade \
      --filter-scripts 0003*
docker-compose -f docker-compose.yml up -d

Alternatively, if you are encountering issues running the upgrade, you can simply generate the new key file with the following OpenSSL command:

docker-compose -f docker-compose-old.yml down --remove-orphans
cd keys/
openssl pkcs8 -topk8 -inform PEM -outform DER -in node-secret.key -out node-secret.pkcs8.der -nocrypt
docker-compose -f docker-compose.yml up -d

From any version < 0.2.0: https://pyaleph.readthedocs.io/en/latest/guides/upgrade.html

Loading

v0.4.1

17 Oct 13:11
@odesenfans odesenfans
v0.4.1
d540f14
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.1

This release fixes a minor issue of v0.4.0 where an improperly formatted message could be retried indefinitely.

What's Changed

  • Fix: do not retry pending messages failing validation by @odesenfans in #348

Full Changelog: v0.4.0...v0.4.1

How to upgrade

0.4.0 introduces a new service for P2P communication. This implies a few modifications to the configuration of your node.

Node private key file

This release includes a change in the private key format. The P2P service requires the key to be in PKCS8 DER format. This change is automatically managed in a migration script that you must run with the upgrade command (see below).

P2P topics

This release introduces new configuration options for the P2P service. We advise to set the following configuration options. Append these options to your config.yml file:

aleph:
    queue_topic: ALEPH-TEST

p2p:
  daemon_host: p2p-service
  http_port: 4024
  port: 4025
  control_port: 4030
  listen_port: 4031
  reconnect_delay: 60
  peers:
    - /dns/api1.aleph.im/tcp/4025/p2p/Qmaxufiqdyt5uVWcy1Xh2nh3Rs3382ArnSP2umjCiNG2Vs
    - /dns/api2.aleph.im/tcp/4025/p2p/QmZkurbY2G2hWay59yiTgQNaQxHSNzKZFt2jbnwJhQcKgV

Note that you are free to configure bootstrap peers as you wish. The two default servers listed above will fit most use cases.

RabbitMQ config

The Core Channel Node communicates with the new P2P service through an instance of RabbitMQ for P2P pubsub topics. We strongly advise to change the default username and password in the config.yml and docker-compose.yml files:

config.yml

rabbitmq:
    host: rabbitmq
    port: 5672
    username: <new-username>
    password: <new-password>

docker-compose.yml

services:
  rabbitmq:
    environment:
      RABBITMQ_DEFAULT_USER: <new-username>
      RABBITMQ_DEFAULT_PASS: <new-password>

Upgrade your node

From any version >= 0.2.0

Download the new Docker Compose file

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.4.1/deployment/samples/docker-compose/docker-compose.yml"

Stop, upgrade and restart the node

docker-compose -f docker-compose-old.yml down --remove-orphans
docker-compose -f docker-compose.yml \
      run \
      --entrypoint /opt/pyaleph/migrations/config_updater.py \
      pyaleph \
      --key-dir /opt/pyaleph/keys \
      --key-file /opt/pyaleph/keys/node-secret.key \
      --config /opt/pyaleph/config.yml \
      upgrade
docker-compose -f docker-compose.yml up -d

From any version < 0.2.0: https://pyaleph.readthedocs.io/en/latest/guides/upgrade.html

Contributors

odesenfans
Loading

v0.4.0

17 Oct 10:25
@odesenfans odesenfans
v0.4.0
949adeb
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.0

This release introduces the use of the Aleph.im P2P service to replace the libp2p daemon.
This is expected to reduce the load of core channel nodes significantly and fixes several performance issues.

The release also fixes several bugs linked to the validation of messages.

What's Changed

Full Changelog: v0.3.3...v0.4.0

How to upgrade

Node private key file

This release includes a change in the private key format. The P2P service requires the key to be in PKCS8 DER format. This change is automatically managed in a migration script that you must run with the upgrade command (see below).

P2P topics

This release introduces new configuration options for the P2P service. We advise to set the following configuration options. Append these options to your config.yml file:

aleph:
    queue_topic: ALEPH-QUEUE

rabbitmq:
    host: rabbitmq
    port: 5672

RabbitMQ config

The Core Channel Node communicates with the new P2P service through an instance of RabbitMQ for P2P pubsub topics. We strongly advise to change the default username and password in the Docker Compose file:

services:
  rabbitmq:
    environment:
      RABBITMQ_DEFAULT_USER: <new-username>
      RABBITMQ_DEFAULT_PASS: <new-password>

And in the config.yml file:

rabbitmq:
  username: <new-username>
  password: <new-password>

From any version >= 0.2.0

Download the new Docker Compose file

mv docker-compose.yml docker-compose-old.yml
wget "https://raw.githubusercontent.com/aleph-im/pyaleph/v0.4.0/deployment/samples/docker-compose/docker-compose.yml"

Stop, upgrade and restart the node

docker-compose down
docker-compose \
      run \
      --entrypoint /opt/pyaleph/migrations/config_updater.py \
      pyaleph \
      --key-dir /opt/pyaleph/keys \
      --key-file /opt/pyaleph/keys/node-secret.key \
      --config /opt/pyaleph/config.yml \
      upgrade
docker-compose up -d

From any version < 0.2.0: https://pyaleph.readthedocs.io/en/latest/guides/upgrade.html

Contributors

hoh and odesenfans
Loading