DNM: build(deps): libz-sys v1.1.8 -> v1.1.19 #32503
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…unctions (MaterializeInc#31097) This PR changes the MySQL source to support ingesting the `bit` type as `uint8`. It also adds two new Postgres functions, `bit_count(bytea)` and `get_bit(bytea, int32)` to making working with byte strings easier. ### Motivation Progress towards: MaterializeInc/database-issues#8891 ### Checklist - [x] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [x] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [x] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [x] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [x] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. --------- Co-authored-by: Dennis Felsing <[email protected]>
Enabled widely by default with zero consequences so far.
Don't need an explicit %N anymore
The materialized binary generates massive debuginfo: 8GB with full debuginfo and 3.5GB with limited debuginfo. (This is not the compressed size of the debuginfo on disk, but the amount of memory required to load the debuginfo into memory so that backtraces can be symbolized.) We've historically shipped the materialized binary with full debuginfo, so that we get rich backtraces for any crashes users run into when running the emulator. Unfortunately, the full debuginfo is so large that it is itself the *cause* of OOMs [0], which is unacceptable. (When a console query encounters certain routine errors, like a connection failing to validate, the adapter attempts to log a backtrace, which requires loading the debuginfo.) Even the limited debuginfo size (3.5GB) is unacceptable for a Docker image that's meant to be run on developer laptops. So, this commit adjusts the materialized image to strip all debuginfo from the binary. If a user reports a crash with an unsymbolized backtrace, it's still *possible* (just painful) to manually symbolize the backtrace as long as they give us the exact version of Materialize they were running. We'll just need to manually run `addr2line` on each address reported in the backtrace. (We do irrevocably lose access to frames for inlined functions, but that seems tolerable.) One silver lining here is that the Docker image will get much smaller. It's currently about 1GB. I expect this change to shave off a huge chunk of that. [0]: https://materializeinc.slack.com/archives/C07FX1W1Y03/p1737414021061139
…alizeInc#31127) This refactors the impl of `Blob` for Azure in a way that should be faster. The `BlobClient` we use from the `azure_storage_blob` crate returns a `Stream` that when `await`-ed sends a ranged GET request for a chunk of a blob. This PR refactors our impl so we await each ranged request in a `tokio::task` which increases the concurrency at which we fetch chunks of a `Part`. It also refactors how we handle the case when the `content-length` header is missing, and adds metrics so we can track how often this occurs. ### Motivation Maybe progress against MaterializeInc/database-issues#8892 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.
Previously, INSPECT SHARD would panic if it was given an ID that didn't belong to any item. This commit fixes the issue by returning an error instead. Fixes #MaterializeInc/database-issues/issues/8910
We built this for the streaming iterator, but it works well here too.
In particular, don't decode K/Vs when we have an override set, and make sure every K/V only gets decoded once.
…Inc#32173) When creating a `PartMigration` if we don't have a `schema_id` but we do have structured only data, we'll fallback to the `deprecated_schema_id` field because this indicates the `persist_record_schema_id` and `persist_batch_columnar_format` flags did not propagate properly. Fix an issue at least one customer has run into. - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.
The `reclock` operator accepts and enqueues all remap updates, and
requires memory proportional to the number of updates supplied. This has
the potential to be much larger than required if the updates when
advanced using `as_of` would consolidate to a much smaller collection.
This PR advances all inbound times using `as_of`, and consolidates them
before enqueueing them in the heap-based holding pen. This does not
guarantee that they will fully consolidate, as it depends on
non-determinism of data arrival, and a more thorough solution might look
more like DD's `MergeBatcher` that uses a continual merge sort to
maintain consolidated priority order. Ideally this temporary fix
relieves some pressure, though.
### Motivation
<!--
Which of the following best describes the motivation behind this PR?
* This PR fixes a recognized bug.
[Ensure issue is linked somewhere.]
* This PR adds a known-desirable feature.
[Ensure issue is linked somewhere.]
* This PR fixes a previously unreported bug.
[Describe the bug in detail, as if you were filing a bug report.]
* This PR adds a feature that has not yet been specified.
[Write a brief specification for the feature, including justification
for its inclusion in Materialize, as if you were writing the original
feature specification.]
* This PR refactors existing code.
[Describe what was wrong with the existing code, if it is not obvious.]
-->
### Tips for reviewer
<!--
Leave some tips for your reviewer, like:
* The diff is much smaller if viewed with whitespace hidden.
* [Some function/module/file] deserves extra attention.
* [Some function/module/file] is pure code movement and only needs a
skim.
Delete this section if no tips.
-->
### Checklist
- [ ] This PR has adequate test coverage / QA involvement has been duly
considered. ([trigger-ci for additional test/nightly
runs](https://trigger-ci.dev.materialize.com/))
- [ ] This PR has an associated up-to-date [design
doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md),
is a design doc
([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)),
or is sufficiently small to not require a design.
<!-- Reference the design in the description. -->
- [ ] If this PR evolves [an existing `$T ⇔ Proto$T`
mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md)
(possibly in a backwards-incompatible way), then it is tagged with a
`T-proto` label.
- [ ] If this PR will require changes to cloud orchestration or tests,
there is a companion cloud PR to account for those changes that is
tagged with the release-blocker label
([example](MaterializeInc/cloud#5021)).
<!-- Ask in #team-cloud on Slack if you need help preparing the cloud
PR. -->
- [ ] If this PR includes major [user-facing behavior
changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note),
I have pinged the relevant PM to schedule a changelog post.
Signed-off-by: Petros Angelatos <[email protected]>
Signed-off-by: Petros Angelatos <[email protected]>
Signed-off-by: Petros Angelatos <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
This PR adds some new queries to `PostgresConsensus` that are tuned
against vanilla postgres to obtain row level locks as opposed to the
table level locks that the current queries get. These new queries are
only used if `persist_use_postgres_tuned_queries` is set to true,
__and__ we detect we're running against vanilla Postgres.
Locks acquired on `main`
```
pid | mode | granted
-------+------------------+---------
21004 | AccessShareLock | t
21004 | RowExclusiveLock | t
```
Locks acquired with the new feature enabled
```
pid | mode | granted
-------+------------------+---------
21004 | RowShareLock | t
21004 | RowExclusiveLock | t
```
It's not entirely clear that this improves performance, but some
simulation testing seems to indicate that it does.
Also included in this PR are new CI workflows that run platform-checks
and maelstrom against Postgres consensus, using the new Postgres
queries.
Improve performance of self-hosted which runs against vanilla postgres
<!--
Leave some tips for your reviewer, like:
* The diff is much smaller if viewed with whitespace hidden.
* [Some function/module/file] deserves extra attention.
* [Some function/module/file] is pure code movement and only needs a
skim.
Delete this section if no tips.
-->
- [ ] This PR has adequate test coverage / QA involvement has been duly
considered. ([trigger-ci for additional test/nightly
runs](https://trigger-ci.dev.materialize.com/))
- [ ] This PR has an associated up-to-date [design
doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md),
is a design doc
([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)),
or is sufficiently small to not require a design.
<!-- Reference the design in the description. -->
- [ ] If this PR evolves [an existing `$T ⇔ Proto$T`
mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md)
(possibly in a backwards-incompatible way), then it is tagged with a
`T-proto` label.
- [ ] If this PR will require changes to cloud orchestration or tests,
there is a companion cloud PR to account for those changes that is
tagged with the release-blocker label
([example](MaterializeInc/cloud#5021)).
<!-- Ask in #team-cloud on Slack if you need help preparing the cloud
PR. -->
- [ ] If this PR includes major [user-facing behavior
changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note),
I have pinged the relevant PM to schedule a changelog post.
…-deadlock cluster: avoid workers becoming stalled after reconnect
…pped-activations cluster: ensure workers get activated on controller reconnect
…r-dataflow compute: persistent worker command channel
compute: handle future epochs in channel adapter
To fix macOS build. Seen failing in https://buildkite.com/materialize/test/builds/103554#0196d301-1245-4a7c-b87f-36fdcd4427f0
def-
added
the
lts-backport-v25.1
|
Confirmed in https://buildkite.com/materialize/test/builds/103592#0196d36f-6edd-4d2d-82f5-5bd634aa0328 that this (the latest commit) fixes the macOS clippy on newer macOS versions. I won't submit the PR, but will cherry-pick the commit when we cut another v0.130 patch release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This exists only to be cherry-picked to v0.130 when we cut a new patch release.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.