Skip to content

Conversation

aaronsteers
Copy link
Contributor

@aaronsteers aaronsteers commented Jul 23, 2025

Combines:

Summary by CodeRabbit

  • Bug Fixes

    • Enhanced compatibility with WebAssembly (emscripten) by dynamically selecting serialization and validation libraries based on the runtime environment.
  • Chores

    • Updated dependency versions and platform-specific markers to improve environment support.
    • Made certain dependencies optional or limited to development use.
    • Removed unused dependencies from the project.
    • Added new ignore pattern for Pyodide build artifacts.

devin-ai-integration bot and others added 8 commits July 23, 2025 21:21
…l for WASM compatibility

- Remove psutil and rapidfuzz from main dependencies (marked as unused)
- Move google-cloud-secret-manager to dev extras group
- Add new 'system' extras group for psutil and rapidfuzz
- Update deptry configuration comments to reflect optional status

This enables airbyte-cdk to be installed in WASM environments like Pyodide
by making problematic binary dependencies optional.

Co-Authored-By: AJ Steers <[email protected]>
Automatic code formatting applied during dependency update process.
Includes line wrapping and copyright header removal.

Co-Authored-By: AJ Steers <[email protected]>
Address PR feedback from @aaronsteers to fully remove psutil and rapidfuzz
rather than making them optional dependencies. These packages are unused
and were marked as TODO for removal.

Changes:
- Remove psutil and rapidfuzz from optional dependencies
- Remove 'system' extras group (no longer needed)
- Update deptry comments to reflect complete removal
- Keep google-cloud-secret-manager in dev extras as requested

Co-Authored-By: AJ Steers <[email protected]>
- Add conditional dependencies in pyproject.toml using environment markers
- Use sys_platform != 'emscripten' for serpyco-rs, sys_platform == 'emscripten' for serpyco
- Update imports in airbyte_protocol_serializers.py, airbyte_protocol.py, entrypoint_wrapper.py, and test_source.py
- Add conditional import logic to detect WASM/Pyodide environments at runtime
- Maintain API compatibility between serpyco-rs and serpyco libraries
- All tests pass with conditional imports

Co-Authored-By: AJ Steers <[email protected]>
@Copilot Copilot AI review requested due to automatic review settings July 23, 2025 22:06
@github-actions github-actions bot added the enhancement New feature or request label Jul 23, 2025
@aaronsteers aaronsteers changed the base branch from devin/1753304881-serpyco-wasm-fallback to main July 23, 2025 22:06
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR makes the CDK (Connector Development Kit) runnable in WebAssembly (WASM) by removing dependencies that are incompatible with WASM environments and making others optional.

  • Removes WASM-incompatible dependencies: psutil and rapidfuzz
  • Makes google-cloud-secret-manager an optional dependency in the 'dev' extra
  • Updates dependency configuration and comments to reflect WASM compatibility changes

@@ -39,20 +39,18 @@ click = "^8.1.8"
dpath = "^2.1.6"
dunamai = "^1.22.0"
genson = "1.3.0"
google-cloud-secret-manager = "^2.17.0"
google-cloud-secret-manager = { version = "^2.17.0", optional = true } # 'dev' extra
Copy link
Preview

Copilot AI Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The comment "'dev' extra" could be clearer. Consider using "Optional dependency included in 'dev' extra" to better explain its purpose and usage.

Suggested change
google-cloud-secret-manager = { version = "^2.17.0", optional = true } # 'dev' extra
google-cloud-secret-manager = { version = "^2.17.0", optional = true } # Optional dependency included in 'dev' extra

Copilot uses AI. Check for mistakes.

@@ -230,14 +228,12 @@ DEP001 = [
DEP002 = [
"setuptools", # Incorrectly used without declaring in some connectors (e.g. source-shopify)
"cryptography", # Constrained as transitive dependency due to a bug in newer versions
"google-cloud-secret-manager", # Deptry can't detect that `google.cloud.secretmanager_v1` uses this package
"google-cloud-secret-manager", # Deptry can't detect that `google.cloud.secretmanager_v1` uses this package - optional 'dev' dependency
Copy link
Preview

Copilot AI Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The comment combines two different pieces of information. Consider splitting into two comments or restructuring: "Deptry can't detect that google.cloud.secretmanager_v1 uses this package. Now optional 'dev' dependency for WASM compatibility."

Suggested change
"google-cloud-secret-manager", # Deptry can't detect that `google.cloud.secretmanager_v1` uses this package - optional 'dev' dependency
"google-cloud-secret-manager", # Deptry can't detect that `google.cloud.secretmanager_v1` uses this package.
# This is an optional 'dev' dependency for WASM compatibility.

Copilot uses AI. Check for mistakes.

Copy link

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@aj/feat/run-cdk-in-wasm#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch aj/feat/run-cdk-in-wasm

Helpful Resources

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /poe <command> - Runs any poe command in the CDK environment

📝 Edit this welcome message.

Copy link

github-actions bot commented Jul 23, 2025

PyTest Results (Fast)

1 tests   - 3 699   0 ✅  - 3 689   2s ⏱️ - 6m 42s
1 suites ±    0   0 💤  -    11 
1 files   ±    0   0 ❌ ±    0   1 🔥 +1 

For more details on these errors, see this check.

Results for commit b90f3f5. ± Comparison against base commit 51cfea5.

This pull request removes 3700 and adds 1 tests. Note that renamed tests count towards both.
unit_tests.cli.airbyte_cdk.test_secrets.TestFetch ‑ test_fetch_with_all_failed_secrets
unit_tests.cli.airbyte_cdk.test_secrets.TestFetch ‑ test_fetch_with_some_failed_secrets
unit_tests.cli.airbyte_cdk.test_secrets.TestWriteSecretFile ‑ test_write_secret_file_with_enabled_version
unit_tests.cli.airbyte_cdk.test_secrets.TestWriteSecretFile ‑ test_write_secret_file_with_no_enabled_versions
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel0-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel1-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel2-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel3-False]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel4-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel5-True]
…
unit_tests.connector_builder.test_connector_builder_handler

♻️ This comment has been updated with latest results.

Copy link
Contributor

coderabbitai bot commented Jul 23, 2025

📝 Walkthrough

Walkthrough

This update introduces conditional imports throughout the codebase to select between the serpyco_rs (Rust-backed) and serpyco (pure Python) libraries depending on whether the platform is emscripten. Associated dependency declarations in pyproject.toml are adjusted to reflect this platform-specific logic, and some unused dependencies are removed.

Changes

File(s) Change Summary
airbyte_cdk/models/airbyte_protocol.py Added platform check to conditionally import Alias from serpyco_rs.metadata or serpyco.metadata based on emscripten platform.
airbyte_cdk/models/airbyte_protocol_serializers.py Added sys import; conditional import of CustomType and Serializer from serpyco_rs or serpyco depending on platform.
airbyte_cdk/test/entrypoint_wrapper.py
unit_tests/sources/test_source.py
Added sys import; conditional import of SchemaValidationError from serpyco_rs or serpyco based on platform.
pyproject.toml Updated dependency declarations for platform-specific serpyco/serpyco-rs; made google-cloud-secret-manager optional; removed psutil and rapidfuzz; loosened pandas and pytz version constraints.
.gitignore Added .pyodide-xbuildenv pattern to ignore Pyodide build artifacts.

Sequence Diagram(s)

sequenceDiagram
    participant App
    participant sys
    participant serpyco_rs
    participant serpyco

    App->>sys: Check platform (sys.platform)
    alt platform != "emscripten"
        App->>serpyco_rs: Import modules/classes
    else platform == "emscripten"
        App->>serpyco: Import modules/classes
    end
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Would you like to consider adding a brief comment near each conditional import to clarify the rationale for future maintainers, or do you prefer to keep the code concise as is? Wdyt?

Suggested reviewers

  • aldogonzalez8

Note

⚡️ Unit Test Generation - Beta

CodeRabbit's unit test generation is now available in Beta! Automatically generate comprehensive unit tests for your code changes, ensuring better test coverage and catching edge cases you might miss. Our AI analyzes your code structure and creates tests that follow best practices and your project's testing patterns. Learn more here, or just try it under ✨ Finishing Touches.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between bda9765 and c5b0767.

📒 Files selected for processing (2)
  • .gitignore (1 hunks)
  • pyproject.toml (4 hunks)
✅ Files skipped from review due to trivial changes (1)
  • .gitignore
🚧 Files skipped from review as they are similar to previous changes (1)
  • pyproject.toml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: SDM Docker Image Build
  • GitHub Check: Analyze (python)
✨ Finishing Touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch aj/feat/run-cdk-in-wasm

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (1)
pyproject.toml (1)

231-231: Consider improving comment clarity as suggested previously.

The past review comment about splitting this into two separate comments still applies. What do you think about making it clearer by separating the deptry detection issue from the WASM compatibility note, wdyt?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 51cfea5 and ca46edf.

⛔ Files ignored due to path filters (1)
  • poetry.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • airbyte_cdk/models/airbyte_protocol.py (1 hunks)
  • airbyte_cdk/models/airbyte_protocol_serializers.py (2 hunks)
  • airbyte_cdk/test/entrypoint_wrapper.py (2 hunks)
  • pyproject.toml (4 hunks)
  • unit_tests/sources/test_source.py (1 hunks)
🧰 Additional context used
🧠 Learnings (6)
📓 Common learnings
Learnt from: ChristoGrab
PR: airbytehq/airbyte-python-cdk#58
File: airbyte_cdk/sources/declarative/yaml_declarative_source.py:0-0
Timestamp: 2024-11-18T23:40:06.391Z
Learning: When modifying the `YamlDeclarativeSource` class in `airbyte_cdk/sources/declarative/yaml_declarative_source.py`, avoid introducing breaking changes like altering method signatures within the scope of unrelated PRs. Such changes should be addressed separately to minimize impact on existing implementations.
Learnt from: aaronsteers
PR: airbytehq/airbyte-python-cdk#58
File: airbyte_cdk/cli/source_declarative_manifest/_run.py:62-65
Timestamp: 2024-11-15T01:04:21.272Z
Learning: The files in `airbyte_cdk/cli/source_declarative_manifest/`, including `_run.py`, are imported from another repository, and changes to these files should be minimized or avoided when possible to maintain consistency.
airbyte_cdk/models/airbyte_protocol.py (4)

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/_run.py:62-65
Timestamp: 2024-11-15T01:04:21.272Z
Learning: The files in airbyte_cdk/cli/source_declarative_manifest/, including _run.py, are imported from another repository, and changes to these files should be minimized or avoided when possible to maintain consistency.

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/spec.json:9-15
Timestamp: 2024-11-15T00:59:08.154Z
Learning: When code in airbyte_cdk/cli/source_declarative_manifest/ is being imported from another repository, avoid suggesting modifications to it during the import process.

Learnt from: ChristoGrab
PR: #58
File: airbyte_cdk/sources/declarative/yaml_declarative_source.py:0-0
Timestamp: 2024-11-18T23:40:06.391Z
Learning: When modifying the YamlDeclarativeSource class in airbyte_cdk/sources/declarative/yaml_declarative_source.py, avoid introducing breaking changes like altering method signatures within the scope of unrelated PRs. Such changes should be addressed separately to minimize impact on existing implementations.

Learnt from: aaronsteers
PR: #174
File: airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py:1093-1102
Timestamp: 2025-01-14T00:20:32.310Z
Learning: In the airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py file, the strict module name checks in _get_class_from_fully_qualified_class_name (requiring module_name to be "components" and module_name_full to be "source_declarative_manifest.components") are intentionally designed to provide early, clear feedback when class declarations won't be found later in execution. These restrictions may be loosened in the future if the requirements for class definition locations change.

airbyte_cdk/test/entrypoint_wrapper.py (4)

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/_run.py:62-65
Timestamp: 2024-11-15T01:04:21.272Z
Learning: The files in airbyte_cdk/cli/source_declarative_manifest/, including _run.py, are imported from another repository, and changes to these files should be minimized or avoided when possible to maintain consistency.

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/spec.json:9-15
Timestamp: 2024-11-15T00:59:08.154Z
Learning: When code in airbyte_cdk/cli/source_declarative_manifest/ is being imported from another repository, avoid suggesting modifications to it during the import process.

Learnt from: ChristoGrab
PR: #58
File: airbyte_cdk/sources/declarative/yaml_declarative_source.py:0-0
Timestamp: 2024-11-18T23:40:06.391Z
Learning: When modifying the YamlDeclarativeSource class in airbyte_cdk/sources/declarative/yaml_declarative_source.py, avoid introducing breaking changes like altering method signatures within the scope of unrelated PRs. Such changes should be addressed separately to minimize impact on existing implementations.

Learnt from: pnilan
PR: airbytehq/airbyte-python-cdk#0
File: :0-0
Timestamp: 2024-12-11T16:34:46.319Z
Learning: In the airbytehq/airbyte-python-cdk repository, the declarative_component_schema.py file is auto-generated from declarative_component_schema.yaml and should be ignored in the recommended reviewing order.

unit_tests/sources/test_source.py (2)

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/spec.json:9-15
Timestamp: 2024-11-15T00:59:08.154Z
Learning: When code in airbyte_cdk/cli/source_declarative_manifest/ is being imported from another repository, avoid suggesting modifications to it during the import process.

Learnt from: ChristoGrab
PR: #221
File: airbyte_cdk/sources/utils/transform.py:0-0
Timestamp: 2025-01-16T00:50:39.069Z
Learning: In the TypeTransformer class, the data being transformed comes from API responses or source systems, so only standard JSON-serializable types are expected. The python_to_json mapping covers all expected types, and it's designed to fail fast (KeyError) on unexpected custom types rather than providing fallbacks.

airbyte_cdk/models/airbyte_protocol_serializers.py (3)

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/_run.py:62-65
Timestamp: 2024-11-15T01:04:21.272Z
Learning: The files in airbyte_cdk/cli/source_declarative_manifest/, including _run.py, are imported from another repository, and changes to these files should be minimized or avoided when possible to maintain consistency.

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/spec.json:9-15
Timestamp: 2024-11-15T00:59:08.154Z
Learning: When code in airbyte_cdk/cli/source_declarative_manifest/ is being imported from another repository, avoid suggesting modifications to it during the import process.

Learnt from: ChristoGrab
PR: #221
File: airbyte_cdk/sources/utils/transform.py:0-0
Timestamp: 2025-01-16T00:50:39.069Z
Learning: In the TypeTransformer class, the data being transformed comes from API responses or source systems, so only standard JSON-serializable types are expected. The python_to_json mapping covers all expected types, and it's designed to fail fast (KeyError) on unexpected custom types rather than providing fallbacks.

pyproject.toml (1)

Learnt from: aaronsteers
PR: #58
File: airbyte_cdk/cli/source_declarative_manifest/_run.py:62-65
Timestamp: 2024-11-15T01:04:21.272Z
Learning: The files in airbyte_cdk/cli/source_declarative_manifest/, including _run.py, are imported from another repository, and changes to these files should be minimized or avoided when possible to maintain consistency.

🪛 GitHub Actions: Linters
airbyte_cdk/models/airbyte_protocol.py

[error] 8-8: ruff formatting check failed: inconsistent use of single vs double quotes around 'emscripten'. Run 'ruff format --fix' to fix code style issues.

airbyte_cdk/test/entrypoint_wrapper.py

[error] 30-30: ruff formatting check failed: inconsistent use of single vs double quotes around 'emscripten'. Run 'ruff format --fix' to fix code style issues.

unit_tests/sources/test_source.py

[error] 12-12: ruff formatting check failed: inconsistent use of single vs double quotes around 'emscripten'. Run 'ruff format --fix' to fix code style issues.

airbyte_cdk/models/airbyte_protocol_serializers.py

[error] 23-23: mypy: Name "CustomType" already defined (possibly by an import) [no-redef]


[error] 23-23: mypy: Name "Serializer" already defined (possibly by an import) [no-redef]

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
  • GitHub Check: Build Python Package
  • GitHub Check: Check: source-intercom
  • GitHub Check: Check: destination-motherduck
  • GitHub Check: Check: source-hardcoded-records
  • GitHub Check: Check: source-shopify
  • GitHub Check: Check: source-pokeapi
  • GitHub Check: Pytest (Fast)
  • GitHub Check: preview_docs
  • GitHub Check: Pytest (All, Python 3.11, Ubuntu)
  • GitHub Check: SDM Docker Image Build
  • GitHub Check: Pytest (All, Python 3.10, Ubuntu)
  • GitHub Check: Analyze (python)
🔇 Additional comments (5)
pyproject.toml (1)

86-87: Great approach for WASM compatibility!

The platform-specific dependency setup using markers is exactly what's needed to support both regular Python environments and WebAssembly. The conditional logic here aligns perfectly with the runtime platform detection in the code files.

airbyte_cdk/test/entrypoint_wrapper.py (1)

20-20: Nice addition for platform detection!

The sys import is properly placed and will be used for the conditional import logic below.

airbyte_cdk/models/airbyte_protocol.py (1)

5-5: Good addition for platform detection.

The sys import is correctly placed for the conditional logic below.

unit_tests/sources/test_source.py (1)

7-7: Proper sys import for platform detection.

Good placement of the sys import for the conditional logic.

airbyte_cdk/models/airbyte_protocol_serializers.py (1)

2-2: Good sys import for platform detection.

Properly placed import for the conditional logic.

Comment on lines 17 to 24
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""

if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer
else:
from serpyco import CustomType, Serializer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Excellent approach with the constant, but fix mypy issue.

I love the USE_RUST_BACKEND constant approach - it makes the code much more readable than inline platform checks! However, mypy is complaining about the name redefinition in the conditional imports.

Could you fix the mypy issue by using type: ignore comments or restructuring the imports, wdyt?

 if USE_RUST_BACKEND:
-    from serpyco_rs import CustomType, Serializer
+    from serpyco_rs import CustomType, Serializer  # type: ignore[misc]
 else:
-    from serpyco import CustomType, Serializer
+    from serpyco import CustomType, Serializer  # type: ignore[misc]

Alternatively, you could import with aliases and then assign to common names:

 if USE_RUST_BACKEND:
-    from serpyco_rs import CustomType, Serializer
+    from serpyco_rs import CustomType as _CustomType, Serializer as _Serializer
 else:
-    from serpyco import CustomType, Serializer
+    from serpyco import CustomType as _CustomType, Serializer as _Serializer
+
+CustomType = _CustomType
+Serializer = _Serializer
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer
else:
from serpyco import CustomType, Serializer
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer # type: ignore[misc]
else:
from serpyco import CustomType, Serializer # type: ignore[misc]
Suggested change
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer
else:
from serpyco import CustomType, Serializer
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType as _CustomType, Serializer as _Serializer
else:
from serpyco import CustomType as _CustomType, Serializer as _Serializer
CustomType = _CustomType
Serializer = _Serializer
🧰 Tools
🪛 GitHub Actions: Linters

[error] 23-23: mypy: Name "CustomType" already defined (possibly by an import) [no-redef]


[error] 23-23: mypy: Name "Serializer" already defined (possibly by an import) [no-redef]

🤖 Prompt for AI Agents
In airbyte_cdk/models/airbyte_protocol_serializers.py around lines 17 to 24,
mypy raises an error due to the redefinition of CustomType and Serializer in the
conditional imports. To fix this, import the modules with distinct aliases based
on the condition and then assign the common names CustomType and Serializer to
the appropriate aliases. This avoids name redefinition and satisfies mypy
without losing readability.

Comment on lines 11 to 14
if sys.platform == 'emscripten':
from serpyco.metadata import Alias
else:
from serpyco_rs.metadata import Alias
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix quote consistency for pipeline compliance.

The conditional import approach for the Alias class is perfect for WASM support! Just need to fix the quote style to match the pipeline requirements.

Could you update the quote style to be consistent, wdyt?

-if sys.platform == 'emscripten':
+if sys.platform == "emscripten":
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if sys.platform == 'emscripten':
from serpyco.metadata import Alias
else:
from serpyco_rs.metadata import Alias
if sys.platform == "emscripten":
from serpyco.metadata import Alias
else:
from serpyco_rs.metadata import Alias
🤖 Prompt for AI Agents
In airbyte_cdk/models/airbyte_protocol.py around lines 11 to 14, the quote style
used in the conditional import statement is inconsistent. Update all string
literals in this block to use the same type of quotes (either single or double)
consistently to comply with the pipeline's style requirements.

Comment on lines 33 to 36
if sys.platform == 'emscripten':
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix quote consistency for pipeline compliance.

The conditional import logic is solid for WASM compatibility! However, there's a formatting issue with quote consistency that's causing the pipeline to fail.

Could you apply this fix to match the codebase's quote style, wdyt?

-if sys.platform == 'emscripten':
+if sys.platform == "emscripten":
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if sys.platform == 'emscripten':
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
if sys.platform == "emscripten":
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
🤖 Prompt for AI Agents
In airbyte_cdk/test/entrypoint_wrapper.py around lines 33 to 36, the import
statements use inconsistent quote styles. To fix the pipeline failure caused by
this, update the quotes in both import lines to match the project's preferred
quote style (either single or double quotes consistently). Ensure both 'serpyco'
and 'serpyco_rs' imports use the same type of quotes.

Comment on lines 15 to 18
if sys.platform == 'emscripten':
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Great test compatibility approach, but fix quotes.

The conditional import ensures tests work correctly with both serialization backends depending on the platform. Just need to fix the quote consistency for pipeline compliance.

Could you update to use double quotes, wdyt?

-if sys.platform == 'emscripten':
+if sys.platform == "emscripten":
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if sys.platform == 'emscripten':
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
if sys.platform == "emscripten":
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
🤖 Prompt for AI Agents
In unit_tests/sources/test_source.py around lines 15 to 18, the import
statements use single quotes for the platform string and module names. Update
all single quotes to double quotes for consistency and pipeline compliance,
changing 'emscripten' to "emscripten" and similarly for the module names in the
import statements.

Copy link

github-actions bot commented Jul 23, 2025

PyTest Results (Full)

15 tests   - 3 688    0 ✅  - 3 692   14s ⏱️ - 17m 57s
 1 suites ±    0    0 💤  -    11 
 1 files   ±    0    0 ❌ ±    0   15 🔥 +15 

For more details on these errors, see this check.

Results for commit b90f3f5. ± Comparison against base commit 51cfea5.

This pull request removes 3703 and adds 15 tests. Note that renamed tests count towards both.
unit_tests.cli.airbyte_cdk.test_secrets.TestFetch ‑ test_fetch_with_all_failed_secrets
unit_tests.cli.airbyte_cdk.test_secrets.TestFetch ‑ test_fetch_with_some_failed_secrets
unit_tests.cli.airbyte_cdk.test_secrets.TestWriteSecretFile ‑ test_write_secret_file_with_enabled_version
unit_tests.cli.airbyte_cdk.test_secrets.TestWriteSecretFile ‑ test_write_secret_file_with_no_enabled_versions
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel0-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel1-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel2-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel3-False]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel4-True]
unit_tests.cli.airbyte_cdk.test_util ‑ test_resolve_airbyte_repo_root_real_fs[start_dir_rel5-True]
…
unit_tests.connector_builder.test_connector_builder_handler
unit_tests.connector_builder.test_message_grouper
unit_tests.connector_builder.test_property_chunking
unit_tests.destinations.test_destination
unit_tests.destinations.vector_db_based.writer_test
unit_tests.sources.declarative.partition_routers.test_parent_state_stream
unit_tests.sources.file_based.test_file_based_scenarios
unit_tests.sources.file_based.test_file_based_stream_reader
unit_tests.sources.file_based.test_scenarios
unit_tests.sources.streams.concurrent.scenarios.test_concurrent_scenarios
…

♻️ This comment has been updated with latest results.

@aaronsteers aaronsteers marked this pull request as draft July 23, 2025 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant