Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/azure-sdk-tools.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ jobs:
with:
python-version: 3.13

- name: Configure UV indexes
run: |
echo "UV_DEFAULT_INDEX=https://pypi.org/simple/" >> "$GITHUB_ENV"
echo "UV_EXTRA_INDEX_URL=https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" >> "$GITHUB_ENV"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We test both pip and uv, so we should set PIP_INDEX_URL as well 👍

shell: bash

- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
Expand Down
69 changes: 69 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,75 @@ SDK performance testing is supported via the custom `perfstress` framework. For

We maintain an [additional document](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/eng_sys_checks.md) that has a ton of detail as to what is actually _happening_ in these executions.

### Package Index Configuration

This repo uses Central Feed Services (CFS) as the default package source for CI, and prefers it for local development over PyPI.

A repo-root `uv.toml` configures uv to use the CFS feed automatically.

To use the feed with standard `pip`, use the `--index-url` flag. For example:

```bash
pip install <pkg> --index-url https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/
```

#### `azpysdk` Configuration

The `azpysdk` tool automatically configures pip and uv subprocesses to use the CFS feed by setting the `PIP_INDEX_URL` and `UV_DEFAULT_INDEX` environment variables. If you have already set these variables to a different feed, `azpysdk` will respect your configuration and won't override them.

If you wish to bypass CFS and install directly from PyPI,use the `--pypi` flag to pull from PyPI:

```
azpysdk --pypi <command>
```

#### Authentication for upstream pull-through
When installing a package version not yet cached in the CFS feed, it needs to be pulled through from PyPI upstream, which requires authentication.

Below are options for authenticating:

**Authenticate in terminal (with uv):**

This requires the Azure CLI. First run:

```bash
az login
```

Then:

```bash
export UV_INDEX_AZURE_SDK_USERNAME=x
export UV_INDEX_AZURE_SDK_PASSWORD=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)
```

**Create a PAT in Azure DevOps:**

1. Go to [https://dev.azure.com/azure-sdk](https://dev.azure.com/azure-sdk)
2. Click the user settings button in the top right corner next to your profile picture
3. Select "Personal access tokens"
4. Click "New Token"
5. Give it a name (e.g., "Azure SDK Python Feed Access")
6. Set expiration as needed
7. Under "Scopes" -> "Packaging", select "Read, write, & manage"
8. Click "Create"
9. Copy the token and use it in your environment:

```bash
export UV_INDEX_AZURE_SDK_USERNAME=x
export UV_INDEX_AZURE_SDK_PASSWORD=<your-pat-token>
```

Or for pip:
```bash
export PIP_INDEX_URL="https://your-azure-username:your-pat-token@pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/"
```

**To override the CFS index and use PyPI with uv:**
```bash
uv pip install <package> --index-url https://pypi.org/simple/
```

### Dev Feed
Daily dev build version of Azure sdk packages for python are available and are uploaded to Azure devops feed daily. Below is the link to Azure devops feed.
[`https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python`](https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python)
Expand Down
17 changes: 17 additions & 0 deletions doc/tool_usage_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,23 @@ To utilize this feature, add `--isolate` to any `azpysdk` invocation:
- The monorepo requires a minimum of `python 3.9`, but `>=3.11` is required for the `sphinx` check due to compatibility constraints with external processes.
- You may optionally use the ["uv"](https://docs.astral.sh/uv/) tool, which is fast and handles Python version and venv creation automatically.

## Package Index (CFS)

This repo defaults to using Central Feed Services (CFS) as the package source instead of PyPI directly. This provides a security layer between the repo and PyPI for all package installs.

**How it works:**
- A repo-root `uv.toml` configures uv to use the CFS feed automatically for all `uv pip install` / `uv sync` commands run within the repo.
- `azpysdk` sets `PIP_INDEX_URL` and `UV_DEFAULT_INDEX` environment variables so pip and uv subprocesses also use the CFS feed. If you have already set these variables to a different feed, `azpysdk` will respect your configuration and won't override them.

**Authentication for upstream pull-through:**
When a package version is not yet cached in the CFS feed, uv/pip pulls it through from PyPI upstream, which requires authentication. See [CONTRIBUTING.md](../CONTRIBUTING.md#authentication-for-upstream-pull-through) for details.

**Bypassing CFS:**
If you need to install directly from PyPI (e.g., for a package not yet in the feed and auth isn't set up), use:
```bash
azpysdk --pypi <command>
```

## Initial setup

- Go to the folder of the package you're working on, for instance `sdk/contoso/azure-contoso`
Expand Down
40 changes: 39 additions & 1 deletion eng/tools/azure-sdk-tools/azpysdk/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,13 @@
from .devtest import devtest
from .optional import optional
from .update_snippet import update_snippet

from ci_tools.logging import configure_logging, logger

__all__ = ["main", "build_parser"]
__version__ = "0.0.0"

CFS_INDEX_URL = "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/"


def build_parser() -> argparse.ArgumentParser:
"""Create and return the top-level ArgumentParser for the CLI."""
Expand All @@ -56,6 +57,12 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--isolate", action="store_true", default=False, help="If set, run in an isolated virtual environment."
)
parser.add_argument(
"--pypi",
action="store_true",
default=False,
help="Use PyPI directly instead of the CFS (Central Feed Services) feed.",
)
parser.add_argument(
"--python",
default=None,
Expand Down Expand Up @@ -155,6 +162,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
Exit code to return to the OS.
"""
original_cwd = os.getcwd()
# Save original env vars so we can restore them when azpysdk finishes
original_pip_index = os.environ.get("PIP_INDEX_URL")
original_uv_index = os.environ.get("UV_DEFAULT_INDEX")

parser = build_parser()
args = parser.parse_args(argv)
Expand All @@ -171,6 +181,25 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
if uv_path:
os.environ["TOX_PIP_IMPL"] = "uv"

# default to CFS feed unless --pypi is specified, but allow explicit env var override (e.g. for CI)
if args.pypi:
# Explicitly set PyPI URLs to override uv.toml CFS default
os.environ["PIP_INDEX_URL"] = "https://pypi.org/simple/"
os.environ["UV_DEFAULT_INDEX"] = "https://pypi.org/simple/"
logger.info("Installing from PyPI (--pypi flag set)")
else:
using_cfs = False
if shutil.which("uv"):
if not os.environ.get("UV_DEFAULT_INDEX"):
os.environ["UV_DEFAULT_INDEX"] = CFS_INDEX_URL
using_cfs = True
else:
if not os.environ.get("PIP_INDEX_URL"):
os.environ["PIP_INDEX_URL"] = CFS_INDEX_URL
using_cfs = True

if using_cfs:
logger.info("Installing from CFS feed: %s", CFS_INDEX_URL)
# --python requires both --isolate and uv
python_version = getattr(args, "python_version", None)
if python_version:
Expand All @@ -196,6 +225,15 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
return 2
finally:
os.chdir(original_cwd)
# Restore original env vars (or remove them if they weren't set before)
if original_pip_index is None:
os.environ.pop("PIP_INDEX_URL", None)
else:
os.environ["PIP_INDEX_URL"] = original_pip_index
if original_uv_index is None:
os.environ.pop("UV_DEFAULT_INDEX", None)
else:
os.environ["UV_DEFAULT_INDEX"] = original_uv_index


if __name__ == "__main__":
Expand Down
4 changes: 4 additions & 0 deletions uv.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[[index]]
name = "azure-sdk"
url = "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/"
default = true
Loading