Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FlashInfer Windows support #964

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

SystemPanic
Copy link

This PR adds support for Windows OS, including AOT and JIT builds.

Due to standard console (cmd.exe) command length limit of 8192 chars, PowerShell is required to do the installation / build.

A Visual Studio 2019 or newer is required to launch the compiler x64 environment. The installation path is referred in the instructions as VISUAL_STUDIO_INSTALL_PATH, just replace it with your Visual Studio installation path.

Instructions for Windows build:

  1. Open PowerShell
  2. Clone the repository
  3. Change the working directory to the cloned repository path, for example: cd C:\flashinfer
  4. Execute the following commands in order:
Import-Module "VISUAL_STUDIO_INSTALL_PATH\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"
Enter-VsDevShell -VsInstallPath "VISUAL_STUDIO_INSTALL_PATH" -DevCmdArguments '-arch=x64'
$env:DISTUTILS_USE_SDK=1;
$env:FLASHINFER_ENABLE_AOT=1;
$env:MAX_JOBS=10;
  1. (Optional) To speed up compilation time, set environment var TORCH_CUDA_ARCH_LIST to match your gpu arch. For example: $env:TORCH_CUDA_ARCH_LIST="8.6";
  2. (Optional) To use the installed torch in the case you have a nightly / built from source version, execute: $env:FLASHINFER_USE_CURRENT_TORCH=1;
  3. Execute: pip install . --no-build-isolation to install or python setup.py bdist_wheel to generate a installable Windows wheel
  4. (Optional) Clean the Windows customized build folder: python setup.py clean

Special mention to changes made on:

  • pod.cu: To solve a C1060 compiler error, the PrefillParams and DecodeParams assignments has been moved to an external function.
  • setup.py: To support current and future compiled kernels object linking, the build directory has been customized on Windows to keep the final linker command below the PowerShell limit of 32767 characters,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant