Skip to content

WIP: Better packages detection #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

WIP: Better packages detection #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 3 additions & 14 deletions flake-module.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,19 +177,7 @@ in
#
# In future, we could just read `cabal.project`. See #76.
let
# Like pkgs.haskell.lib.haskellPathsInDir' but with a few differences
# - Allows top-level .cabal files
haskellPathsInDir' = path:
lib.filterAttrs (k: v: v != null) (lib.mapAttrs'
(k: v:
if v == "regular" && lib.strings.hasSuffix ".cabal" k
then lib.nameValuePair (lib.strings.removeSuffix ".cabal" k) path
else
if v == "directory" && builtins.pathExists (path + "/${k}/${k}.cabal")
then lib.nameValuePair k (path + "/${k}")
else lib.nameValuePair k null
)
(builtins.readDir path));

errorNoDefault = msg:
builtins.throw ''
haskell-flake: A default value for `packages` cannot be auto-detected:
Expand All @@ -199,7 +187,8 @@ in
'';
cabalPaths =
let
cabalPaths = haskellPathsInDir' self;
haskellPathsInDir = import ./nix/haskellPathsInDir.nix { inherit lib; };
cabalPaths = haskellPathsInDir self;
in
if cabalPaths == { }
then
Expand Down
46 changes: 46 additions & 0 deletions nix/haskellPathsInDir.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Like pkgs.haskell.lib.haskellPathsInDir' but with a few differences
# - Allows top-level .cabal files
# - Recurses into subdirectories
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recursing into arbitrary subdirectories hurts performance. This will be exacerbated by projects that rely on lazy fetching of git submodules.

make haskellPathsInDir perfect

Not sure if perfect exists. I'd prefer a cabal.project or stack.yaml parser.

Copy link
Owner Author

@srid srid Feb 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recursing into arbitrary subdirectories hurts performance.

Doesn't nix copy the source tree to the store anyway? I ran this on a large internal mono repo with 31 packages using package.yaml which all lie as far as 5 sub-directories deep, and didn't notice a perceptible difference (to be fair, this was on my M1 Max). But yes performance degradation is a valid concern. I'll benchmark it properly before we stamp this PR for approval.

This will be exacerbated by projects that rely on lazy fetching of git submodules.

Lazy fetching of git submodules?

I'd prefer a cabal.project or stack.yaml parser.

I don't know about stack (why would someone use stack if there is Nix?) - but cabal.project parsing is indeed better (ie. #76); here's our cabal.project

packages:
  lib/*/*.cabal
  app/*/*/*.cabal
  app/*/*/*/*.cabal
  test/*.cabal

(cabal.project doesn't support hpack, so we would also have to look for package.yaml)

I suppose we can just write a Haskell executable (put it in pkgs.haskellPackages.<whatever>) and then use that to get a JSON back. This Haskell executable can also expose the hpack YAML data as JSON to Nix. In a way, it would be like cabal2nix. Maybe I should just go this route instead of this PR ...

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haskell executable

Like https://github.com/NorfairKing/cabal2json#readme

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Haskell executable can even do the traversal and return packages information.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lazy fetching of git submodules?

Sorry, I should have explained myself. I'm considering unimplemented lazy-trees features.

why would someone use stack if there is Nix?

The stack repl multi-module hack is rather useful. You can load a repl where all local packages are quick-reloadable.

There's also horizon-haskell, which is like Nixpkgs Haskell, but more stackage based.

I suppose we can just write a Haskell executable (put it in pkgs.haskellPackages.<whatever>) and then use that to get a JSON back.

It would be really nice to avoid IFD for this purpose. Ideally, the set of flake attribute names can be determined without doing any IFD. This keeps evaluation quick, especially on monorepos, and especially when you don't always need a haskell package in an evaluation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# - Allows package.yaml files
{ lib, ... }:

path':

let
parseCabalName = file:
lib.strings.removeSuffix ".cabal" file;
parseHpackName = file:
# TODO: this is a hack, we should use a proper parser
let
line = builtins.head (lib.strings.split "[[:space:]]*\n[[:space:]]*" (builtins.readFile file));
extract = xs: builtins.head (builtins.tail (builtins.tail xs));
in
extract (lib.strings.split "[[:space:]]*name[[:space:]]*:[[:space:]]*" line);
f = path: lib.mapAttrsToList
(k: v:
let
pass = [ ];
one = x: [ x ];
in
if v == "regular"
then
if lib.strings.hasSuffix ".cabal" k
then one (lib.nameValuePair (parseCabalName k) path)
else if k == "package.yaml"
then one (lib.nameValuePair (parseHpackName k) path)
else pass
else
if v == "directory"
then
if builtins.pathExists (path + "/${k}/${k}.cabal")
then
one (lib.nameValuePair k (path + "/${k}"))
else
if builtins.pathExists (path + "/${k}/package.yaml")
then one (lib.nameValuePair (parseHpackName (path + "/${k}/package.yaml")) (path + "/${k}"))
else f (path + "/${k}")
else pass
)
(builtins.readDir path);
in
lib.listToAttrs (lib.flatten (f path'))