Using clash binaries within projects

[kleinreact]

There currently is a confusing mix of options in the wild to use the clash compiler binary within a hardware design project:

• system: the binary might be installed on the system level (independently of a particular project)
• cabal: the binary might be pulled in via the cabal build system taking the project and its dependencies into account
• bogus copy: a bogus copy of clash can be created within the project as a new executable to link it with the utilized build system (as for example utilized by clash-startes)

Additionally, all these variants can be executed differently within the cabal and stack environments leading to a bunch of possible options.

Hence, to get a better overview, I did some experiments with this simple project in order to get some first overview. The results are summarized below. Note that I also utilized a cabal install + nix shell setup, in order to work around some problems with my system setup, where I listed both variants for comparison. Binaries that are installed with cabal are put into a local bin folder (cabal install --installdir=bin clash-ghc) to keep them local within the project. You might also note that all of these variants utilize slightly different configurations of GHC and the Clash compiler to see how bad interferences can be.

The experiments are split into two stages: one with the cabal.project of the referenced project and one without. Note that this cabal.project only enforces to always write GHC environment files and pulls in a particular version of clash.

The PATH column indicates either a direct call of the given executable or having it put into the PATH environment variable first.

Without the cabal.project

clash binary source	PATH	cabal run	cabal exec	stack run	stack exec
system	❌	❌	(✔)	❌	(✔)*
cabal install	❌	-	-	-	-
cabal install + nix shell	❌	❌	(✔)	❌	(✔)*
bogus copy	❌	❌	(✔)	☑️	(✔)*

Error messages of the different ❌s    (click to expand)

$ clash src/TopEntity.hs --verilog    # system [PATH]
GHC: Setting up GHC took: 0.097s

[1 of 1] Compiling TopEntity        ( src/TopEntity.hs, src/TopEntity.o, src/TopEntity.dyn_o )

src/TopEntity.hs:16:1: error:
    Could not find module ‘OtherModule’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
16 | import OtherModule
   | ^^^^^^^^^^^^^^^

$ cabal run -- clash src/TopEntity.hs --verilog    # system [cabal run]
Resolving dependencies...
Error: cabal: Cannot run the package clash, it is not in this project (either
directly or indirectly). If you want to add it to the project then edit the
cabal.project file.

$ stack run -- clash src/TopEntity.hs --verilog    # system [stack run]
GHC: Setting up GHC took: 1.232s

<no location info>: error:
    Other error:
    target ‘clash’ is not a module name or a source file

$ cabal install --installdir=bin clash-ghc    # cabal install [*]
Wrote tarball sdist to
/home/user/demo-project/dist-newstyle/sdist/demo-1.0.tar.gz
Resolving dependencies...
Error: cabal: Cannot build the executables in the package clash-ghc because it
does not contain any executables. Check the .cabal file for the package and
make sure that it properly declares the components that you expect.

$ clash src/TopEntity.hs --verilog    # cabal install + nix-shell [PATH]
<command line>: Could not find module ‘GHC.TypeLits.Normalise’
Use -v (or `:set -v` in ghci) to see a list of the files searched for.

$ cabal run -- clash src/TopEntity.hs --verilog    # cabal install + nix-shell [cabal run]
Resolving dependencies...
Error: cabal: Cannot run the package clash, it is not in this project (either
directly or indirectly). If you want to add it to the project then edit the
cabal.project file.

$ stack run -- clash src/TopEntity.hs --verilog    # cabal install + nix-shell [stack run]
GHC: Setting up GHC took: 1.682s

<no location info>: error:
    Other error:
    target ‘clash’ is not a module name or a source file

$ ./dist-newstyle/build/x86_64-linux/ghc-9.4.5/demo-1.0/x/bogus-clash/build/bogus-clash/bogus-clash src/TopEntity.hs --verilog    # bogus copy [PATH]
GHC: Setting up GHC took: 1.190s

src/TopEntity.hs:16:1: error:
    Could not find module ‘OtherModule’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
16 | import OtherModule
   | ^^^^^^^^^^^^^^^

$ cabal run -- bogus-clash src/TopEntity.hs --verilog    # bogus copy [cabal run]
Resolving dependencies...
...
GHC: Setting up GHC took: 1.151s

src/TopEntity.hs:16:1: error:
    Could not find module ‘OtherModule’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
16 | import OtherModule
   | ^^^^^^^^^^^^^^^

With the cabal.project

clash binary source	PATH	cabal run	cabal exec	stack run	stack exec
system	❌	(✔)*	(✔)*	❌	(✔)*
cabal install	(✔)	(✔)*	(✔)*	❌	(✔)*
cabal install + nix shell	❌	(✔)*	(✔)*	❌	(✔)*
bogus copy	(✔)	☑️	(✔)	☑️	(✔)*

Error messages of the different ❌s    (click to expand)

$ cabal build && clash src/TopEntity.hs --verilog    # system [PATH]
Loaded package environment from /home/user/demo/.ghc.environment.x86_64-linux-9.4.5
GHC: Setting up GHC took: 0.301s

<no location info>: error:

    Clash only works with the exact clash-prelude it was built with.
    Clash was built with: clash-prelude-1.8.1-84Px0bEzBrcALnkZXJ9ivy
    So can't run with:    clash-prelude-1.9.0-380a9aaaedfa505a9c0838771a3e8961b36a0456cb1ddf1de0c97c79e0fc7ab3

$ stack run -- clash src/TopEntity.hs --verilog    # system [stack run]
GHC: Setting up GHC took: 1.167s

<no location info>: error:
    Other error:
    target ‘clash’ is not a module name or a source file

$ stack run -- clash src/TopEntity.hs --verilog    # cabal-install [stack run]
GHC: Setting up GHC took: 1.167s

<no location info>: error:
    Other error:
    target ‘clash’ is not a module name or a source file

$ ./bin/clash src/TopEntity.hs --verilog    # cabal install + nix shell [PATH]

<no location info>: error:
    Other error:
    <command line>: Could not find module ‘GHC.TypeLits.Normalise’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.

$ stack run -- clash src/TopEntity.hs --verilog    # cabal install + nix shell [stack run]
GHC: Setting up GHC took: 1.889s

<no location info>: error:
    Other error:
    target ‘clash’ is not a module name or a source file

(✔): works, but requires cabal build / stack build first
__*: uses the clash binary pulled in by the utilized build system, not the one in PATH

---

The following observations might also be noteworthy:

• cabal repl and stack repl work without problems for the referenced setup, being a prerequisite.

• cabal exec and cabal run behave completely differently with and without the cabal.project being present:

  Without the cabal.project, cabal run only has access to executables within the project, while cabal exec basically can use any executable in PATH. Nonetheless, cabal exec is special, as it runs them in the scope of the package's environment files, causing the same effect as putting the respective option to the cabal.project file.

  With the cabal.project in scope, every clash binary in PATH turns obsolete and is not available any more in that context. The cabal.project references it's own copy of clash such that both, cabal run and cabal exec, consider this reference as the only option available. Both commands now only behave differently on the bogus copy, where run automatically rebuilds the dependencies, while exec does not.

• Automatic rebuilding of the dependencies really only works for the bogus copy in combination with cabal / stack run.

• It seems to be hard/impossible to use clash in PATH under the scope of cabal, if a cabal.project is present.

---

The main problem that I see with the currently available mix is that it can turn development to be very confusing for users. Not only for starters, but also for experts. Having multiple clash binaries around makes it hard to track the one that is currently utilized. Also starters usually don't know anything about stack or cabal, so the get confused quickly with these subtleties.

A more standard way would be to share the binary via PATH instead, which would conform to a standard POSIX work environment at least. Then inspecting PATH with whereis clash for example makes it transparent, which clash binary is used and no bogus copies have to be around. Using this in combination with cabal install --instaldir=<local dir> or stack install --local-bin-path=<local-dir> still would allow for the option to pull project specific versions of clash, where utilities like nix shell or direnv could help in pushing them to your PATH automatically (see here for a reference). The only difference against the bogus copy is that the user is responsible on his own to cabal build first, which is a standard build system dependency tracking issue however.

Another minor that I observed is that it might be useful if clash --version would also output the commit hash of the given build. This would make distinguishing multiple binaries around easier at least.

I am interested on your opinions on this, because I think that in terms of user experience it would be more useful if we could offer a more straightforward solution that what we currenlty have.

[DigitalBrains1]

We have been bitten by unexpected issues in the past. What I'm worried about, is that if we go around fiddling with this, we forget about an edge case and make the experience worse without noticing it until someone is assertive enough to open an issue or discussion about it (many users will simply work around it instead of contacting a project).

I don't see a compelling reason to change what we do until upstream tools make significant changes.

We could assuage your objections by renaming the binary we build in a starter project; if we don't call it clash and clashi, you won't get confused and can more easily find where it is defined.

[edit]
I think upstream GHC recommends against system-wide installations. If I'm right about that, I cannot imagine us recommending such a setup either anyway.
[/edit]

[martijnbastiaan]

Related issue: #2107.

As you've noticed all the proposed solutions have some pretty big downsides over the current one. Either they don't invalidate caches, or they don't work at all.

In any case, I don't think the issues are quite as bad as being presented here: you only run into "real" issues if you're stubborn enough to ignore the installation instructions on Hackage and clash-lang.org/install - at which point I don't care too much anymore. There is an issue if you add clash-compiler as a dependency to cabal.project, but even that I'd brush up as minor: Cabal pretty much tells you what to do.

If we'd want it to be an experience similar to vanilla Rust/Haskell, we'd have to host our own version of ghcup and quite possibly cabal.

[leonschoorl]

Conceptually I don't like the (bogus) copies of clash binaries in each project either.

But the fact that it is the only row in your table that has any ☑️, is the reason that it (via clash-starters) is the recommended way of running clash right now.
It solves a number of problems in a convenient, albeit kind of hacky, way.

Here are some of the issues I'm aware of around using a clash installed globally (both at the system level or at the user level):

1. GHC has the concept of boot libraries, these come shipped with the compiler and generally can't be upgraded.
   GHC, cabal and stack have special ways of dealing with these libraries.
   These includes things like: base, ghc, ghc-prim,template-haskell.
   Clash sort of extends this set with: clash-lib, clash-prelude and ghc-typelits-[extra,knownnat,natnormalise].
   But we don't have a way of building into the clash binary where to find these. (This is why outside of project cabal install clash-ghc && clashi will fail because it can't find the type checker plugins.)
2. If we did have a way to bake in the locations of the extended boot libraries into clash(i), you could use a global clash standalone for simple projects, without stack or cabal.
   But when your project grows more complex and you want to use libraries other then the extended boot libraries you'd have to:
   • a. Install them globally too, with all the "cabal hell" problems of the cabal v1-style of doing things
   • b. Or start using some build tool like stack or cabal, but then we'd then need a way of telling them about the extended boot libraries. Or they will want to build their own versions of them, and that breaks things.
3. When build with the wrong flavour of linking clash can be very slow to start, the starter project fixes this here. This doesn't look like a major issue, and looking at it now, seems like we should just make -dynamic the default in the clash-ghc. But maybe there is a good reason we don't.
4. When going the 2b route, cabal or stack is now in charge of at least building and bringing into scope your dependencies. And you could make a choice:

• a. Run clash in this environment directly on the source files of your design.
  This can work but clash isn't aware of what you may have put in your cabal file.
  So if you store your source files in the src dir, you'd have to tell it to look there(with -isrc). Forgetting this is the cause your:

src/TopEntity.hs:16:1: error:
    Could not find module ‘OtherModule’

• b. Use stack/cabal to build your code with GHC like a "normal" Haskell project.
  And then run clash with a module name (instead of a source file), it'll read in the modules precompiled by GHC and do its thing to change that into HDL.
  When doing this you'd have to be careful that clash doesn't see you source files, so it won't know if you changed them, and you have to make sure to ask cabal/stack recompile the sources, so clash can see the new changes you made.

---

The way this all works with the current starter project is it chooses options 2b and 4b.
And it has the bogus copies of the clash binaries to automatically solve the problem described in 4b. Because we define the local package as a dependency of the local copies of the clash binaries, that causes the local project to automatically get (re)compiled when attempting to run the clash binaries.

And the passing of the library environment to clash is slightly different depending on whether you're using stack or cabal:

Cabal

We use a cabal.project file to force the setting write-ghc-environment-files: always.
This causes cabal to generate a file called .ghc.environment.arch-os-version.
This file is loaded automatically by ghc/clash.

Stack

Via some combination of stack.yml and my-project-name.cabal stack determines what libraries to build and use and packs that information into the GHC_PACKAGE_PATH environment variable.