-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: Embed import path to binary to enable rebuilding #16814
Comments
I just discovered the tool gorebuild which extracts import paths from the binary. However it seems to derive it from a filesystem path since it's printing absolute paths for me. The independent existence of that tool is another reason why storing this would be useful and necessary. Maybe instead of a manifest file this information should in fact be embedded in the binary itself? That would have the additional benefit of carrying to another system than where the binary was built. If already filesystem paths are stored, import path should be even less sensitive to add. Not sure if there exists a cross-platform place to embed such meta-information into binary and retrieve it. |
+1, I think this information should be embedded directly in the binary and object files. |
I'm going to mark this as Go1.8 as I think it is a prereq for having build caching (since we need to know what went into a file to know if it needs to be rebuilt). |
What I would suggest is to include in binaries a kind of a "birth certificate" / "bill of materials" describing where it came from (import path, vcs version?) and what it consists of (packages, vcs version?). Here's an example of a very similar idea in @hlandau's acmetool:
|
The go tool already embeds a build id into the binary so that it
can track when updating the binary is required.
I don't understand why we must bake the entire list of import
paths and revisions into the binary.
|
It would be helpful (to me, at least) if those who are in favor of this proposal would elaborate on what problem they're trying to solve with this information. |
@minux Initially my need was to find out where the binaries in my bin/ came from (import path). This is useful for many things, including updating installed binaries (think Hopefully the first part is uncontroversial and you can agree that at least the path that could be used with go get should be reliably available to enable a better gorebuild, regardless whether the binary came from a binary distribution, scp'ed from another machine or go get. My argument for the imported packages would be about transparency and auditability, secure development practices. For example it could be used to detect whether the binary is affected by a bug fix or vulnerability, and as a way of e.g. re-creating the build artifact (e.g. validate that a build is not compromised). It would be great if it was possible to independently re-create a binary build, all the way to matching checksum. Maybe this would come with an opt-out. |
I see you've provided more motivation for including actual import path in the binary in FiloSottile/gorebuild#7. I do think the import path of the command could be nice to always include, but I'm not yet sure if going beyond that is worthwhile. |
If the vcs information is available in go build, would it be ok to include
at least commit hash of the package with main? I think at least go get
knows vcs stuff, right?
Then the rest would be a problem solved with vendoring manifests like gvt,
outside of go build.
If e.g. the repo has a gvt manifest of imports, then commit hash would be
sufficient to enable what I suggested for projects that choose to do so.
|
does gcc embed the file path into the binary (except for the debugging
info)?
Why should go tool does that?
To put it another way, if I compile random C programs and put them into my
$PATH, and then forget where do they come from, I won't ask gcc to embed
such paths into the binary to make my live easier.
Managing binary assets is a much larger problem and I don't think the go
tool should solve it. For example, I'd argue that having paths and revision
is not enough, because some of the packages might have custom patches
applied. How could you represent that?
|
I don't think a mechanism would be foolproof and as such the rebuild Including the path just makes it reliable to parse this, rather than I don't believe go tools should manage binaries but in this case enabling I'm now convinced that for the used packages it's better to have external I still want the path needed for go get, and unless there are strong
|
The required file path information is recorded in the DWARF debugging info,
and that's what's in the strings(1) output.
Using debug/dwarf you can easily get such information. And by looking at
the file that contains main.main, you can know the path of the main package.
|
That's what gorebuild (and But it doesn't work reliably if the GOPATH is not set (or is different) when trying to reconstruct the import path. Imagine if main.main is located at path |
The binary itself should be in $GOPATH/bin, and that
gives enough information for the $GOPATH.
|
/cc @rsc |
This issue is asking for a particular implementation, not new functionality. Like @shurcooL said, this needs to be in service of a concrete larger goal in order to be evaluated. I'm not opposed to putting the import path into the binary next to the build ID, as long as there's some limit to how long it is (build IDs are designed to be limited size). We certainly don't want to try to record the specific version of every file. None of this is necessary for proper build caching. The build IDs in the binary today already suffice. In any event, this is vague enough that we can put any decisions off to the Go 1.9 cycle. |
What kind of a limit would you have in mind? Would it be something large enough to cover vast majority of realistic import path lengths (like 1024 bytes)? Or something else? What happens if an import path exceeds that length, doesn't that defeat the purpose of this field, since it would no longer accurately answer the question of "what is the import path" in some cases? Either that, or it puts a hard limitation on the length of an import path (something that was previously not there). Is the limit needed to prevent someone from modifying Go binaries in a malicious way, by placing an unbounded number of bytes into the import path field? |
To help this get evaluated, I can share what goals I have that I think this would be helpful for. I currently want it for 2 use cases:
|
Perhaps this could be included in the go tool. Something like |
vgo does this already. When it moves into the go command, the go command will too (for module-based builds). Closing this issue - we're not going to make a special case for non-module-based builds at this point. |
For posterity, this has happened by now. This functionality is available in Go 1.12 under the Function |
Edit: After more thought and discussion embedding the import path in binary seems like the way to go. This would enable upgrading an installed binary application in many cases.
Original submission below:
I'd like to have go get and go install produce a manifest along with the binary that indicates when the binary was compiled, what package produced it.
This is useful to understand why I have binaries in my bin/ and would allow e.g. updating them in bulk later through some mechanism (TBD outside of this issue).
Why this should be enabled by default? Many existing tools (e.g. editors) have a mechanism to install tools (e.g. go guru). To have a complete manifest, including already shipping editors and external applications, this should be updated by default (if someone insists, with opt-out flag).
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version
)?What operating system and processor architecture are you using (
go env
)?Uh, isn't this redundant to 1.?
What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
What did you expect to see?
A manifest file listing where the binary came from being generated along with the binary
E.g.
This could be a simple text file in $GOPATH/bin/.manifest or something more fancy. KISS would support a simple log format. The exact fields open to discussion, but I feel that timestamp, package and hash of the binary generated are minimal; possibly VCS version could be nice.
What did you see instead?
No manifest file generated.
The text was updated successfully, but these errors were encountered: