Fix writing vendor/modules.txt to deal with subpackages

[owen-mc]

The file format isn't clearly documented, but this example from the go project shows that what we were producing before wasn't correct.

Note that a lot of this code was written by copilot, so should perhaps merit greater scrutiny.

[Copilot]

Pull Request Overview

This PR fixes the generation of vendor/modules.txt by updating the code to deal with subpackages while replacing deprecated ioutil functions with equivalent os functions.

• Replaces ioutil functions with os functions across multiple files.
• Introduces helper functions to scan Go source files and identify used submodules.
• Updates import statements to remove unused packages and add necessary ones (e.g. go/parser, regexp).

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
vendor.go	Implements submodule extraction and writes explicit module/submodule entries.
reflect.go	Replaces temporary file and directory creation methods using ioutil with os.
util.go	Uses os.ReadDir instead of ioutil.ReadDir for improved file reading handling.
depstubber.go	Updates file reading to use os.ReadFile, removing the dependency on ioutil.

Tip: If you use Visual Studio Code, you can request a review from Copilot before you push from the "Source Control" tab. Learn more

[owen-mc]

I don't think depstubber has any tests. I've tried this on one folder in github/codeql and it fixed the problem I was having. Before merging I should try it in more (all, if possible) to check that it doesn't cause any regressions.

[smowton]

Looks plausible

[owen-mc]

I reran all the depstubber commands stored as go-generate comments in codeql/go. This PR contains the results. After reverting changes which overrode manual edits all the tests passed.

[owen-mc]

One case where this new logic would fall over: nested go modules. If you have a go.mod in a child dir of a go.mod then it makes a new module. For a project which imports packages from both modules, we would then attribute a package of the inner module to both modules. However, I'm not really sure why you'd have nested go modules., and I have never seen this done in a library. And it should be relatively easy to fix manually. I propose that we don't try to fix depstubber for this case until we actually see it happen at least once in the wild.

[owen-mc]

I added a commit to sort the packages. This matches what go mod vendor does, making it easier to compare. And it also just makes it easier to read. The effect on our stubs for codeql/go can be seen in this commit.