Add functionality to Xcode template to bundle dylibs

[artificiel]

Problem: moving an app between machines requires that (1) all required dylibs to be bundled within the .app and (2) their paths to be tweaked in order to be found at runtime (including paths in dylibs-depending-on-other-dylibs). It's a hairy process to do manually, and also hairy to automate in a manner that survives a PG project update (as all tweaks to the project are lost).

This leverages https://github.com/auriamg/macdylibbundler (commonly available as brew install dylibbundler). The script first scans the Xcode project for all referenced dylibs that would be provided by addons and passes them to dylibbundler, who will copy and work recursively from these dylibs, automating the install_name_tool dance (and also signs the dylibs). A Project.xcconfig flag enables the script, which is disabled by default.

So as an end user: make a new project in PG; choose an addon that provides macOS .dylibs; activate the flag; build the project; and the dylibs are copied and tweaked within the .app bundle, that you can then copy on another machine. (of how far that translates depend on many factors (are the dylibs ARM+intel, specific project compile flags, OS version, etc).

As an addon writer: collect the required .dylibs inside your addon libs/ directory and doctor them once to play well together (the case of custom dylibs relying on custom dylibs will be correctly tracked by macdylibbundler, however some install_name_tool'ing of the "leaf" dylibs might be required to handle both bundled and not bundled).

more at #7277

NOTE: the problem this solves could have been addressed differently by augmenting the PG/addon with a post-processing step. I recently came about this PR openframeworks/projectGenerator#217 that could solve the problem by enabling the execution of a script that would "inject" another script step into the pbxproj. however it would rely on some less-common software to manipulate the pbxproj. It would also make it brittle against future template modifications. Likewise, a custom template could be made ("Xcode-dylibbundle") but would require future double-management of the Xcode template.

Having a disabled script tucked within the template is the best compromise. Tested between M1 and M2 machines with macOS12 and 13.

#changelog #osx

[dimitre]

+1 for this. I think it is good to merge!

One suggestion (I can make another PR after this one)
We can group some of the scripts into one, like having one with all functionality of

• Copy Resources
• Bundle Data Folder
• Code Sign
• Bundle Dylibs

This way XCode write less different files and scripts output

[artificiel]

@dimitre the drawback I see with your suggestion of combining script steps is that the Xcode script editor is annoying to work with, so longer scripts translates to more effort in scrolling/locating things... with separate steps the enabling flag conditional is right there at the top for all of them; makes it easier to figure out than a switch 25 lines down?

[dimitre]

@artificiel yeah I agree. in any case to read / edit I always copy to an editor.
For me it is the same. a litte tiny bit of time to build or a little time to read the code.

for the average OF user only uncommenting defines in Project.xcconfig will be all that is. if they need some of the flags.

[artificiel]

or: the Xcode script step is a single-liner that calls another script which is held in it's own .sh file? it augments readability and simplifies maintenance (fiddling with the json pbxproj is not exactly enjoyable...)

[dimitre]

It is the best! we can use the variable $OF_PATH to point to some path in OF project.
I've made a test here and it is working.
so we can have a nicely formatted script with separators on different actions.

Edit: Another advantage of a fixed script, is if we update and fix scripts in one place in OF, all projects are up to date automatically, no need to regenerate it using PG.

[dimitre]

@ofTheo @danoli3 any opinion on this?
I really like the idea of making a separate .sh file with all steps except OF building.

[ofTheo]

Hmm I think a preferable solution would be one that didn't rely on installing brew and a 3rd party tool.
I think this could all be doable without dylibbundler if my understanding of install_name_tool and rpaths is correct.

• Run install_name_tool for all dylibs that are bundled in the app and set the rpath to @executable_path ( or up a level if needed )

I have done this manually before and I think it could be fairly easy to automate. But maybe I am missing something here?

Another script build phase which has been on the TODO list is App Notarization which with a few defines in the xcconfig could be automated too.

Edit: this would also require that any dylibs that are linked to the project are copied to the Frameworks folder ( if that isn't happening currently it should ) so that should allow things to survive PG rebuilds.

[artificiel]

@ofTheo it is certainly possible to reproduce the functionality but macdylibbundler is ~1000 lines of C++ so I don't know if it can be said to be "fairly easy to automate".

I initially started like everyone else trying to script it within Xcode (like this which is the basic starting point https://stackoverflow.com/questions/42022884/making-xcode-embed-necessary-dylibs), then after hitting many little friction zones looked at various more "complete" bash and python scripts (as documented in #7277) but they did not cover all aspects of my requirements, notably being recursive (my use case is a library that relies on another library). macdylibbundler worked correctly and completely (while being malleable with flags to cover different scenarios).

After spending a bunch of semi~enjoyable hours with otool & company and (reading other's purpose-specific scripts) I don't see the value of re-implementing what is existing.

I can re-state that the approach of having a post-process step in the addon_config (as implemented here openframeworks/projectGenerator#217) is a more generic solution that I might have harnessed without touching the Xcode template as the equivalent script and instructions would be provided within the addon and injected by PG -- but that opens another can of worms as Apple does not provide an API for manipulating the .pbxproj, so potential fragility/maintenance.

Also re-state the goal: making use of an addon that provides .dylibs and produce a self-contained .app in a generic workflow (no pre-determined paths or dylib names) that survives PG. It's a bit specialized and currently not possible in a fluid manner, and at that point I find the requirement of brew a minor hurdle, considering the audience it is addressed to.

From your comment I understand it's also a pre-requisite for notarization; I haven't been looking at notarization as it's out of the current project scope (can take a look a bit later) but yes things are copied (from #7277) :

(Note that the config signs and puts the .dylibs by default in a ./Content/libs/ directory (and not ./Content/Frameworks/) -- that's changeable, but apparently better not to clutter the Frameworks/ with non-frameworks)

[dimitre]

@ofTheo what about starting with a solution which uses homebrew?
As it is an optional process, few people will use and it is better than not having it.
Later it can be ported to one with no dependencies if somebody has the time and intent to do this effort.

[ofTheo]

@dimitre
Yeah I was sort of tempted to do that too. 🙂
I think this could be an intemediatary step with the goal of doing a much better job packing dylibs / frameworks etc

Will have another look over the changes, but I think that could be a good approach.
I have def dealt with install_name_tool hell in the past and can't imagine it being fun for that many dylibs @artificiel

[artificiel]

well dylibbundler handles it all automatically! you just need to point it to your .app and a starting point (which are found by looking for *.dylibs in LD_FLAGS), then it hunts the dependencies, intelligently copies things (deduplicating if needed, and excludes system libs), and rewrites the paths (all that takes 1-2 seconds). some of them surprised me... and i was also surprised that it «just worked» when deploying on the other machines...

[dimitre]

I think a really great test case would be using ofxHapPlayer in other machine.
it uses a lot of dylibs, we used to install in macOS through a script in other machines but nowadays macOS forbids that. or erase the libs after reboot.
So packing those dylibs and signing would be great

[artificiel]

heheh well grab the PR, brew install dylibbundler, uncomment the flag in .xcconfig and compile!

curious to see it in action in another env.

[artificiel]

so had a couple minutes to try this: downloaded ofxHapPlayer, built the example on a M1 macOS13 (all architectures) and it runs copied straight on an i7 macOS12.

[dimitre]

I can work in joining scripts after this one is merged