How to remove Attach <Model> button

[GlennBags]

New to Nova and taking over project from another dev.
I have Users and Organizations in a many to many.
Users can be attached to many organizations.
When I click to view a User, I get the Attach Organization button.
I want to be able to restrict access to be able to Attach based on a user's role.
In looking at the documentation, there is the attach function.
In my UserPolicy I defined attachOrganization() with a return of FALSE.
I still see the Attach Organization button.
When I click through, the drop down for Choose Organization doesn't show any organizations, so I suppose you could say it's restricting me from adding any, since I can't select any. However, if the user doesn't have access to attach, I would think use shouldn't see the Attach Organization in the first place.