The "View for all" permission makes all schemas by default visible to all users. You only have to configure the restrictions - so the schemas which you don’t want all users to see. This simplifies the permission management massively.
Relationships and Fieldsets have the setting "View for all" applied by default, to ensure that users can see relationship information of a linked Content Item. This cannot be changed and the button is grayed out, but you can restrict access to the main menu fieldset and relations. You must protect fields of type fieldsets and relationships by restricting access to the Layers using those fields.