UINavigationController method setToolbarHidden bug in Xcode 9: infinite calculation of auto-layout constraints leads to OOM

Question

I have an instance of UINavigationController nested in UITabBarController. I use the navigation controller to reach some view controller (tab bar still visible), from which I segue to a second view controller (tab bar no longer visible).

In the second view controller, as soon as I make a call to: [self.navigationController setToolbarHidden:NO] the app freezes and memory grows until OOM exception crashes it.

I acknowledge that nesting the navigation controller inside the tab bar is not recommended, but this setup seemed to work fine until iOS 11.

EDIT: when stopping execution, I see a lot of calls to:

UIView(UIConstraintBasedLayout)

UIView(AdditionalLayerSupport)

NSLayoutConstraint

Here's the full stack trace

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP   * frame #0: 0x0000000106dd895c libobjc.A.dylib`objc_msgSend
+ 28
    frame #1: 0x00000001067b6b9b Foundation`-[NSConcreteMapTable removeObjectForKey:] + 138
    frame #2: 0x00000001069e6019 Foundation`_substituteOutAllOccurencesOfBodyVar + 1282
    frame #3: 0x00000001067f3c5b Foundation`-[NSISEngine tryAddingDirectly:] + 144
    frame #4: 0x00000001067f332f Foundation`-[NSISEngine tryToAddConstraintWithMarker:expression:integralizationAdjustment:mutuallyExclusiveConstraints:]
+ 440
    frame #5: 0x00000001069f2067 Foundation`-[NSLayoutConstraint _addLoweredExpression:toEngine:integralizationAdjustment:lastLoweredConstantWasRounded:mutuallyExclusiveConstraints:]
+ 273
    frame #6: 0x00000001067ea601 Foundation`-[NSLayoutConstraint _addToEngine:integralizationAdjustment:mutuallyExclusiveConstraints:] + 240
    frame #7: 0x0000000109c9488d UIKit`__57-[UIView(AdditionalLayoutSupport)
_switchToLayoutEngine:]_block_invoke_2 + 452
    frame #8: 0x00000001067f0de1 Foundation`-[NSISEngine withBehaviors:performModifications:] + 131
    frame #9: 0x0000000109c946a2 UIKit`__57-[UIView(AdditionalLayoutSupport)
_switchToLayoutEngine:]_block_invoke + 604
    frame #10: 0x0000000109c9441e UIKit`-[UIView(AdditionalLayoutSupport) _switchToLayoutEngine:] + 223
    frame #11: 0x00000001091ed84f UIKit`__45-[UIView(Hierarchy) _postMovedFromSuperview:]_block_invoke + 112
    frame #12: 0x00000001067f0de1 Foundation`-[NSISEngine withBehaviors:performModifications:] + 131
    frame #13: 0x00000001091ed778 UIKit`-[UIView(Hierarchy) _postMovedFromSuperview:] + 855
    frame #14: 0x00000001091fe031 UIKit`-[UIView(Internal) _addSubview:positioned:relativeTo:] + 1927
    frame #15: 0x0000000109b507e1 UIKit`-[_UILayoutArrangement insertItem:atIndex:] + 502
    frame #16: 0x0000000109ca1b4d UIKit`__50-[_UIOrderedLayoutArrangement insertItem:atIndex:]_block_invoke + 50
    frame #17: 0x0000000109ca18df UIKit`-[_UIOrderedLayoutArrangement _trackChangesAffectingExternalBaselineConstraints:] + 320
    frame #18: 0x0000000109ca1aea UIKit`-[_UIOrderedLayoutArrangement insertItem:atIndex:] + 478
    frame #19: 0x000000010982edea UIKit`-[UIStackView insertArrangedSubview:atIndex:] + 283
    frame #20: 0x0000000109b29972 UIKit`-[_UIButtonBar _layoutBar] + 3639
    frame #21: 0x0000000109b2bb44 UIKit`-[_UIButtonBarStackView updateConstraints] + 48
    frame #22: 0x0000000109c958b6 UIKit`-[UIView(AdditionalLayoutSupport)
_sendUpdateConstraintsIfNecessaryForSecondPass:] + 161
    frame #23: 0x0000000109c95ed2 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 1296
    frame #24: 0x0000000109c95d51 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 911
    frame #25: 0x0000000109c95d51 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 911
    frame #26: 0x0000000109c95d51 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 911
    frame #27: 0x00000001067f0de1 Foundation`-[NSISEngine withBehaviors:performModifications:] + 131
    frame #28: 0x0000000109c96703 UIKit`__100-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededWithViewForVariableChangeNotifications:]_block_invoke
+ 90
    frame #29: 0x0000000109c94f61 UIKit`-[UIView(AdditionalLayoutSupport)
_withUnsatisfiableConstraintsLoggingSuspendedIfEngineDelegateExists:] + 104
    frame #30: 0x0000000109c96272 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededWithViewForVariableChangeNotifications:] + 160
    frame #31: 0x0000000109c9738c UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsAtEngineLevelIfNeededWithViewForVariableChangeNotifications:]
+ 401
    frame #32: 0x00000001091ef1b6 UIKit`-[UIView(Hierarchy) layoutBelowIfNeeded] + 1517
    frame #33: 0x000000010957b35e UIKit`-[_UIButtonBarButton willMoveToWindow:] + 63
    frame #34: 0x00000001091ec996 UIKit`-[UIView(Hierarchy) _willMoveToWindow:] + 861
    frame #35: 0x00000001091eb493 UIKit`__UIViewWillBeRemovedFromSuperview + 484
    frame #36: 0x00000001091eb0ea UIKit`-[UIView(Hierarchy) removeFromSuperview] + 95
    frame #37: 0x0000000109b295d3 UIKit`-[_UIButtonBar _layoutBar] + 2712
    frame #38: 0x0000000109b2bb44 UIKit`-[_UIButtonBarStackView updateConstraints] + 48
    frame #39: 0x0000000109c958b6 UIKit`-[UIView(AdditionalLayoutSupport)
_sendUpdateConstraintsIfNecessaryForSecondPass:] + 161
    frame #40: 0x0000000109c95ed2 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 1296
    frame #41: 0x0000000109c95d51 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 911
    frame #42: 0x0000000109c95d51 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 911
    frame #43: 0x0000000109c95d51 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededCollectingViews:forSecondPass:] + 911
    frame #44: 0x00000001067f0de1 Foundation`-[NSISEngine withBehaviors:performModifications:] + 131
    frame #45: 0x0000000109c96703 UIKit`__100-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededWithViewForVariableChangeNotifications:]_block_invoke
+ 90
    frame #46: 0x0000000109c94f61 UIKit`-[UIView(AdditionalLayoutSupport)
_withUnsatisfiableConstraintsLoggingSuspendedIfEngineDelegateExists:] + 104
    frame #47: 0x0000000109c96272 UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsIfNeededWithViewForVariableChangeNotifications:] + 160
    frame #48: 0x0000000109c9738c UIKit`-[UIView(AdditionalLayoutSupport)
_updateConstraintsAtEngineLevelIfNeededWithViewForVariableChangeNotifications:]
+ 401
    frame #49: 0x00000001091efa5b UIKit`-[UIView(Hierarchy) _updateConstraintsAsNecessaryAndApplyLayoutFromEngine] + 159
    frame #50: 0x00000001095742d5 UIKit`-[UILayoutContainerView layoutSubviews] + 270
    frame #51: 0x0000000109204551 UIKit`-[UIView(CALayerDelegate) layoutSublayersOfLayer:] + 1331
    frame #52: 0x00000001064db4ba QuartzCore`-[CALayer layoutSublayers] + 153
    frame #53: 0x00000001064df5a9 QuartzCore`CA::Layer::layout_if_needed(CA::Transaction*) + 401
    frame #54: 0x00000001064681cd QuartzCore`CA::Context::commit_transaction(CA::Transaction*) + 365
    frame #55: 0x0000000106493ae4 QuartzCore`CA::Transaction::commit() + 500
    frame #56: 0x0000000109160687 UIKit`_afterCACommitHandler + 272
    frame #57: 0x00000001080f8db7 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__
+ 23
    frame #58: 0x00000001080f8d0e CoreFoundation`__CFRunLoopDoObservers + 430
    frame #59: 0x00000001080dd324 CoreFoundation`__CFRunLoopRun + 1572
    frame #60: 0x00000001080dca89 CoreFoundation`CFRunLoopRunSpecific + 409
    frame #61: 0x000000010dc429c6 GraphicsServices`GSEventRunModal + 62
    frame #62: 0x0000000109135d30 UIKit`UIApplicationMain + 159
    frame #63: 0x0000000101ff6bf9 MyAppName`main(argc=1, argv=0x00007fff5de3e0a8) at main.m:23
    frame #64: 0x000000010f453d81 libdyld.dylib`start + 1
    frame #65: 0x000000010f453d81 libdyld.dylib`start + 1

Answer 1

This is the answer from a developer from the same team as OP is from.

After one more round of research we have found that the problem was on our side:

- (NSArray *)toolbarItems
{
  return [self toolbarItemsWithRunningAdditionalAnimation:NO];
}

Original developer made the method toolbarItemsWithRunningAdditionalAnimation to return new objects each time the method was called. When going to this controller iOS calls toolbarItems at least 3 times so giving it new objects every time we made iOS confused so it was recalculating auto-layout constraints in an infinite loop. Our original fix below is also working, however it becomes obsolete as we start always returning the same array of items like:

- (NSArray *)toolbarItems {
  if (_cachedToolbarItems) { return _cachedToolbarItems; }

  _cachedToolbarItems = [self toolbarItemsWithRunningAdditionalAnimation:NO];

  return _cachedToolbarItems'
}

---

We are doing the following in our app (pseudocode):

UIToolbar *toolbar = self.navigationController.toolbar;

NSArray <UIBarButtonItem *> *items = @[
  flexibleSpace, 
  share, 
  flexibleSpace, 
  play, 
  flexibleSpace, 
  stats, 
  flexibleSpace
];
[toolbar setItems:items animated:animated];

[self.navigationController setToolbarHidden:NO animated:animated];

The problematic items that caused infinite calculation of auto-layout constraints were share and stats. The thing they both had in common - they both were created using -[UIBarButtonItem initWithImage:style:target:action:] initializer. When we started using another initializer:

 UIButton *customShareButton = ... // we create button ourselves.
 UIBarButtonItem *shareItem = [[UIBarButtonItem alloc] initWithCustomView: customShareButton];

The problem went away.

Answer 2

Besides the accepted answer, we found that our problem was related to the fact that in our class, we override the method - (NSArray *)toolbarItems.

Every time our method is called, a new set of toolbarItems is created. Seems like from iOS 11, every time new toolbar items are returned to this method, UINavigationController:layoutIfNeeded is called again, which in turn calls toolbarItems which returns new items because of our implementation. This causes an infinite loop.

If you encounter this problem, check if you are overriding - (NSArray *)toolbarItems