NSTextView not showing red misspelling underlines when on a layer

Question

When an NSTextView is a subview of an NSView that is layer-backed (-wantsLayer == YES), it does not render the squiggly red underlines for misspelled words. All it takes to reproduce this is to make an empty Cocoa project, open the nib, drag NSTextView into the window, and toggle the window's content view to want a layer. Boom - no more red underlines.

I've done some searching, and this appears to be a known situation and has been true since 10.5. What I cannot find, though, is a workaround for it. Is there no way to get the underlines to render when NSTextView is in a layer-backed view?

I can imagine overriding NSTextView's drawRect: and using the layout manager to find the proper rects with the proper temporary attributes set that indicate misspellings and then drawing red squiggles myself, but that is of course a total hack. I also can imagine Apple fixing this in 10.7 (perhaps) and suddenly my app would have double underlines or something.

---

[update] My Workaround

My current workaround was inspired by nptacek's mentioned spell checking delegate method which prompted me to dig deeper down a path I didn't notice before, so I'm going to accept that answer but post what I've done for posterity and/or further discussion.

I am running 10.6.5. I have a subclass of NSTextView which is the document view of a custom subclass of NSClipView which in turn is a subview of my window's contentView which has layers turned on. In playing with this, I eventually had all customizations commented out and still the spelling checking was not working correctly.

I isolated what, I believe, are two distinct problems:

#1 is that NSTextView, when hosted in a layer-backed view, doesn't even bother to draw the misspelling underlines. (I gather based on Google searches that there may have been a time in the 10.5 days when it drew the underlines, but not in the correct spot - so Apple may have just disabled them entirely to avoid that problem in 10.6. I am not sure. There could also be some side effect of how I'm positioning things, etc. that caused them not to appear at all in my case. Presently unknown.)

#2 is that when NSTextView is in this layer-related situation, it appears to not correctly mark text as misspelled while you're typing it - even when -isContinuousSpellCheckingEnabled is set to YES. I verified this by implementing some of the spell checking delegate methods and watching as NSTextView sent messages about changes but never any notifying to set any text ranges as misspelled - even with obviously misspelled words that would show the red underline in TextEdit (and other text views in other apps). I also overrode NSTextView's -handleTextCheckingResults:forRange:types:options:orthography:wordCount: to see what it was seeing, and it saw the same thing there. It was as if NSTextView was actively setting the word under the cursor as not misspelled, and then when the user types a space or moves away from it or whatever, it didn't re-check for misspellings. I'm not entirely sure, though.

Okay, so to work around #1, I overrode -drawRect: in my custom NSTextView subclass to look like this:

- (void)drawRect:(NSRect)rect
{
    [super drawRect:rect];
    [self drawFakeSpellingUnderlinesInRect:rect];
}

I then implemented -drawFakeSpellingUnderlinesInRect: to use the layoutManager to get the text ranges that contain the NSSpellingStateAttributeName as a temporary attribute and render a dot pattern reasonably close to the standard OSX misspelling dot pattern.

- (void)drawFakeSpellingUnderlinesInRect:(NSRect)rect
{
    CGFloat lineDash[2] = {0.75, 3.25};

    NSBezierPath *underlinePath = [NSBezierPath bezierPath];
    [underlinePath setLineDash:lineDash count:2 phase:0];
    [underlinePath setLineWidth:2];
    [underlinePath setLineCapStyle:NSRoundLineCapStyle];

    NSLayoutManager *layout = [self layoutManager];
    NSRange checkRange = NSMakeRange(0,[[self string] length]);

    while (checkRange.length > 0) {
        NSRange effectiveRange = NSMakeRange(checkRange.location,0);
        id spellingValue = [layout temporaryAttribute:NSSpellingStateAttributeName atCharacterIndex:checkRange.location longestEffectiveRange:&effectiveRange inRange:checkRange];

        if (spellingValue) {
            const NSInteger spellingFlag = [spellingValue intValue];

            if ((spellingFlag & NSSpellingStateSpellingFlag) == NSSpellingStateSpellingFlag) {
                NSUInteger count = 0;
                const NSRectArray rects = [layout rectArrayForCharacterRange:effectiveRange withinSelectedCharacterRange:NSMakeRange(NSNotFound,0) inTextContainer:[self textContainer] rectCount:&count];

                for (NSUInteger i=0; i<count; i++) {
                    if (NSIntersectsRect(rects[i], rect)) {
                        [underlinePath moveToPoint:NSMakePoint(rects[i].origin.x, rects[i].origin.y+rects[i].size.height-1.5)];
                        [underlinePath relativeLineToPoint:NSMakePoint(rects[i].size.width,0)];
                    }
                }
            }
        }

        checkRange.location = NSMaxRange(effectiveRange);
        checkRange.length = [[self string] length] - checkRange.location;
    }

    [[NSColor redColor] setStroke];
    [underlinePath stroke];
}

So after doing this, I can see red underlines but it doesn't seem to update the spelling state as I type. To work around that problem, I implemented the following evil hacks in my NSTextView subclass:

- (void)setNeedsFakeSpellCheck
{
    if ([self isContinuousSpellCheckingEnabled]) {
        [NSObject cancelPreviousPerformRequestsWithTarget:self selector:@selector(forcedSpellCheck) object:nil];
        [self performSelector:@selector(forcedSpellCheck) withObject:nil afterDelay:0.5];
    }
}

- (void)didChangeText
{
    [super didChangeText];
    [self setNeedsFakeSpellCheck];
}

- (void)updateInsertionPointStateAndRestartTimer:(BOOL)flag
{
    [super updateInsertionPointStateAndRestartTimer:flag];
    [self setNeedsFakeSpellCheck];
}

- (void)forcedSpellCheck
{
    [self checkTextInRange:NSMakeRange(0,[[self string] length]) types:[self enabledTextCheckingTypes] options:nil];
}

It doesn't work quite the same way as the real, expected OSX behavior, but it's sorta close and it gets the job done for now. Hopefully this is helpful for someone else, or, better yet, someone comes here and tells me I was missing something incredibly simple and explains how to fix it. :)

Answer 1

Core Animation is awesome, except when it comes to text. I experienced this firsthand when I found out that subpixel antialiasing was not a given when working with layer-backed views (which you can technically get around by setting an opaque backgroundColor and making sure to draw the background). Subpixel anti-aliasing is just one of the many caveats encountered while working with text and layer-backed views.

In this case, you've got a couple of options. If at all possible, move away from layer-backed views for the parts of your program that utilize the text views. If you've already tried this, and can't avoid it, there is still hope!

Without going so far as overriding drawRect, you can achieve something that is close to the standard behavior with the following code:

- (NSArray *)textView:(NSTextView *)view didCheckTextInRange:(NSRange)range types:(NSTextCheckingTypes)checkingTypes options:(NSDictionary *)options results:(NSArray *)results orthography:(NSOrthography *)orthography wordCount:(NSInteger)wordCount
{
     for(NSTextCheckingResult *myResult in results){
        if(myResult.resultType == NSTextCheckingTypeSpelling){
            NSMutableDictionary *attr = [[NSMutableDictionary alloc] init];
            [attr setObject:[NSColor redColor] forKey:NSUnderlineColorAttributeName];
            [attr setObject:[NSNumber numberWithInt:(NSUnderlinePatternDot | NSUnderlineStyleThick | NSUnderlineByWordMask)] forKey:NSUnderlineStyleAttributeName];
            [[inTextView layoutManager] setTemporaryAttributes:attr forCharacterRange:myResult.range];
            [attr release];
        }
    }
    return results;
}

We're basically doing a quick-and-dirty delegate method for NSTextView (make sure to set the delegate in IB!) which checks to see if a word is flagged as incorrect, and if so, sets a colored underline.

Note that there are some issues with this code -- Namely that characters with descenders (g, j, p, q, y, for example) won't display the underline correctly, and it's only been tested for spelling errors (no grammar checking here!). The underline dot pattern (NSUnderlinePatternDot) does not match Apple's style for spellchecking, and the code is still enabled even when layer backing is disabled for the view. Additionally, I'm sure there are other problems, as this code is quick and dirty, and hasn't been checked for memory management or anything else.

Good luck with your endeavor, file bug reports with Apple, and hopefully this will someday be a thing of the past!

Answer 2

This is also a bit of a hack, but the only thing I could get working was to put an intermediate delegate on the NSTextView's layer, so that all selectors are passed through, but drawLayer:inContext: then calls the NSTextView's drawRect:. This works, and is probably a little more future proof, although I'm not sure if it will break any CALayer animations. It also seems you have to fix the CGContextRef's CTM (based on the backing layer frame?).

Edit: You can get the drawing rect as in the drawInContext: documentation, with CGContextGetClipBoundingBox(ctx), but there might be an issue with flipped coordinates in the NSTextView.

I'm not entirely sure how to fix this as calling drawRect: as I did is a bit hackish, but I'm sure someone on the net has a tutorial on doing it. Perhaps I can make one if/when I have time and work it out.

It might be worthwhile looking for an NSCell backing the NSTextView, as it's probably a lot more appropriate to use this instead.