Implementing Rich Text with Images on OS X and iOS

Our goal when we started was to create an application that would allow users to enter data on any device, a Mac, an iPad or an iPhone.  Working on large complex change programs usually required capturing complex hierarchies of information, including organisation structure, process models, application and system models and work breakdown structures.

In many work situations we need a tool that would allow easy capture of information, including images during workshops or information discussions.  In some cases we wanted to be able to prepare notes and images to be used for the workshops and if necessary zoom in and pan around detailed images embedded in the notes.

So basically we were looking for an app that combined elements of a word processor, a presentation tool, a spreadsheet, a drawing tool and a database all into one application !

It did not have to be the worlds best word processor, nor did it have to be the worlds best spreadsheet but by combining elements from all of the traditional productivity tools we felt we could offer a unique productivity tool that provides users with better navigation and organisation ability than is available any other way.

One of the key challenges was creating a text editor that provided good support for creating formatted text and for embedding images.  OS X already offered great basic capabilities with the NSTextView class and finally with iOS 7 UITextView offered virtually the same capabilities.  So we set about enhancing these capabilities to provide the following:

Paragraph Style Sheets, similar to the way MS Word provides styles that can be applied to paragraphs.  By selecting a style the format of the current paragraph can be instantly changed by the user.  We wanted to support at least the following styles:

Heading 1

Heading 2

Heading 3

Normal text

  • Bullet 1
    • Bullet 2

Embedded Images, the ability to copy and paste images in the text, scale the images to correctly fit the view without loosing the original resolution of the image and by selecting the image view the image in an image viewer that allows zooming in and panning the image.

TV001

iPad TextView

TV002

iPhone TextView

TV003

Formatting Toolbar, an easy to use formatting toolbar and keyboard shortcuts for quickly applying formatting to text.

TV004

TV005

TV006

The ability to open the image in an image viewer and zoom and pan the image or copy the image in its original resolution

TV007

TV008

TV009

OK enough on requirements!

Implementation

Nothing fancy, we did not want to end up building a word processor or graphics manipulation program, so the aim was to just use standard classes and APIs.

The core of the ability to display rich text with embedded images comes for  the NSAttributedString class.  Images are embedded using the NSTextAttachment class and not all image formats are supported on iOS, most notably PDF attachments are not supported so it’s not possible to display PDF images using standard classes on iOS.

In addition the OS X NSTextAttachment class is different to the iOS NSTextAttachment class.

So first things first, let’s get the text formatting out of the way.

Paragraph Styles
To apply text styles to paragraphs we need to do the following:

  • Create a paragraph style comprising attributes specific to the paragraph, such as left right and first line indent, left, right or centered alignment, line and paragraph spacing, and attributes specific to the text, such as font, font size, bold, italic or underlined.
  • Apply this style to the paragraph or paragraphs selected by the user, and in the case of bulleted styles check for the presence of a bullet and tab character and if not present add to the beginning of each paragraph.
  • If characters were added then adjust the text selection by the number of added characters so that the selection beginning and end points don’t change relative to the characters they are next to.
  • Finally set the typing attributes for the UITextView so that the next characters typed use the same attributes of the current paragraph style.

All this is done in the UITextView and NSTextView subclasses.  We have to implement separate classes for iOS and OS X because there are some small code differences required for each platform. Here are some of the basics.
To set up the Style toolbar we use this code to create the UISegmentedControl which uses different labels in order to fit the screen depending on whether the device is an iPhone or an iPad.

    UISegmentedControl *_formatControl;
    if ([[UIDevice currentDevice] userInterfaceIdiom] == UIUserInterfaceIdiomPhone) {
        _formatControl = [[UISegmentedControl alloc] initWithItems:@[@"H1",@"H2",@"H3",@"N",@"B1",@"B2"]];
    } else  {
        _formatControl = [[UISegmentedControl alloc] initWithItems:@[@"Heading 1",@"Heading 2",@"Heading 3",@"Normal",@"Bullet 1",@"Bullet 2"]];
    }
    self.inputAccessoryView = _formatControl;
    [_formatControl addTarget:self action:@selector(formatText:) forControlEvents:UIControlEventValueChanged];

The formatText method gets called when the UISegmentedControl pressed.

- (void)formatText:(id)sender {
    UISegmentedControl *segmentedControl = (UISegmentedControl *)sender;
    switch ([segmentedControl selectedSegmentIndex]) {
        case 0: //Heading 1
            [self styleHeading1:sender];
            break;

        case 1: //Heading 2
            [self styleHeading2:sender];
            break;

        case 2: //Heading 3
            [self styleHeading3:sender];
            break;

        case 3: //Normal
            [self styleNormal:sender];
            break;

        case 4: //Bullet 1
            [self styleBullet1:sender];
            break;

        case 5: //Bullet 2
            [self styleBullet2:sender];
            break;

        default:
            break;
    }
}

And within each style method we use the following basic pattern. Note that I no longer use [textStorage setAttributes:range] because it causes image attachments to disappear, whereas [textStorage addAttributes:range:] seems to work correctly. I have filed a bug report with Apple on this.

- (IBAction) styleHeading1:(id)sender
{
    NSRange charRange = [self rangeForUserParagraphAttributeChange];
    if (charRange.location == NSNotFound) return;
    NSTextStorage *myTextStorage = [self textStorage];

    if ([self isEditable] && charRange.location != NSNotFound)
    {
        [myTextStorage addAttributes:[self heading1Style] range:charRange];
    }
    [self setTypingAttributes:[self heading1Style]];
}

To make sure we have correctly selected the beginning and end points of the paragraphs and not just the selected text we use the following method (note that NSTextView on OSX provides this method but UITextView does not so we create our own equivalent). Bear in mind the user may have the cursor positioned anywhere on a line when they apply a paragraph style and they would expect the entire paragraph to be affected. If you just want to format selected text then you need a different approach because some of the attributes we are setting will be applied to the paragraph (left indent, right indent, first line indent etc.).

- (NSRange)rangeForUserParagraphAttributeChange {
    NSRange paragraphRange = [self.textStorage.string paragraphRangeForRange: self.selectedRange];
    return paragraphRange;
}

So that’s pretty straight forward. When applying a Bullet style we also check for the presence of a leading bullet and tab character in each paragraph and insert them if they are not present. Also because we are inserting additional characters we need to reset the actual cursor or text selection position so that it looks to the user as if their selection has not changed. See the full source code for the details.

Creating the STYLE
To create the style we need to combine a number of different attributes into a dictionary

- (NSDictionary*)normalStyle
{
    return [self createStyle:[self getDefaultParagraphStyle] font:[self normalFont] fontColor:[UIColor blackColor] underlineStyle:NSUnderlineStyleNone];

}

We use a helper function to make this a bit easier

- (NSDictionary*)createStyle:(NSParagraphStyle*)paraStyle font:(UIFont*)font fontColor:(UIColor*)color underlineStyle:(int)underlineStyle
{
    NSMutableDictionary *style = [[NSMutableDictionary alloc] init];
    [style setValue:paraStyle forKey:NSParagraphStyleAttributeName];
    [style setValue:font forKey:NSFontAttributeName];
    [style setValue:color forKey:NSForegroundColorAttributeName];
    [style setValue:[NSNumber numberWithInt: underlineStyle] forKey:NSUnderlineStyleAttributeName];

    return style;
}

To get the basic paragraph settings you need something like this

- (NSMutableParagraphStyle*)getDefaultParagraphStyle
{
    NSMutableParagraphStyle *para;
    para = [[NSParagraphStyle defaultParagraphStyle]mutableCopy];
    [para setTabStops:nil];
    [para setAlignment:NSTextAlignmentLeft];
    [para setBaseWritingDirection:NSWritingDirectionLeftToRight];
    [para setDefaultTabInterval:[self ptsFromCMF:3.0]];
    [para setFirstLineHeadIndent:0];
    [para setHeadIndent:0.0];
    [para setHyphenationFactor:0.0];
    [para setLineBreakMode:NSLineBreakByWordWrapping];
    [para setLineHeightMultiple:1.0];
    [para setLineSpacing:0.0];
    [para setMaximumLineHeight:0];
    [para setMinimumLineHeight:0];
    [para setParagraphSpacing:6];
    [para setParagraphSpacingBefore:3];
    [para setTailIndent:0.0];
    return para;
}

And then for each variation start with the default paragraph style and just replace the elements that are different.

- (NSParagraphStyle*)getBullet1ParagraphStyle
{
    NSMutableParagraphStyle *para;
    para = [self getDefaultParagraphStyle];
    NSMutableArray *tabs = [[NSMutableArray alloc] init];
    [tabs addObject:[[NSTextTab alloc] initWithTextAlignment:NSTextAlignmentLeft location:[self ptsFromCMF:1.0] options:nil]];
    [para setTabStops:tabs];
    [para setDefaultTabInterval:[self ptsFromCMF:2.0]];
    [para setFirstLineHeadIndent:[self ptsFromCMF:0.0]];
    [para setHeadIndent:[self ptsFromCMF:1.0]];
    [para setParagraphSpacing:3];
    [para setParagraphSpacingBefore:3];
    return para;
}
- (NSMutableParagraphStyle*)getRightParagraphStyle
{
    NSMutableParagraphStyle *para = [self getDefaultParagraphStyle];
    [para setAlignment:NSTextAlignmentRight];
    return para;
}

I prefer working in centimetres because I find it easier to visualise the layout so use a couple of conversion routines to convert cm back to points required by the API.

-(NSNumber*)ptsFromCMN:(float)cm
{
    return [NSNumber numberWithFloat:[self ptsFromCMF:cm]];
}
-(float)ptsFromCMF:(float)cm
{
    return cm * 28.3464567;
}

So, while it may initially seem quite daunting to tackle this, once you have the basic formatting methods set up it is quite quick and easy to build on that to create a number of standard styles for your users.

Image Attachment Handing

Our object with image handling is to allow the user to embed images in the text by pasting them in from other applications. However the default image display behaviour provided by NSTextView and UITextView is to display the image at full resolution, which in most cases results in the image being very large relative to the text.
So we wanted to make sure that the image is scaled proportionately to ensure its width is never wider that width of the text line (and the right border of the text view). To keep things simple we don’t aim to provide drag points so the user can scale the image to whatever size they prefer, nor do we aim to allow the image to float over the text with text wrapping.
So the first challenge is to actually get access to the image and figure out how we can modify its display behaviour without altering the resolution of the original image.
OS X
On OS X it turns out there are a number of ways of achieving this, including creating our own NSTextAttachmentCell subclass or overriding NSLayoutManager. For a number of reasons we need to subclass the NSTextAttachment on iOS and this more or less forces us to create our own NSTextAttachment subclass on OS X.
The basic issue revolves around a few key issues:

  • The need to be able to archive and unarchive the NSAttributedString including any NSTextAttachments on both the iOS and OS X platforms. To do this we must have classes of the same name that we can archive to and from
  • On iOS the NSTextAttachments .image property is not populated during unarchiving even though the image is displayed in the UITextView. If this image property is not populated then the user interaction with the image is not activated so we have to make sure that the image property gets populated when unarchiving the attributed string from the Core Data store

So we create our own subclass of NSTextAttachment for OS X using the same name, OSTextAttachment, as we use for the iOS NSTextAttachment subclass. However we don’t need any new functionality so this class contains no new or override properties or methods. We need this class to unarchive any attachments created on the iOS platform because they will be OSTextAttachment classes. We should probably just make sure all the attachments we add on the OS X platform are inserted as OSTextAttachments as well but for now we don’t bother because iOS can unarchive NSTextAttachment objects.
More importantly we need to subclass NSTextAttachmentCell on OS X in order to modify the default display behaviour. We need to override the cellFrameForTextContainer method on this subclass to return the required frame size as shown below. We also override the drawWithFrame method to simply draw the cells image in the frame.

@implementation OSxTextAttachmentCell

- (NSRect)cellFrameForTextContainer:(NSTextContainer *)textContainer proposedLineFragment:(NSRect)lineFrag glyphPosition:(NSPoint)position characterIndex:(NSUInteger)charIndex {

    float width = lineFrag.size.width;

    NSRect rect = [self scaleImageSizeToWidth:width];

    return rect;
}
// Scale the image to fit the line width less 10%
- (NSRect)scaleImageSizeToWidth:(float)width {

    float scalingFactor = 1.0;

    NSSize imageSize = [self.image size];

    if (width < imageSize.width)
        scalingFactor = (width*0.9) / imageSize.width;

    NSRect rect = NSMakeRect(0, 0, imageSize.width * scalingFactor, imageSize.height * scalingFactor);

    return rect;

}
- (void)drawWithFrame:(NSRect)cellFrame inView:(NSView *)aView {
    if (self.image != nil) {
        [self.image drawInRect:cellFrame fromRect:NSZeroRect operation:NSCompositeSourceOver fraction:1.0 respectFlipped:YES hints:nil];
    }

}
@end

Now we need to make sure we replace any NSTextAttachmentCell instances with OSxTextAttachmentCell subclasses containing our overridden methods.
Fortunately the NSTextStorage delegate methods include one method for exactly this purpose. It provides your app with the last opportunity to modify the attributes of the attributed string. In this method we simply iterate over any attachments and replace their cells with our own cell subclass.

- (void)textStorageWillProcessEditing:(NSNotification *)aNotification {
    //FLOG(@"textStorageWillProcessEditing: called");
    NSRange range = [self.textStorage editedRange];

    [self.textStorage enumerateAttributesInRange:range options:NSAttributedStringEnumerationLongestEffectiveRangeNotRequired usingBlock:
     ^(NSDictionary *attributes, NSRange range, BOOL *stop) {

         // Iterate over each attribute and look for a Font Size
         [attributes enumerateKeysAndObjectsUsingBlock:^(id key, id obj, BOOL *stop) {
             if ([[key description] isEqualToString:NSAttachmentAttributeName]) {
                 //FLOG(@" attachment of class %@ found", [obj class]);

                 NSTextAttachment *attachment = (NSTextAttachment *)obj;
                 //LOG(@" attachment found");

                 //Replace the default attachment cell with our own
                 [self replaceAttachmentCell:attachment];
             }

         }];
     }];
}

- (void)replaceAttachmentCell:(NSTextAttachment*)attachment {

    // If it is one we have already replaced then return
    if ([attachment.attachmentCell isKindOfClass:[OSxTextAttachmentCell class]]) {
        return;
    }

    // Get the current attachment cell
    NSTextAttachmentCell *cell = (NSTextAttachmentCell *)[attachment attachmentCell] ;

    // Get the image from it
    NSImage * image = [cell image];

    // Create a new one with the image
    OSxTextAttachmentCell *newCell = [[OSxTextAttachmentCell alloc] initImageCell:image];

    // Now replace the attachment cell
    [attachment setAttachmentCell:newCell];

    return;
}

Now any new images pasted into the NSTextView on OS X will always be displayed in a frame sized to 0.9 times the line width.

iOS
On iOS we need to take a slightly different approach because of slight differences in the classes, for example, there is no NSTextAttachmentCell class on iOS. Fortunately the NSTextAttachment class on iOS provides a number of methods for us to modify the display behaviour of embedded image attachments.
Firstly we need to fix the default behaviour which does not populate the image attribute when the NSTextAttachment is unarchived so when we initialise the attachment we create the UIImage using the attachment data.

- (id)initWithData:(NSData *)contentData ofType:(NSString *)uti {
    //FLOG(@"initWithData called");
    //FLOG(@"uti is %@", uti);
    self = [super initWithData:contentData ofType:uti];

    if (self) {
        if (self.image == nil) {
            //FLOG(@" self.image is nil");
            self.image = [UIImage imageWithData:contentData];
        } else {
            FLOG(@" self.image is NOT nil");
        }
    }
    return self;
}

And then we override the method that is called by the Layout Manager to get the drawing rectangle for the attachment

- (CGRect)attachmentBoundsForTextContainer:(NSTextContainer *)textContainer proposedLineFragment:(CGRect)lineFrag glyphPosition:(CGPoint)position characterIndex:(NSUInteger)charIndex {

    float width = lineFrag.size.width;

    return [self scaleImageSizeToWidth:width];
}

// Scale the image to fit the line width less 10%
- (CGRect)scaleImageSizeToWidth:(float)width {

    float scalingFactor = 1.0;

    CGSize imageSize = [self.image size];

    if (width < imageSize.width)
        scalingFactor = (width*0.9) / imageSize.width;

    CGRect rect = CGRectMake(0, 0, imageSize.width * scalingFactor, imageSize.height * scalingFactor);

    return rect;

}

Finally we need to replace the default NSTextAttachment instance in the attributed string with our subclass. This is performed in the NSTextStorage delegate method shown below which we implement in our UITextView subclass, so we must also remember to set our UITextView subclass instance as a delegate to its text storage object.

- (void)textStorage:(NSTextStorage *)textStorage didProcessEditing:(NSTextStorageEditActions)editedMask range:(NSRange)editedRange changeInLength:(NSInteger)delta {
    //FLOG(@"textStorage:didProcessEditing:range:changeInLength: called");
    __block NSMutableDictionary *dict;

    [textStorage enumerateAttributesInRange:editedRange options:NSAttributedStringEnumerationLongestEffectiveRangeNotRequired usingBlock:
     ^(NSDictionary *attributes, NSRange range, BOOL *stop) {

         dict = [[NSMutableDictionary alloc] initWithDictionary:attributes];

         // Iterate over each attribute and look for attachments
         [dict enumerateKeysAndObjectsUsingBlock:^(id key, id obj, BOOL *stop) {

             if ([[key description] isEqualToString:NSAttachmentAttributeName]) {
                 //FLOG(@" textAttachment found");
                 //FLOG(@" textAttachment class is %@", [obj class]);

                 NSTextAttachment *attachment = obj;
                 OSTextAttachment *osAttachment;

                 if (attachment.image) {
                     //FLOG(@" attachment.image found");
                     osAttachment = [[OSTextAttachment alloc] initWithData:UIImagePNGRepresentation(attachment.image) ofType:attachment.fileType];
                 }
                 else {
                     //FLOG(@" attachment.image is nil");
                     osAttachment = [[OSTextAttachment alloc] initWithData:attachment.fileWrapper.regularFileContents ofType:attachment.fileType];
                 }

                 [dict setValue:osAttachment forKey:key];
             }

         }];

         [textStorage setAttributes:dict range:range];

     }];

}

Now in order to implement popup menu options for the image attachments we need to implement the UITextView delegate method in our detail view controller (the one that contains the UITextView) to display the action sheet.

- (BOOL)textView:(UITextView *)textView shouldInteractWithTextAttachment:(NSTextAttachment *)textAttachment inRange:(NSRange)characterRange {

    // save in ivar so we can access once action sheet option is selected
    _attachment = textAttachment;

    [self attachmentActionSheet:(UITextView *)textView range:characterRange];

    return NO;
}

If you wish to make sure your archived NSAttributedString remains compatible with other apps that may not have the custom attachment subclass available you should implement a method to replace any custom attachment subclasses in the attributed string before storing it.

Feel free to send donations via PayPal to duncan.groenewald@ossh.com.au.

Swift Versions for iOS

Code Listings for OS X

Code Listings for iOS

Related Articles

14 thoughts on “Implementing Rich Text with Images on OS X and iOS

  1. Hi Duncan,

    Can you please help me out with this stackoverflow question.I am using your approach for adding NSTextAttachment’s but it slowing down the UItextview a lot.

    http://stackoverflow.com/questions/23026975/ios-7-0-uitextview-gettings-terribly-slow-after-adding-images-to-it

    http://stackoverflow.com/questions/23300349/ios-7-uitextview-size-of-nstextattachment-getting-2x-after-reopening-the-applic

    I am stuck with this problem from many days.I would really appreciate if you help me out here.I have tried to override the attachmentSizeforGlyphIndex: method of nslayoutmanger but it is not helping out as the rect of text attachment increases.and trying to resize the linefragment of image messes up whole textview.I am quite new to this.Ignore my naiveness

    • With my approach I always keep a copy of the original image which means that the displayed image has to be scaled for whatever device it is being shown on – obviously this will have some performance impact, especially on slower devices.

      However I guess you could overcome this by using a custom attachment which stores a copy of the images scaled for each device and uses them for display. A bit like a thumbnail. This will still have a bit of a performance impact the first time the app has to create a scaled version of the image for a device.

      • Thanks a lot Duncan.I followed a similar approach to solve the issue.

  2. How about if instead of adding image-based attachments, I wanted to add a piece of tagified text? It seems inefficient to have to build an image of the text before displaying it.

    • I can’t recall the details of how you use attachments to tag text but I believe it is possible. You might want to look at the WWDC2013 example of customising the textview.

  3. How do you check for the presence of a bullet and tab character for bullet style? And how do you insert bullet character before tab?

  4. I don’t usually leave comments on blog posts (especially when I’m looking for an answer to a code-related question), but I gotta give props when props are due. So … mad props. Thank you. And thank you for taking the time to mention the Mac side of things. So many sources are iOS-focused now, I can’t ever find in-depth info for OS X things. I really appreciate it.

  5. Hi,

    This is really useful. Cannot thank you enough. However I want to ask a question.

    > How to center the image in an attachment based on the device size for iOS?

    I tried setting the rect in scaleImageSizeToWidth and that did not help.

    • Well I am not sure exactly what you are asking here but to centre on the page you would first need to set things up such that your page width matches the width of the device (orientation) so that text wrapping occurs correctly and then set the paragraph alignment as centered for the attachment paragraph. You would also need to make sure the image is sized to fit the width as appropriate.

  6. Generally I don’t learn article on blogs, but I wish to say that this write-up very
    compelled me to take a look at and do it! Your writing style has been surprised me.
    Thank you, quite great post.

  7. Pingback: Add image on UITextView with custom spacing and text - DexPage

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s