stringByReplacingOccurrencesOfString not giving me desired result

Question

I have some NSString like :

test = @"this is %25test%25 string";

I am trying to replace test with some arabic text , but it is not replacing exactly as it is :

[test stringByReplacingOccurrencesOfString:@"test" withString:@"اختبار"];

and the result is :

this is %25 اختبار %25 string

Some where I read there could be some problem with encoding or text alignment.Is there extra adjustment needed to be done for arabic string operations .

EDIT : I have used NSMutable string insert property but still the same result .

Answer 1

EDIT 2:

One other thing that occurs to me that is causing most of your trouble in this specific example. You have a partially percent-encoded string above. You have spaces, but you also have %25. You should avoid doing that. Either percent-encode a string or don't. Convert it all at once when required (using stringByAddingPercentEscapesUsingEncoding:). Don't try to "hard-code" percent-encoding. If you just used "this is a %اختبار% string" (and then percent-encoded the entire thing at the end), all your directional problems would go away (see how that renders just fine?). The rest of these answers address the more general question when you really need to deal with directionality.

---

EDIT:

The original answer after the line relates to human-readable strings, and is correct for human-readable strings, but your actual question (based on your followups) is about URLs. URLs are not human-readable strings, even if they occasionally look like them. They are a sequence of bytes that are independent of how they are rendered to humans. "اختبار" cannot be in the path or fragment parts of an URL. These characters are not part of the legal set of characters for those sections (اختبار is allowed to be part of the host, but you have to follow the IDN rules for that).

The correct URL encoding for this is a %25<arabic>%25 string is:

this%20is%20a%20%2525%D8%A7%D8%AE%D8%AA%D8%A8%D8%A7%D8%B1%2525%20string

If you decode and render this string to the screen, it will appear like this:

this is a %25اختبار%25 string

But it is in fact exactly the string you mean (and it is the string you should pass to the browser). Follow the bytes (like the computer will):

this   - this    (ALPHA)
%20    - <space> (encoded)
is     - is      (ALPHA)
%20    - <space> (encoded)
a      - a       (ALPHA)
%20    - <space> (encoded)
%25    - %       (encoded)
25     - 25      (DIGIT)
%D8%A7 - ا       (encoded)
%D8%AE - خ       (encoded)
%D8%AA - ت       (encoded)
%D8%A8 - ب       (encoded)
%D8%A7 - ا       (encoded)
%D8%B1 - ر       (encoded)
%25    - %       (encoded) 
25     - 25      (DIGIT)
%20    - <space> (encoded)
string - string  (ALPHA)

The Unicode BIDI display algorithm is doing what it means to do; it just isn't what you expect. But those are the bytes and they're in the correct order. If you add any additional bytes (such as LRO) to this string, then you are modifying the URL and it means something different.

So the question you need to answer is, are you making an URL, or are you making a human-readable string? If you're making an URL, it should be URL-encoded, in which case you will not have this display problem (unless this is part of the host, which is a different set of rules, but I don't believe that's your problem). If this is a human-readable string, see below about how to provide hints and overrides to the BIDI algorithm.

It's possible that you really need both (a human-friendly string, and a correct URL that can be pasted). That's fine, you just need to handle the clipboard yourself. Show the string, but when the user goes to copy it, replace it with the fully encoded URL using UIPasteboard or by overriding copy:. See Copy, Cut, and Paste Operations. This is fairly common (note how in Safari, it displays just "stackoverflow.com" in the address bar but if you copy and paste it, it pastes "https://stackoverflow.com/" Same thing.

---

Original answer related to human-readable strings.

Believe it or not, stringByReplacingOccuranceOfString: is doing the right thing. It's just not displaying the way you expect. If you walk through characterAtIndex:, you'll find that it's:

% 2 5 ا ...

The problem is that the layout engine gets very confused around all the "neutral direction" characters. The engine doesn't understand whether you meant "%25" to be attached to the left to right part or right to left part. You have to help it out here by giving it some explicit directional characters to work with.

There are a few ways to go about this. First, you can do it the Unicode 6.3 tr9-29 way with Explicit Directional Isolates. This is exactly the kind of problem that Isolates are meant to solve. You have some piece of text whose direction you want to be considered completely independently of all other text. Unicode 6.3 isn't actually supported by iOS or OS X as best I can tell, but for many (though not all) uses, it "works."

You want to surround your Arabic with FSI (FIRST STRONG ISOLATE U+2068) and PDI (POP DIRECTIONAL ISOLATE U+2069). You could also use RLI (RIGHT-TO-LEFT ISOLATE) to be explicit. FSI means "treat this text as being in the direction of the first strong character you find."

So you could ideally do this:

NSString *test = @"this is a %25\u2068test\u2069%25 string";
NSString *arabic = @"اختبار";
NSString *result = [test stringByReplacingOccurrencesOfString:@"test" withString:arabic];

That works if you know what you're going to substitute before hand (so you know where to put the FSI and PDI). If you don't, you can do it the other way and make it part of the substitution:

NSString * const FSI = @"\u2068";
NSString * const PDI = @"\u2069";

NSString *test = @"this is %25test%25 string";
NSString *arabic = @"اختبار";
NSString *replaceString = [@[FSI, arabic, PDI] componentsJoinedByString:@""];
NSString *result = [test stringByReplacingOccurrencesOfString:@"test" withString:replaceString];

I said this "mostly" works. It's fine for UILabel, and it probably is fine for anything using Core Text. But in NSLog output, you'll get these extra "placeholder" characters:

You might get this other places, too. I haven't checked UIWebView for instance.

So there are some other options. You can use directional marks. It's a little awkward, though. LRM and RLM are zero-width strongly directional characters. So you can bracket the arabic with LRM (left to right mark) so that the arabic doesn't disturb the surrounding text. This is a little ugly since it means the substitution has to be aware of what it's substituting into (which is why isolates were invented).

NSString * const LRM = @"\u200e";
NSString *test = @"this is a %25test%25 string";
NSString *replaceString = [@[LRM, arabic, LRM] componentsJoinedByString:@""];
NSString *result = [test stringByReplacingOccurrencesOfString:@"test" withString:replaceString];

BTW, Directional Marks are usually the right answer. They should always be the first thing you try. This particular problem is just a little too tricky.

One more way is to use Explicit Directional Overrides. These are the giant "do what I tell you to do" hammer of the Unicode world. You should avoid them whenever possible. There are some security concerns with them that make them forbidden in certain places (<RLO>elgoog<PDF>.com would display as google.com for instance). But they will work here.

You bracket the whole string with LRO/PDF to force it to be left-to-right. You then bracket the substitution with RLO/PDF to force it to the right-to-left. Again, this is a last resort, but it lets you take complete control over the layout:

NSString * const LRO = @"\u202d";
NSString * const RLO = @"\u202e";
NSString * const PDF = @"\u202c";

NSString *test = [@[LRO, @"this is a %25test%25 string", PDF] componentsJoinedByString:@""];
NSString *arabic = @"اختبار";
NSString *replaceString = [@[RLO, arabic, PDF] componentsJoinedByString:@""];
NSString *result = [test stringByReplacingOccurrencesOfString:@"test" withString:replaceString];

I would think you could solve this problem with the Explicit Directional Embedding characters, but I haven't really found a way to do it without at least one override (for instance, you could use RLE instead of RLO above, but you still need the LRO).

Those should give you the tools you need to figure all of this out. See the Unicode TR9 for the gory details. And if you want a deeper introduction to the problem and solutions, see Cal Henderson's excellent Understanding Bidirectional (BIDI) Text in Unicode.