How do you remove invalid characters when creating a friendly url (ie how do you create a slug)?

Question

Say I have this webpage:
http://ww.xyz.com/Product.aspx?CategoryId=1

If the name of CategoryId=1 is "Dogs" I would like to convert the URL into something like this:
http://ww.xyz.com/Products/Dogs

The problem is if the category name contains foreign (or invalid for a url) characters. If the name of CategoryId=2 is "Göra äldre", what should be the new url?

Logically it should be:
http://ww.xyz.com/Products/Göra äldre
but it will not work. Firstly because of the space (which I can easily replace by a dash for example) but what about the foreign characters? In Asp.net I could use the URLEncode function which would give something like this:
http://ww.xyz.com/Products/G%c3%b6ra+%c3%a4ldre
but I can't really say it's better than the original url (http://ww.xyz.com/Product.aspx?CategoryId=2)

Ideally I would like to generate this one but how can I can do this automatically (ie converting foreign characters to 'safe' url characters):
http://ww.xyz.com/Products/Gora-aldre

Answer 1

One other thing worth considering:

If a user provides a string such as 好听的音乐 which you want to convert to a URL friendly title then you should consider using IdnMapping

For example:

string urlFriendlyTitle = Slugify(url);

public static string Slugify(string text)
{
    IdnMapping idnMapping = new IdnMapping();
    text = idnMapping.GetAscii(text);

    text = RemoveAccent(text).ToLower();

    //  Remove all invalid characters.  
    text = Regex.Replace(text, @"[^a-z0-9\s-]", "");

    //  Convert multiple spaces into one space
    text = Regex.Replace(text, @"\s+", " ").Trim();

    //  Replace spaces by underscores.
    text = Regex.Replace(text, @"\s", "_");

    return text;
}

public static string RemoveAccent(string text)
{
    byte[] bytes = Encoding.GetEncoding("Cyrillic").GetBytes(text);

    return Encoding.ASCII.GetString(bytes);
}

Without this, 好听的音乐 will be converted to string.Empty. With this, xn--fjqr6lw2ek78az68a which is punycode

Answer 2

I've come up with the 2 following extension methods (asp.net / C#):

     public static string RemoveAccent(this string txt)
    {
        byte[] bytes = System.Text.Encoding.GetEncoding("Cyrillic").GetBytes(txt);
        return System.Text.Encoding.ASCII.GetString(bytes);
    }

    public static string Slugify(this string phrase)
    {
        string str = phrase.RemoveAccent().ToLower();
        str = System.Text.RegularExpressions.Regex.Replace(str, @"[^a-z0-9\s-]", ""); // Remove all non valid chars          
        str = System.Text.RegularExpressions.Regex.Replace(str, @"\s+", " ").Trim(); // convert multiple spaces into one space  
        str = System.Text.RegularExpressions.Regex.Replace(str, @"\s", "-"); // //Replace spaces by dashes
        return str;
    }