Characters Å Ä Ö is not getting displayed in my DDL, how can I tell restclient to utilize a specific charset?

Question

Before I start here is the problem. It should be like this:

Björn Nilsson, instead its displaying strange special characters, all values that have characters Å, Ä and Ö becomes like this.

I fill my DDL with values from API in XML format that has all the values, and we are also using Linq2Rest for that.

This how the process looks like

 private readonly RestContext<ADConsultants> restContext;

public ConsultantContext(Uri uri, Format format)
{
    restContext = new RestContext<ADConsultants>(GetRestClient(uri, format), GetSerializerFactory(format));
}
public enum Format
{
    Pox,
    Json
}

private static readonly IEnumerable<Type> knownTypes = new[] {typeof (ADConsultants)};

public static IRestClient GetRestClient(Uri uri, Format format)
{
    switch (format)
    {
        case Format.Pox:
            return new XmlRestClient(uri);
        case Format.Json:
            return new JsonRestClient(uri);
        default:
            throw new NotImplementedException();
    }
}
private static ISerializerFactory GetSerializerFactory(Format format)
{
    switch (format)
    {
        case Format.Pox:
            return new XmlSerializerFactory(knownTypes);
        case Format.Json:
            return new JsonNetSerializerFactory();
        default:
            throw new NotImplementedException();
    }

}
public IQueryable<ADConsultants> Consultant
{
    get { return restContext.Query; }
}

}

This is my JsonNetSerializerFactory class:

public class JsonNetSerializerFactory :ISerializerFactory 
{
    public ISerializer<T> Create<T>()
    {
        return new JsonNetSerializer<T>();
    }
    public class JsonNetSerializer<T> : ISerializer<T>
    {
        public T Deserialize(string input)
        {
            return JsonConvert.DeserializeObject<T>(input);
        }

        public IList<T> DeserializeList(string input)
        {
            return JsonConvert.DeserializeObject<IList<T>>(input);
        }
    }
}

And this is inside my controller:

var consultants = new ConsultantContext(
        new Uri("http://adress:port/api/consultants"),
                ConsultantContext.Format.Json)
                    .Consultant
                    .Where(x => x.Office == "Örebro")  
                    .OrderBy(x => x.DisplayName)  
                    .ToList() 
                    .Select(x => new
                    {
                        name = x.DisplayName
                    });

I did a test by doing this:

name = "åäö"

and it worked fine, ddl values was "åäö"

Any help is appreciated on how to fix so characters Å Ä Ö works fine as values in my DDL.

HTTP header is utf-8, my html content have it aswell. It must be the XML that need to be set to a specific charset, How can I do that?

Thanks in advance!

Answer 1

in the theory you got an charset encoding/decoding problem.

the Cause: the content you try to read has been encoded using a charset like iso-8859-1 or iso-8859-15. and you'll try to read (decode) it directly to an "UTF-8" character Model. Of course it won't work because UTF-8 because UTF-8 won't in a miracle recognize your special chars (Ä,Ü, Ö, and so on..). UTF-8 is no guesser for character coding.

Solution:

1- (Re)encode your content( e.g "Björn Nilsson") with its corresponding charset (iso-8859-1/iso-8859-15) into Byte collection.

2- Decode your content with into "UTF-8" based charset.

here an Helper Class as example:

using System;
using System.Collections.Generic;
using System.Text;

    namespace csharp.util.charset
    {
        public class SysUtil
        {
            /// <summary>
            /// Convert a string from one charset to another charset
            /// </summary>
            /// <param name="strText">source string</param>
            /// <param name="strSrcEncoding">original encoding name</param>
            /// <param name="strDestEncoding">dest encoding name</param>
            /// <returns></returns>
            public static String StringEncodingConvert(String strText, String strSrcEncoding, String strDestEncoding)
            {
                System.Text.Encoding srcEnc = System.Text.Encoding.GetEncoding(strSrcEncoding);
                System.Text.Encoding destEnc = System.Text.Encoding.GetEncoding(strDestEncoding);
                byte[] bData=srcEnc.GetBytes(strText);
                byte[] bResult = System.Text.Encoding.Convert(srcEnc, destEnc, bData);
                return destEnc.GetString(bResult);
            }

        }
    }

Usage:

in your (JSON-, XML, other) serializer/deserializer classes just convert your content like that

String content = "Björn Nilsson";
SysUtil.StringEncodingConvert(content, "ISO-8859-1","UTF-8");

you could try to make your calls in your deserializer (if they really do what they mean):

public class JsonNetSerializerFactory :ISerializerFactory 
{
    public ISerializer<T> Create<T>()
    {
        return new JsonNetSerializer<T>();
    }
    public class JsonNetSerializer<T> : ISerializer<T>
    {
        public T Deserialize(string input, String fromCharset, String toCharset)

        {
           String changedString = SysUtil.StringEncodingConvert(input, fromCharset,toCharset);

            return JsonConvert.DeserializeObject<T>(changedString  );
        }

        public IList<T> DeserializeList(string input, String fromCharset, String toCharset)
        {
         String changedString =  SysUtil.StringEncodingConvert(input, fromCharset,toCharset);

            return JsonConvert.DeserializeObject<IList<T>>(changedString);
        }
    }
}

that's for your

JsonNetSerializerFactory  

please try to do same for other factories like

XmlSerializerFactory  

and don't forget your setting in your HTML-page

<meta charset="utf-8"> <!--HTML 5 -->

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <!-- if HTML version < 5-->