What encoding used when invoke fopen or open?

When we invoke system call in linux like 'open' or stdio function like 'fopen' we must provide a 'const char * filename'. My question is what is the encoding used here? It's utf-8 or ascii or iso8859-x? Does it depend on the system or environment setting?

I know in MS Windows there is a _wopen which accept utf-16.

What type of encoding is UTF-8?

UTF-8 is a Unicode character encoding method. This means that UTF-8 takes the code point for a given Unicode character and translates it into a string of binary. It also does the reverse, reading in binary digits and converting them back to characters.

It's a byte string, the interpretation is up to the particular filesystem.