How does Semantic Highlighting work in VS Code with custom Xtext LS

Question

I'm currently testing how to create a VS Code extension that would allow me to write files using my own DSL defined in Xtext (*.sample). So far I've been able to create the VS Code extension and use it:

• Open a file (*.sample) and load the extension.
• See syntax errors, text prediction, renaming, etc (communication with Xtext LS works fine).
• See syntax color highlighting by means of a TextMate grammar and a couple of VS Code UI themes I've created.

Now I would like to go a step further and enable semantic highlighting. So far I understand that this is a capability of the LS (see LSP specification 3.16) and the highlighting information comes from the server. However the only information I can find about semantic highlighting and VS Code is the creation of a DocumentSemanticTokensProvider in the VS Code extension, that is, client side (see VS Code documentation). As I understand the Semantic Token Provider returns the tokens with its classification (class, namespace, enum, ...) and modifier (declaration, declaration, ...). From my understanding this would imply some file analysis on the client side, which from my point of view is not the way to go.

On the other side, I'm not able to find any documentation on how to "activate" the semantic highlighting in the server side (Xtext) so the semantic tokens are returned when the client requests them.

As you can see I have lots of doubts on how the semantic highlighting would work in my scenario. Therefore I would like to ask the following questions.

• Where do I need to define the logic that decides which are the semantic tokens of a file?
• How does the VS Code extension triggers the calculation of semantic tokens in the server?

Any other comment on whether my assumptions are correct (or not) are welcome.

Many thanks in advance.

Answer 1

As stated on Cristian Dietrich's comment, the Xtext LSP implementation does no implement the semantic token feature of the LSP protocol yet (see issue). In my case, in order to be able to get "hardcoded" semantic tokens from the LSP (instead of using the DocumentSemanticTokensProvider from the VS Code extension), I had to extend the Xtext LSP implementation to set a semanticTokensProvider as part of the LSP capabilities:

public class CustomLanguageServerImpl extends LanguageServerImpl {
    
    @Override
    protected ServerCapabilities createServerCapabilities(InitializeParams params) {
        ServerCapabilities capabilities = super.createServerCapabilities(params);
        // Create server capabilities for semantic tokens.
        SemanticTokensWithRegistrationOptions options = new SemanticTokensWithRegistrationOptions();

        List<String> tokenTypes = new ArrayList<String>();
        tokenTypes.add(SemanticTokenTypes.Namespace);
        ...
        
        List<String> tokenModifiers = new ArrayList<String>();
        tokenModifiers.add(SemanticTokenModifiers.Declaration);
        ...

        options.setLegend(new SemanticTokensLegend(tokenTypes, tokenModifiers));
        options.setRange(false);
        options.setFull(true);

        capabilities.setSemanticTokensProvider(options);

        return capabilities;
    }
}

By using the new LSP implementation, I'm able to see hardcoded semantic tokens in my VS Code extension. My guess is that the "handshaking" process executed by the VS Code extension and the LSP (during initialization) makes the VS Code extension to request the semantic tokens based on the configuration (full/range). This can be seen in the messages exchanged between client and server:

   {jsonrpc: '2.0', id: 0, method: 'initialize', params: {…}}
   {jsonrpc: '2.0', id: 0, result: {…}}
   {jsonrpc: '2.0', method: 'initialized', params: {…}}
   {jsonrpc: '2.0', method: 'textDocument/didOpen', params: {…}}
   {jsonrpc: '2.0', method: 'textDocument/publishDiagnostics', params: {…}}
   {jsonrpc: '2.0', id: 1, method: 'textDocument/documentSymbol', params: {…}}
   {jsonrpc: '2.0', method: 'textDocument/publishDiagnostics', params: {…}}
   {jsonrpc: '2.0', id: 1, result: Array(1)}
   {jsonrpc: '2.0', id: 2, method: 'textDocument/documentSymbol', params: {…}}
   {jsonrpc: '2.0', id: 2, result: Array(1)}
-> {jsonrpc: '2.0', id: 3, method: 'textDocument/semanticTokens/range', params: {…}}
   {jsonrpc: '2.0', id: 4, method: 'textDocument/foldingRange', params: {…}}
   {jsonrpc: '2.0', id: 3, result: {…}}
   {jsonrpc: '2.0', id: 4, result: Array(2)}

Using the default Xtext LSP implementation, the textDocument/semanticTokens/* request is not even triggered.