Generic code completion library/tool

Question

I'm trying to set up proper code completion in my favorite editor, let's call it AnEditor to avoid program-specific answers which the internets are full of. (And the language is ALanguage, you know.) The editor has two features which I like it for: it works as well in console as in GUI so I can use it over a network, and it's vastly extensible. So I'm reluctant to use a full-blown IDE. But the editor doesn't have solid code completion though it can be plugged in if I find a decent solution.

I've googled up a whole bunch of questions and solutions to "[language X] completion in [editor/IDE Y]". It seems that every new IDE is implementing its code completion for every language from scratch, parser and all. And every simpler editor (including AnEditor) does one of the following:

• complete only standard library function names,
• or use ctags which offers a retarded regex-based "parsing" (for non-C programs) and is not supposed to tell the type of a variable you've typed in just now so not very useful for real code completion,
• there are other ways if the editor is extensible with plugins but they usually boil down to a more or less pervert combination of the above, with a good deal of custom regexes.

Now, the question is, why can't we have a sound code completion library that I can plug in to AnEditor and someone else to ABigIDE? As far as I can tell (deciphering C pointer jungle not being my objective), the answer should look somewhat like this:

• a generic parser in the style of yacc/lex/bison (or a static analyzer), somehow relaxed to tolerate code in the process of writing, able to make sense of JavaDoc-style comments. And fast, preferably, so it can be used on the fly
• an index that can tell a class' members, methods' signatures and their positions in files (ctags doing this now), return values and other documentation from the JavaDoc comments
• another index that knows a variable's type, and a function that tells the type based on a position in a file or code being currently written

So, to get completion for some language working, you make up parser rules for the language, build indexes over the standard library and your project, summon the type-telling function and look up the class' members and documentation. Or just list classes and members if you're dealing with object construnction or static calls.

If Eclipse, Netbeans and JetBrains have successfully done it in Java (so that I'm supposed to plug Eclipse to AnEditor), why can't anyone do it in a less bloated and more universal fashion? Or am I missing something and the future is already lurking somewhere?

Answer 1

So: by now, pretty much this exact problem was tackled by Microsoft, of all people. ‘Language servers’ provide a protocol to plug language-specific semantics-aware completion and other features to different editors and IDEs.

Notably, ‘language servers’ tend to be developed in the same languages that they parse. This might be a dubious decision in cases of slower languages such as Python.

For some languages, similar solutions were available for some time, independently of Langservers—e.g. Tern for JS, Scion and ghc-mod for Haskell, etc. These tend to serve Emacs and Vim primarily. Predictably, such solutions mainly appear for non-mainstream languages, like all the compiled-to-JS ones—which seem to pop up faster than it would take to develop individual plugins for each editor/IDE.