What are options to communicate between VB6 COM server on Windows XP and Python/D on Linux, excluding .NET?

Question

I have a complex and mature scientific data-crunching project that consists of an entangled web of COM servers written mostly in VB6 that use only very little UI (no forms nor controls), as most common users so far write short VBA programs with it.

We have two future goals:

1. Create a web UI by means of web-serving component living on Linux.

2. Gradually migrate the project to D/Python and away from Windows and VB6.

We are at the stage of designing the messaging framework that would serve as best common nominator for both goals.

The projects does heavy number crunching tasks, that involves running custom R scripts and LaTeX/MS Word/Excel reports. We need to expose some of the functionality to broad range of customers, and web UI seems the best option. On the other hand, as usage of Windows in our environment gradually declines, we'd like to eventually port our project to Linux, or at least keep that option open and not lock ourselves in Windows-specific technology.

• From this perspective, the natural .NET-based solution is something that we would like to avoid in the first place.

• DCOM/MSRPC seems natural from VB6-point of view, but this technology is depreciated and insecure. Besides, we would like to keep the web-serving part of the project on Linux.

• So the best option would be to write the web-serving part on Linux (using e.g. Python or D). The question remains: how to efficiently do communication between VB6 and Python (or D). It is obvious it will be some sort of network protocol. What we need, is efficient and easy to use protocol, that allows passing events and messages between two parties. The volume of data transferred will be small, and high latency not important.

  • DCOM/MSRPC seems fine to do local communication, but will it really be easy to do on the Linux part? We have little experience with Python so far, and there is no support for it in D.
  • Custom IP protocol. As our needs are small, that might just work. But isn't it reinventing the wheel?
  • We like the idea of JSON-RPC, as part of our project already uses JSON, and with publicly available clients for Python and VB6.

I guess, that there are many more feasible solutions to our task. We will be hiring people for help with that, but before that we need to know what technologies are available and where (who) to seek help.

Answer 1

First thing that comes to mind are serialization protocols like Google Protocol Buffers, Thrift, MessagePack, Avro, etc. These are so popular that one of them MUST have already been implemented in VB6. As far as I know, all, except perhaps Avro are implemented in D as well, so you may have few options depending what is the case in the VB6 world. My suggestion is to pick one of them and use it, or if there is REST support for VB6, then perhaps VibeD on Linux side could do the job for you.

Answer 2

Here is my private conversation I had between 12 and 16 November with Adam D. Ruppe about this topic, pasted with permission:

The Windows XP TLS bug strikes again... and it might actually be a showstopper, depends on how you feel about avoiding allocations.

I can hack around it, but it'd mean the druntime is never loaded. The code in here does this if you compile with -version=skip_rt_init

So anything that calls into it won't work - no string append, array resize, most of phobos is right out, etc. It would be more like writing C than writing D.

The problem btw is that the Windows XP dll loader can't handle thread local static variables. It is explained in detail here: http://www.nynaeve.net/?p=180

It works if the dll is loaded with the exe, but since VB and JScript call LoadLibrary in an already running process, we hit this problem.

D, by default, uses implicit TLS. We can avoid it in our own code, but not so easily in druntime. Calls to allocate memory and such in D overwrite the parent exe's memory, causing it to crash unpredictably.

The problem is fixed in Windows Vista, but that doesn't help XP users...

But, I got this code more or less working with -version=skip_rt_init. As you can see in com helpers, I jumped through a few hoops to avoid most allocations - some druntime and phobos work, just much of it doesn't.

On the bright side though, look at dclient.d, chello.d and ihello.d. Fairly beautiful code! dserver.d is no longer necessary, I was able to make all of it generic.

Most the mess is in comhelpers.d. BTW, returning values to the script still isn't completed, I just spent all day fighting with this thread thing.

Bottom line, if you can avoid anything that needs the D runtime to be initialized, or can use win vista +, this is going to be pretty. Otherwise, the tls bug might be a showstopper for you.

and here's a link to the zip file

http://arsdnet.net/dcode/com.zip

check the build.txt file for the command line i used; I don't use Visual D but it can't be too much more different than that.

This solution isn't perfect on Win XP due to the bug, but it might work well enough for you to use.

---

Adam Ryczkowski wrote: That's a really dire news. I thought intuitively, that since D allows much of the low-level hacking features of C/C++, it could be made compatible with COM on any platform...

Yeah, the problem is the TLS thing. You could make it work, but since druntime uses the default thread local storage it gets problematic.

A.R.: Is there any simple way of telling which part of phobos and druntime will work, other than by inspecting the source for memory allocations?

I don't think so. Any access to a thread local variable (mainly module level globals not marked shared or gshared) could be a problem.

A.R.: How about out of proc server? There shouldn't be much problem with the memory anymore, at the expense of the execution speed, but I might live with that.

That's a good idea. I don't know how to do an out of proc server, but it can't be too much different, and being its own exe avoids trouble.

I probably won't have a chance to play with this myself again for a few days though.

A.R.: Thank you very much for the solution. Would you agree if I post the information you gave me into StackExchange? I want to give it as much spin as I can.

go ahead, I moved to email since stack overflow isn't good for more extended talks, but feel free to post anything said in here.

---

edit: Adam D. Ruppe send me another email just now:

I just stumbled upon something that might help that Windows XP problem: the module core.sys.windows.dll has a function:

        if( !dll_fixTLS( hInstance, tlsstart, tlsend, tls_callbacks_a ....

It is used from here:

http://dlang.org/dll.html

in the dll_process_attach function. This might just work for our problem too. I'm in the middle of something else right now but i can try it later if you don't do it first.