Compile VB6 as background process on Server 2016

Question

We're having a very strange behaviour that I'm unable to identify a root cause for. We use TFS (2017.U2) to compile our legacy system, and are trying to update our build farm from 2008R2 up to 2016. The build system uses PowerShell (v5) to cycle through a list of VBP projects and run a VBS script to compile the projects.

First a bit of basics. UAC is totally disabled (in the registry, not just the slider control), VB6.EXE is also set to XP SP3 compatability, and also to run as the administrator.

Unfortunately, while we can see VB6.EXE start in task manager - it just hangs. Zero activity. Running the same compile interactively works just fine with the same user. This led me to theorize it was an environment problem, however process explorer shows me a valid user environment on the VB6.EXE process.

I don't believe this is due to VB6 throwing an error, as (at least in previous versions of Windows Server) when a background process opens a UI element, the OS indicates to the foreground that the background wants to break in. We dont see that.

We've stubbed this back to a bare minimum code example which I call "test.ps1":

$vb6="C:\Program Files (x86)\Microsoft Visual Studio\VB98\vb6.exe"
Set-Location D:\Builds\27\s\path\prjdir
start-process $vb6 -ArgumentList "/make /out errors.txt project.vbp"  -wait

We've been using "start-process" to trigger the VB6 compiles because direct invokation via PowerShell doesn't ingest the parameters properly (they're actually built out of strings passed into the master script in the full blown process... this is the simplified version).

When run interactively (.\test.ps1) this functions properly. The project compiled and I get an errors.txt file written.

When started as a process (start-process .\test.ps1) this again functions properly.

When triggered via a TFS "PowerShell Script" task, this fails to complete the VB6 step - the VB6.EXE can be seen in the Task viewer with the appropriate arguments, and no CPU or IO is associated with the task. No errors.txt file is written. No new DLL is created.

I was able to dummy this down even further and remove TFS from the stack by running the test script from another machine. I ran a remote invokation of the script, and duplicated the result using this command:

PS C:\Users\svc_build> Invoke-Command –ComputerName TestBuild02 –ScriptBlock {powershell  C:\Users\svc_build\desktop\test.ps1 }

Again, no errors.txt, and no new DLL. VB6.EXE starts up and just sits there. Process monitor doesnt provide any help in diagnosing what might be the issue.

This now smells of a security door being shut on me - even though the same domain user is running the same job, the difference is that this is a background process... and something is preventing a background process of executing in the same context as a foreground process.

Assuming this assumption is correct, can someone point me at the reason a remotely triggered (background) session isn't able to run VB6.EXE? (and of course, a work around for the situation would be appreciated :) )

If this is not a security issue - can someone identify the real culprit, and the solution to getting VB6 running as a background process, like we're used to seeing it run on W2K8R2?

Answer 1

I'm a bit late to the party, but this sounds like a very similar scenario to what I've just encountered.

• Windows 10 v2004
• UAC disabled
• Compiling by running VB6.exe via a PowerShell script.
• Using Bamboo as the build server, running as a Managed Service Account.

When running the build on the server via Bamboo, it hangs. When logging into the build server and running the build manually, it succeeded.

After cutting down the code I was able to reproduce a minimal failing case. The hang was caused by code in a UserControl's UserControl_Initialize method that was manipulating UI controls, but only when that UserControl was placed on a Form in the same project.

During compilation, the compiler will create an instance of the Form (why, I don't know), which in turn creates an instance of the UserControl, which in turn runs the UserControl_Initialize method. I can only assume that running the code at that point resulted in an error of some sort, and that resulted in the compiler hanging.

The same error can be caused by the UserControl_Resize event. That case is reasonably easy to fix by checking if Ambient.UserMode is true before trying to resize the child controls.

Private Sub UserControl_Resize()
    If UserControl.Ambient.UserMode Then
        ' Position the child controls
    End If
End Sub

Fixing the UserControl_Initialize methods required the code in those methods to be run at a different point (for example, when the UserControl is given the data to display, we now run the code that was previously in UserControl_Initialize).

Also worth noting is the compatibility settings for VB6.exe that we had to use. Using "Windows XP SP3" compatibility mode resulted in VB6.exe hanging immediately. We had to set it to not use any compatibility mode, but we did have to set Run this program as an adminisrtator, and had that applying for all users.