Managing very large codebases in Delphi using a Library of Debug and Regular DCUs I built myself

Question

I am trying to resolve this compile error, occuring only in Debug config, and only in the case described below:

[dcc32 Fatal Error] MyIndyTCPChannel.pas(22): F2051 Unit IdIOHandlerSocket was compiled with a different version of IdGlobal.IdDisposeAndNil

I am working on a very large Delphi codebase, with 2.5 million lines of in-house code, and 3 million lines of component code, which includes several large commercial Delphi component suites (Developer Express, TeeChart, and others), and a large number of open source delphi components as well, plus a fairly large in-house developed set of components, numbering 252 packages, of which about 140 are designtime+runtime or designtime, and the others are runtime packages (which are also loaded, into the IDE at runtime, by DLL-dependencies in their associated designtime package).

Our main library path has been optimized down to be small as can be, and it contains the paths that Delphi ships with as standard, plus three more we added, the primary one is a single "OurCompanyLibraryDCU" folder, which contains underneath it folders for the two platforms and two configurations that we use:

c:\dev\OurCompanyLibraryDCU\Win32\Release
c:\dev\OurCompanyLibraryDCU\Win32\Debug
c:\dev\OurCompanyLibraryDCU\Win64\Release
c:\dev\OurCompanyLibraryDCU\Win32\Debug

Each of the above folders contains the set of BPL, DCP, and DCU files in a single folder, for that platform/config combination.

A macro like the following, in the project options is used, so we can change platform, and config, and have the directories resolve correctly:

$(OURCOMPANYLIBRARYDCU)\$(Platform)\$(Config)

OURCOMPANYLIBRARYDCU is an environment variable and $(X) is the syntax to expand an environment variable, in the context of the Delphi IDE.

I am trying to get the most important and largest VCL Application project (call it BigApp.dproj) to build so that the project search directory only contains our APPLICATION source folders and does not need the project search path to contain all our third party component LIBRARY source code. To do that, we need to link against the debug DCUs, or release DCUs.

So far we have everything working except for the case where you have both Debug and Release DCUs available. The release DCUs are in the library path, and the debug DCUs are in the Debug DCU path, in the IDE settings. Confronted with the choice between these two libraries, Delphi's linker appears to fail, whenever both sets of DCUs exist, with errors in this form, when I click Build, and the Build Configuration is set to Release, I get F2051 errors. The ordinary cause of an F2051 error is that multiple incompatible binary DCUs exist and are both accessible, and the linker is failing to make it all work. However, when you want both Debug and Release DCUs both in the library path, I thought that this sort of thing would not occur, due to the Linker selecting the debug or release DCUs for you.

If I have not build the Debug DCUs, the above problem does not occur. I suspect that my Debug DCUs are subtly "invalid" or that the Debug-DCU-selection algorithm inside Delphi is not working, but have no idea why, or how to fix this.

Multi-part-Question:

A. Is having a single folder for each platform/config combination, containing the DCU, BPL, and DCP in a single folder, and then added to the IDE Library Path known to cause problems? Do I need three sub-folders, making a total of 12 folders for every platform+config+filetype, or can I keep them together by platform+config?

B. In a package compilation situation, is it okay to have the IDE Library path contain the OurCompanyLibraryDCU folder, and also have that folder configured as the DCP Output Directory, Package Output Directory, and Unit Output Directory? My concern is that by having input folder and output folders the same, there is a case where the compiler could be failing to rebuild a Unit from .pas source, and simply linking the prior compile's DCU.

C. If I'm going about this wrong, how instead, shall I prevent the over 2.5 million lines of component LIBRARY code from being compiled from source each time I build my BigApp, instead only link them via DCU, and still have the debug and release dcus work properly?

D. I can get past the original error if I go to the Win32\Debug folder and delete IdGlobal.dcu. This suggests to me that my package compilation (for debug config) is producing an INVALID IdGlobal.dcu. Is that even possible? Can delphi silently output garbled DCUs?

Notes: I'm not using, and can not use Runtime Packages, to deal with the application size problems.

Update: The first thing I should have done here is verify that ZERO additional DCU files are ANYWHERE on my harddrive, ANYWHERE. That's the standard F2051 error advice. I'll update this question after I've taken care of that. It appears possible that Delphi will itself COPY a DCU from one place to another, or that a bogus DCU that is NOT in the CURRENT search path might have been in some other project's search path. A kind of bucket-brigade of bad-DCU-copies can occur. I'll update the question once I'm sure what kind of bad-DCU-generations-or-copies are occuring.

Update 2: I have now guaranteed that no additional copies of IdGlobal.dcu exist before building, and the problem still reproduces. So the question then turns on the compiler options used when building the IdGlobal.dcu, versions the compiler options used when building BigApp.dproj in Debug build.

Update 3: Although all my package compiles appear to complete without error, it seems they were not using a correct library search path, during the time when the DCC32.exe or MSBUILD.exe is being launched to build the packages. This library path inconsistency issue appears to be the core issue, thanks to Sir Rufo for pointing that out.

Answer 1

Perhaps I can shed some light on the order of search paths presented to the compiler, which should make clear why the problem happens in the first place and can be cured (at least in your situation) by adding the Debug DCU path at that specific location. All these observations were made with XE7.

There are several places in the IDE where you can specify search paths:

1. Library path (Delphi-Options - Library)
2. Translated Library path (Delphi-Options - Library-Translated)
3. Debug DCU path (Delphi-Options - Library)
4. Translated Debug DCU path (Delphi-Options - Library-Translated)
5. Search path (via Project Options)

When the Library language is set to English, those pathes are given to the compiler in the order 5,1 or 3,5,1 depending of the setting of Use debug .dcus. This is already a bit weird as the debug dcu path takes precedence over the project search path.

So f.i. to make the compiler find our own dcu files of a newer Indy version, we have to place the corresponding paths in front of the paths under 1 and 3.

Now things get complicated when the Library language is set to something different than English. In this case the translated paths come into play resulting in the order 2,5,1 or 4,3,2,5,1 depending of the setting of Use debug .dcus.

To make the above example with a newer Indy version work, you have to tweak the translated paths, too.

The culprit lies in CodeGear.Delphi.Targets, which places the paths in this order. I was able to modify this file so that the natural order of paths is used: 5,2,1 or 5,4,3,2,1. If anyone can confirm that I am allowed to show these changes here I will do. Perhaps I can provide a patch only.

Update: Here are the changes of CodeGear.Delphi.Targets from XE7 as shown by Mercurial

@@ -122,20 +122,19 @@
     <DcpFilename Condition="'$(DcpFilename)'!='' And !HasTrailingSlash('$(DcpFilename)')">$(DcpFilename)\</DcpFilename>
     <DcpFilename Condition="'$(DcpFilename)'!=''">$(DcpFilename)$(MSBuildProjectName).dcp</DcpFilename>

-    <UnitSearchPath Condition="'$(DCC_UnitSearchPath)' != ''">$(DCC_UnitSearchPath);$(DelphiLibraryPath)</UnitSearchPath>
-    <UnitSearchPath Condition="'$(DCC_UnitSearchPath)' == ''">$(DelphiLibraryPath)</UnitSearchPath>
-
+    <UnitSearchPath>$(DelphiLibraryPath)</UnitSearchPath>
     <UnitSearchPath Condition="'$(DCC_TranslatedLibraryPath)' != ''">$(DCC_TranslatedLibraryPath);$(UnitSearchPath)</UnitSearchPath>
     <UnitSearchPath Condition="'$(DCC_DebugDCUs)'=='true' And '$(DelphiDebugDCUPath)'!=''">$(DelphiDebugDCUPath);$(UnitSearchPath)</UnitSearchPath>
     <UnitSearchPath Condition="'$(DCC_DebugDCUs)'=='true' And '$(DCC_TranslatedDebugLibraryPath)' != ''">$(DCC_TranslatedDebugLibraryPath);$(UnitSearchPath)</UnitSearchPath>
-
+    <UnitSearchPath Condition="'$(DCC_UnitSearchPath)' != ''">$(DCC_UnitSearchPath);$(UnitSearchPath)</UnitSearchPath>
+    
     <___ResourcePath Condition="'$(DCC_ResourcePath)' != ''">$(DCC_ResourcePath);$(DelphiLibraryPath)</___ResourcePath>
     <___ResourcePath Condition="'$(DCC_ResourcePath)' == ''">$(DelphiLibraryPath)</___ResourcePath>
+    <___ResourcePath Condition="'$(DCC_TranslatedResourcePath)' != ''">$(DCC_TranslatedResourcePath);$(___ResourcePath)</___ResourcePath>
     <__ResourcePath Condition="'$(DCC_UnitSearchPath)' != ''">$(DCC_UnitSearchPath);$(___ResourcePath)</__ResourcePath>
     <__ResourcePath Condition="'$(DCC_UnitSearchPath)' == ''">$(___ResourcePath)</__ResourcePath>
     <ResourcePath Condition="'$(BRCC_OutputDir)' != ''">$(BRCC_OutputDir);$(__ResourcePath)</ResourcePath>
     <ResourcePath Condition="'$(BRCC_OutputDir)' == ''">$(__ResourcePath)</ResourcePath>
-    <ResourcePath Condition="'$(DCC_TranslatedResourcePath)' != ''">$(DCC_TranslatedResourcePath);$(ResourcePath)</ResourcePath>

     <NameSpace Condition="'DelphiNamespaceSearchPath'!=''">$(NameSpace);$(DelphiNamespaceSearchPath)</NameSpace>

Answer 2

Now I understand a source for this problem. Please upvote Sir Rufo as he put me in mind of the solution.

It is this: I was invoking DCC32.exe to compile packages (using .dpk, but no .dproj file, and not invoking msbuild to compile these packages). When I built these, I was not inserting the Debug DCU path to the head of the library path passed in via -I parameters to DCC32.exe.

Once the DCC32.exe package compilation Library Search path has the Debug DCU folders FIRST, it works.

If anyone is interested in such a package system, I am planning to open source this package build system, as part of a relaunch of the WANT project originally built by Juancarlo Anez, which I will probably call by a new name. I'll update this answer once a working demo of a component build system is available.

A brief outline of a working system to meet the requirements I asked in my question:

1. You will need a file (could be xml, ini, json file) that defines a list of packages to build.

2. You will need to invoke MSBUILD or DCC32.exe on each of these. You could write your own code, or you could use mine, which I will open source when I can.

3. You will need to include the Debug DCU DPROJ into the library path as the first items, ONLY when invoking the Debug item builds.

4. You will want to use the $(OURCOMPANYLIBRARYDCU)\$(Platform)\$(Config) macro in your project search paths and library paths.

5. In your Delphi IDE, you will want to hard-code $(OURCOMPANYLIBRARYDCU)\$(Platform)\Release as a path within the Library path.

6. In your Delphi IDE, you will want to hard-code $(OURCOMPANYLIBRARYDCU)\$(Platform)\Debug as a path within the Debug DCU path.