Building solution made up of different projects located in different projects in DevOps

Question

I've just started playing around with Azure DevOps as I want to migrate all my personal projects over to it as I really like using it at the office but I've never had to set it up and I'm having my first problem and hope someone can help.

I created a project in DevOps and I pushed my main solution under into its repo. All good. This solution has a dependency library from another project so I created another project in DevOps and pushed this library to its repo as well.

My main solution (.sln) which includes the project for my main project also includes a reference to library and compiling the solution locally works as expected but when I try this in DevOps, I get the following error:

D:\a\1\s\MyProject.sln.metaproj(0,0): Error MSB3202: The project file  
"D:\a\1\s\..\..\..\Libraries\MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj"
was not found.

and as a result of that I get a bunch of other errors similar to the below as it failed to compile the library that was referenced in the solution but is in a completely different project in DevOps:

MyProject\Controllers\MembersController.cs(8,7): Error CS0246: The type or namespace name 
'MyLibraryAPI' could not be found (are you missing a using directive or an assembly 
reference?) 

Clearly can't find the project when in DevOps but how can I resolve this? I still want to make sure that this library gets compiled every time I compile my main project.

I've only read a little bit of YAML and I assume that I may have to change something in there but I'm not sure what to be honest. Anyway, I'll wait for someone's feedback and I hope the above makes sense and that someone can clarify how I can resolve this problem.

Thanks.

UPDATE-1

Thanks for the detailed feedback @MerlinLiang. I have followed the instruction from the link provided by @FrankAlvaro and yours and they are indeed the same but I appreciate the additional information as I did face this problem regarding the error:

The Git repository with name or identifier {repos name}does not exist
or you do not have permissions for the operation you are attempting

But I still haven't resolved the issue. I'm now getting the following error:

 ##[error]d:\a\1\s\MyProject\MyProject.sln.metaproj(0,0): 
 Error MSB3202: The project file "d:\a\1\s\MyProject\..\..\..\Libraries\
 MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj" was not found.

Remember that MyProject is my main solution which has a reference to my MyLibrary but it's weird to see that it's trying to put MyLibrary\MyLibraryAPI inside the MyProject as locally they are in different location, so while I understand it would different in Azure DevOps, I would have thought that when checking out the project, it would have tried to respect the folder structure used locally, no?

I don't know if this will help solve my problem but this is how the projects are defined in my .sln file:

Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MyProject",
"MyProject\MyProject.csproj", "{CD02E7AD-97EE-4831-A18D-A57E2CC92E08}"
EndProject

Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MyLibraryAPI", 
"..\..\..\Libraries\MyLibrary\MyLibraryAPI\MyLibraryAPI.csproj", 
"{82A7849D-22EF-4889-A7A6-2AE7D3F98F77}"
EndProject

UPDATE-2

I've tried what @MerlinLiang suggested by setting a path but I'm not sure I'm doing right as it still isn't working as expected. Here's what I've done:

• Set a path for MyProject

• Set a path for MyLibraryAPI

  variables: solution: '**/*.sln' buildPlatform: 'Any CPU' buildConfiguration: 'Release' MyProjectCheckoutPath: 'Repos\Work\Company\MyProject' MyLibraryApiCheckoutPath: 'Repos\Libraries\MyLibrary'

  steps:

  • checkout: self path: '$(MyProjectCheckoutPath)'
  • checkout: MyLibrary path: '$(MyLibraryApiCheckoutPath)'

Note: MyLibrary is my solution folder which contains 2 folders, one of them called MyLibraryAPI which contains MyLibraryAPI.csproj while the other is just a console application used to test the library but isn't required. I'm not sure how to specifically specify to use the MyLibraryAPI project only as it is contained within a repo that contains 2 folders. I hope this makes sense.

When I specify the above path in YAML and I run the pipeline, I get the following warnings:

Repository is current at 'D:\a\1\s\My Project', move to 
'D:\a\1\Repos\Work\Company\MyProject'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\Repos\Work\Company\MyProject'.

Repository is current at 'D:\a\1\s\MyLibrary', move to 
'D:\a\1\Repos\Libraries\MyLibrary'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\Repos\Libraries\MyLibrary'.

As you can see, both are giving a warning when I'm trying to use the paths defined in my variables.

As these are only warning, it continues with the build but eventually I get an error in the VSBuild section:

##[error]Solution not found using search pattern 'D:\a\1\s\**\*.sln'.

I've tried changing the solution variable in my YAML to MyProject

variables:
   solution: '**/MyProject.sln'

But I still get an error as the path is clearly different:

##[error]Solution not found using search pattern 
'D:\a\1\s\**\MyProject.sln'.

I'm not sure if I'm defining the path correctly as I'm getting warnings but assuming, it somehow did work, the solution path is still a problem.

Any ideas?

UPDATE-3

For a second, I thought I had it sorted as it took longer than usual for VSBuild to fail but to no avail. I changed the solution path to use the full path of MyProject to:

but ended up getting the following error:

"D:\a\_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\vswhere.exe" 
-version [16.0,17.0) -latest -format json
"C:\Program Files (x86)\Microsoft Visual 
Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe" 
"D:\a\1\s\Repos\Work\Company\MyProject\MyProject.sln" /nologo /nr:false 
/dl:CentralLogger,"D:\a\_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\
Microsoft.TeamFoundation.DistributedTask.
MSBuild.Logger.dll";"RootDetailId=7d46ac00-083a-434d-9729- 
be159b80eea4|SolutionDir=D:\a\1\s\Repos
\Work\Company\MyProject"*ForwardingLogger,"D:\a\_tasks\
VSBuild_71a9a2d3-a98a-4caa-96ab- 
affca411ecda\1.166.0\ps_modules\MSBuildHelpers\
Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll"
/p:DeployOnBuild=true /p:WebPublishMethod=Package 
/p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true 
/p:PackageLocation="D:\a\1\a" /p:platform="Any CPU" 
/p:configuration="Release" /p:VisualStudioVersion="16.0" 
/p:_MSDeployUserAgent="VSTS_aa0b7157-846e-49d9-8668-38efaf1e9745_build_5_0"
MSBUILD : error MSB1009: Project file does not exist.
Switch: D:\a\1\s\Repos\Work\Company\MyProject\MyProject.sln
##[error]Process 'msbuild.exe' exited with code '1'.
Finishing: VSBuild

UPDATE-4:

I've tried to create the directory via a Powerscript as suggested in

Warning when fetching additional repositories

but this didn't solve the problem. I've also added a script to list the content of the directory where it is supposed to have extracted the files from the repositories and both directories are showing as empty.

Here is my full YAML:

resources:
 repositories:
   - repository: MyLibrary
     type: git 
     name: MyLibrary/MyLibrary

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
  solution: 'Repos\Work\Company\MyProject\MyProject.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'
  myLibraryCheckoutPath: 'Repos\Libraries\MyLibrary'
  myProjectCheckoutPath: 'Repos\Work\Company\MyProject'

steps:
- task: PowerShell@2
  displayName:  "Create repository folder $(myProjectCheckoutPath)"
  inputs:
    targetType: inline
    script: |
      New-Item -Path . -Name "$(myProjectCheckoutPath)" -ItemType "directory"
    pwsh: true

- checkout: self
  path: '$(myProjectCheckoutPath)'

- script: dir
  workingDirectory: $(myProjectCheckoutPath)
  displayName: List contents of a folder for MyProject

- task: PowerShell@2
  displayName:  "Create repository folder $(myLibraryCheckoutPath)"
  inputs:
    targetType: inline
    script: |
      New-Item -Path . -Name "$(myLibraryCheckoutPath)" -ItemType "directory"
    pwsh: true

- checkout: MyLibrary
  path: '$(myLibraryCheckoutPath)'

- script: dir
  workingDirectory: $(myLibraryCheckoutPath)
  displayName: List contents of a folder for MyLibrary

- task: NuGetToolInstaller@1

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package 
                  /p:PackageAsSingleFile=true 
                  /p:SkipInvalidConfigurations=true 
                  /p:PackageLocation="$(build.artifactStagingDirectory)"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

# - task: VSTest@2
#   inputs:
#     platform: '$(buildPlatform)'
#     configuration: '$(buildConfiguration)'

UPDATE 5

I'm still failing to understand what's going on.

When I have 2 projects being checked out, I get the following:

- Directory created: D:\a\1\s\Repos\Work\Company\MyProject
- Directory created: D:\a\1\s\Repos\Libraries\MyLibrary

And I get the following output in the logs:

- Repository is current at 
  'D:\a\1\s\MyProject', move to 
  'D:\a\1\s\Repos\Work\Company\MyProject' 
  (##[warning]Unable move and reuse existing repository to required location)

- Repository is current at 
  'D:\a\1\s\MyLibrary', move to 
  'D:\a\1\s\Repos\Libraries\MyLibrary' 
  (##[warning]Unable move and reuse existing repository to required location)

and as you can see if still fails to pull the relevant repos and move them to the relevant directory.

Yet, when I just pull a single repo i.e. MyProject, the following is outputted to the log:

Repository is current at 
'D:\a\1\s', move to 'D:\a\1\s\Repos\Work\Company\MyProject'. 
Repository will be located at 
'D:\a\1\s\Repos\Work\Company\MyProject'.

And this works as expected.

In both scenarios, the root is 'D:\a\1\s' but when extracting one, the current directory is the same but when used with 2, it is the root directory and the project name. You would think this should be ok but it clearly isn't since it's failing to move the files.

One thing I don't get as well is according to Check out multiple repositories in your pipeline, there is a Note that states:

If you are using default paths, adding a second repository checkout step changes the default path of the code for the first repository. For example, the code for a repository named tools would be checked out to C:\agent_work\1\s when tools is the only repository, but if a second repository is added, tools would then be checked out to C:\agent_work\1\s\tools. If you have any steps that depend on the source code being in the original location, those steps must be updated.

What is meant by

"If you have any steps that depend on the source code being in the original location, those steps must be updated"

What steps and update them to what??

I've read so much documentation, I've been googling for hours and try so many YAML combinations but to no avail. I just don't seem to able to find a solution. Sorry if I'm missing something but I just don't understand why it works with one repo and doesn't work with 2.

Answer 1

I eventually figured out the missing piece!

While the Microsoft Azure DevOps documentation is excellent overall, I feel, it totally failed to explain the multi-checkout requirements, not to mention, a clear flaw in the system.

Note: My answer is based on the path requirements from my question.

In short, when checking out multiple projects, it simply doesn't create the source folders for each of the projects that are being checked out.

2 Issues:

1) Misleading warning message:

Repository is current at 'D:\a\1\s\MyProject', move to 
'D:\a\1\s\Repos\Work\Company\MyProject'.
##[warning]Unable move and reuse existing repository to required location.
Repository will be located at 'D:\a\1\s\Repos\Work\Company\MyProject'.

Why not display what the actual problem is i.e.

The folder 'D:\a\1\s\MyProject' not found.

and instead of creating a warning, it should create an error and stop the process.

2) Does not automatically create the source folder: Maybe there is an underlying reason behind this but I can't think of what it might be, but I personally think it should always create the source directory where the files are going to be checked out before they are moved.

Ideally, it should just create them directly in the specified path. I don't get either why you have to create it in one folder, only to move it to another one.

So to resolve your problem, simply make sure that you create the source folder for each of your checkouts and everything should work as expected.

My final script ended up looking like this:

resources:
 repositories:
   - repository: MyProject
     type: git 
     name: My Project/My Project
   - repository: MyLibrary
     type: git 
     name: MyLibrary/MyLibrary

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

  myProjectCheckoutSource: 'My Project'
  myProjectCheckoutTarget: 's/Repos/Work/Company/MyProject'
  myProjectCheckoutTargetFolder: '$(Agent.BuildDirectory)\s\Repos\Work\Company\MyProject'

  myLibraryCheckoutSource: 'MyLibrary'
  myLibraryCheckoutTarget: 's/Repos/Libraries/MyLibrary'
  myLibraryCheckoutTargetFolder: '$(Agent.BuildDirectory)\s\Repos\Libraries\MyLibrary'  

steps:

###################
# Pull My Project #
###################

# Create checkout source folder as when using multiple checkouts, 
# it doesn't get created automatically.
- task: PowerShell@2
  displayName: "Create repository folder: $(myProjectCheckoutSource)"
  inputs:
    targetType: inline
    script: |
      New-Item -Path . -Name "$(myProjectCheckoutSource)" -ItemType "directory"
    pwsh: true

# The target checkout folder defined in the 'path' is created automatically
- checkout: self
  displayName: checking out $(myProjectCheckoutSource) to $(myProjectCheckoutTarget)
  path: $(myProjectCheckoutTarget)

##################
# Pull MyLibrary #
##################

# Create checkout source folder as when using multiple checkouts, 
# it doesn't get created automatically.
- task: PowerShell@2
  displayName: "Create repository folder: $(myLibraryCheckoutSource)"
  inputs:
    targetType: inline
    script: |
      New-Item -Path . -Name "$(myLibraryCheckoutSource)" -ItemType "directory"
    pwsh: true

# The target checkout folder defined in the 'path' is created automatically
- checkout: myLibrary
  displayName: checking out $(myLibraryCheckoutSource) to $(myLibraryCheckoutTarget)
  path: $(myLibraryCheckoutTarget)

- task: NuGetToolInstaller@1

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package 
    /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true 
    /p:PackageLocation="$(build.artifactStagingDirectory)"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'