How do I set up TeamCity CI so that it unpacks Xamarin components?

Question

In Visual Studio everything works and a Components directory is created with the appropriate dlls. However, TeamCity is not able to retrieve the Android Support Library dlls because the trigger for the restore is a Xamarin VS plugin that runs when loading the solution. The equivalent of nuget package restore for Xamarin is xamarin-component. I have placed the xamarin-component.exe in my C:\Windows directory. To configure TeamCity, I prepended a Command Line build step with

Command executable: xamarin-component 
Command parameters: restore mysolution.sln

TeamCity runs as NT Authority\System. So using PsExec,

psexec -i -s %SystemRoot%\system32\cmd.exe

If I then run 'xamarin-component login'

INFO (login): Computed cookie jar path: C:\Windows\system32\config\systemprofile\.xamarin-credentials 
INFO (login): Computed cookie jar path: C:\Windows\system32\config\systemprofile\.xamarin-credentials 
INFO (login): Credentials successfully stored.

When I go to my solution in cmd and attempt the restore, I get an attempt to download the componet, and then a Json parsing error. This is the same error I get in TeamCity.

I get the error if I use 'Administrator' (which stores the credential in C:\Users\Administrator. Earlier when I was using my personal account, it did work. However, once I deleted the C:\Users\tim\AppData\Local\Xamarin\Cache\Components, the same issue emerged. Fiddler shows that rather than getting Json back (as we do when we enter an invalid token) we are getting a 302 redirect that says Object moved here. And here is the xamarin login page - obviously not Json.

Tried. 1. Set COOKIE_JAR_PATH to C:\Users\tim.xamarin-credentials - xpkg picks up but same error 2. Copy .xamarin-credentials from Config\system32 to D:\, set COOKIE_JAR_PATH to D:.xamarin-credentials - xpkg picks up but same error 3. Move .xamarin-credentials to C:\, set COOKIE_JAR_PATH - same error 4. Re-login in NT Authority with COOKIE_JAR_PATH to C:.xamarin-credentials - same error

My temporary idea now is to figure out where the NT Authority xamarin-component looks for Cache and put the files there.

C:\Windows\system32\config\systemprofile\AppData\Local\Xamarin\Cache\Components\xamandroidsupportv4-18-4.18.1.xam

The version of my xamarin-component is 0.99 - for 100, we try harder...

Answer 1

I’ve had trouble actually getting the cookie jar to load correctly from the system32 path. I think this is a path virtualization issue that I just don't understand well enough to make heads or tails of.

I ended up adding an environment variable that the tool will read from (I'm its principal author at Xamarin :-) that specifies the cookie jar path to read from, and this solved the problem for others using TeamCity. The environment variable is COOKIE_JAR_PATH.

You can set it from TeamCity's environment settings to point to a cookie jar path outside of the system32 profile directory (I think in my original testing, I put it in the root of the C: drive, but it can be anywhere, really).

Answer 2

As a hack, I copied the Cache folder from

C:\Users\tim\AppData\Local\Xamarin 

to

C:\Windows\system32\config\systemprofile\AppData\Local\Xamarin\

That bypassed communication with the Xamarin server.

Update. I suspect it might be a bad link or setup on their server side. When xamarin-component restore is called, a call is made to

GET /api/available_versions?alias=xamandroidsupportv4-18 HTTP/1.1

which returns "Object moved to here" where "here" is nowhere.

If you start Visual Studio after deleting the Cache and Components folder (next to the solution), Xamarin makes a call to

GET /api/download/xamandroidsupportv4-18/4.18.1 HTTP/1.0

which has a similar looking Object moved to, but this time it directs you to xamarin-components.s3.amazonaws.com/

GET /fdca922d2b77799fe208a08c9f3444fe/xamandroidsupportv4-18-4.18.1.xam HTTP/1.0

Perhaps something changed, or the available_versions API has changed.

Answer 3

Thanks very much for this question and your answers to it. I didn't really like the idea of storing an auth cookie on the build node or having to copy a cache there manually, so I came up with my own solution so I hacked around this problem with a quick Powershell script that mimics the behaviour of the xamarin-component.exe restore action:

param
(
    [Parameter(Mandatory=$true)]
    $authCookie,

    [Parameter(Mandatory=$true)]
    $componentDirectory,

    [Parameter(Mandatory=$true)]
    $project
)

[void]([System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression.FileSystem'))
$xml = [xml] $(cat $project);
$components = $xml.Project.ItemGroup.XamarinComponentReference | ? { $_.Include.Length -gt 0 } | % { $_.Include };

if (!(test-path $componentDirectory))
{
    echo "$componentDirectory didn't exist, so it was created.";
    [void](mkdir $componentDirectory);
}

foreach ($component in $components)
{
    $source = "http://components.xamarin.com/download/$component";
    $destination = "$componentDirectory\$component.zip";

    if (test-path $destination)
    {
        echo "$destination already exists, skipping...";
        continue;
    }

    echo "Downloading $component from $source to $destination...";

    $client = New-Object System.Net.WebClient
    $client.Headers.Add([System.Net.HttpRequestHeader]::Cookie, "XAM_AUTH=$authCookie");

    try
    {
        $client.DownloadFile($source, $destination);
    }
    catch
    {
        # The error message will be on one of these lines hopefully:
        write-error "Failed to download! Errors are below:";
        write-error $_
        write-error $_.Exception
        write-error $_.Exception.InnerException
        write-error $_.Exception.InnerException.InnerException
        exit 1;
    }

    if (!(test-path $destination))
    {
        write-error "$destination doesn't exist - the download must have failed!";
        exit 1;
    }

    echo "Decompressing $source to $componentDirectory"
    [System.IO.Compression.ZipFile]::ExtractToDirectory($destination, $componentDirectory)
    echo ""
}

echo "Done!";

The -authCookie parameter can be extracted from either the XAM_AUTH cookie in your browser or from the .xamarin-credentials "cookiejar" in your home directory. It's nice to have it parameterised like this so you can store it as a secret variable in TeamCity.

The componentDirectory parameter must be the full path to the component directory - it will be created if it doesn't exist.

The project parameter should be the path to your project that you want to restore packages for - if you have multiple projects that need this then you'll have to execute the script for each one. Don't specify your solution as it won't work.

Unfortunately, this isn't very resilient to Xamarin's whims - a simple API change could render this useless, so obviously the best solution is to wait for Xamarin to fix this. I e-mailed Xamarin support to complain about this problem but I don't imagine I'll get a timely response (they seem very very busy these days). I hope this is useful!