How to test new Azure DevOps extension version before publishing it for everyone

Question

I am using Azure DevOps (not TFS on-prem) and have my extension already published as public and many people using it. How can I push out a new version for just myself (or specific organizations) to test it, before making it public for everyone?

I know that I could roll all of the tasks in my Build/Release pipeline extension to v2, so if there are issues it wouldn't break in my users' systems right away until they flipped the task to use v2. However, I typically only want to increment the task version when there is a breaking change to it. Also, that still wouldn't address the issue of "I don't want ANY users to use the new version until I've tested it out first", as that could result in problems for my users and them giving bad ratings/reviews.

My initial thought was to flip the galleryFlags attribute in the vss-extension.json file from "Public" to "Preview" and push a new version, but I'm not certain if that will remove my extension from the marketplace, or if it will just publish the new version a "Preview" and leave the existing version available in the marketplace.

Before migrating to Azure DevOps, I was able to install new versions on the extension in our on-prem TFS instance from a local .vsix file without needing to publish them to the marketplace. It seems that running in the cloud does not offer this feature though, and Azure DevOps can only install extensions from the marketplace.

I posed a new GitHub issue to have the MS documentation updated to give some instructions or recommendations on this. I also found this similar SO post, and this one, and the recommendation there was to create a 2nd publisher account and publish the same extension as private and share it with my organization. This would work, but seems very hacky to have to setup 2 separate publishing accounts and muck with the extension IDs before publishing each time for testing new versions of extensions.

I've created this new SO post (instead of following up on those existing ones) at the request of Microsoft so they can comment on it directly here.

Answer 1

When testing a new version of your extension, you need to use either use a different ExtensionID or a different PublisherID. And the test extension must be marked public: false.

There are multiple ways to make this process easy. I personally use the Azure DevOps Extension Tasks in different ways.

For my own private extension I have a build definition that builds either the public version or the private version. In the past I used to have 2 separate build definitions, but with YAML available I've started to consolidate this into a single definition. extensionTag is appended to the existing extensionId.

steps:
- task: ms-devlabs.vsts-developer-tools-build-tasks.package-extension-build-task.PackageVSTSExtension@1
  displayName: 'Package Extension: $(Build.SourcesDirectory)'
  inputs:
    rootFolder: '$(Build.SourcesDirectory)'
    outputPath: '$(Build.BinariesDirectory)\vsix\jessehouwing.azure-pipelines-snyk-task.vsix'
    outputVariable: CreateExtension.OutputPath
    publisherId: jessehouwing
    extensionId: 'vsts-snyk'
    extensionVersion: '$(Build.BuildNumber)'
    updateTasksVersion: true
    updateTasksVersionType: patch
    extensionVisibility: public

- task: ms-devlabs.vsts-developer-tools-build-tasks.publish-extension-build-task.PublishExtension@1
  displayName: 'Publish Extension Private'
  inputs:
    connectedServiceName: 'Jesse Houwing'
    fileType: vsix
    vsixFile: '$(CreateExtension.OutputPath)'
    extensionTag: '-develop'
    extensionVisibility: private
  condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/master'))

- task: ms-devlabs.vsts-developer-tools-build-tasks.publish-extension-build-task.PublishExtension@1
  displayName: 'Publish Extension Public'
  inputs:
    connectedServiceName: 'Jesse Houwing'
    fileType: vsix
    vsixFile: '$(CreateExtension.OutputPath)'
    extensionVisibility: public
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

I use the conditions to trigger either the public or the private publish function.

The end result looks like this in my publisher:

On the ALM Rangers account we use a build definition that builds a single vsix during build and then we use binary promotion to publish the vsix with different overrides. You don't need to use a different publisher in this case, but the Rangers do. The reason for this is that the Rangers used to publish to the MsDevDiv publisher and Microsoft didn't want to give every contributor access to that publisher and all of its extensions. So the separate publisher is used more to separate the concerns of developing the extension from providing support, answering questions and responding to reviews.

In order to test I publish the extension to a different Azure DevOps organisation. This is because you can't have two extensions installed to the same Azure DevOps organisation if both extensions contain the same Build Task Ids. In my case I use dev.azure.com/jessehouwing to build my extensions and use dev.azure.com/jessehouwing-dev to test the changes before making them publicly available.

For some extensions I publish a second private extension for early adopters:

• Extension ID: jessehouwing.snyk-develop privately shared to jessehouwing-dev for testing.
• Extension ID: jessehouwing.snyk-canary privately shared to a couple of select users for early adopters.
• Extension ID: jessehouwing.snyk for public use.

A couple of my customers have a special case where they work on an extension pack with multiple developers at the same time. To not have to provide each developer with a separate Azure DevOps organisation and build agents, they publish the test and public extension to a single account. In this case:

• Extension ID: publisher.extension private for standard usage.
• Extension ID: publisher.extension-branch, private, preview for internal development and canary releases. There can be multiple of these active at the same time.

To allow this each build task must have a unique Task ID for the Build tasks in in the extension. The Azure DevOps Extension Tasks have a special feature to generate unique IDs based on the publisher, extension-id, taskname. This feature is detailed in these release notes.

I recently presented at the MVP summit on these usage patterns. The presentation is shared here.

If you want to roll your own scripts then you can follow the following pattern:

• vss-extension.json - store all common properties of the extension in this file. Don't specify extensionid nor galleryflags nor public.
• vss-extension-test.json - store the values unique to the test version. These include: extensionid, galleryflags: preview, public: false.
• vss-extension-release.json - store values unique to release version. These include: extensionid, galleryflags: public, public: true.

Then invoke:

// deploy test
tfx extension publish --manifest-globs vss-extension.json vss-extension-test.json
// deploy release
tfx extension publish --manifest-globs vss-extension.json vss-extension-release.json

To publish the combined manifests.

Or use an override manifest:

• vss-extension.json - store all the details for the public extension
• vss-extension-override-test.json - store a json-patch file with the values you want to override. Again: extensionid, galleryflags: preview, public.

Then use

// deploy test
tfx extension publish --manifest-globs vss-extension.json --override-file vss-extension-override-test.json
// deploy release: 
tfx extension publish --manifest-globs vss-extension.json

If you're rolling your own scripts, then you can use vsts-bump to auto-increase the version of your build tasks.