What are Visual Studio Code experiments?

Question

Today I was surprised to find an "Enable Experiments" option under VSCode's Workbench settings, turned on by default.

The setting's description is "Fetches experiments to run from a Microsoft online service" which seems rather vague to me. I tried googling this but didn't find any clear answers.

So, does anybody know what those "experiments" are and if it would probably be better to turn this off?

Answer 1

This is one of the case where using open-source software is a good idea. Because the source code of visual studio code is published in https://github.com/Microsoft/vscode. We could try to search in where the code would be used.

First, we could try to search the string Enable Experiments. And see, to which action the option is tied to. From there, I see that, the file src/vs/workbench/contrib/experiments/node/experimentService.ts is using it. Specifically, when trying to load an experiment in line 173

if (!product.experimentsUrl || this.configurationService.getValue('workbench.enableExperiments') === false) {

We see that, the code would check for "experiment URL". this could be seen in product.json which @Joey mentioned in the comment. In my case, the text looks like this.

"experimentsUrl": "https://az764295.vo.msecnd.net/experiments/vscode-experiments.json",

From there, we could see the content of the JSON file by making a GET request to that URL. And, it returns this (at least, at the time I make the request)

{
    "experiments": [
        {
            "id": "cdias.searchForAzure",
            "enabled": true,
            "action": {
                "type": "ExtensionSearchResults",
                "properties": {
                    "searchText": "azure",
                    "preferredResults": [
                        "ms-vscode.vscode-node-azure-pack",
                        "ms-azuretools.vscode-azureappservice",
                        "ms-azuretools.vscode-azurestorage",
                        "ms-azuretools.vscode-cosmosdb"
                    ]
                }
            }
        }
    ]
}

Based on the response, I could see that, it try to alter my search result if I search using "azure" key word. Which I tried, and the search result shows the 4 items there on top of the result search.

As to whether to disable it or not. On safe side (if you don't want for it to alter your experience using vscode) I think you would want to disable it. But, I don't think microsoft would do something crazy.

Answer 2

I just noticed this one and was curious about it as well. A search through the VS Code release notes finds one reference to it in July 2018. workbench.enableExperiments is listed as one of the settings for VS Code's "Offline mode": https://code.visualstudio.com/updates/v1_26#_offline-mode

The description of offline mode suggests that this settings is for "A/B experiments":

To support this offline mode, we have added new settings to turn off features such as automatic extension update checking, querying settings for A/B experiments, and fetching of online data for auto-completions.

As mentioned by others, the source code for VS Code shows this setting being used in experimentService.ts: https://github.com/microsoft/vscode/blob/93bb67d7efb669b4d1a7e40cd299bfefe5e85574/src/vs/workbench/contrib/experiments/common/experimentService.ts

If you look at the code of experimentService.ts, the stuff it's fetching seems to be related to extension recommendations, notifications about new features, and similar things. So it looks like the experiment service is for fetching data to do A/B testing of feature and extension recommendations to users.