[REGRESSION]: vstest@3 fails with Visual Studio 18.3.0 installed

[jessehouwing]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

vstest@3

Breaking task version

3.272.0

Last working task version

3.271.0

Regression Description

The new vstest version parsing fails for vstest version 18.3.0, which seems to return a different format version string.

The test fails to run and render error:

Error: Unexpected version number detected for vstest.console.exe

The reported version is:

VSTest version 18.3.0 (x64)

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

Error: Unexpected version number detected for vstest.console.exe: VSTest version 18.3.0 (x64)

Full task logs with system.debug enabled

No response

Repro steps

[jessehouwing]

Related:

• [BUG]: VsTest versionfinder.ts uses deprecated wmic #21123
• Fixing the Vstest issue with Powershell GetItemProperty #21480

[virekumar]

Hi @jessehouwing,

I was not able to reproduce issue with self hostest agents having 272 version of VsTest task.
Can you please share logs and pipeline task definition?
Please remove any specific details from uploaded files before sharing. If you want to share privately, please share at virekumar@microsoft.com

Also VsTestVersion we provide high level version only like 18.0, you seems to be using minor version which we are not able to pick at Task. We have specific checks like 18.0, 17.0. Let me know if error gets fixed with this.

[jessehouwing]

This is set with version latest. The new code in the linked PR then tries to read the exact resolved version from the binary to do feature-detection. It's not an input provided by the user.

The error is thrown here:

azure-pipelines-tasks/Tasks/VsTestV2/versionfinder.ts

Lines 55 to 58 in e9c9b1a

	if (isNaN(majorVersion) || isNaN(minorVersion) || isNaN(patchNumber)) {
	tl.warning(tl.loc('UnexpectedVersionNumber', output));
	throw new Error(tl.loc('UnexpectedVersionNumber', output));
	}

After resolving the requested version range (from the input) to the file on disk and then using that to detect that minor and build versions (in this case 18.3.0).

It looks like some older builds (or files installed by the testplatforminstaller) do not return the expected version value.

[virekumar]

I have reverted this buggy version.
Once we have correct repro details, I will add a fix.