Test cases not running after upgrading to 1.42.0 in Azure DevOps
erenkocyigit opened this issue · comments
After upgrading Playwright from version 1.40.0 to 1.42.0, my test cases are no longer running in Azure DevOps. No error codes are displayed, but the test suite simply hangs and does not execute any of the tests.
In my local everything is working as expected
I have checked the Playwright documentation and release notes for any breaking changes between 1.40.0 and 1.42.0, but I could not find anything that seems to be relevant to this issue.
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Autofac.Extensions.DependencyInjection" Version="8.0.0" />
<PackageReference Include="coverlet.collector" Version="3.2.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Docker.DotNet" Version="3.125.12" />
<PackageReference Include="JetBrains.Annotations" Version="2022.3.1" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Options" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="7.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.4.0" />
<PackageReference Include="Microsoft.Playwright" Version="1.42.0" />
<PackageReference Include="Microsoft.Playwright.NUnit" Version="1.42.0" />
<PackageReference Include="NUnit.Analyzers" Version="3.5.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Polly" Version="7.2.3" />
<PackageReference Include="SpecFlow" Version="3.9.74" />
<PackageReference Include="SpecFlow.Autofac" Version="3.9.74" />
<PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.9.57" />
<PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.9.74" />
<PackageReference Include="SpecFlow.xUnit" Version="3.9.74" />
<PackageReference Include="xunit" Version="2.7.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.7">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<None Remove="specflow.json" />
<Content Include="specflow.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<None Remove="appsettings.json" />
<Content Include="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
<ItemGroup>
<Compile Update="TestScenarios\Features\ApiTokens.feature.cs">
<DependentUpon>ApiTokens.feature</DependentUpon>
</Compile>
</ItemGroup>
Additional Information:
Added test output. The output shows that the tests are started and one test file is found, but it hangs without any errors or test results.
Test Output: (1.42.0 version) TIME OUT FAILED
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.7+8f2703126a (64-bit .NET 7.0.10)
[xUnit.net 00:00:00.08] Discovering: Manager.UITests.PW
[xUnit.net 00:00:00.16] Discovered: Manager.UITests.PW
[xUnit.net 00:00:00.16] Starting: Manager.UITests.PW
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\SpecFlow.Autofac.SpecFlowPlugin.dll
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\Manager.UITests.PW.dll
##[error]The operation was canceled.
Test Output: (1.40.0 version) PASSED
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.7+8f2703126a (64-bit .NET 7.0.10)
[xUnit.net 00:00:00.09] Discovering: Manager.UITests.PW
[xUnit.net 00:00:00.18] Discovered: Manager.UITests.PW
[xUnit.net 00:00:00.18] Starting: Manager.UITests.PW
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\SpecFlow.Autofac.SpecFlowPlugin.dll
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Users\bcknd-win-bld-agt\buildAgents\agent1\_work\8\s\tests\Manager.UITests.PW\bin\Debug\net7.0\Manager.UITests.PW.dll
-> Using specflow.json
Passed Hint when regenerating Channel Access [11 s]
Passed Regenerate PIN Code [10 s]
Passed Matching stereo input sources(leftInput: "Analog 1 with gain (In 1)", rightInput: "Analog 2 with gain (In 2)", exampleTags: []) [9 s]
Passed....
Possible Causes:
An error is occurring during the tests but no error message is displayed.
There is an issue with a plugin or test library.
I would appreciate any help in diagnosing and resolving this issue. I am happy to provide any additional information that may be helpful.
and here is my azure-pipelines.yml file:
parameters:
- name: registryEndpoint
type: string
default: "*******"
trigger: none
pr: none
schedules:
- cron: "0 3 * * *"
displayName: Nightly UI Tests
branches:
include:
- develop
stages:
- stage: run_ui_tests
displayName: "Run UI Tests"
condition: |
or(
eq(variables['Build.Reason'], 'Schedule'),
eq(variables['Build.Reason'], 'Manual')
)
jobs:
- job: run_ui_tests
displayName: "Run UI Tests"
timeoutInMinutes: 5 # we might need to increase this if the test suite becomes big and slow
cancelTimeoutInMinutes: 1
pool:
name: *******
demands: UI.Testing
steps:
- checkout: self
submodules: "recursive"
persistCredentials: true
- task: NuGetAuthenticate@1
inputs:
forceReinstallCredentialProvider: true
- task: UseDotNet@2
displayName: "Install or update dotnet sdk"
inputs:
packageType: "sdk"
useGlobalJson: true
- task: Docker@2
displayName: Docker login
inputs:
command: login
containerRegistry: ${{ parameters.registryEndpoint }}
- task: DotNetCoreCLI@2
displayName: New Manifest for tool
inputs:
command: custom
custom: 'new '
arguments: tool-manifest --force
- task: DotNetCoreCLI@2
displayName: Install Playwright CLI
inputs:
command: custom
custom: 'tool '
arguments: install Microsoft.Playwright.CLI
- task: DotNetCoreCLI@2
inputs:
command: 'build'
- task: DotNetCoreCLI@2
displayName: Run Playwright Install
inputs:
command: custom
custom: 'tool '
arguments: run playwright install
- task: DockerCompose@0
displayName: Pull images
inputs:
action: Run a Docker Compose command
containerRegistryType: Container Registry
dockerRegistryEndpoint: ${{ parameters.registryEndpoint }}
projectName: "uitests_manager_pw"
dockerComposeFile: $(Build.SourcesDirectory)/tests/Manager.UITests.PW/installation-bundle/docker-compose.yml
dockerComposeCommand: pull
- script: docker pull *******
displayName: Pull DockerCS:latest
- task: DockerCompose@0
displayName: Start Manager
inputs:
action: Run a Docker Compose command
containerRegistryType: Container Registry
dockerRegistryEndpoint: ${{ parameters.registryEndpoint }}
projectName: "uitests_manager_pw"
dockerComposeFile: $(Build.SourcesDirectory)/tests/Manager.UITests.PW/installation-bundle/docker-compose.yml
dockerComposeCommand: up -d
detached: true
- task: DotNetCoreCLI@2
displayName: "Run UI tests in Chrome"
inputs:
command: test
arguments: '--collect:"XPlat Code Coverage" --verbosity detailed'
projects: "$(Build.SourcesDirectory)/tests/Manager.UITests.PW"
publishTestResults: true
testRunTitle: "UI Tests in Chrome"
env:
UiTesting_UserCredentialOptions__QA1Password: *****
UiTesting_UserCredentialOptions__QA2Password: *****
UiTesting_UserCredentialOptions__QA3Password: *****
UiTesting_UserCredentialOptions__QA4Password: *****
UiTesting_EnvironmentOptions__Browser: Chrome
UiTesting_EnvironmentOptions__DisplayBrowser: true
- task: DotNetCoreCLI@2
displayName: "Run UI tests in Firefox"
inputs:
command: test
arguments: '--collect:"XPlat Code Coverage"'
projects: "$(Build.SourcesDirectory)/tests/Manager.UITests.PW"
publishTestResults: true
testRunTitle: "UI Tests in Firefox"
env:
UiTesting_UserCredentialOptions__QA1Password: *****
UiTesting_UserCredentialOptions__QA2Password: *****
UiTesting_UserCredentialOptions__QA3Password: *****
.......
From the error log it appears to be failing outside of playwright while loading some plugins.
In my local everything is working as expected
Then it has to do with your environment. Unfortunately we can't help much without a repro. Feel free to file a new bug if you have a reproduction.