Running concurrent jobs on Jenkins with fastlane/scan puts CoreSimulatorService in a deadlock state

[Fabien Lydoire]

Duplicate of fastlane/fastlane#14528 (comment)

Issue Description

We rely on fastlane/scan to perform the tests on our Jenkins CI server.

• When running non concurrent jobs (1 Jenkins job at a time), we got no issue.
• When running concurrent jobs (Multiple fastlane/scan at the same time), jobs get blocked.

We rely on fastlane-plugin-simctl to execute xcrun simctl commands.

We execute up to 12 concurrent Jenkins jobs.

Most of the time, the Jenkins jobs run fine.
Sometimes the xcrun simctl commands are not responding at some point, so the jobs are stuck during simulator creation or deletion.

The process com.apple.CoreSimulator.CoreSimulatorService seems stuck in a deadlock.
We ran sample on the process and the sample analysis did exhibit :

Dispatch Thread Soft Limit: 64 reached in 8117 of 8121 samples -- too many dispatch threads blocked in synchronous operations

Killing the simulators does not end the deadlock. We have to
killall -9 com.apple.CoreSimulator.CoreSimulatorService

Command executed

private_lane :jenkins_test do |options|
	setup_jenkins(result_bundle: false)
	sh 'defaults write com.apple.iphonesimulator ConnectHardwareKeyboard 0'
	simctl(type: 'iPhone Xʀ',
		   block: lambda { |action, device|
		   action.scan(
					   skip_detect_devices: true,
					   destination: ["id=#{device.udid}"],
					   workspace: "#{Dir.pwd}/MyApp.xcworkspace",
					   scheme: options[:scheme],
					   xcargs: "-UseModernBuildSystem=NO",
					   clean: true,
					   code_coverage:true,
					   derived_data_path: '.',
					   output_directory: "./test_output/",
					   fail_build: false,
					   result_bundle: "TestResults" # To generate test reports
					   )
		   })
end

$ bundle exec fastlane jenkins_test scheme:"MyApp" configuration:"Debug"

Environment

🚫 fastlane environment 🚫

Stack

Key	Value
OS	10.14.4
Ruby	2.3.7
Bundler?	false
Git	git version 2.20.1 (Apple Git-117)
Installation Source	/usr/local/bin/fastlane
Host	Mac OS X 10.14.4 (18E226)
Ruby Lib Dir	/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib
OpenSSL Version	LibreSSL 2.6.5
Is contained	false
Is homebrew	false
Is installed via Fabric.app	false
Xcode Path	/Applications/Xcode.app/Contents/Developer/
Xcode Version	10.2

System Locale

Variable	Value
LANG	en_US.UTF-8	✅
LC_ALL	en_US.UTF-8	✅
LANGUAGE	en_US.UTF-8	✅