beringresearch / macpine

Whenever I try to create an instance I am getting the following error message:

qemu-system-x86_64: Property 'host-x86_64-cpu.pdpe1gb=on' not found

I also tried to use -a aarch64 but this time I got ssh handshake failed error. I rather prefer to use host CPU (x86_64) though anyways. Do I have to enable cpu.pdpe1gb feature somehow or there's something else that's causing the issue. Any pointers would be appreciated!

My host machine is: MacBook Pro 2019 (Intel model)

Hi- thanks for raising this. Could you check your qemu version number?

Hi- thanks for raising this. Could you check your qemu version number?

Sure, here you go:

❯ qemu-system-x86_64 --version
QEMU emulator version 7.2.1
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

❯ brew upgrade qemu
==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################## 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
######################################################################## 100.0%
Warning: qemu 7.2.1 already installed

Yep, that's definitely up to date. Interesting! I don't personally have an Intel mac to test on. Some intel architectures don't support 1GB pages -- I wonder if qemu makes that show up as this "property not found" error.

The ssh handshake error could be related to qemu not parsing or handling remaining options and therefore not opening the port for SSH forwarding.

Thanks! I’ll be able to take a look at this, but will be early-ish next week.

I suspect this may be due to a potential bug in how qemu cpuType string is constructed for host cpus using x86 (

macpine/qemu/ops.go

Line 282 in d65fa84

cpuType := map[string]string{

)

But will investigate in greater detail.

I played a bit with those options.

I tried reverting changes been made using those commits:

It worked with both host,-pdpe1gb and host values, but not with the host,-pdpe1gb=on value which is the current state.

I also tried with the host,pdpe1gb=on value, which also worked.

Flags	Works?	pdpe1gb support
`host,-pdpe1gb=on`	No	N/A
`host`	Yes	Yes
`host,pdpe1gb`	Yes	Yes
`host,pdpe1gb=on`	Yes	Yes
`host,pdpe1gb=off`	Yes	No

In conclusion, I am not sure how this change may affect the current user base. Also, I am not that good with qemu stuff. So, I am leaving to you guys to choose what next needs to be done.

if you use host,pdpe1gb (the second in the table -- thanks for such a thorough report!) and then ssh into the guest and run cat /proc/cpuinfo | grep -i pdpe1gb in the guest, does the guest indicate it supports 1GB pages?

Same question for the other working combinations in the table?

Same question for the other working combinations in the table?

I updated the table with regarding pdpe1gb support information for each configuration. Looks like unless explicitly disabled with pdpe1gb=off flag it's enabled by default.

Awesome! It looks like the flag format may be incorrect in macpine, and while the commit that fixed some x64 architectures (which didn't support 1GB pages, I guess?) broke it for x64 machines which do support it.

@idroz when you get a chance, we should figure out when 1GB pages need to be disabled (if ever any more?) and use the =off format rather than the - format to achieve it.

Thanks again @themisir for a truly excellent bug report with the additional detail.

@themisir I've created a pull request #123 . Would you be able to test?

It looks like somewhere along the line "x86_64_host": "host,-pdpe1gb" was overwritten with "x86_64_host": "host,-pdpe1gb=on", which is clearly not right :)

I've reverted this to explicitly say "host,pdpe1gb=off"

Intel host CPUs on Macs do not seem to support 1gb pages. See #63 for more info on this.

@idroz Per the table above it looks like maybe @themisir's modern (2019 macbook pro) intel machine can run guests with 1GB page support? Possibly only older intel cpus can't?

1GB pages can be a significant optimization for VMs as they spend less time in emulated VMM, depending on their memory usage. If it doesn't need to be disabled it'd be ideal to have it on when available.

Is there a way we can determine if the x86_64 host supports 1GB pages for VMs, and use it if it does?

@maxzinkus looking into it now.

@themisir could you run the following on your host to check if it supports supper pages (mac's version of huge pages)

echo "#include <mach/vm_statistics.h>" | gcc -dM -E - | grep VM_FLAGS_SUPERPAGE_SIZE_

Host

❯ echo "#include <mach/vm_statistics.h>" | gcc -dM -E - | grep VM_FLAGS_SUPERPAGE_SIZE_
#define VM_FLAGS_SUPERPAGE_SIZE_2MB (SUPERPAGE_SIZE_2MB<<VM_FLAGS_SUPERPAGE_SHIFT)
#define VM_FLAGS_SUPERPAGE_SIZE_ANY (SUPERPAGE_SIZE_ANY << VM_FLAGS_SUPERPAGE_SHIFT)

VM

localhost:~# cat /proc/cpuinfo | grep -i pdpe1gb
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase bmi1 avx2 smep bmi2 erms rdseed adx clflushopt xsaveopt xgetbv1 arat
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase bmi1 avx2 smep bmi2 erms rdseed adx clflushopt xsaveopt xgetbv1 arat
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase bmi1 avx2 smep bmi2 erms rdseed adx clflushopt xsaveopt xgetbv1 arat
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase bmi1 avx2 smep bmi2 erms rdseed adx clflushopt xsaveopt xgetbv1 arat

I also tested with host,pdpe1gb=off previously, which did worked. But, 1gb pages were not enabled (see above table).

On 2021 M1 and macOS Ventura, same thing:

$ echo "#include <mach/vm_statistics.h>" | gcc -dM -E - | grep VM_FLAGS_SUPERPAGE_SIZE_
#define VM_FLAGS_SUPERPAGE_SIZE_2MB (SUPERPAGE_SIZE_2MB<<VM_FLAGS_SUPERPAGE_SHIFT)
#define VM_FLAGS_SUPERPAGE_SIZE_ANY (SUPERPAGE_SIZE_ANY << VM_FLAGS_SUPERPAGE_SHIFT)

What about using sysctl -a to determine if the system supports 1 GB pages then setting the flag accordingly?

❯ sysctl -a | grep machdep.cpu.extfeatures
machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI

Also, when I tested without setting pdpe1gb flag, the VM was created with 1gb pages support. I am not 100% sure, but what if qemu enables it by default if the device supports 1gb pages?

Also, when I tested without setting pdpe1gb flag, the VM was created with 1gb pages support.

This is really odd because according to QEMU docs on pdpe1gb

Not included by default in any Intel CPU model.
Should be explicitly turned on for all Intel CPU models.

sysctl could work. I'll try this option with syscall.Sysctl from the standard library.

Unless, QEMU's host CPU just passes through all CPU features without the need to explicitly set flags...?

I think the qemu docs mean that the flag is not enabled by default, not that the technology is unavailable.

It looks like it does based on the table posted above! Just gotta make sure pdpe1gb=off happens by default on old Intel machines.

Unless, QEMU's host CPU just passes through all CPU features without the need to explicitly set flags...?

I'm not sure if this is related but, https://manpages.ubuntu.com/manpages/kinetic/en/man7/qemu-cpu-models.7.html

Libvirt supports a third way to configure CPU models known as "Host model". This uses the QEMU "Named model" feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible. This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with "Host passthrough", but gives much of the benefit of passthrough, while making live migration safe.

I couldn't find qemu documentation regarding host cpu model, though some other resources also pointed out that by default, the host model tries to approximate the host cpu.

I've updated #123 with automated checks for huge page support using sysctl.

I don't have access to an Intel Mac, so am kind of going in blind into this. @themisir would you be able to check if the pull request works for you?

@maxzinkus any other thoughts on the code?

Reviewed and +1, but yeah same limitation.

Unable to create an instance

Host

VM