vmd: VMs are stopping with: vcpu 0 run ioctl failed: Invalid argument

Post by Olivier Cherrier
Hi,
I am running ~ current on amd64 (dmesg attached) and am experiencing
vmd crashes. The processes responsible for operating the VMs are
Dec 20 12:26:32 stanislas vmd[38693]: vcpu_run_loop: vm 14 / vcpu 0 run ioctl failed: Invalid argument
Dec 20 12:26:32 stanislas vmd[98389]: vcpu_run_loop: vm 15 / vcpu 0 run ioctl failed: Invalid argument
It happens randomly. Generally the VMs are staying up for a few hours.
Is there anything I can do to troubleshoot ?

Could be anything. What guest VM are you running here? And how much memory are
you assigning?

A useful option is to run vmd manually (rcctl stop vmd && vmd -dvvvv), and/or
to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel.
Obviously that kernel goes on the host.

-ml

Post by Olivier Cherrier
Thank you,
Best
--
Olivier Cherrier
Phone: +352691754777
OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017
real mem = 4188946432 (3994MB)
avail mem = 4054945792 (3867MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0: vendor American Megatrends Inc. version "5.11" date 06/03/2016
bios0: Notebook N24_25JU
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT SSDT DBGP DBG2 SSDT UEFI SSDT DMAR ASF!
acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PXSX(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) PXSX(S4) RP01(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 1 (application processor)
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 1, core 0, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus -1 (RP17)
acpiprt5 at acpi0: bus -1 (RP18)
acpiprt6 at acpi0: bus -1 (RP19)
acpiprt7 at acpi0: bus -1 (RP20)
acpiprt8 at acpi0: bus 1 (RP01)
acpiprt9 at acpi0: bus 2 (RP02)
acpiprt10 at acpi0: bus -1 (RP03)
acpiprt11 at acpi0: bus -1 (RP04)
acpiprt12 at acpi0: bus -1 (RP05)
acpiprt13 at acpi0: bus -1 (RP06)
acpiprt14 at acpi0: bus -1 (RP07)
acpiprt15 at acpi0: bus -1 (RP08)
acpiprt16 at acpi0: bus 3 (RP09)
acpiprt17 at acpi0: bus -1 (RP10)
acpiprt18 at acpi0: bus -1 (RP11)
acpiprt19 at acpi0: bus -1 (RP12)
acpiprt20 at acpi0: bus -1 (RP13)
acpiprt21 at acpi0: bus -1 (RP14)
acpiprt22 at acpi0: bus -1 (RP15)
acpiprt23 at acpi0: bus -1 (RP16)
acpiec0 at acpi0
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpitz0 at acpi0: critical temperature is 120 degC
"INT3F0D" at acpi0 not configured
"MSFT0001" at acpi0 not configured
"SYN1218" at acpi0 not configured
"INT33A1" at acpi0 not configured
"PNPC000" at acpi0 not configured
acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
acpibtn2 at acpi0: LID0
acpiac0 at acpi0: AC unit online
acpibat0 at acpi0: BAT0 model "BAT" serial 0001 type LION oem "Notebook"
"PNP0C14" at acpi0 not configured
"INT340E" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
cpu0: Enhanced SpeedStep 2095 MHz: speeds: 2101, 2100, 2000, 1900, 1700, 1600, 1500, 1400, 1200, 1100, 1000, 800, 700, 600, 500, 400 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 6G Host" rev 0x08
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 510" rev 0x07
drm0 at inteldrm0
inteldrm0: msi
error: [drm:pid0:i915_firmware_load_error_print] *ERROR* failed to load firmware i915/skl_dmc_ver1.bin (-22)
error: [drm:pid0:i915_gem_init_hw] *ERROR* Failed to initialize GuC, error -8 (ignored)
inteldrm0: 1366x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x21: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
"Intel 100 Series MEI" rev 0x21 at pci0 dev 22 function 0 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x21: msi, AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
ahci0: PHY offline on port 1
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, KINGSTON SA400S3, SBFK> SCSI3 0/direct fixed naa.5000000000000000
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
ppb0 at pci0 dev 28 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci1 at ppb0 bus 1
rtsx0 at pci1 dev 0 function 0 "Realtek RTL8411B Card Reader" rev 0x01: msi
sdmmc0 at rtsx0: 4-bit, dma
re0 at pci1 dev 0 function 1 "Realtek 8168" rev 0x12: RTL8411B (0x5c80), msi, address 80:fa:5b:38:37:27
rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
ppb1 at pci0 dev 28 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
iwm0 at pci2 dev 0 function 0 "Intel Dual Band Wireless AC 7265" rev 0x61, msi
ppb2 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
nvme0 at pci3 dev 0 function 0 "Samsung SM961/PM961 NVMe" rev 0x00: msi, NVMe 1.2
nvme0: Samsung SSD 960 EVO 250GB, firmware 2B7QCXE7, serial S3ESNX0J904388B
scsibus2 at nvme0: 1 targets
sd1 at scsibus2 targ 0 lun 0: <NVMe, Samsung SSD 960, 2B7Q> SCSI4 0/direct fixed
sd1: 238475MB, 512 bytes/sector, 488397168 sectors
pcib0 at pci0 dev 31 function 0 "Intel 100 Series LPC" rev 0x21
"Intel 100 Series PMC" rev 0x21 at pci0 dev 31 function 2 not configured
azalia0 at pci0 dev 31 function 3 "Intel 100 Series HD Audio" rev 0x21: msi
azalia0: codecs: Realtek ALC269, Intel/0x2809, using Realtek ALC269
audio0 at azalia0
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x21: apic 2 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pms0: Synaptics touchpad, firmware 8.2, 0x1e2b1 0x840300
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhidev0 at uhub0 port 3 configuration 1 interface 0 "TypeMatrix.com USB Keyboard" rev 1.10/1.50 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub0 port 3 configuration 1 interface 1 "TypeMatrix.com USB Keyboard" rev 1.10/1.50 addr 2
uhidev1: iclass 3/1, 3 report ids
uhid0 at uhidev1 reportid 1: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=2, output=0, feature=0
uhid2 at uhidev1 reportid 3: input=2, output=1, feature=0
ugen0 at uhub0 port 5 "Intel Bluetooth" rev 2.01/0.01 addr 3
uvideo0 at uhub0 port 6 configuration 1 interface 0 "Chicony Electronics Co.,Ltd. Chicony USB2.0 Camera" rev 2.00/10.19 addr 4
video0 at uvideo0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd1a (4c28d48edb6ee300.a) swap on sd1b dump on sd1b
iwm0: hw rev 0x210, fw ver 16.242414.0, address 7c:b0:c2:50:11:e4

Olivier Cherrier

2018-01-16 09:40:54 UTC

Post by Mike Larkin

Could be anything. What guest VM are you running here? And how much memory are
you assigning?
A useful option is to run vmd manually (rcctl stop vmd && vmd -dvvvv), and/or

Here is what I typically get :

...
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument
vmm_sighdlr: handling signal 20
vmm_sighdlr: attempting to terminate vm 4
terminate_vm: terminating vmid 23
vmm_sighdlr: calling vm_remove
vm_remove: removing vm id 4 from running config
vm_remove: calling vm_stop
vm_stop: stopping vm 4
vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22
vmd_dispatch_vmm: about to stop vm id 4
vm_stop: stopping vm 4

Post by Mike Larkin
to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel.

Will try that.

Thank you!
Best
Olivier

Carlos Cardenas

2018-01-16 14:56:04 UTC

Post by Mike Larkin

Could be anything. What guest VM are you running here? And how much memory are
you assigning?
A useful option is to run vmd manually (rcctl stop vmd && vmd -dvvvv), and/or

...
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument
vmm_sighdlr: handling signal 20
vmm_sighdlr: attempting to terminate vm 4
terminate_vm: terminating vmid 23
vmm_sighdlr: calling vm_remove
vm_remove: removing vm id 4 from running config
vm_remove: calling vm_stop
vm_stop: stopping vm 4
vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22
vmd_dispatch_vmm: about to stop vm id 4
vm_stop: stopping vm 4

Post by Mike Larkin
to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel.

Will try that.
Thank you!
Best
Olivier

Thanks for reporting the symptoms Olivier.

In order for us to reproduce this and find the root cause, we're going
to need the info that Mike was asking:
* What guest VM are you running here?
* How much memory are you assigning?

I would add:
* What was your "vmctl start ..." command?

Thanks in advance.

+--+
Carlos

Olivier Cherrier

2018-01-16 18:24:18 UTC

Post by Olivier Cherrier
...
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vionet queue notify - no space, dropping packet
vcpu_run_loop: vm 23 / vcpu 0 run ioctl failed: Invalid argument
vmm_sighdlr: handling signal 20
vmm_sighdlr: attempting to terminate vm 4
terminate_vm: terminating vmid 23
vmm_sighdlr: calling vm_remove
vm_remove: removing vm id 4 from running config
vm_remove: calling vm_stop
vm_stop: stopping vm 4
vmd_dispatch_vmm: handling TERMINATE_EVENT for vm id 4 ret 22
vmd_dispatch_vmm: about to stop vm id 4
vm_stop: stopping vm 4

Post by Mike Larkin
to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel.

Will try that.
Thank you!
Best
Olivier

Thanks for reporting the symptoms Olivier.
In order for us to reproduce this and find the root cause, we're going
* What guest VM are you running here?
* How much memory are you assigning?

Hi Carlos,

Only OpenBSD involved here. The physical machine's dmesg is attached.
My /etc/vm.conf configuration file is also attached.

I don't think it is related to memory exhaustion since it sometimes
happens when the machine is not loaded (cpu and memory).

Post by Carlos Cardenas
* What was your "vmctl start ..." command?

Here, I did use : vmd -dvvvv

Will try by recompiling with VMM_DEBUG.

Thanks a lot,
Best
Olivier

Carlos Cardenas

2018-01-17 05:20:57 UTC

Post by Mike Larkin
to enable VMM_DEBUG in sys/arch/amd64/amd64/vmm.c and rebuild the kernel.

Will try that.
Thank you!
Best
Olivier

Thanks for reporting the symptoms Olivier.
In order for us to reproduce this and find the root cause, we're going
* What guest VM are you running here?
* How much memory are you assigning?

Hi Carlos,
Only OpenBSD involved here. The physical machine's dmesg is attached.
My /etc/vm.conf configuration file is also attached.
I don't think it is related to memory exhaustion since it sometimes
happens when the machine is not loaded (cpu and memory).

Post by Carlos Cardenas
* What was your "vmctl start ..." command?

Here, I did use : vmd -dvvvv
Will try by recompiling with VMM_DEBUG.
Thanks a lot,
Best
Olivier

Ok...see comments inline

Post by Olivier Cherrier
OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017

This looks like a custom kernel, can you run the latest snapshot, as-is?

Post by Olivier Cherrier
real mem = 4188946432 (3994MB)
avail mem = 4054945792 (3867MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0: vendor American Megatrends Inc. version "5.11" date 06/03/2016
bios0: Notebook N24_25JU
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT MCFG HPET SSDT SSDT DBGP DBG2 SSDT UEFI SSDT DMAR ASF!
acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PXSX(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) PXSX(S4) RP01(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 1 (application processor)
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 1, core 0, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,ERMS,INVPCID,RDSEED,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus -1 (RP17)
acpiprt5 at acpi0: bus -1 (RP18)
acpiprt6 at acpi0: bus -1 (RP19)
acpiprt7 at acpi0: bus -1 (RP20)
acpiprt8 at acpi0: bus 1 (RP01)
acpiprt9 at acpi0: bus 2 (RP02)
acpiprt10 at acpi0: bus -1 (RP03)
acpiprt11 at acpi0: bus -1 (RP04)
acpiprt12 at acpi0: bus -1 (RP05)
acpiprt13 at acpi0: bus -1 (RP06)
acpiprt14 at acpi0: bus -1 (RP07)
acpiprt15 at acpi0: bus -1 (RP08)
acpiprt16 at acpi0: bus 3 (RP09)
acpiprt17 at acpi0: bus -1 (RP10)
acpiprt18 at acpi0: bus -1 (RP11)
acpiprt19 at acpi0: bus -1 (RP12)
acpiprt20 at acpi0: bus -1 (RP13)
acpiprt21 at acpi0: bus -1 (RP14)
acpiprt22 at acpi0: bus -1 (RP15)
acpiprt23 at acpi0: bus -1 (RP16)
acpiec0 at acpi0
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpitz0 at acpi0: critical temperature is 120 degC
"INT3F0D" at acpi0 not configured
"MSFT0001" at acpi0 not configured
"SYN1218" at acpi0 not configured
"INT33A1" at acpi0 not configured
"PNPC000" at acpi0 not configured
acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
acpibtn2 at acpi0: LID0
acpiac0 at acpi0: AC unit online
acpibat0 at acpi0: BAT0 model "BAT" serial 0001 type LION oem "Notebook"
"PNP0C14" at acpi0 not configured
"INT340E" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
cpu0: Enhanced SpeedStep 2095 MHz: speeds: 2101, 2100, 2000, 1900, 1700, 1600, 1500, 1400, 1200, 1100, 1000, 800, 700, 600, 500, 400 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 6G Host" rev 0x08
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 510" rev 0x07
drm0 at inteldrm0
inteldrm0: msi
error: [drm:pid0:i915_firmware_load_error_print] *ERROR* failed to load firmware i915/skl_dmc_ver1.bin (-22)
error: [drm:pid0:i915_gem_init_hw] *ERROR* Failed to initialize GuC, error -8 (ignored)
inteldrm0: 1366x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x21: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
"Intel 100 Series MEI" rev 0x21 at pci0 dev 22 function 0 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x21: msi, AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
ahci0: PHY offline on port 1
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, KINGSTON SA400S3, SBFK> SCSI3 0/direct fixed naa.5000000000000000
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
ppb0 at pci0 dev 28 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci1 at ppb0 bus 1
rtsx0 at pci1 dev 0 function 0 "Realtek RTL8411B Card Reader" rev 0x01: msi
sdmmc0 at rtsx0: 4-bit, dma
re0 at pci1 dev 0 function 1 "Realtek 8168" rev 0x12: RTL8411B (0x5c80), msi, address 80:fa:5b:38:37:27
rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
ppb1 at pci0 dev 28 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
iwm0 at pci2 dev 0 function 0 "Intel Dual Band Wireless AC 7265" rev 0x61, msi
ppb2 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
nvme0 at pci3 dev 0 function 0 "Samsung SM961/PM961 NVMe" rev 0x00: msi, NVMe 1.2
nvme0: Samsung SSD 960 EVO 250GB, firmware 2B7QCXE7, serial S3ESNX0J904388B
scsibus2 at nvme0: 1 targets
sd1 at scsibus2 targ 0 lun 0: <NVMe, Samsung SSD 960, 2B7Q> SCSI4 0/direct fixed
sd1: 238475MB, 512 bytes/sector, 488397168 sectors
pcib0 at pci0 dev 31 function 0 "Intel 100 Series LPC" rev 0x21
"Intel 100 Series PMC" rev 0x21 at pci0 dev 31 function 2 not configured
azalia0 at pci0 dev 31 function 3 "Intel 100 Series HD Audio" rev 0x21: msi
azalia0: codecs: Realtek ALC269, Intel/0x2809, using Realtek ALC269
audio0 at azalia0
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x21: apic 2 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pms0: Synaptics touchpad, firmware 8.2, 0x1e2b1 0x840300
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhidev0 at uhub0 port 3 configuration 1 interface 0 "TypeMatrix.com USB Keyboard" rev 1.10/1.50 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub0 port 3 configuration 1 interface 1 "TypeMatrix.com USB Keyboard" rev 1.10/1.50 addr 2
uhidev1: iclass 3/1, 3 report ids
uhid0 at uhidev1 reportid 1: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=2, output=0, feature=0
uhid2 at uhidev1 reportid 3: input=2, output=1, feature=0
ugen0 at uhub0 port 5 "Intel Bluetooth" rev 2.01/0.01 addr 3
uvideo0 at uhub0 port 6 configuration 1 interface 0 "Chicony Electronics Co.,Ltd. Chicony USB2.0 Camera" rev 2.00/10.19 addr 4
video0 at uvideo0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd1a (4c28d48edb6ee300.a) swap on sd1b dump on sd1b
iwm0: hw rev 0x210, fw ver 16.242414.0, address 7c:b0:c2:50:11:e4
# $OpenBSD: vm.conf,v 1.7 2017/03/25 16:28:25 reyk Exp $
#
# Macros
#
sets="/data/VMs/"
#
# Virtual machines
#
#switch "uplink" {
## This interface will default to bridge0, but switch(4) is supported
##interface switch0
#
## Add additional members
#interface em0
#}
switch "local" {
interface bridge0
}
vm "openbsd.vm" {
disable
boot $sets "bsd.rd"
# First disk from 'vmctl create "/home/vm/OpenBSD.img" -s 4G'
disk "/home/vm/OpenBSD.img"
# Second disk from OpenBSD contains the install sets
disk $sets "install60.fs"
# Interface will show up as tap(4) on the host and as vio(4) in the VM
interface { switch "uplink" }
interface { switch "local" }
}

Might want to either comment out the interface for "uplink" or remove it
altogether.

Post by Olivier Cherrier
vm "puffy1" {
owner oc
disable
memory 512M
boot $sets "bsd"
disk $sets "puffy1.img"
# Use a specific tap(4) interface with a hardcoded MAC address
# local interface tap
interface {
switch "local"
lladdr 00:11:22:33:44:55
}
}
vm "puffy2" {
owner oc
disable
memory 1024M
# boot $sets "current/bsd.rd"
disk $sets "puffy2.img"
disk $sets "puffy2_data0.img"
# Use a specific tap(4) interface with a hardcoded MAC address
# local interface tap
interface {
switch "local"
lladdr 00:11:22:33:44:56
}
}
vm "am-azure" {
owner oc
memory 1024M
# boot $sets "current/bsd.rd"
disk $sets "am-azure.img"
disk $sets "am-azure_data0.img"
# Use a specific tap(4) interface with a hardcoded MAC address
# local interface tap
interface {
switch "local"
lladdr 00:11:22:33:44:57
}
}

Ok...it's unclear which version the guest(s) are running. Can you
have the guest run the latest snapshot (same as the host) so we can
narrow the variables?

Also, which vm was giving issue?

+--+
Carlos

Olivier Cherrier

2018-01-17 09:30:18 UTC

Post by Olivier Cherrier
Only OpenBSD involved here. The physical machine's dmesg is attached.
My /etc/vm.conf configuration file is also attached.
I don't think it is related to memory exhaustion since it sometimes
happens when the machine is not loaded (cpu and memory).

Post by Carlos Cardenas
* What was your "vmctl start ..." command?

Here, I did use : vmd -dvvvv
Will try by recompiling with VMM_DEBUG.
Thanks a lot,
Best
Olivier

Ok...see comments inline

Post by Olivier Cherrier
OpenBSD 6.2-current (GENERIC.MP) #2: Thu Dec 7 10:13:35 CET 2017

This looks like a custom kernel, can you run the latest snapshot, as-is?

The only change I made is related to /sys/dev/ic/pckbc.c in order to
have a working trackpad.

Post by Olivier Cherrier
vm "openbsd.vm" {
disable
boot $sets "bsd.rd"
# First disk from 'vmctl create "/home/vm/OpenBSD.img" -s 4G'
disk "/home/vm/OpenBSD.img"
# Second disk from OpenBSD contains the install sets
disk $sets "install60.fs"
# Interface will show up as tap(4) on the host and as vio(4) in the VM
interface { switch "uplink" }
interface { switch "local" }
}

Might want to either comment out the interface for "uplink" or remove it
altogether.

OK, commented out.

Ok...it's unclear which version the guest(s) are running. Can you
have the guest run the latest snapshot (same as the host) so we can
narrow the variables?

Guests are running various versions of current. I have this problem with
the latest snapshot as well:
$ sysctl kern.version
kern.version=OpenBSD 6.2-current (GENERIC) #359: Mon Jan 15 22:32:02 MST 2018
***@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC

It happens if I run only puffy1, puffy1 and puffy2 or only am-azure.

Post by Carlos Cardenas
Also, which vm was giving issue?

All of them.
I am upgrading the physical machine with VMM_DEBUG.

BTW, congratulations for the hardware support (ACPI, ...). This laptop
goes to suspend multiple times a day and everything still works after
more than a month:
10:23:43 ***@stanislas $ uptime
10:23AM up 41 days, 9 mins, 16 users, load averages: 1.01, 1.23, 0.97
10:23:44 ***@stanislas $

Thanks

--
Olivier Cherrier
Phone: +352691754777
mailto:***@symacx.com

Olivier Cherrier

2018-01-17 13:45:15 UTC

Post by Carlos Cardenas
This looks like a custom kernel, can you run the latest snapshot, as-is?

The only change I made is related to /sys/dev/ic/pckbc.c in order to
have a working trackpad.

Post by Carlos Cardenas
Might want to either comment out the interface for "uplink" or remove it
altogether.

OK, commented out.
I am upgrading the physical machine with VMM_DEBUG.

Now, everything is running current.
After reboot, vmd started the am-azure VM. It worked fine for a few
hours and it just stopped. Here are the kernel messages:

Jan 17 14:35:29 stanislas /bsd: vmx_handle_exit: unhandled exit 0x31 (EPT misconfiguration)
Jan 17 14:35:29 stanislas vmd[86737]: vcpu_run_loop: vm 1 / vcpu 0 run ioctl failed: Invalid argument
Jan 17 14:35:30 stanislas /bsd: vmm_free_vpid: freed VPID/ASID 1

I hope it helps.
Thanks

--
Olivier Cherrier
Phone: +352691754777
mailto:***@symacx.com

Mike Larkin

2018-01-17 17:26:14 UTC