Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

issue(wireguard-go): page fault - tun,tls,cloud_init #1860

Open
rinor opened this issue May 24, 2023 · 2 comments
Open

issue(wireguard-go): page fault - tun,tls,cloud_init #1860

rinor opened this issue May 24, 2023 · 2 comments

Comments

@rinor
Copy link
Contributor

rinor commented May 24, 2023

While deploying and testing with real traffic (wireguard-go in this case) on azure (will test in other clouds in the following days), I'm getting the following identical Page fault crashes.

Will provide more info and exact setup configurations later on. Just posting this in case anyone can help and point me in the right direction to troubleshoot it.

The issue manifests itself after some time (traffic) and initially everything works fine.

Page fault 
wg2: assigned 172.16.0.1
en1: assigned FE80::20D:3AFF:FE55:70A4
en1: assigned 10.0.0.7

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000034
   rbx: 0000000000000020
   rcx: 0000000000000076
   rdx: 00000000ffffffde
   rsi: 0000000000000054
   rdi: ffffc00004a145b0
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc00004a145e0
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805a00
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc00004a145ee
ffffc00000a37c90:   ffffc00004a145b0
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   ffffc00000a37cc8
ffffc00000a37cb0:   1400000000000006
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805a00
ffffc00000a37cd0:   ffffc00000805a00
ffffc00000a37cd8:   ffffc00004a145ee
ffffc00000a37ce0:   ffffc00004a06c90
ffffc00000a37ce8:   0000000000034470
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   f78d992c2d819138
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc00004a145b0
ffffc00000a37d40:   ffffc00000805a00
ffffc00000a37d48:   ffffc00004a145b0
ffffc00000a37d50:   ffffc00000401b80
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   0000000000000042
ffffc00000a37d70:   0000000000000042
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020fab80

wg2: assigned 172.16.0.1
en1: assigned FE80::20D:3AFF:FE55:70A4
en1: assigned 10.0.0.7

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000040
   rbx: 0000000000000014
   rcx: 000000000000008e
   rdx: 00000000ffffffc6
   rsi: 0000000000000054
   rdi: ffffc00004a0b4f0
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc00004a0b520
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805a00
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc00004a0b52e
ffffc00000a37c90:   ffffc00004a0b4f0
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   ffffc00000a37cc8
ffffc00000a37cb0:   1400000000000006
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805a00
ffffc00000a37cd0:   ffffc00000805a00
ffffc00000a37cd8:   ffffc00004a0b52e
ffffc00000a37ce0:   ffffc00004a00000
ffffc00000a37ce8:   0000000000036360
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   f1d015a7eb33e362
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc00004a0b4f0
ffffc00000a37d40:   ffffc00000805a00
ffffc00000a37d48:   ffffc00004a0b4f0
ffffc00000a37d50:   ffffc00000401b80
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   000000000000004e
ffffc00000a37d70:   000000000000004e
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020f738c

wg2: assigned 172.16.0.1
en1: assigned FE80::20D:3AFF:FE55:70A4
en1: assigned 10.0.0.7

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000034
   rbx: 0000000000000020
   rcx: 0000000000000076
   rdx: 00000000ffffffde
   rsi: 0000000000000054
   rdi: ffffc00004a34050
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc00004a34080
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805a00
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc00004a3408e
ffffc00000a37c90:   ffffc00004a34050
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   3320646e79622d32
ffffc00000a37cb0:   14ffc00000a37d00
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805a00
ffffc00000a37cd0:   ffffc00000805a00
ffffc00000a37cd8:   ffffc00004a3408e
ffffc00000a37ce0:   24674b2041522959
ffffc00000a37ce8:   0000000000024730
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   c3414a2fe2ff8cb6
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc00004a34050
ffffc00000a37d40:   ffffc00000805a00
ffffc00000a37d48:   ffffc00004a34050
ffffc00000a37d50:   ffffc00000401b80
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   0000000000000042
ffffc00000a37d70:   0000000000000042
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020f4280

en1: assigned 10.0.0.7
wg2: assigned 172.16.0.1
ERROR: (wg2) 2023/05/24 17:28:31 peer(OBt8…ifUI) - Failed to send handshake initiation: no known endpoint for peer
en1: assigned FE80::20D:3AFF:FE55:70A4

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000034
   rbx: 0000000000000020
   rcx: 0000000000000076
   rdx: 00000000ffffffde
   rsi: 0000000000000054
   rdi: ffffc0000460d920
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc0000460d950
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805a00
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc0000460d95e
ffffc00000a37c90:   ffffc0000460d920
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   3320646e79622d32
ffffc00000a37cb0:   1400000000000006
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805a00
ffffc00000a37cd0:   ffffc00000805a00
ffffc00000a37cd8:   ffffc0000460d95e
ffffc00000a37ce0:   ffffc0000460d920
ffffc00000a37ce8:   0000000000027fe0
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   5ccfb88e23576feb
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc0000460d920
ffffc00000a37d40:   ffffc00000805a00
ffffc00000a37d48:   ffffc0000460d920
ffffc00000a37d50:   ffffc00000401b80
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   0000000000000042
ffffc00000a37d70:   0000000000000042
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020fc780

wg2: assigned 172.16.0.1
en1: assigned FE80::20D:3AFF:FE55:70A4
en1: assigned 10.0.0.7

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000034
   rbx: 0000000000000020
   rcx: 0000000000000076
   rdx: 00000000ffffffde
   rsi: 0000000000000054
   rdi: ffffc00004a04860
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc00004a04890
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805a00
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc00004a0489e
ffffc00000a37c90:   ffffc00004a04860
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   3320646e79622d32
ffffc00000a37cb0:   14ffc00000a37d00
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805a00
ffffc00000a37cd0:   ffffc00000805a00
ffffc00000a37cd8:   ffffc00004a0489e
ffffc00000a37ce0:   adf2fca0d36c43ef
ffffc00000a37ce8:   0000000000019248
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   d571d8c8928486de
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc00004a04860
ffffc00000a37d40:   ffffc00000805a00
ffffc00000a37d48:   ffffc00004a04860
ffffc00000a37d50:   ffffc00000401b80
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   0000000000000042
ffffc00000a37d70:   0000000000000042
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020fc780

wg2: assigned 172.16.0.1
en1: assigned FE80::20D:3AFF:FE55:70A4
en1: assigned 10.0.0.7

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000034
   rbx: 0000000000000020
   rcx: 0000000000000076
   rdx: 00000000ffffffde
   rsi: 0000000000000054
   rdi: ffffc00004a0b4f0
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc00004a0b520
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805400
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc00004a0b52e
ffffc00000a37c90:   ffffc00004a0b4f0
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   3320646e79622d32
ffffc00000a37cb0:   14ffc00000a37d00
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805400
ffffc00000a37cd0:   ffffc00000805400
ffffc00000a37cd8:   ffffc00004a0b52e
ffffc00000a37ce0:   54197793501bf262
ffffc00000a37ce8:   000000000001a240
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   208352a8e3ea839e
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc00004a0b4f0
ffffc00000a37d40:   ffffc00000805400
ffffc00000a37d48:   ffffc00004a0b4f0
ffffc00000a37d50:   ffffc00000401c00
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   0000000000000042
ffffc00000a37d70:   0000000000000042
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020fc780

wg2: assigned 172.16.0.1
en1: assigned FE80::20D:3AFF:FE55:70A4
en1: assigned 10.0.0.7

no fault handler
cpu 0000000000000000, state kernel, vector 000000000000000e
lastvector: 000000000000000e (Page fault)
     frame: ffffc00000a30000
      type: kernel
active_cpu: 0000000000000000
 stack top: ffffc00000a37ff0
error code: 0000000000000000
   address: 0000000000000012

   rax: 0000000000000034
   rbx: 0000000000000020
   rcx: 0000000000000076
   rdx: 00000000ffffffde
   rsi: 0000000000000054
   rdi: ffffc00004a090c0
   rbp: ffffc00000a37c98
   rsp: ffffc00000a37c88
    r8: 0000000000000000
    r9: 0d00000000000000
   r10: ffffc00004a090f0
   r11: 0000000000000030
   r12: 0000000000000000
   r13: 0000000000000054
   r14: ffffc00000805a00
   r15: 0000000000000014
   rip: ffffffff80122603	(pbuf_realloc + 0000000000000033/00000000000000cc)
rflags: 0000000000010097
    ss: 0000000000000008
    cs: 0000000000000010
    ds: 0000000000000000
    es: 0000000000000000
fsbase: 0000000000000000
gsbase: 0000000000000000

frame trace:
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d90:   ffffffff800fe274	(netvsc_recv + 0000000000000214/0000000000000277)
ffffc00000a37e00:   ffffffff800ff80f	(hv_rf_on_receive + 000000000000017f/0000000000000273)
ffffc00000a37e80:   ffffffff800ff0c9	(hv_nv_on_channel_callback + 0000000000000339/0000000000000579)
ffffc00000a37f20:   ffffffff800fb069	(vmbus_chan_task + 0000000000000039/0000000000000072)
ffffc00000a37f50:   ffffffff8005256c	(runloop_internal + 00000000000002cc/0000000000000a0d)
ffffc00000a37fc0:   ffffffff80040fef	(context_switch_finish + 000000000000007f/00000000000001b4)

loaded klibs: tun@0xffffffff80200000/0x3000 tls@0xffffffff80300000/0x97000 cloud_init@0xffffffff80208000/0x8000 

stack trace:
ffffc00000a37c88:   ffffc00004a090fe
ffffc00000a37c90:   ffffc00004a090c0
ffffc00000a37c98:   ffffc00000a37d58
ffffc00000a37ca0:   ffffffff80111e52	(ip4_input + 00000000000000d2/00000000000004f6)
ffffc00000a37ca8:   3320646e79622d32
ffffc00000a37cb0:   1400000000000006
ffffc00000a37cb8:   ffffc00000a37cc8
ffffc00000a37cc0:   0000000000000000
ffffc00000a37cc8:   ffffc00000805a00
ffffc00000a37cd0:   ffffc00000805a00
ffffc00000a37cd8:   ffffc00004a090fe
ffffc00000a37ce0:   ffffc00004a0b4f0
ffffc00000a37ce8:   0000000000032c10
ffffc00000a37cf0:   000000000003f000
ffffc00000a37cf8:   ffffc00000804e80
ffffc00000a37d00:   ffffc00000804e80
ffffc00000a37d08:   ffffc00000a37d98
ffffc00000a37d10:   0000000000000002
ffffc00000a37d18:   0000000000000000
ffffc00000a37d20:   11d9cfd3cb9b7b63
ffffc00000a37d28:   0000000000000002
ffffc00000a37d30:   0000000000000008
ffffc00000a37d38:   ffffc00004a090c0
ffffc00000a37d40:   ffffc00000805a00
ffffc00000a37d48:   ffffc00004a090c0
ffffc00000a37d50:   ffffc00000401b80
ffffc00000a37d58:   ffffc00000a37d88
ffffc00000a37d60:   ffffffff8012c04e	(ethernet_input + 000000000000011e/0000000000000185)
ffffc00000a37d68:   0000000000000042
ffffc00000a37d70:   0000000000000042
ffffc00000a37d78:   0000000000000000
ffffc00000a37d80:   ffffc000020fab80
@francescolavra
Copy link
Member

I tried deploying and stress-testing a simple web server running on Azure, but couldn't reproduce this issue, which by looking at your logs looks like a bug in the network interface driver.
However, I checked the driver code and I found a bug that could potentially cause this kind of issues. It's fixed in #1863. Let us know if this fix solves the issue.
Thanks

@rinor
Copy link
Contributor Author

rinor commented Jun 3, 2023 •
edited

Still failing. I will test on other clouds and try to reproduce it locally, but it looks like it's related to the tunnel/forwarding.

  • GCP - using f1-micro - ok - (no pagefault)
  • DO - using s-1vcpu-1gb - ok - (no pagefault)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rinor @francescolavra