openbmc / linux

OpenBMC Linux kernel source tree

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

WARNING: CPU: 0 PID: 142 at kernel/dma/mapping.c:148 dma_map_page_attrs

shenki opened this issue · comments

dev-5.9 (wip) on witherspoon:

[   53.724209] ------------[ cut here ]------------
[   53.728913] WARNING: CPU: 0 PID: 142 at kernel/dma/mapping.c:148 dma_map_page_attrs+0x17c/0x228
[   53.737616] Modules linked in:
[   53.740696] CPU: 0 PID: 142 Comm: systemctl Not tainted 5.9.9-00137-g6adb2e629a5b #274
[   53.748602] Hardware name: Generic DT based system
[   53.753385] Backtrace: 
[   53.755873] [<8091a910>] (dump_backtrace) from [<8091abb4>] (show_stack+0x20/0x24)
[   53.763451]  r7:00000009 r6:60000193 r5:00000000 r4:80e40efc
[   53.769132] [<8091ab94>] (show_stack) from [<8091fc8c>] (dump_stack+0xc8/0xe4)
[   53.776390] [<8091fbc4>] (dump_stack) from [<80124f04>] (__warn+0xd0/0x148)
[   53.783356]  r7:00000009 r6:00000094 r5:8019aeb4 r4:80b1da6c
[   53.789037] [<80124e34>] (__warn) from [<8091b370>] (warn_slowpath_fmt+0x6c/0xe4)
[   53.796519]  r7:8019aeb4 r6:00000094 r5:80b1da6c r4:00000000
[   53.802195] [<8091b308>] (warn_slowpath_fmt) from [<8019aeb4>] (dma_map_page_attrs+0x17c/0x228)
[   53.810900]  r8:00000002 r7:9ee6c9a0 r6:00000800 r5:00000008 r4:9e55c400
[   53.817640] [<8019ad38>] (dma_map_page_attrs) from [<8069243c>] (usb_gadget_map_request_by_dev.part.0+0xe8/0x1d8)
[   53.827904]  r8:00000008 r7:000139cd r6:9e55c400 r5:00000000 r4:939c4180
[   53.834626] [<80692354>] (usb_gadget_map_request_by_dev.part.0) from [<80692584>] (usb_gadget_map_request+0x28/0x34)
[   53.845153]  r9:8069e868 r8:939c4180 r7:93960000 r6:95e38e40 r5:95e4a0e4 r4:939c4180
[   53.852908] [<8069255c>] (usb_gadget_map_request) from [<806958c0>] (ast_vhub_epn_queue+0xa4/0x208)
[   53.861963] [<8069581c>] (ast_vhub_epn_queue) from [<8069151c>] (usb_ep_queue+0x48/0x110)
[   53.870146]  r10:951f6268 r9:8069e868 r8:939c4180 r7:93960000 r6:00000001 r5:939c4180
[   53.877971]  r4:95e4a0e4
[   53.880527] [<806914d4>] (usb_ep_queue) from [<8069ed18>] (hidg_set_alt+0x11c/0x2ac)
[   53.888276]  r9:8069e868 r8:938fab00 r7:93960000 r6:00000001 r5:939c4180 r4:93960084
[   53.896032] [<8069ebfc>] (hidg_set_alt) from [<8068ba3c>] (composite_setup+0x7a0/0x1cc4)
[   53.904129]  r10:939600e4 r9:951f6268 r8:95068cdc r7:93960084 r6:00000000 r5:95e41060
[   53.911954]  r4:8069ebfc r3:00000004
[   53.915549] [<8068b29c>] (composite_setup) from [<8068e930>] (configfs_composite_setup+0x74/0x8c)
[   53.924425]  r10:9528e000 r9:00000000 r8:951f62dc r7:a0000193 r6:9528fdbc r5:95e41060
[   53.932249]  r4:951f6268
[   53.934803] [<8068e8bc>] (configfs_composite_setup) from [<80694980>] (ast_vhub_ep0_handle_setup+0xc8/0x1f8)
[   53.944635]  r9:00000000 r8:00000015 r7:0000000e r6:00000009 r5:00000002 r4:95e41288
[   53.952392] [<806948b8>] (ast_vhub_ep0_handle_setup) from [<80696614>] (ast_vhub_dev_irq+0x58/0x84)
[   53.961433]  r5:00000001 r4:95e41048
[   53.965024] [<806965bc>] (ast_vhub_dev_irq) from [<80693b78>] (ast_vhub_irq+0x148/0x210)
[   53.973117]  r5:00000200 r4:95e38e40
[   53.976724] [<80693a30>] (ast_vhub_irq) from [<8017fab0>] (__handle_irq_event_percpu+0x54/0x224)
[   53.985518]  r10:9528e000 r9:00000000 r8:00000015 r7:00000001 r6:9e563e68 r5:80d6ad58
[   53.993340]  r4:95e2ea80
[   53.995890] [<8017fa5c>] (__handle_irq_event_percpu) from [<8017fd90>] (handle_irq_event+0x74/0xec)
[   54.004939]  r10:00000129 r9:9528e000 r8:9e40ea00 r7:00000001 r6:9e563e68 r5:80d6ad58
[   54.012762]  r4:9e563e00
[   54.015313] [<8017fd1c>] (handle_irq_event) from [<80184ad8>] (handle_level_irq+0xc0/0x1c0)
[   54.023663]  r6:00000000 r5:80d6ad58 r4:9e563e00
[   54.028294] [<80184a18>] (handle_level_irq) from [<8017f2e4>] (__handle_domain_irq+0x84/0xe4)
[   54.036816]  r5:80d6ad58 r4:00000000
[   54.040414] [<8017f260>] (__handle_domain_irq) from [<8010146c>] (avic_handle_irq+0x68/0x70)
[   54.048852]  r9:9528e000 r8:80100284 r7:9528ff24 r6:ffffffff r5:9528fef0 r4:9e483280
[   54.056606] [<80101404>] (avic_handle_irq) from [<80100b6c>] (__irq_svc+0x6c/0x90)
[   54.064176] Exception stack(0x9528fef0 to 0x9528ff38)
[   54.069231] fee0:                                     00000003 7ef4c86c 40000040 00000000
[   54.077408] ff00: 7ef4c898 01b89928 7ef4c864 00000129 80100284 9528e000 00000129 9528ffa4
[   54.085576] ff20: 9528ffa8 9528ff40 80100080 80786570 80000013 ffffffff
[   54.092185]  r5:80000013 r4:80786570
[   54.095785] [<80786560>] (sys_recvmsg) from [<80100080>] (ret_fast_syscall+0x0/0x54)
[   54.103523] Exception stack(0x9528ffa8 to 0x9528fff0)
[   54.108581] ffa0:                   7ef4c898 01b89928 00000003 7ef4c86c 40000040 00000000
[   54.116754] ffc0: 7ef4c898 01b89928 7ef4c864 00000129 7ef4c860 00000001 00000000 7ef4daa4
[   54.124922] ffe0: 470499e0 7ef4c820 46f5c4e8 46c231b4
[   54.129975]  r6:7ef4c864 r5:01b89928 r4:7ef4c898
[   54.134592] ---[ end trace 552986e33242a1ea ]---
[   54.139219]  1e6a0000.usb-vhub:p1: failed to map buffer
[   54.144456] aspeed_vhub 1e6a0000.usb-vhub: Request mapping failure -14
[   54.150986] configfs-gadget gadget: ep2 queue req --> -14

https://github.com/shenki/linux/blob/dev-5.9/kernel/dma/mapping.c#L148

if (WARN_ON_ONCE(!dev->dma_mask))

So the no dma mask it doesn't point to the correct dma device, or the dma mask now needs to be setup explictly in the platform for the device because its no longer default.

@geissonator saw this with v5.10.

When attempting to debug this the issue goes away, so there is some aspect of timing or build configuration that affects reproducibility.

69c9e82 fixes a similar looking warning. I suspect we're setting this on the wrong device. Reliably reproducing is still difficult which makes it hard to test.

dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,
                size_t offset, size_t size, enum dma_data_direction dir,
                unsigned long attrs)
{
        const struct dma_map_ops *ops = get_dma_ops(dev);
        dma_addr_t addr;

        BUG_ON(!valid_dma_direction(dir));

        if (WARN_ON_ONCE(!dev->dma_mask))
                return DMA_MAPPING_ERROR;

@shenki USB-DMA issue observed in 5.10 based kernel

I could able to recreate this issue with AST2600 eval kit i.e.., whenever I connect the USB cable to my host setup (laptop/server) such dma triggers were observed through USB interrupt which in turn lead to a crash.
Also in our setup we've enabled the network over usb (USB_ETH) during our testing. Whereas this issue is not seen when I swithc back to 5.8 kernel. Looks like there seems to be a compatiblitiy issue with Aspeed USB gadget driver on latest kernel.

The errors logs observed in my setup is shown below:

[  228.370578] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
[  228.377631] ------------[ cut here ]------------
[  228.377655] WARNING: CPU: 0 PID: 14 at kernel/dma/mapping.c:149 dma_map_page_attrs+0x2ac/0x39c
[  228.377666] CPU: 0 PID: 14 Comm: kworker/0:1 Not tainted 5.10.20-74acb73 #1
[  228.377669] Hardware name: Generic DT based system
[  228.377680] Workqueue: events linkwatch_event
[  228.377687] Backtrace:
[  228.377701] [<80976638>] (dump_backtrace) from [<8097688c>] (show_stack+0x20/0x24)
[  228.377708]  r7:00000095 r6:600b0193 r5:00000000 r4:80da2358
[  228.377718] [<8097686c>] (show_stack) from [<80982584>] (dump_stack+0x9c/0xb0)
[  228.377729] [<809824e8>] (dump_stack) from [<80121888>] (__warn+0xf8/0x154)
[  228.377735]  r7:00000095 r6:00000009 r5:80199d3c r4:80b21528
[  228.377744] [<80121790>] (__warn) from [<80977014>] (warn_slowpath_fmt+0x8c/0xc0)
[  228.377750]  r7:00000095 r6:80b21528 r5:00000000 r4:81116000
[  228.377759] [<80976f8c>] (warn_slowpath_fmt) from [<80199d3c>] (dma_map_page_attrs+0x2ac/0x39c)
[  228.377768]  r9:00000001 r8:00000200 r7:80a01764 r6:00000000 r5:81116000 r4:811cf400
[  228.377780] [<80199a90>] (dma_map_page_attrs) from [<806d0560>] (usb_gadget_map_request_by_dev.part.0+0xe4/0x1d0)
[  228.377788]  r10:80d05d10 r9:00000001 r8:00000008 r7:00005540 r6:811cf400 r5:00000000
[  228.377792]  r4:8553d900
[  228.377802] [<806d047c>] (usb_gadget_map_request_by_dev.part.0) from [<806d1ac0>] (usb_gadget_map_request+0x28/0x34)
[  228.377810]  r9:81116000 r8:855406f4 r7:855406f4 r6:811cfa40 r5:856b0188 r4:8553d900
[  228.377820] [<806d1a98>] (usb_gadget_map_request) from [<806d5938>] (ast_vhub_epn_queue+0xa4/0x208)
[  228.377828] [<806d5894>] (ast_vhub_epn_queue) from [<806d09e4>] (usb_ep_queue+0x48/0x100)
[  228.377837]  r10:80d05d10 r9:81116000 r8:855406f4 r7:855406f4 r6:8553d900 r5:85586d7c
[  228.377840]  r4:856b0188
[  228.377850] [<806d099c>] (usb_ep_queue) from [<806dc18c>] (rndis_response_available+0x6c/0xb4)
[  228.377857]  r7:855406f4 r6:81143e80 r5:85586d7c r4:85586cc0
[  228.377866] [<806dc120>] (rndis_response_available) from [<806dd2ac>] (rndis_msg_parser+0x190/0x630)
[  228.377872]  r7:855406f4 r6:00000000 r5:855406c0 r4:8553b000
[  228.377881] [<806dd11c>] (rndis_msg_parser) from [<806dafe0>] (rndis_command_complete+0x28/0x48)
[  228.377888]  r8:00000000 r7:81143bb8 r6:00000000 r5:85582288 r4:81143b80
[  228.377898] [<806dafb8>] (rndis_command_complete) from [<806d1c38>] (usb_gadget_giveback_request+0x30/0x9c)
[  228.377902]  r5:85582288 r4:81143b80
[  228.377913] [<806d1c08>] (usb_gadget_giveback_request) from [<806d3ed8>] (ast_vhub_done+0xbc/0xe4)
[  228.377918]  r5:85582288 r4:81143b80
[  228.377928] [<806d3e1c>] (ast_vhub_done) from [<806d4d94>] (ast_vhub_ep0_handle_ack+0x1a0/0x288)
[  228.377936]  r9:81116000 r8:00000001 r7:00000000 r6:00000018 r5:81143b80 r4:85582288
[  228.377945] [<806d4bf4>] (ast_vhub_ep0_handle_ack) from [<806d665c>] (ast_vhub_dev_irq+0x60/0x7c)
[  228.377953]  r9:81116000 r8:00000200 r7:000002f8 r6:00000001 r5:00000012 r4:85582048
[  228.377962] [<806d65fc>] (ast_vhub_dev_irq) from [<806d3d58>] (ast_vhub_irq+0x110/0x1d4)
[  228.377966]  r5:80000200 r4:811cfa40
[  228.377978] [<806d3c48>] (ast_vhub_irq) from [<8017d830>] (__handle_irq_event_percpu+0x68/0x204)
[  228.377986]  r10:80d05d10 r9:81116000 r8:00000027 r7:81117990 r6:00000000 r5:81116000
[  228.377990]  r4:8567f140 r3:806d3c48
[  228.377999] [<8017d7c8>] (__handle_irq_event_percpu) from [<8017dad8>] (handle_irq_event+0x78/0xe4)
[  228.378008]  r10:80df0fa0 r9:81116000 r8:81043800 r7:811e056c r6:81117bd0 r5:81116000
[  228.378011]  r4:811e0500
[  228.378020] [<8017da60>] (handle_irq_event) from [<80182b4c>] (handle_fasteoi_irq+0xa8/0x218)
[  228.378026]  r7:00000001 r6:81117bd0 r5:80d063f4 r4:811e0500
[  228.378034] [<80182aa4>] (handle_fasteoi_irq) from [<8017cf3c>] (__handle_domain_irq+0x90/0xe4)
[  228.378041]  r7:00000001 r6:81117bd0 r5:00000000 r4:80c764cc
[  228.378050] [<8017ceac>] (__handle_domain_irq) from [<80101348>] (gic_handle_irq+0x7c/0x90)
[  228.378058]  r9:81116000 r8:bf80200c r7:80c75bc8 r6:bf802000 r5:81117a28 r4:80d063f4
[  228.378066] [<801012cc>] (gic_handle_irq) from [<80100b0c>] (__irq_svc+0x6c/0x90)
[  228.378070] Exception stack(0x81117a28 to 0x81117a70)
[  228.378078] 7a20:                   bd7d3dc0 00000000 ffffe406 000009e3 bd7d3dc0 00200000
[  228.378086] 7a40: 81117a90 ffffe3f3 81116000 00000000 80df0fa0 81117ae4 ffffe450 81117a78
[  228.378092] 7a60: 801a11e0 801a113c a00b0113 ffffffff
[  228.378100]  r9:81116000 r8:81116000 r7:81117a5c r6:ffffffff r5:a00b0113 r4:801a113c
[  228.378109] [<801a0fdc>] (__run_timers.part.0) from [<801a139c>] (run_timer_softirq+0x70/0x78)
[  228.378117]  r10:81116000 r9:81116000 r8:00000082 r7:00000100 r6:00000001 r5:00000002
[  228.378121]  r4:80d03d00
[  228.378129] [<801a132c>] (run_timer_softirq) from [<80101514>] (__do_softirq+0x144/0x3d0)
[  228.378133]  r5:00000002 r4:80d03084
[  228.378141] [<801013d0>] (__do_softirq) from [<80128ca4>] (irq_exit+0xbc/0x114)
[  228.378149]  r10:80dca1f8 r9:81116000 r8:81043800 r7:00000001 r6:00000000 r5:00000000
[  228.378153]  r4:ffffe000
[  228.378161] [<80128be8>] (irq_exit) from [<8017cf40>] (__handle_domain_irq+0x94/0xe4)
[  228.378165]  r5:00000000 r4:80c764cc
[  228.378174] [<8017ceac>] (__handle_domain_irq) from [<80101348>] (gic_handle_irq+0x7c/0x90)
[  228.378182]  r9:81116000 r8:bf80200c r7:80c75bc8 r6:bf802000 r5:81117bd0 r4:80d063f4
[  228.378190] [<801012cc>] (gic_handle_irq) from [<80100b0c>] (__irq_svc+0x6c/0x90)
[  228.378193] Exception stack(0x81117bd0 to 0x81117c18)
[  228.378199] 7bc0:                                     600b0093 3cb64000 00000000 200b0013
[  228.378207] 7be0: 80dcab28 00000000 81117c4c 80d05d6c 80d13a98 81116000 80dca1f8 81117ce4
[  228.378214] 7c00: 81117c20 81117c20 801793e8 801793f0 600b0013 ffffffff
[  228.378222]  r9:81116000 r8:80d13a98 r7:81117c04 r6:ffffffff r5:600b0013 r4:801793f0
[  228.378231] [<80178fd4>] (console_unlock) from [<8017980c>] (vprintk_emit+0x130/0x21c)
[  228.378239]  r10:600b0013 r9:80bbdef4 r8:00000000 r7:00000037 r6:00000000 r5:80d05d6c
[  228.378242]  r4:80dca1f8
[  228.378250] [<801796dc>] (vprintk_emit) from [<80179928>] (vprintk_default+0x30/0x38)
[  228.378258]  r10:8567d800 r9:00000004 r8:00000000 r7:80d97700 r6:81117d94 r5:80bbdef4
[  228.378262]  r4:80c6b450
[  228.378270] [<801798f8>] (vprintk_default) from [<8017b05c>] (vprintk_func+0x7c/0x100)
[  228.378279] [<8017afe0>] (vprintk_func) from [<809777c4>] (printk+0x40/0x5c)
[  228.378285]  r7:80d97700 r6:85687c00 r5:8567d800 r4:81116000
[  228.378296] [<80977784>] (printk) from [<8090450c>] (addrconf_notify+0xa68/0xa88)
[  228.378303]  r3:80000000 r2:81117e60 r1:8567d800 r0:80bbdef4
[  228.378306]  r4:81116000
[  228.378316] [<80903aa4>] (addrconf_notify) from [<80149f64>] (raw_notifier_call_chain+0x44/0x78)
[  228.378325]  r10:8567d800 r9:8567da8c r8:80d985c4 r7:81117e60 r6:00000004 r5:fffffff5
[  228.378328]  r4:80da149c
[  228.378337] [<80149f20>] (raw_notifier_call_chain) from [<807ded5c>] (call_netdevice_notifiers_info+0x5c/0xa8)
[  228.378343]  r7:00000000 r6:80d97700 r5:00000004 r4:81117e60
[  228.378352] [<807ded00>] (call_netdevice_notifiers_info) from [<807e3394>] (netdev_state_change+0x70/0x8c)
[  228.378357]  r6:00000000 r5:8567d800 r4:81116000
[  228.378367] [<807e3324>] (netdev_state_change) from [<80805110>] (linkwatch_do_dev+0x64/0x90)
[  228.378372]  r6:81117eb4 r5:80df0394 r4:8567d800
[  228.378382] [<808050ac>] (linkwatch_do_dev) from [<808053e0>] (__linkwatch_run_queue+0x138/0x240)
[  228.378386]  r5:80df0394 r4:00000063
[  228.378393] [<808052a8>] (__linkwatch_run_queue) from [<80805524>] (linkwatch_event+0x3c/0x44)
[  228.378402]  r10:00000000 r9:00000000 r8:00000000 r7:bd7db800 r6:bd7da540 r5:81005f80
[  228.378405]  r4:80d985c4
[  228.378414] [<808054e8>] (linkwatch_event) from [<8013f4e0>] (process_one_work+0x1dc/0x530)
[  228.378424] [<8013f304>] (process_one_work) from [<8013f89c>] (worker_thread+0x68/0x57c)
[  228.378432]  r10:bd7da540 r9:00000008 r8:80d03d00 r7:bd7da558 r6:81005f94 r5:bd7da540
[  228.378436]  r4:81005f80
[  228.378445] [<8013f834>] (worker_thread) from [<80146db8>] (kthread+0x168/0x16c)
[  228.378454]  r10:810fde84 r9:81005f80 r8:8013f834 r7:81116000 r6:00000000 r5:810cd8c0
[  228.378457]  r4:810cd900
[  228.378465] [<80146c50>] (kthread) from [<80100148>] (ret_from_fork+0x14/0x2c)
[  228.378468] Exception stack(0x81117fb0 to 0x81117ff8)
[  228.378474] 7fa0:                                     00000000 00000000 00000000 00000000
[  228.378481] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  228.378488] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  228.378496]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80146c50
[  228.378499]  r4:810cd8c0
[  228.378503] ---[ end trace 0ed0f73ab1520d45 ]---
[  228.378511]  1e6a0000.usb-vhub:p1: failed to map buffer
[  228.378519] aspeed_vhub 1e6a0000.usb-vhub: Request mapping failure -14
[  233.386352]  1e6a0000.usb-vhub:p2: failed to map buffer
[  233.392210] aspeed_vhub 1e6a0000.usb-vhub: Request mapping failure -14
[  233.399496] configfs-gadget gadget: ep2 queue req --> -14

We have exactly the same issue on 6687842 on g220a.

Also encountering this on my e3c246d4i port (100% reproducible so far).

I'm facing the similar backtrace when moving to kernel 5.10, and I have applied below 1-line patch in my tree to fix it. So far it works for Facebook AST2400 and AST2500 platforms but I didn't get chance to try it on AST2600.

facebook/openbmc-linux@83153bd#diff-d44e373511270207f6d3dc1f1081834d0b7cd91ffa086c86a999a3dd753cc744

Thanks @tao-ren. I think this indicates the kernel is using the wrong device pointer somewhere along the lines. I've been reading through the code trying to work out where this is.

Can someone boot using CONFIG_DMA_API_DEBUG and reproduce the issue?

Full dmesg with CONFIG_DMA_API_DEBUG=y:

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.10.23-6687842-dirty-12a1b95 (oe-user@oe-host) (arm-openbmc-linux-gnueabi-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1.20210209) #1 Thu Mar 25 05:17:00 UTC 2021
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] OF: fdt: Machine model: ASRock e3c246d4i v1.00
[    0.000000] Memory policy: Data cache writeback
[    0.000000] cma: Reserved 16 MiB at 0x9d800000
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x000000009effffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009effffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009effffff]
[    0.000000] On node 0 totalpages: 126976
[    0.000000]   Normal zone: 992 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 126976 pages, LIFO batch:31
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 125984
[    0.000000] Kernel command line: console=ttyS4,115200n8 root=/dev/ram rw
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 474636K/507904K available (7168K kernel code, 660K rwdata, 1496K rodata, 1024K init, 279K bss, 16884K reserved, 16384K cma-reserved)
[    0.000000] random: get_random_u32 called from cache_alloc_refill+0x420/0x960 with crng_init=0
[    0.000000] ftrace: allocating 22921 entries in 45 pages
[    0.000000] ftrace: allocated 45 pages with 4 groups
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] i2c controller registered, irq 17
[    0.000000] clocksource: FTTMR010-TIMER2: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 77222644334 ns
[    0.000024] sched_clock: 32 bits at 24MHz, resolution 40ns, wraps every 86767015915ns
[    0.000108] Switching to timer-based delay loop, resolution 40ns
[    0.001257] Calibrating delay loop (skipped), value calculated using timer frequency.. 49.50 BogoMIPS (lpj=247500)
[    0.001331] pid_max: default: 32768 minimum: 301
[    0.002109] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.002175] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.004051] CPU: Testing write buffer coherency: ok
[    0.006259] Setting up static identity map for 0x80100000 - 0x80100038
[    0.007529] ASPEED AST2500 rev A2 (04030303)
[    0.008500] devtmpfs: initialized
[    0.045581] DMA-API: preallocated 65550 debug entries
[    0.045630] DMA-API: debugging enabled by kernel config
[    0.045674] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.045731] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.048992] pinctrl core: initialized pinctrl subsystem
[    0.050509] NET: Registered protocol family 16
[    0.052919] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.055482] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[    0.055533] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.112242] mc: Linux media interface: v0.10
[    0.112456] videodev: Linux video capture interface: v2.00
[    0.116043] clocksource: Switched to clocksource FTTMR010-TIMER2
[    0.171894] NET: Registered protocol family 2
[    0.173216] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.173333] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.173447] TCP bind hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.173556] TCP: Hash tables configured (established 4096 bind 4096)
[    0.173838] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.173925] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.174421] NET: Registered protocol family 1
[    0.175343] Unpacking initramfs...
[    1.150251] Freeing initrd memory: 1128K
[    1.152878] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    1.153513] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.153568] jffs2: version 2.2. (SUMMARY)  © 2001-2006 Red Hat, Inc.
[    1.154282] NET: Registered protocol family 38
[    1.154994] gpio-864 (bmc-ready-hog): hogged as output/high
[    1.163626] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
[    1.168136] 1e787000.serial: ttyS5 at MMIO 0x1e787000 (irq = 32, base_baud = 1546875) is a 16550A
[    1.170317] 1e784000.serial: ttyS4 at MMIO 0x1e784000 (irq = 31, base_baud = 1500000) is a 16550A
[    1.587272] printk: console [ttyS4] enabled
[    1.593627] timeriomem_rng 1e6e2078.hwrng: 32bits from 0x(ptrval) @ 1us
[    1.602152] random: fast init done
[    1.612861] random: crng init done
[    1.630788] loop: module loaded
[    1.677294] aspeed-smc 1e620000.spi: Using 100 MHz SPI frequency
[    1.683549] aspeed-smc 1e620000.spi: w25q256 (32768 Kbytes)
[    1.689268] aspeed-smc 1e620000.spi: CE0 window [ 0x20000000 - 0x22000000 ] 32MB
[    1.696774] aspeed-smc 1e620000.spi: CE1 window [ 0x22000000 - 0x2a000000 ] 128MB
[    1.704280] aspeed-smc 1e620000.spi: read control register: 203c0641
[    1.938130] 5 fixed-partitions partitions found on MTD device bmc
[    1.944264] Creating 5 MTD partitions on "bmc":
[    1.948956] 0x000000000000-0x000000060000 : "u-boot"
[    1.956739] 0x000000060000-0x000000080000 : "u-boot-env"
[    1.964706] 0x000000080000-0x0000004c0000 : "kernel"
[    1.972494] 0x0000004c0000-0x000001c00000 : "rofs"
[    1.980160] 0x000001c00000-0x000002000000 : "rwfs"
[    1.990896] libphy: Fixed MDIO Bus: probed
[    1.996790] ftgmac100 1e660000.ethernet: Read MAC address 70:85:c2:fb:73:13 from chip
[    2.013196] libphy: ftgmac100_mdio: probed
[    2.018398] RTL8211E Gigabit Ethernet 1e660000.ethernet--1:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=1e660000.ethernet--1:00, irq=POLL)
[    2.034275] ftgmac100 1e660000.ethernet eth0: irq 20, mapped at 2bc13f8b
[    2.044670] aspeed_vhub 1e6a0000.usb-vhub: Initialized virtual hub in USB2 mode
[    2.052910] Mass Storage Function, version: 2009/09/11
[    2.058212] LUN: removable file: (no medium)
[    2.062572] no file given for LUN0
[    2.066143] g_mass_storage 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
[    2.074503] i2c /dev entries driver
[    2.079975] aspeed-i2c-bus 1e78a080.i2c-bus: i2c bus 1 registered, irq 33
[    2.090276] at24 3-0057: 16384 byte 24c128 EEPROM, writable, 16 bytes/write
[    2.097557] aspeed-i2c-bus 1e78a100.i2c-bus: i2c bus 3 registered, irq 34
[    2.107697] Driver for 1-wire Dallas network protocol.
[    2.125918] NET: Registered protocol family 10
[    2.132978] Segment Routing with IPv6
[    2.137569] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    2.145052] NET: Registered protocol family 17
[    2.149681] 8021q: 802.1Q VLAN Support v1.8
[    2.165531] printk: console [netcon0] enabled
[    2.170051] netconsole: network logging started
[    2.175668] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    2.189867] Freeing unused kernel memory: 1024K
[    2.197788] Checked W+X mappings: passed, no W+X pages found
[    2.203510] Run /init as init process
[    2.207277]   with arguments:
[    2.207300]     /init
[    2.207314]   with environment:
[    2.207325]     HOME=/
[    2.207335]     TERM=linux
[    3.127249] jffs2: notice: (79) jffs2_build_xattr_subsystem: complete building xattr subsystem, 56 of xdatum (43 unchecked, 13 orphan) and 147 of xref (31 dead, 21 orphan) found.
[    3.195121] overlayfs: upper fs does not support tmpfile.
[    3.201410] overlayfs: upper fs does not support RENAME_WHITEOUT.
[    5.867533] systemd[1]: Failed to look up module alias 'autofs4': Function not implemented
[    6.058966] systemd[1]: systemd 247.3+ running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK +SYSVINIT -UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS -ACL +XZ -LZ4 -ZSTD -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid)
[    6.083436] systemd[1]: Detected architecture arm.
[    6.127628] systemd[1]: Set hostname to <asrock-e3c246d4i>.
[    7.993443] systemd[1]: unit_file_find_fragment: obmc-host-reset@.target+0 → obmc-host-reset@0.target
[    8.110833] systemd[1]: unit_file_find_fragment: obmc-host-reset-running@.target+0 → obmc-host-reset-running@0.target
[    8.192400] systemd[1]: unit_file_find_fragment: obmc-host-stop@.target+0 → obmc-host-stop@0.target
[    8.298532] systemd[1]: unit_file_find_fragment: obmc-host-stopped@.target+0 → obmc-host-stopped@0.target
[    8.328240] systemd[1]: unit_file_find_fragment: obmc-power-stop-pre@.target+0 → obmc-power-stop-pre@0.target
[    8.369142] systemd[1]: unit_file_find_fragment: obmc-chassis-poweron@.target+0 → obmc-chassis-poweron@0.target
[    8.447101] systemd[1]: unit_file_find_fragment: obmc-chassis-powered-off@.target+0 → obmc-chassis-powered-off@0.target
[    8.489939] systemd[1]: unit_file_find_fragment: obmc-chassis-powerreset@.target+0 → obmc-chassis-powerreset@0.target
[    8.595745] systemd[1]: unit_file_find_fragment: obmc-power-on@.target+0 → obmc-power-on@0.target
[    8.624649] systemd[1]: unit_file_find_fragment: obmc-power-start@.target+0 → obmc-power-start@0.target
[    8.746779] systemd[1]: unit_file_find_fragment: obmc-power-start-pre@.target+0 → obmc-power-start-pre@0.target
[    8.860470] systemd[1]: unit_file_find_fragment: obmc-host-stopping@.target+0 → obmc-host-stopping@0.target
[    8.891346] systemd[1]: unit_file_find_fragment: obmc-host-starting@.target+0 → obmc-host-starting@0.target
[    8.921542] systemd[1]: unit_file_find_fragment: obmc-host-start-pre@.target+0 → obmc-host-start-pre@0.target
[    8.962096] systemd[1]: unit_file_find_fragment: obmc-host-stop-pre@.target+0 → obmc-host-stop-pre@0.target
[    9.002540] systemd[1]: unit_file_find_fragment: obmc-host-startmin@.target+0 → obmc-host-startmin@0.target
[    9.185905] systemd[1]: unit_file_find_fragment: obmc-host-quiesce@.target+0 → obmc-host-quiesce@0.target
[    9.218750] systemd[1]: unit_file_find_fragment: obmc-host-started@.target+0 → obmc-host-started@0.target
[    9.248530] systemd[1]: unit_file_find_fragment: obmc-chassis-poweroff@.target+0 → obmc-chassis-poweroff@0.target
[    9.339508] systemd[1]: unit_file_find_fragment: obmc-power-off@.target+0 → obmc-power-off@0.target
[    9.368786] systemd[1]: unit_file_find_fragment: obmc-power-stop@.target+0 → obmc-power-stop@0.target
[   10.397759] systemd[1]: /lib/systemd/system/phosphor-ipmi-net@.socket:3: Invalid interface name, ignoring: sys-subsystem-net-devices-%i.device
[   10.551028] systemd[1]: Queued start job for default target Multi-User System.
[   10.571866] systemd[1]: Created slice system-getty.slice.
[   10.610381] systemd[1]: Created slice system-mapper\x2dwait.slice.
[   10.650150] systemd[1]: Created slice system-modprobe.slice.
[   10.690837] systemd[1]: Created slice system-obmc\x2dled\x2dgroup\x2dstart.slice.
[   10.730352] systemd[1]: Created slice system-phosphor\x2dcertificate\x2dmanager.slice.
[   10.770436] systemd[1]: Created slice system-phosphor\x2ddiscover\x2dsystem\x2dstate.slice.
[   10.820290] systemd[1]: Created slice system-phosphor\x2dipmi\x2dkcs.slice.
[   10.860373] systemd[1]: Created slice system-phosphor\x2dipmi\x2dnet.slice.
[   10.900209] systemd[1]: Created slice system-phosphor\x2dreset\x2dhost\x2dcheck.slice.
[   10.940441] systemd[1]: Created slice system-phosphor\x2dreset\x2dhost\x2drunning.slice.
[   10.990592] systemd[1]: Created slice system-phosphor\x2dreset\x2dsensor\x2dstates.slice.
[   11.040445] systemd[1]: Created slice system-serial\x2dgetty.slice.
[   11.078414] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[   11.118366] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   11.160352] systemd[1]: Reached target Paths.
[   11.197474] systemd[1]: Reached target Remote File Systems.
[   11.237116] systemd[1]: Reached target Slices.
[   11.277846] systemd[1]: Reached target Swap.
[   11.329470] systemd[1]: Listening on Syslog Socket.
[   11.400841] systemd[1]: Listening on Process Core Dump Socket.
[   11.438328] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   11.478773] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
[   11.490504] systemd[1]: Listening on Journal Socket (/dev/log).
[   11.530310] systemd[1]: Listening on Journal Socket.
[   11.570820] systemd[1]: Listening on Network Service Netlink Socket.
[   11.615898] systemd[1]: Listening on udev Control Socket.
[   11.659588] systemd[1]: Listening on udev Kernel Socket.
[   11.700229] systemd[1]: Condition check resulted in Huge Pages File System being skipped.
[   11.711573] systemd[1]: Condition check resulted in POSIX Message Queue File System being skipped.
[   11.732653] systemd[1]: Mounting Kernel Debug File System...
[   11.797126] systemd[1]: Mounting Kernel Trace File System...
[   11.860223] systemd[1]: Mounting Temporary Directory (/tmp)...
[   11.920931] systemd[1]: Condition check resulted in Create list of static device nodes for the current kernel being skipped.
[   11.948001] systemd[1]: Starting Load Kernel Module configfs...
[   12.019044] systemd[1]: Starting Load Kernel Module fuse...
[   12.058588] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[   12.104321] systemd[1]: Starting Journal Service...
[   12.173591] systemd[1]: Condition check resulted in Load Kernel Modules being skipped.
[   12.218508] systemd[1]: Starting Remount Root and Kernel File Systems...
[   12.288748] systemd[1]: Starting Apply Kernel Variables...
[   12.378475] systemd[1]: Starting Coldplug All udev Devices...
[   12.706006] systemd[1]: Mounted Kernel Debug File System.
[   12.845850] systemd[1]: Mounted Kernel Trace File System.
[   12.978901] systemd[1]: Mounted Temporary Directory (/tmp).
[   13.003226] systemd[1]: modprobe@configfs.service: Succeeded.
[   13.081315] systemd[1]: Finished Load Kernel Module configfs.
[   13.154397] systemd[1]: modprobe@fuse.service: Succeeded.
[   13.220885] systemd[1]: Finished Load Kernel Module fuse.
[   13.316586] systemd[1]: Finished Remount Root and Kernel File Systems.
[   13.396393] systemd[1]: Finished Apply Kernel Variables.
[   13.443272] systemd[1]: Condition check resulted in FUSE Control File System being skipped.
[   13.527715] systemd[1]: Mounting Kernel Configuration File System...
[   13.597289] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[   13.633941] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[   13.668979] systemd[1]: Condition check resulted in Create System Users being skipped.
[   13.738570] systemd[1]: Starting Create Static Device Nodes in /dev...
[   13.838126] systemd[1]: Started Journal Service.
[   14.319288] systemd-journald[104]: Received client request to flush runtime journal.
[   29.481872] ------------[ cut here ]------------
[   29.486596] WARNING: CPU: 0 PID: 213 at kernel/dma/mapping.c:149 dma_map_page_attrs+0x2cc/0x3e0
[   29.495318] CPU: 0 PID: 213 Comm: env Not tainted 5.10.23-6687842-dirty-12a1b95 #1
[   29.502893] Hardware name: Generic DT based system
[   29.507688] Backtrace: 
[   29.510187] [<8078b134>] (dump_backtrace) from [<8078b388>] (show_stack+0x20/0x24)
[   29.517768]  r7:00000009 r6:00000095 r5:8015f93c r4:808f6240
[   29.523462] [<8078b368>] (show_stack) from [<80794c00>] (dump_stack+0x28/0x30)
[   29.530715] [<80794bd8>] (dump_stack) from [<80115d08>] (__warn+0xe4/0x150)
[   29.537686]  r5:8015f93c r4:808ea578
[   29.541286] [<80115c24>] (__warn) from [<8078ba30>] (warn_slowpath_fmt+0x6c/0xe0)
[   29.548774]  r7:8015f93c r6:00000095 r5:808ea578 r4:00000000
[   29.554463] [<8078b9c8>] (warn_slowpath_fmt) from [<8015f93c>] (dma_map_page_attrs+0x2cc/0x3e0)
[   29.563171]  r8:00000008 r7:9ecab4a0 r6:00000008 r5:816a4e00 r4:ffffffff
[   29.569894] [<8015f670>] (dma_map_page_attrs) from [<8053a23c>] (usb_gadget_map_request_by_dev.part.0+0xf8/0x204)
[   29.580159]  r10:85280260 r9:00000002 r8:00000008 r7:00004ca5 r6:816a4e00 r5:00000000
[   29.587982]  r4:84ccf620
[   29.590541] [<8053a144>] (usb_gadget_map_request_by_dev.part.0) from [<8053b264>] (usb_gadget_map_request+0x28/0x34)
[   29.601073]  r9:80547c0c r8:84ccfbc0 r7:85bcc100 r6:8565c220 r5:816c40c4 r4:84ccf620
[   29.608839] [<8053b23c>] (usb_gadget_map_request) from [<8053f078>] (ast_vhub_epn_queue+0xa0/0x1e8)
[   29.617900] [<8053efd8>] (ast_vhub_epn_queue) from [<8053a5c0>] (usb_ep_queue+0x48/0xc4)
[   29.625996]  r10:85280260 r9:80547c0c r8:84ccfbc0 r7:85bcc100 r6:84ccf620 r5:84ccf620
[   29.633830]  r4:816c40c4 r3:8053efd8
[   29.637423] [<8053a578>] (usb_ep_queue) from [<80548090>] (hidg_set_alt+0x11c/0x2a0)
[   29.645171]  r7:85bcc100 r6:00000001 r5:84ccf620 r4:85bcc174
[   29.650867] [<80547f74>] (hidg_set_alt) from [<8053410c>] (composite_setup+0x784/0x1cfc)
[   29.658965]  r10:816b9040 r9:85280260 r8:00000000 r7:84c532dc r6:85bcc1d4 r5:85bcc174
[   29.666796]  r4:84caa290 r3:80547f74
[   29.670404] [<80533988>] (composite_setup) from [<805380dc>] (configfs_composite_setup+0x50/0x58)
[   29.679290]  r10:80b8e388 r9:816bc540 r8:84c79bf8 r7:00000200 r6:8565c220 r5:00000011
[   29.687120]  r4:a0000193
[   29.689686] [<8053808c>] (configfs_composite_setup) from [<8053e204>] (ast_vhub_ep0_handle_setup+0xa8/0x1b0)
[   29.699520]  r5:00000011 r4:816b91a8
[   29.703121] [<8053e15c>] (ast_vhub_ep0_handle_setup) from [<8053fcf8>] (ast_vhub_dev_irq+0x54/0x80)
[   29.712169]  r4:816b9028
[   29.714722] [<8053fca4>] (ast_vhub_dev_irq) from [<8053d544>] (ast_vhub_irq+0x110/0x1b8)
[   29.722814]  r5:00000001 r4:00000200
[   29.726419] [<8053d434>] (ast_vhub_irq) from [<80155858>] (__handle_irq_event_percpu+0x4c/0x1a8)
[   29.735215]  r10:80b8e388 r9:816bc540 r8:84c79bf8 r7:00000015 r6:00000000 r5:80b8aad0
[   29.743038]  r4:80d01740 r3:8053d434
[   29.746634] [<8015580c>] (__handle_irq_event_percpu) from [<80155aa4>] (handle_irq_event+0x58/0xc4)
[   29.755684]  r10:00000013 r9:84c78000 r8:80c17600 r7:00000001 r6:00000000 r5:80b8aad0
[   29.763508]  r4:816bc540
[   29.766065] [<80155a4c>] (handle_irq_event) from [<80159cb8>] (handle_level_irq+0xac/0x180)
[   29.774414]  r5:80b8aad0 r4:816bc540
[   29.778013] [<80159c0c>] (handle_level_irq) from [<80154fe0>] (__handle_domain_irq+0x80/0xd8)
[   29.786539]  r5:80b8aad0 r4:00000000
[   29.790142] [<80154f60>] (__handle_domain_irq) from [<80101210>] (avic_handle_irq+0x68/0x70)
[   29.798591]  r9:84c78000 r8:84c79e30 r7:84c79ca4 r6:ffffffff r5:84c79c70 r4:80c04320
[   29.806345] [<801011a8>] (avic_handle_irq) from [<80100acc>] (__irq_svc+0x6c/0x90)
[   29.813915] Exception stack(0x84c79c70 to 0x84c79cb8)
[   29.818974] 9c60:                                     84c79d68 00000041 00000000 0000002f
[   29.827158] 9c80: 84c79d68 00000041 8514d010 84cdc000 84c79e30 84c79d68 00000013 84c79ce4
[   29.835336] 9ca0: 84c79ce8 84c79cc0 802931cc 8028fde0 20000013 ffffffff
[   29.841951]  r5:20000013 r4:8028fde0
[   29.845562] [<8028fdac>] (path_init) from [<802931cc>] (path_openat+0x7c/0xde8)
[   29.852883]  r9:84c79d68 r8:84c79e30 r7:84cdc000 r6:84c79e30 r5:84c79d68 r4:ffffe000
[   29.860639] [<80293150>] (path_openat) from [<80293fbc>] (do_filp_open+0x84/0x130)
[   29.868220]  r10:00000013 r9:84ca58c0 r8:00000013 r7:00000001 r6:84c79e30 r5:84c79d68
[   29.876044]  r4:8514d000
[   29.878597] [<80293f38>] (do_filp_open) from [<8028aa18>] (do_open_execat+0x80/0x224)
[   29.886433]  r7:84ca2a40 r6:85265c5c r5:8514d000 r4:8514d000
[   29.892110] [<8028a998>] (do_open_execat) from [<8028b948>] (open_exec+0x34/0x4c)
[   29.899596]  r5:85265c00 r4:8514d000
[   29.903202] [<8028b914>] (open_exec) from [<802f7560>] (load_elf_binary+0x4ac/0x1414)
[   29.911036]  r5:85265c00 r4:84ca2a00
[   29.914636] [<802f70b4>] (load_elf_binary) from [<8028ae44>] (bprm_execve+0x288/0x50c)
[   29.922560]  r10:80b64338 r9:00000000 r8:85265c5c r7:00000006 r6:00000000 r5:80b5f140
[   29.930382]  r4:85265c00
[   29.932937] [<8028abbc>] (bprm_execve) from [<8028b57c>] (do_execveat_common+0x154/0x178)
[   29.941125]  r10:00000000 r9:7eac5e70 r8:7eac5e68 r7:ffffff9c r6:85265c00 r5:8532a000
[   29.948949]  r4:00000000
[   29.951503] [<8028b428>] (do_execveat_common) from [<8028c7d0>] (sys_execve+0x40/0x48)
[   29.959428]  r9:84c78000 r8:80100224 r7:0000000b r6:7eac5e68 r5:7eac5e70 r4:7eac5e68
[   29.967181] [<8028c790>] (sys_execve) from [<80100040>] (ret_fast_syscall+0x0/0x54)
[   29.974840] Exception stack(0x84c79fa8 to 0x84c79ff0)
[   29.979903] 9fa0:                   7eac5f4b 00000001 7eac5c28 7eac5e68 7eac5e70 0000006e
[   29.988088] 9fc0: 7eac5f4b 00000001 7eac5e68 0000000b 7eac5f96 7eac5f9e 0000003d 7eac5cbc
[   29.996265] 9fe0: 00000000 7eac5c24 76e47540 76e46e3c
[   30.001324]  r5:00000001 r4:7eac5f4b
[   30.004902] ---[ end trace e1f25ad1cb8de2a0 ]---
[   30.009532]  1e6a0000.usb-vhub:p1: failed to map buffer
[   30.014774] aspeed_vhub 1e6a0000.usb-vhub: Request mapping failure -14
[   30.021312] configfs-gadget gadget: ep2 queue req --> -14
[   40.595783] ftgmac100 1e660000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   40.660424] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   40.739040] ------------[ cut here ]------------
[   40.743752] WARNING: CPU: 0 PID: 161 at kernel/dma/debug.c:973 check_unmap+0x40c/0x92c
[   40.751695] DMA-API: ftgmac100 1e660000.ethernet: device driver frees DMA memory with different size [device address=0x0000000084ea37a2] [map size=1536 bytes] [unmap size=357 bytes]
[   40.767868] CPU: 0 PID: 161 Comm: dbus-broker Tainted: G        W         5.10.23-6687842-dirty-12a1b95 #1
[   40.777524] Hardware name: Generic DT based system
[   40.782316] Backtrace: 
[   40.784811] [<8078b134>] (dump_backtrace) from [<8078b388>] (show_stack+0x20/0x24)
[   40.792400]  r7:00000009 r6:000003cd r5:80164b5c r4:808f6240
[   40.798095] [<8078b368>] (show_stack) from [<80794c00>] (dump_stack+0x28/0x30)
[   40.805344] [<80794bd8>] (dump_stack) from [<80115d08>] (__warn+0xe4/0x150)
[   40.812309]  r5:80164b5c r4:808eaad4
[   40.815908] [<80115c24>] (__warn) from [<8078ba64>] (warn_slowpath_fmt+0xa0/0xe0)
[   40.823394]  r7:80164b5c r6:000003cd r5:808eaad4 r4:808eb3a4
[   40.829075] [<8078b9c8>] (warn_slowpath_fmt) from [<80164b5c>] (check_unmap+0x40c/0x92c)
[   40.837178]  r8:00000600 r7:80b030ac r6:8533fc28 r5:80bbadfc r4:80d05f20
[   40.843892] [<80164750>] (check_unmap) from [<80165114>] (debug_dma_unmap_page+0x98/0xa8)
[   40.852087]  r10:85662440 r9:84dc0000 r8:80800a30 r7:816bf810 r6:84ea37a2 r5:00000165
[   40.859917]  r4:00000002
[   40.862492] [<8016507c>] (debug_dma_unmap_page) from [<8015fb0c>] (dma_unmap_page_attrs+0xbc/0x138)
[   40.871549]  r7:85662000 r6:00000165 r5:00000000 r4:8010b43c
[   40.877243] [<8015fa50>] (dma_unmap_page_attrs) from [<8053196c>] (ftgmac100_poll+0x2f0/0x468)
[   40.885858]  r10:85662440 r9:84dc0000 r8:9d850000 r7:85662000 r6:00000165 r5:00000000
[   40.893685]  r4:85662498
[   40.896249] [<8053167c>] (ftgmac100_poll) from [<8060a5d8>] (net_rx_action+0x160/0x3f4)
[   40.904258]  r10:80b460e0 r9:00000000 r8:0000012c r7:8533fd44 r6:80b8dc80 r5:00000040
[   40.912082]  r4:85662498
[   40.914644] [<8060a478>] (net_rx_action) from [<801012f4>] (__do_softirq+0xdc/0x334)
[   40.922398]  r10:00000008 r9:8533e000 r8:00000004 r7:0000000c r6:8533e000 r5:00000003
[   40.930223]  r4:80ba66ec
[   40.932785] [<80101218>] (__do_softirq) from [<8011a488>] (irq_exit+0xc4/0x128)
[   40.940104]  r10:00004040 r9:8533e000 r8:80c17600 r7:00000001 r6:00000000 r5:80b8aad0
[   40.947927]  r4:00000000
[   40.950488] [<8011a3c4>] (irq_exit) from [<80154fe4>] (__handle_domain_irq+0x84/0xd8)
[   40.958337] [<80154f60>] (__handle_domain_irq) from [<80101210>] (avic_handle_irq+0x68/0x70)
[   40.966789]  r9:8533e000 r8:80100224 r7:8533fe54 r6:ffffffff r5:8533fe20 r4:80c04320
[   40.974542] [<801011a8>] (avic_handle_irq) from [<80100acc>] (__irq_svc+0x6c/0x90)
[   40.982112] Exception stack(0x8533fe20 to 0x8533fe68)
[   40.987174] fe20: 0000001c 8533fea0 8533fe9c 00000000 8533fea0 8533fe9c 019472a0 00000176
[   40.995360] fe40: 80100224 8533e000 00004040 8533fe84 8533fe88 8533fe70 805e2bc0 802a509c
[   41.003535] fe60: 80000013 ffffffff
[   41.007027]  r5:80000013 r4:802a509c
[   41.010644] [<805e2ba0>] (sockfd_lookup_light) from [<805e6478>] (__sys_sendmmsg+0x50/0x1a0)
[   41.019085]  r5:7eee679c r4:00000001
[   41.022687] [<805e6428>] (__sys_sendmmsg) from [<805e65ec>] (sys_sendmmsg+0x24/0x2c)
[   41.030439]  r10:00000000 r9:8533e000 r8:80100224 r7:00000176 r6:019472a0 r5:76f8ee30
[   41.038270]  r4:019730e0
[   41.040824] [<805e65c8>] (sys_sendmmsg) from [<80100040>] (ret_fast_syscall+0x0/0x54)
[   41.048655] Exception stack(0x8533ffa8 to 0x8533fff0)
[   41.053720] ffa0:                   019730e0 76f8ee30 0000001c 7eee679c 00000001 00004040
[   41.061905] ffc0: 019730e0 76f8ee30 019472a0 00000176 7eee679c 019472b0 00000001 00000000
[   41.070079] ffe0: 00000000 7eee6770 0049caa8 76e49150
[   41.075133] ---[ end trace e1f25ad1cb8de2a1 ]---
[   41.079747] DMA-API: Mapped at:
[   41.082909]  debug_dma_map_page+0x58/0xf4
[   41.086949]  dma_map_page_attrs+0x1c0/0x3e0
[   41.091160]  ftgmac100_alloc_rx_buf.constprop.0+0xb8/0x1d8
[   41.096659]  ftgmac100_init_all+0xd8/0x2e0
[   41.100764]  ftgmac100_reset_task+0x134/0x1d0

Thanks @tao-ren. I think this indicates the kernel is using the wrong device pointer somewhere along the lines. I've been reading through the code trying to work out where this is.

My feeling was: the "dummy" device (one for each port, created in "ast_vhub_init_dev") was not fully initialized (no dma mask). The driver used be working, but it fails after commit f959dcd ("dma-direct: Fix potential NULL pointer dereference").

Inspecting the code, this is our call stack:

ast_vhub_epn_queue(struct usb_ep* u_ep, ...)
  struct ast_vhub_ep *ep = to_ast_ep(u_ep);

  usb_gadget_map_request(&ep->dev->gadget, ...)
    usb_gadget_map_request_by_dev(gadget->dev.parent, ...)
      dma_map_single(dev, ...)
        dma_map_single_attrs(dev, ...)
          dma_map_page_attrs(dev, ...)

The dma buffer is mapped with

drivers/usb/gadget/udc/aspeed-vhub/epn.c:       ep->buf = dma_alloc_coherent(&vhub->pdev->dev,

So we need to know, does &vhub->pdev->dev == gadget->dev.parent?

Following Tao's thread, the fake dummy device has it's parent set to vhub->pdev->dev. In usb_initialize_gadget gadget->dev.parent = parent;, which takes me to a dead end!

@tao-ren Can you send your patch to the upstream mailing list? cc the people in this thread, and those from the dma-direct patch you linked to. If it's not the right thing to do I'm sure we will be told!

I can't speak to whether or not it's entirely the right fix, but experimentally I've just tested @tao-ren's patch, and with it applied I no longer see the aspeed-vhub WARN() and the openbmc iKVM's K and M components are functional. (I still get the ftgmac100 check_unmap() warning, but I presume that's unrelated.)

@zevweiss Thanks for testing. The ftgmac100 warning is #201.

Inspecting the code, this is our call stack:

ast_vhub_epn_queue(struct usb_ep* u_ep, ...)
  struct ast_vhub_ep *ep = to_ast_ep(u_ep);

  usb_gadget_map_request(&ep->dev->gadget, ...)
    usb_gadget_map_request_by_dev(gadget->dev.parent, ...)
      dma_map_single(dev, ...)
        dma_map_single_attrs(dev, ...)
          dma_map_page_attrs(dev, ...)

The dma buffer is mapped with

drivers/usb/gadget/udc/aspeed-vhub/epn.c:       ep->buf = dma_alloc_coherent(&vhub->pdev->dev,

So we need to know, does &vhub->pdev->dev == gadget->dev.parent?

Following Tao's thread, the fake dummy device has it's parent set to vhub->pdev->dev. In usb_initialize_gadget gadget->dev.parent = parent;, which takes me to a dead end!

@tao-ren Can you send your patch to the upstream mailing list? cc the people in this thread, and those from the dma-direct patch you linked to. If it's not the right thing to do I'm sure we will be told!

I was about to learn more before sending out the patch but didn't get chance to do so (struggling with ast2600 these days). Sure I will send out the patch to upstream list soon.

I'm facing the similar backtrace when moving to kernel 5.10, and I have applied below 1-line patch in my tree to fix it. So far it works for Facebook AST2400 and AST2500 platforms but I didn't get chance to try it on AST2600.

facebook/openbmc-linux@83153bd#diff-d44e373511270207f6d3dc1f1081834d0b7cd91ffa086c86a999a3dd753cc744

Tried the patch on g220a and it works great!

FYI I just sent the patch to upstream list. I couldn't cc all the people in the thread because I don't have all the email addresses, but I added openbmc email alias, and you could search "[PATCH] usb: gadget: aspeed: set port_dev dma mask".

Cheers,
Tao

This has been fixed in mainline with bd4d607, and backported to the openbmc dev-5.10 kenrel with bbfc8e2.