Repository navigation
[BUG] Various IPC timed out with multi-core run simultaneously #7774
Description
Describe the bug
We enabled multi-core tplg in CI and observed that it doesn't support 3 cores run at the same time. It failed with IPC timed out 0x40030004|MOD_INIT_INSTANCE.
dmesg
[ 2398.696058] kernel: snd_sof:sof_ipc4_widget_setup: sof-audio-pci-intel-mtl 0000:00:1f.3: Create widget dai-copier.SSP.NoCodec-2.playback instance 3 - pipe 6 - core 2
[ 2398.696064] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx : 0x40030004|0x203002f: MOD_INIT_INSTANCE [data size: 188]
[ 2399.201595] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc timed out for 0x40030004|0x203002f
[ 2399.201612] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Attempting to prevent DSP from entering D3 state to preserve context
[ 2399.201618] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump start ]------------
[ 2399.201643] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Host IPC initiator: 0xc0030004|0x203002f|0x0, target: 0x40030004|0x203002f|0x0, ctl: 0x3
[ 2399.201650] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump end ]------------
[ 2399.201655] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 2399.201659] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: IPC timeout
[ 2399.201663] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_COMPLETE (7)
[ 2399.201677] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM status: 0x5, ROM error: 0x0
[ 2399.201682] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[ 2399.201689] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM feature bit enabled
[ 2399.201693] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
To Reproduce
TPLG=/lib/firmware/intel/sof-ace-tplg/sof-mtl-nocodec-multicore.tplg MODEL=MTLP_RVP_NOCODEC_MULTICORE ~/sof-test/test-case/multiple-pipeline.sh -f p -l 50
Reproduction Rate
30%
Environment
- Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
- Kernel: {0efaf69a10d7}
- SOF: {5f118b761bf0}
- Name of the topology file
- Topology: {sof-ace-tplg/sof-mtl-nocodec-multicore.tplg}
- Name of the platform(s) on which the bug is observed.
- Platform: {MTL-NOCODEC-MULTICORE}
Activity
keqiaozhang commentedon Jun 21, 2023
New errors in CI. Test ID:28022
fredoh9 commentedon Jul 26, 2023
Instead of MOD_INIT_INSTANCE, now IPC timeout with MOD_BIND.
[-][BUG] Multi-core feature doesn't support 3 cores run at the same time[/-][+][BUG] Various IPC timed out with multi-core run simultaneously[/+]keqiaozhang commentedon Jul 31, 2023
The highest probability of IPC timed out is "MOD_BIND" when testing multiple pipelines, The reproduction rate is about 90%.
The second one is "GLB_SET_PIPELINE_STATE", the reproduction rate is about 10%.
lgirdwood commentedon Jul 31, 2023
@keqiaozhang can you retry Tuesday, a multicore fix PR has been merged today (after you ran this test). Thanks !!
18 remaining items
RanderWang commentedon Aug 28, 2023
I did many tests and find that if IPC4 logging is disabled, multi-core test can pass. I found one infinite loop in fw on one device but it can't be reproduced on another one. I will update the status when I get any hint
RanderWang commentedon Aug 28, 2023
For the HW hang with multicore feature, it can't be reproduced by python test and only found with Linux audio test. Finally found the root reason by HW team
lgirdwood commentedon Aug 28, 2023
@RanderWang can you fix infinite loop with a timout check and log error. We think some were introduced when drivers were added to Zephyr. @kv2019i @abonislawski fyi.
kv2019i commentedon Aug 28, 2023
@RanderWang Which infinite loop did you see? Let's add a timeout to these, so we can exclude them out. I pushed first such fix now -> zephyrproject-rtos/zephyr#61967
Another break-from-inifinite-loop fix at zephyrproject-rtos/zephyr#61978
RanderWang commentedon Aug 29, 2023
Maybe it is cause by #8115
lyakh commentedon Aug 30, 2023
Can we test the same on TGL?
keqiaozhang commentedon Sep 6, 2023
This issue has been improved a lot after #8143 merged. Only multiple-pause-resume test failed in today's CI testing.
mengdonglin commentedon Sep 8, 2023
After PR #8149 got merged, this issue cannot be reproduced on ww36.4
wszypelt commentedon Sep 8, 2023
@keqiaozhang request for confirmation and closing the bug
fredoh9 commentedon Sep 8, 2023
this issue cannot be reproduced on ww36.5 either. Let me do some manually test before closing the issue.
fredoh9 commentedon Sep 8, 2023
re-run multicore test with daily build(planresultdetail/31515) and manually ran multiple-pipeline.sh. Looks very stable.
I think we can close this.