Skip to content

[BUG][MTL] Firmware boot failure due to timeout during suspend/resume stress test (ROM status 0x50000005, ROM error 0x0) #8148

Closed
Closed
[BUG][MTL] Firmware boot failure due to timeout during suspend/resume stress test (ROM status 0x50000005, ROM error 0x0)#8148
Labels
BootFirmware boot or code signing related.MTLApplies to Meteor Lake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedsuspend-resumeIssues observed when doing system suspend and resume

Description

@keqiaozhang
keqiaozhang
opened on Sep 5, 2023

Describe the bug
This issue happened when testing suspend/resume with audio. It only happened once with stress test.

dmesg

[51369.325401] kernel: snd_sof_intel_hda_common:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:00:1f.3: Primary core power up successful
[51369.325408] kernel: snd_sof_intel_hda_common:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x73214]=0x80000000 successful
[51369.325420] kernel: snd_sof_intel_hda_common:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x1800]=0x41 successful
[51369.325429] kernel: snd_sof_intel_hda_common:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x1140]=0x1 successful
[51369.337199] kernel: nvme nvme0: 8/0/0 default/read/poll queues
[51371.343143] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[51371.343150] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware boot failure due to timeout
[51371.343153] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[51371.343197] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM status: 0x0, ROM error: 0x0
[51371.343200] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[51371.343205] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM feature bit enabled
[51371.343207] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[51371.343209] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed to boot DSP firmware after resume -5
[51371.343213] kernel: snd_sof:sof_set_fw_state: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state change: 3 -> 4
[51371.343216] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -5
[51371.343230] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: PM: failed to resume async: error -5
[51371.348200] kernel: OOM killer enabled.
[51371.348203] kernel: Restarting tasks ... done.

To Reproduce
~/sof-test/test-case/check-suspend-resume-with-audio.sh -l 100 -m playback

Reproduction Rate
TBD, need more tests to confirm it.

Environment

  1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
  2. Name of the topology file
    • Topology: {development/sof-mtl-nocodec.tplg}
  3. Name of the platform(s) on which the bug is observed.

dmesg.txt

mtrace.txt

Activity

  1. keqiaozhang
    added
    bugSomething isn't working as expected
    suspend-resumeIssues observed when doing system suspend and resume
    MTLApplies to Meteor Lake platform
    on Sep 5, 2023
  2. mengdonglin
    added
    BootFirmware boot or code signing related.
    on Sep 5, 2023

  3. keqiaozhang commented on Sep 5, 2023

    @keqiaozhang
    keqiaozhang
    on Sep 5, 2023
    CollaboratorAuthor

    A similar boot failure like #7866 (comment)

  4. mengdonglin commented on Sep 13, 2023

    @mengdonglin
    mengdonglin
    on Sep 13, 2023
    Collaborator

    @kv2019i This issue cannot be reproduced with v2.7-rc1 on both MTL and cavs2.5 platforms, confirmed by @keqiaozhang by running more stress test on MTLP_RVP_NOCODEC and ADLP_RVP_NOCODEC.

    v2.7-rc1 disabled IMR context save by default.

    @keqiaozhang If we disable IMR context save on main branch, can we reproduce this issue?

  5. keqiaozhang commented on Sep 15, 2023

    @keqiaozhang
    keqiaozhang
    on Sep 15, 2023
    CollaboratorAuthor

    If we disable IMR context save on main branch, can we reproduce this issue?

    Confirmed that this issue cannot be reproduced on main branch when IMR is disabled.

  6. keqiaozhang
    added
    P2Critical bugs or normal features
    on Sep 18, 2023

  7. fredoh9 commented on Oct 5, 2023

    @fredoh9
    fredoh9
    on Oct 5, 2023
    Contributor

    Today daily test has same issue with MTLP_RVP_NOCODEC multicore.

    [ 1048.207985] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 1048.207994] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware boot failure due to timeout
[ 1048.207997] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[ 1048.208039] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM status: 0x0, ROM error: 0x0
[ 1048.208042] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[ 1048.208048] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM feature bit enabled
[ 1048.208091] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 1048.208094] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed to boot DSP firmware after resume -5
[ 1048.208102] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -5
[ 1048.208119] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: PM: failed to resume async: error -5
[ 1048.322203] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc4_tx_msg_unlocked: ipc message send for 0x11000005|0x0 failed: -19
[ 1048.322233] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: failed to create module pipeline.1
[ 1048.322252] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Failed to set up connected widgets
[ 1048.322271] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed widget list set up for pcm 0 dir 0
[ 1048.322291] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: set pcm hw_params after resume
[ 1048.322308] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_prepare on 0000:00:1f.3: -19
[ 1048.322330] kernel:  Port0: ASoC: error at __soc_pcm_prepare on Port0: -19
[ 1048.322346] kernel:  Port0: ASoC: error at dpcm_fe_dai_prepare on Port0: -19

    Internal Intel Daily test link:
    planresultdetail/32700?model=MTLP_RVP_NOCODEC-ace1_0-multicore-3cores&testcase=check-suspend-resume-with-playback-5

  8. mengdonglin
    changed the title [-][BUG][MTL] Firmware boot failure due to timeout during suspend/resume stress test[/-] [+][BUG][MTL] Firmware boot failure due to timeout during suspend/resume stress test (ROM status 0x50000005, ROM error 0x0)[/+] on Oct 22, 2023

  9. lrudyX commented on Nov 24, 2023

    @lrudyX
    lrudyX
    on Nov 24, 2023

    @tmleman Please check

  10. tmleman commented on Nov 24, 2023

    @tmleman
    tmleman
    on Nov 24, 2023
    Contributor

    @keqiaozhang it looks very similar to the #7866 and probably have the same root cause. Why we tract it in separate issue?

  11. keqiaozhang commented on Dec 25, 2023

    @keqiaozhang
    keqiaozhang
    on Dec 25, 2023
    CollaboratorAuthor

    This issue cannot be reproduced on main branch. Closing this bug.

  12. tmleman commented on Jan 19, 2024

    @tmleman
    tmleman
    on Jan 19, 2024
    Contributor

    @keqiaozhang Is there any update? The latest comment mentions that the problem is not reproducing, but the issue is still open.

  13. keqiaozhang
    closed this as completedon Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BootFirmware boot or code signing related.MTLApplies to Meteor Lake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedsuspend-resumeIssues observed when doing system suspend and resume

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions