Skip to content

[BUG][MTL] S0ix long run failure, Firmware boot failure due to timeout (ROM status: 0x50000005, ROM error: 0x0) #7866

Description

@RDharageswari

EDIT: the kernel part of this issue has been transferred to newer:


We see this error reported in multiple scenario:
One was long run S0ix testing.

[  352.971033] sof-audio-pci-intel-mtl 0000:00:1f.3: ipc timed out for 0x47000000|0x0
[  352.979537] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump start ]------------
[  352.988991] sof-audio-pci-intel-mtl 0000:00:1f.3: Host IPC initiator: 0xc7000000|0x0|0x0, target: 0x67000000|0x0|0x0, ctl: 0x3
[  353.001735] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump end ]------------
[  353.010979] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[  353.020423] sof-audio-pci-intel-mtl 0000:00:1f.3: IPC timeout
[  353.026868] sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_COMPLETE (7)
[  353.035552] sof-audio-pci-intel-mtl 0000:00:1f.3: ROM status: 0xffffffff, ROM error: 0xffffffff
[  353.045287] sof-audio-pci-intel-mtl 0000:00:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[  353.055497] sof-audio-pci-intel-mtl 0000:00:1f.3: ROM feature bit not enabled
[  353.063477] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[  353.072721] sof-audio-pci-intel-mtl 0000:00:1f.3: ctx_save IPC error: -110, proceeding with suspend
[  689.764439] sof-audio-pci-intel-mtl 0000:00:1f.3: GAIN (UUID: 61BCA9A8-18D0-4A18-8E7B-2639219804B7): No CPC match in the firmware file's manifest (ibs/obs: 768/768)
[  692.922594] sof-audio-pci-intel-mtl 0000:00:1f.3: ipc timed out for 0x47000000|0x0
[  692.931113] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump start ]------------
[  692.940571] sof-audio-pci-intel-mtl 0000:00:1f.3: Host IPC initiator: 0xc7000000|0x0|0x0, target: 0x67000000|0x0|0x0, ctl: 0x3
[  692.953325] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump end ]------------
[  692.962576] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]---------

cc:

Activity

  1. added
    bugSomething isn't working as expected
    MTLApplies to Meteor Lake platform
    on Jun 27, 2023
  2. marc-hb commented on Jun 27, 2023

    @marc-hb
    Collaborator

    @RDharageswari can you please not ignore the entire bug template? It has a number of specific and important fields.

  3. RDharageswari commented on Jun 29, 2023

    @RDharageswari
    Author

    Hi Marc,
    Sure.I am trying to get the exact scenario, hence did not update it right.Will update the entire configuration once available

  4. added this to the v2.7 milestone on Jul 5, 2023
  5. mengdonglin commented on Jul 27, 2023

    @mengdonglin
    Collaborator

    @RDharageswari Can you still reproduce this issue with mtl-005.0.3 (Hot Fix 3) release?

  6. RDharageswari commented on Aug 14, 2023

    @RDharageswari
    Author

    @mengdonglin: No reports of this in the recent days..But @yongzhi1 was able to repro this issue in the PnP set up with mtl-005-hotfix3 release

  7. macchian commented on Aug 16, 2023

    @macchian
    Contributor

    @mengdonglin , I also found the ipc time out during suspend resume test on customer MTL board.
    The issue can be reproduced on fw branch mtl-005-drop-stable and mtl-005-hotfix3 as well.

    @yongzhi1 , I saw that your linux PR about page fault
    ASoC: SOF: Intel: MTL: catch invalid IRQ IP ptr #4344
    Is it relative to this?

    ...
    <3>[  124.809909] sof-audio-pci-intel-mtl 0000:00:1f.3: ipc timed out for 0x47000000|0x0
    <3>[  124.818406] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump start ]------------
    <3>[  124.830259] sof-audio-pci-intel-mtl 0000:00:1f.3: Host IPC initiator: 0xc7000000|0x0|0x0, target: 0xe7000000|0x0|0x0, ctl: 0x3
    <3>[  124.856211] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump end ]------------
    <3>[  124.874460] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
    <3>[  124.893443] sof-audio-pci-intel-mtl 0000:00:1f.3: IPC timeout
    <3>[  124.906717] sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_COMPLETE (7)
    <3>[  124.924700] sof-audio-pci-intel-mtl 0000:00:1f.3: ROM status: 0xffffffff, ROM error: 0xffffffff
    <3>[  124.944330] sof-audio-pci-intel-mtl 0000:00:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
    <3>[  124.966038] sof-audio-pci-intel-mtl 0000:00:1f.3: ROM feature bit not enabled
    <3>[  124.982946] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
    <4>[  125.001601] sof-audio-pci-intel-mtl 0000:00:1f.3: ctx_save IPC error: -110, proceeding with suspend
    <1>[  125.199974] BUG: unable to handle page fault for address: ffffbffbcb40001f
    <1>[  125.207668] #PF: supervisor read access in kernel mode
    
  8. macchian commented on Aug 16, 2023

    @macchian
    Contributor

    attach the crash logs.
    kernel.20230815.094120.64594.txt

  9. mengdonglin commented on Aug 16, 2023

    @mengdonglin
    Collaborator

    @macchian @RDharageswari @yongzhi1 This issue seems similar to #7990 and #8028 that happens with mtl-005.0.3 when CPC-based clock selection and IMR context save are both enabled.

    Now for main branch, IMR context save is already enabled and PR of CPC-based clock selection is under review #8019

    We found on main branch, with both IMR context save and CPC-based clock selection (PR) enabled, we cannot reproduce #7990 and #8028. Please find details and recipe in #7990 (comment)

    Maybe you can try this FW recipe of main branch as well?

    Or you may try this kernel test PR #8007 with MTL-005.0.3 FW? This kernel PR will make FW always run at highest clock by always sending ZERO CPC values to FW?

  10. 74 remaining items

  11. tmleman commented on Dec 22, 2023

    @tmleman
    Contributor

    @lgirdwood @ujfalusi @RDharageswari I proposed a fix (pull request to Zephyr zephyrproject-rtos/zephyr#66135), it's one of many possible solutions. I encourage you to discuss it.

    I don't have confirmation that this fix resolves this issue. I would need assistance in verifying this.

  12. tmleman commented on Jan 11, 2024

    @tmleman
    Contributor

    @RDharageswari can you check if the issue still reproduces on the main branch?

  13. lgirdwood commented on Jan 11, 2024

    @lgirdwood
    Member

    @RDharageswari can you check if the issue still reproduces on the main branch?

    @tmleman I guess we need a west update to pick up the Zephyr commit now its merged ?

  14. tmleman commented on Jan 19, 2024

    @tmleman
    Contributor

    @lgirdwood The patch I'm interested in has already been integrated. The kernel version in SOF has also been updated earlier.

  15. abonislawski commented on Jan 23, 2024

    @abonislawski
    Member

    @macchian could you help in fix verification on main branch?

  16. wszypelt commented on Feb 9, 2024

    @wszypelt

    Due to lack of response, reduces to P2
    @RDharageswari Can you confirm that the issue no longer occurs?

  17. added
    P2Critical bugs or normal features
    and removed
    P1Blocker bugs or important features
    on Feb 9, 2024
  18. removed this from the v2.9 milestone on Mar 13, 2024
  19. wszypelt commented on Apr 12, 2024

    @wszypelt

    @RDharageswari Can you confirm that the issue no longer occurs?

  20. wszypelt commented on Jul 19, 2024

    @wszypelt

    Due to the lack of response from the reporting person, I am closing the task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BootFirmware boot or code signing related.IMRRelated to IMR (Isolated Memory Region) on Intel platformsIMR context saveIPC timeoutIPC timeout observedMTLApplies to Meteor Lake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedmtl-005mtl-006suspend-resumeIssues observed when doing system suspend and resume

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions