Skip to content

[BUG] FW reported error: 13 - Requested power transition failed to complete #7969

Closed
Closed
[BUG] FW reported error: 13 - Requested power transition failed to complete#7969
Assignees
tmleman
Labels
I2SApplies to I2S bus for codec connectionIPC errorIPC error is observedIntel Linux Daily testsThis issue can be found in internal Linux daily testsMTLApplies to Meteor Lake platformP1Blocker bugs or important featuresSDWSoundWirebugSomething isn't working as expectedsuspend-resumeIssues observed when doing system suspend and resume
Milestone
v2.7

Description

@keqiaozhang
keqiaozhang
opened on Jul 20, 2023

Describe the bug
Observed this issue in CI daily test, this issue only happens on MTL-NOCODEC so far. The reproduce rate is 100% when doing the suspend/resume test w/ audio. It should be a FW regression. Will do further checks.

[ 5405.125019] snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx      : 0x47000000|0x0: MOD_SET_DX [data size: 8]
[ 5405.125608] snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx reply: 0x6700000d|0x0: MOD_SET_DX
[ 5405.125616] sof-audio-pci-intel-mtl 0000:00:1f.3: FW reported error: 13 - Requested power transition failed to complete
[ 5405.125798] sof-audio-pci-intel-mtl 0000:00:1f.3: ipc error for msg 0x47000000|0x0
[ 5405.125806] sof-audio-pci-intel-mtl 0000:00:1f.3: ctx_save IPC error: -22, proceeding with suspend

To Reproduce
check-suspend-resume-with-audio.sh -l 50 -m playback

Reproduction Rate
100%

Environment

  1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
  2. Name of the topology file
    • Topology: {sof-ace-tplg/sof-mtl-nocodec.tplg}
  3. Name of the platform(s) on which the bug is observed.
    • Platform: {mtl-nocodec}

dmesg.txt
mtrace.txt

Activity

  1. keqiaozhang
    added
    bugSomething isn't working as expected
    P1Blocker bugs or important features
    IPC errorIPC error is observed
    Intel Linux Daily testsThis issue can be found in internal Linux daily tests
    suspend-resumeIssues observed when doing system suspend and resume
    on Jul 20, 2023
  2. keqiaozhang
    assigned
    aborisovich
    on Jul 20, 2023

  3. keqiaozhang commented on Jul 20, 2023

    @keqiaozhang
    keqiaozhang
    on Jul 20, 2023
    CollaboratorAuthor

    This is a FW regression and caused by #7325.
    The commit ID should be 965e1c1.

  4. mengdonglin
    added
    MTLApplies to Meteor Lake platform
    I2SApplies to I2S bus for codec connection
    on Jul 26, 2023
  5. lgirdwood
    added this to the v2.7 milestone on Jul 26, 2023
  6. wszypelt
    assigned
    tmleman
    on Jul 28, 2023
  7. mengdonglin
    added
    IMRRelated to IMR (Isolated Memory Region) on Intel platforms
    regression identifiedIdentified the commit or PR that introduced a regression
    on Jul 31, 2023

  8. mengdonglin commented on Jul 31, 2023

    @mengdonglin
    mengdonglin
    on Jul 31, 2023
    Collaborator

    Commit series on this D3 /IMR context save support:
    Commit 53d471b andrey.borisovich@intel.com ipc4: notify Host on the failed IPC Device power transition
    Commit 965e1c1 andrey.borisovich@intel.com board: intel_adsp_ace15_mtpm: Enabled MTL IMR context save
    Commit 015488b andrey.borisovich@intel.com ipc4: added D3 support using Zephyr Power Manager API
    Commit 80f27b2 andrey.borisovich@intel.com board: intel_adsp_ace15_mtpm: disabled CONFIG_PM_DEVICE_RUNTIME_EXCLUSIVE
    Commit 4e8040c marcin.szkudlinski@intel.com platform: ace: Add pm notifiers to support Zephyr's D3 transition

    image

  9. mengdonglin commented on Jul 31, 2023

    @mengdonglin
    mengdonglin
    on Jul 31, 2023
    Collaborator

    @keqiaozhang If we remove CONFIG_ADSP_IMR_CONTEXT_SAVE=y to disable IMR context save, can we reproduce this issue with tip of SOF main branch? commit 53d471b is kept and can still report device power transition failure.

  10. 5 remaining items

  11. mengdonglin commented on Aug 1, 2023

    @mengdonglin
    mengdonglin
    on Aug 1, 2023
    Collaborator

    @lgirdwood @mwasko @tmleman @wszypelt @keqiaozhang @fredoh9 @alex-cri
    Summary of this issue in ww31:

    • NOT reproduced on cavs2.5 platforms.
    • Only reproduced on MTL RVP with SDW and NOCODEC (both single core and multi-core).
    • Not reproduced with MTL with HD-Audio till now.
    • Seems higher reproduction rate with multi-core (MTLP_RVP_NOCODEC_MULTICORE).
    • Disabling IMR context save on MTL doesn't help.

    Recent occurrences:

    • ww31.1 on MTLP_RVP_NOCODEC_MULTICORE (report ID: 29577)
    • ww30.5 on MTLP_RVP_SDW, MTLP_RVP_NOCODEC , MTLP_RVP_NOCODEC_MULTICORE (report https://sof-ci.ostc.intel.com/#/result/planresultdetail/29547)
    • ww30.4 on MTLP_RVP_SDW and MTLP_RVP_NOCODEC_MULTICORE (report ID 29509)
    • ww30.3 on MTLP_RVP_NOCODEC_MULTICORE, (report ID 29436)
  12. mengdonglin
    removed
    regression identifiedIdentified the commit or PR that introduced a regression
    on Aug 1, 2023

  13. mengdonglin commented on Aug 1, 2023

    @mengdonglin
    mengdonglin
    on Aug 1, 2023
    Collaborator

    @lgirdwood @ujfalusi @mwasko @abonislawski @keqiaozhang @tmleman @wszypelt
    It seems PR #7995 fixed this issue.
    From latest PR test result https://sof-ci.ostc.intel.com/#/result/planresultdetail/29601
    Let's wait for new daily test result.

  14. keqiaozhang commented on Aug 1, 2023

    @keqiaozhang
    keqiaozhang
    on Aug 1, 2023
    CollaboratorAuthor

    It seems PR #7995 fixed this issue.

    This issue still exists on MTL platforms. I still can reproduce this issue with tip main branch on ww31.2 (daily test report ID 29617), on MTLP_RVP_NOCODEC_MULTICORE. @lgirdwood @tmleman

  15. lgirdwood commented on Aug 1, 2023

    @lgirdwood
    lgirdwood
    on Aug 1, 2023
    Member

    It seems PR #7995 fixed this issue.

    This issue still exists on MTL platforms. I still can reproduce this issue with tip main branch on ww31.2 (daily test report ID 29617), on MTLP_RVP_NOCODEC_MULTICORE. @lgirdwood @tmleman

    @keqiaozhang did the repro rate change after #7995 was merged ? Any other difference to before ?

  16. keqiaozhang commented on Aug 2, 2023

    @keqiaozhang
    keqiaozhang
    on Aug 2, 2023
    CollaboratorAuthor

    @lgirdwood the repro rate is the same, no differences. This issue happened on 3 different platforms in today's daily test.

  17. tmleman
    unassigned
    aborisovich
    on Aug 3, 2023

  18. tmleman commented on Aug 3, 2023

    @tmleman
    tmleman
    on Aug 3, 2023
    Contributor

    23ww31.5: I don't see any error in mtrace but I guess the problem comes from ipc device suspension:
    https://github.com/thesofproject/sof/blob/main/src/ipc/ipc4/handler.c#L1120-L1123

    This could easily be confirmed by finding one of these logs:
    https://github.com/thesofproject/sof/blob/main/src/ipc/ipc-zephyr.c#L102-L117

    I need to reproduce this behavior in a local environment.

  19. tmleman commented on Aug 8, 2023

    @tmleman
    tmleman
    on Aug 8, 2023
    Contributor

    23ww32.3: I'm unable to reproduce this issue, work in progress.

  20. tmleman commented on Aug 11, 2023

    @tmleman
    tmleman
    on Aug 11, 2023
    Contributor

    23ww32.6: root cause: SOF_IPC4_NOTIFY_LOG_BUFFER_STATUS notification added to the IPC queue. FW did not manage to send the message before the D3 entry procedure.

    @keqiaozhang thanks for help with confirmation.

    I see two possible solutions:

    • vA if only SOF_IPC4_NOTIFY_LOG_BUFFER_STATUS is in IPC queue, we continue the D3 transition procedure. The message will be sent when the DSP wakes up (PR8029).
    • vB we try to send pending notification.

    Initially, I would go with version A. In version B, I do not know how HOST will behave when waiting for a SET_DX response when it will receive notification.

  21. kv2019i commented on Aug 14, 2023

    @kv2019i
    kv2019i
    on Aug 14, 2023
    Collaborator

    Option A merged via #8029

  22. fredoh9 commented on Aug 15, 2023

    @fredoh9
    fredoh9
    on Aug 15, 2023
    Contributor

    This problem is not found for two consecutive days

  23. lgirdwood commented on Aug 16, 2023

    @lgirdwood
    lgirdwood
    on Aug 16, 2023
    Member

    This problem is not found for two consecutive days

    @fredoh9 pls close when happy resolved.

  24. fredoh9 commented on Aug 16, 2023

    @fredoh9
    fredoh9
    on Aug 16, 2023
    Contributor

    no issue found in today's daily test too. Closing now

  25. fredoh9
    closed this as completedon Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

I2SApplies to I2S bus for codec connectionIPC errorIPC error is observedIntel Linux Daily testsThis issue can be found in internal Linux daily testsMTLApplies to Meteor Lake platformP1Blocker bugs or important featuresSDWSoundWirebugSomething isn't working as expectedsuspend-resumeIssues observed when doing system suspend and resume

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions