Skip to content

Check for a valid trace for ALL tests #297

Closed
#662
Closed
Check for a valid trace for ALL tests#297
#662
Assignees
marc-hb
Labels
False Pass / green failureP1Blocker bugs or important featuresarea:logsLog and results collection, storage, etc.state:blockedtype:enhancementNew framework feature or request

Description

@plbossart
plbossart
opened on Jul 28, 2020

We need to double-check that the trace is functional and that sof-logger reports at least the firmware details and a dai trigger.

If there is no trace then there's no point in checking results and debugging further.

cc:

  • many others, follow the links.

Activity

  1. mengdonglin
    added
    P1Blocker bugs or important features
    type:enhancementNew framework feature or request
    on Jul 31, 2020
  2. mengdonglin
    assigned
    marc-hb
    on Jul 31, 2020

  3. marc-hb commented on Sep 11, 2020

    @marc-hb
    marc-hb
    on Sep 11, 2020
    Collaborator

    First part submitted in #373, please help review.

  4. marc-hb commented on Sep 15, 2020

    @marc-hb
    marc-hb
    on Sep 15, 2020
    Collaborator

    From @xiulipan in #373

    it that behavior right? Do we need to fail the case if we could not get the logger? As we have standalone test case for logger.

  5. marc-hb commented on Sep 15, 2020

    @marc-hb
    marc-hb
    on Sep 15, 2020
    Collaborator

    Do we need to fail the case if we could not get the logger?

    I think what you're asking is: "is the logger a low priority feature"? I read the description of this request as a pretty clear "no" but I'd like to hear more from others.

    As we have standalone test case for logger.

    It's risky to rely on a single test, if the test is broken then the feature can be broken and no one will notice. Even if the test is not broken, the logger could be broken only in some scenarios not tested by the logger test. I think the links above are some real-world examples of either.

    A test is never meant to test only one thing, the more extras each test can test (in no extra time) and the better. Unexpected failures mean extra bugs are found which is great.

    These questions sounds strange to me at a time when the logger seems to be working fine, I mean all tests in #373 pass right now. I mean it sounds very strange to worry about failures we don't even have right now (this reminds me of recent #372) It almost sounds like like a desire to find FEWER bugs / and have more "green failures"?

  6. xiulipan commented on Sep 15, 2020

    @xiulipan
    xiulipan
    on Sep 15, 2020
    Contributor

    @marc-hb What I want to make clarify about is the criterion for our test case pass/fail.
    Should we make aplay test fail if a debug feature is not working?

    A test is never meant to test only one thing, the more extras each test can test (in no extra time) and the better.

    I do not think we are adding test cases in this way. Each test case in this repo is focusing in a single feature. This will help us to narrow down the issue.

  7. marc-hb commented on Sep 15, 2020

    @marc-hb
    marc-hb
    on Sep 15, 2020
    Collaborator

    I do not think we are adding test cases in this way. Each test case in this repo is focusing in a single feature. This will help us to narrow down the issue.

    This is :

    • neither possible because sof-test are not unit tests. They all run from user space so each test involves many lines of code
    • nor desirable unless you want to find fewer bugs - as it happened in the links above and elsewhere.

  8. marc-hb commented on Sep 18, 2020

    @marc-hb
    marc-hb
    on Sep 18, 2020
    Collaborator

    I mean all tests in #373 pass right now.

    They didn't pass, they only appear to pass because I misused exit_failure=2 in #373 where 2 means "Not Applicable" and Not Applicable is green. I didn't know 2 was a special value.

    I mean it sounds very strange to worry about [logger] failures we don't even have right now

    Now we have one: pretty much all tests involving the logger fail to kill -INT it at the end, see new bug https://github.com/thesofproject/sof/issues/3433 [BUG] sof-logger resisted pkill -INT, using -KILL

  9. marc-hb commented on Sep 18, 2020

    @marc-hb
    marc-hb
    on Sep 18, 2020
    Collaborator

    multipipeline tests returning an empty trace: #380 (comment)

  10. marc-hb
    added 2 commits that reference this issue on Sep 18, 2020

    hijack.sh: fail if the logger trace is empty

    b43659e

    hijack.sh: fail if the logger trace is empty

    88adae1

  11. marc-hb commented on Sep 18, 2020

    @marc-hb
    marc-hb
    on Sep 18, 2020
    Collaborator

    multiple-pipeline-capture.sh and multiple-pipeline-playback.sh have a consistently empty trace on about half the platforms: https://sof-ci.01.org/softestpr/PR383/build194/devicetest/

    2020-09-18 05:13:41 UTC [REMOTE_INFO] nlines=1 /home/ubuntu/sof-test/logs/multiple-pipeline-playback/2020-09-18-05:13:29-16938/slogger.txt
2020-09-18 05:13:41 UTC [REMOTE_ERROR] Empty logger trace
2020-09-18 05:13:41 UTC [REMOTE_INFO] Test Result: FAIL!

    @aiChaoSONG does this seem related to your #359 PR?

  12. marc-hb
    added a commit that references this issue on Sep 18, 2020

    hijack.sh: fail if the logger trace is empty

    c3982e1

  13. aiChaoSONG commented on Sep 18, 2020

    @aiChaoSONG
    aiChaoSONG
    on Sep 18, 2020

    @marc-hb Yes, they are related. without #359, these two case are actually testing nothing, and always pass. It is the sof-logger issue which makes the multiple-pipeline-playback/capture failed after your #373 patch

  14. 15 remaining items

  15. kv2019i
    added 2 commits that reference this issue on Nov 12, 2021

    debugability: logger: add note on DMA trace limitations

    ff4ff8d

    debugability: logger: add note on DMA trace limitations

    9cf4ded
  16. lgirdwood
    added a commit that references this issue on Nov 15, 2021

    debugability: logger: add note on DMA trace limitations

    17ca3fe

  17. marc-hb commented on Jan 21, 2022

    @marc-hb
    marc-hb
    on Jan 21, 2022
    Collaborator

    More DMA unreliability:

  18. mengdonglin commented on Mar 20, 2022

    @mengdonglin
    mengdonglin
    on Mar 20, 2022
    Contributor

    Low priority to P2 as SOF is going to use native Zephyr log implementation. @marc-hb @plbossart

  19. mengdonglin
    added
    P2Critical bugs or normal features
    and removed
    P1Blocker bugs or important features
    on Mar 20, 2022

  20. marc-hb commented on Mar 29, 2022

    @marc-hb
    marc-hb
    on Mar 29, 2022
    Collaborator

    I don't understand, Zephyr bugs can be solved without any logs?

  21. marc-hb
    added
    P1Blocker bugs or important features
    and removed
    P2Critical bugs or normal features
    on Mar 31, 2022

  22. marc-hb commented on Jul 26, 2022

    @marc-hb
    marc-hb
    on Jul 26, 2022
    Collaborator

    ... should finish this but it's currently blocking, triggering some failures.

  23. mengdonglin
    added
    state:blocked
    on Sep 21, 2022

  24. mengdonglin commented on Sep 21, 2022

    @mengdonglin
    mengdonglin
    on Sep 21, 2022
    Contributor

    Blocked by mtrace enabling atm.

  25. marc-hb commented on Sep 28, 2023

    @marc-hb
    marc-hb
    on Sep 28, 2023
    Collaborator

    This has been finally fixed by @kv2019i:

    And it does catch mtrace issues like thesofproject/linux#4618 and others.

    The test works, closing.

  26. marc-hb
    closed this as completedon Sep 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

False Pass / green failureP1Blocker bugs or important featuresarea:logsLog and results collection, storage, etc.state:blockedtype:enhancementNew framework feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions