Skip to content

Unsanitized IPC input is used for memory flags #8832

Closed
Closed
Unsanitized IPC input is used for memory flags#8832
Assignees
marc-hb
Labels
P3Low-impact bugs or featuresbugSomething isn't working as expected
Milestone
v2.10

Description

@marc-hb
marc-hb
opened on Feb 2, 2024

./scripts/fuzz.sh -o fuzz-stdout.txt -t 600 fails systematically after L3_HEAP commit 58a42e5

Example: https://github.com/thesofproject/sof/actions/runs/7763189541/job/21174929628

This may take a few minutes but never much longer.

P1 because this is affecting daily tests and our ability to fuzz.

Originally posted by @marc-hb in #8632 (comment)

PR #8632 is probably just the messenger but this fuzzing failure looks like a "good catch" to me.

Since #8632 was merged, one of the new k_panic() gets triggered because caps & SOF_MEM_CAPS_L3 is true even when CONFIG_L3_HEAP is false.

I think the reason caps & SOF_MEM_CAPS_L3 is true is because... caps comes directly from untrusted IPC input!? Why would IPCs be able to set caps directly?

The panic happens when ipc_glb_tplg_buffer_new() does this:

ret = ipc_buffer_new(ipc, (struct sof_ipc_buffer *)ipc->comp_data);

At this point comp_data looks like it came straight from the fuzzer's untrusted input:

(gdb) p /x *desc

$5 = {comp = {hdr = {size = 0x66, cmd = 0x3020ffff}, id = 0xb6b6b600, type = 0xb6b6b6b6, pipeline_id = 0xb6b6b6b6, core = 0xffffb236, 
    ext_data_length = 0xffffffff}, size = 0xffffffff, caps = 0xffffffff, flags = 0xffffffff, reserved = 0xffffffff}

#0  rballoc_align (flags=0, caps=4294903040, bytes=86376703, align=64) at sof/zephyr/lib/alloc.c:398
#1  0x0831d38c in buffer_alloc (size=86376703, caps=4294903040, flags=4294967295, align=64, is_shared=false)
    at sof/src/audio/buffer.c:58
#2  0x082e4d6b in buffer_new (desc=0x8b184c0 <heapmem+1728>, is_shared=false) at sof/src/ipc/ipc-helper.c:48
#3  0x082cc78e in ipc_buffer_new (ipc=0x8b183c0 <heapmem+1472>, desc=0x8b184c0 <heapmem+1728>) at sof/src/ipc/ipc3/helper.c:459
#4  0x082b7249 in ipc_glb_tplg_buffer_new (header=807469055) at sof/src/ipc/ipc3/handler.c:1305
#5  0x082b090c in ipc_glb_tplg_message (header=807469055) at sof/src/ipc/ipc3/handler.c:1416
#6  0x082aff8a in ipc_cmd (_hdr=0x8b184c0 <heapmem+1728>) at sof/src/ipc/ipc3/handler.c:1651
#7  0x082fcb1b in ipc_platform_do_cmd (ipc=0x8b183c0 <heapmem+1472>) at sof/src/platform/posix/ipc.c:162
#8  0x082e2827 in ipc_do_cmd (data=0x8b183c0 <heapmem+1472>) at sof/src/ipc/ipc-common.c:328
#9  0x0836a27a in task_run (task=0x8b183e8 <heapmem+1512>) at sof/zephyr/include/rtos/task.h:94

Activity

  1. marc-hb commented on Feb 2, 2024

    @marc-hb
    marc-hb
    on Feb 2, 2024
    CollaboratorAuthor

    Debugging this is much easier with zephyrproject-rtos/zephyr#68494 and a bit easier with #8831

  2. marc-hb
    changed the title [-]I debugged this a bit. This PR is probably just the messenger but this fuzzing failure looks like a "good catch" to me.[/-] [+]Unsanitized IPC input is used for memory flags[/+] on Feb 2, 2024
  3. marc-hb
    added
    bugSomething isn't working as expected
    P1Blocker bugs or important features
    on Feb 2, 2024

  4. cujomalainey commented on Feb 4, 2024

    @cujomalainey
    cujomalainey
    on Feb 4, 2024
    Contributor

    What is the reason we are just not reverting the change?

  5. marc-hb commented on Feb 5, 2024

    @marc-hb
    marc-hb
    on Feb 5, 2024
    CollaboratorAuthor

    The change looks like just the messenger to me. Commit 58a42e5 adds a k_panic() which stops the fuzzer now but I believe the unsanitized IPC input has always been used for memory flags even before that k_panic().

    Fuzzing is great but it's not a silver bullet. In this case, fuzzing never seemed to notice the unsanitized flags because they never caused any obvious corruption?

  6. cujomalainey commented on Feb 6, 2024

    @cujomalainey
    cujomalainey
    on Feb 6, 2024
    Contributor

    Agreed, I thought the commit added the fallthrough case not just expressing already broken logic, I was under a bad assumption.

  7. lgirdwood commented on Feb 7, 2024

    @lgirdwood
    lgirdwood
    on Feb 7, 2024
    Member

    @marc-hb can you fix and add validation checks around the memory types passed to IPC. Thanks.

  8. lgirdwood
    assigned
    marc-hb
    on Feb 7, 2024
  9. marc-hb
    added 3 commits that reference this issue on Feb 9, 2024

    ipc-helper.c: reject invalid SOF_MEM_CAPS_* bits

    0add0a6

    ipc-helper.c: reject invalid SOF_MEM_CAPS_* bits

    cd7f82c

    ipc-helper.c: reject invalid SOF_MEM_CAPS_* bits

    12aff94

  10. marc-hb commented on Feb 10, 2024

    @marc-hb
    marc-hb
    on Feb 10, 2024
    CollaboratorAuthor

    I have a quick and dirty hack that is passing all the tests in #8850 but it will break again whenever we add a new MEM_CAPS bit.

    What is the reason we are just not reverting the change?

    I still don't think this L3 heap should be reverted but @jxstelter I think it should be reworked to better handle invalid inputs. k_panic() is simply too extreme to handle invalid inputs (and fuzzing is just the messenger).

    In the meantime I submitted a major rework of fuzz.sh because it was good enough for CI but really too inconvenient and too slow for interactive use. With #8851 it's great for both, please review.

  11. jxstelter
    added 2 commits that reference this issue on Feb 12, 2024

    zephyr: lib: alloc: k_panic is not required in failure scenario

    5f267da

    lib: alloc: k_panic is not required in failure case

    6fc162f

  12. 3 remaining items

  13. marc-hb
    added
    P3Low-impact bugs or features
    on Feb 27, 2024
  14. marc-hb
    added 3 commits that reference this issue on Feb 27, 2024

    zephyr/wrapper.c: dereference NULL in POSIX for better k_panic() trace

    aafa56b

    ipc-helper.c: reject invalid SOF_MEM_CAPS_* bits

    9cd5f71

    zephyr/wrapper.c: dereference NULL in POSIX for better k_panic() trace

    dcf6b6b
  15. lgirdwood
    added a commit that references this issue on Feb 28, 2024

    zephyr/wrapper.c: dereference NULL in POSIX for better k_panic() trace

    884305b
  16. marc-hb
    added a commit that references this issue on Feb 28, 2024

    ipc-helper.c: reject invalid SOF_MEM_CAPS_* bits

    10cba1f

  17. marc-hb commented on Feb 29, 2024

    @marc-hb
    marc-hb
    on Feb 29, 2024
    CollaboratorAuthor

    One last open PR on this topic and then we can close:

  18. kv2019i commented on Mar 4, 2024

    @kv2019i
    kv2019i
    on Mar 4, 2024
    Collaborator

    Stable-v2.9 branched, this didn't make the cut, bumping to 2.10.

  19. kv2019i
    modified the milestones: v2.9, v2.10 on Mar 4, 2024
  20. kv2019i
    added a commit that references this issue on Mar 5, 2024

    ipc-helper.c: reject invalid SOF_MEM_CAPS_* bits

    4f59ee8

  21. cujomalainey commented on Mar 5, 2024

    @cujomalainey
    cujomalainey
    on Mar 5, 2024
    Contributor

    @kv2019i given this is a security issue, can we not hotfix?

  22. kv2019i commented on Mar 6, 2024

    @kv2019i
    kv2019i
    on Mar 6, 2024
    Collaborator

    @cujomalainey wrote:

    @kv2019i given this is a security issue, can we not hotfix?

    I actually thought this was a follow-up and the primary issue was already fixed (and thus the P3 priority).

    But if not, let's indeed backport. The main PR is merged yesterday, @marc-hb can you submit a backport to stable-v2.9?

    I think we can then close this issue, right? Or anything else pending?

  23. marc-hb commented on Mar 7, 2024

    @marc-hb
    marc-hb
    on Mar 7, 2024
    CollaboratorAuthor

    I think this was a potential security issue. But it's most likely not as long as unknown flags are ignored or rejected. #8853 is already in stable-v2.9 so I think it's enough.

  24. marc-hb
    closed this as completedon Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P3Low-impact bugs or featuresbugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions