fiber default BodyLimit (4MB) causes 413 for large external-dns payloads

[ResamVi]

Is your feature request related to a problem? Please describe.

We are running external-dns to create CNAME records
and our issue is that a POST /records currently returns a status 413.

A lot of dnsendpoints resources have built up and our external-dns instance is trying to catch up and create 8121 dnsendpoints:

The payload that our external-dns sends to the webhook is 5MB but because your fiber is initialized using no explicit BodyLimit it uses the default of 4MB

Can you please disable it or set a bigger limit that fits our scale?

Describe the solution you'd like

Set BodyLimit to higher value here

Describe alternatives you've considered

external-dns is not source code that we maintain and this webhook should be able to handle higher loads or at least be able to configure a limit for users

Search

• I did search for other open and closed issues before opening this.

Code of Conduct

• I agree to follow this project's Code of Conduct

Additional context

No response

[slm0n87]

Hi @ResamVi , thx for the report and the PR.
I tried to reproduce the payload > 4MB but had no success yet.
For example with having a zone > 8000 records managed by external-dns with quite long record names, I could not reproduce the issue.

[ResamVi]

Okay. so the actual conditions to reproduce are:

A) The external-dns needs to run after all dnsendpoints are created (to simulate the "traffic jam")
B) Not 8000 but 18'500 changes need to occur (creates, updates, deletes)

So that the 4MB threshhold is reached.

Expecting 18500 changes at once is a bit extreme but once you reach that state you're not getting out of it without manual intervention.

I would leave the decision up to you to decide how to proceed