Repository navigation
Kafka: Application maximum poll interval exceeded #893
Description
My consumer
consumer:
dsn: "..."
global:
### Make sure this is unique for each application / consumer group and does not change
### Otherwise, Kafka won't be able to track your last offset and will always start according to
### `auto.offset.reset` setting.
### See Kafka documentation regarding `group.id` property if you want to know more
group.id: 'backend'
metadata.broker.list: '...'
security.protocol: sasl_ssl
ssl.ca.location: '%kernel.project_dir%/config/kafka/ca.pem'
sasl.username: '...'
sasl.password: '...'
sasl.mechanisms: "SCRAM-SHA-256"
max.poll.interval.ms: '25000'
enable.auto.commit: 'false'
enable.auto.offset.store: 'false'
topic:
auto.offset.reset: beginning
### Commit async is true by default since version 0.9.9.
### It is suggested to set it to true in earlier versions since otherwise consumers become extremely slow,
### waiting for offset to be stored on Kafka before continuing.
commit_async: true
Is always producing these messages:
%4|1560006029.619|MAXPOLL|rdkafka#consumer-21| [thrd:main]: Application maximum poll interval (25000ms) exceeded by 333ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1560006054.153|MAXPOLL|rdkafka#consumer-23| [thrd:main]: Application maximum poll interval (25000ms) exceeded by 333ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Do you have an idea, how to fix this?
Activity
Steveb-p commentedon Jun 9, 2019
@KonstantinCodes you're running a long running producer of some sort?
What phprdkafka / librdkafka combination you're running?
Overall, it is kind of related to #749 (not running
pollspecifically). They are warnings, so your application should work despite them (confirm?), but obviously we should do something about it.EDIT: I've looked up the warning message and - as I was kinda expecting - that this message is coming from a change introduced for librdkafka 1.0.
KonstantinCodes commentedon Jun 10, 2019
@Steveb-p yeah it's kind of interesting.
I'm running a long running consumer, but the messages start popping up after 25 seconds.
So from what I read, you just have to run
KafkaConsumer::consume($timeout). Where the $timeout is smaller than your kafka config valuemax.poll.interval.ms.I have tried configuring the enqueue transport this way, but for some reason it's not working. I debugged it but didn't come to an conclusion. So I built my own KafkaTransport for Symfony Messenger, and there it's working fine. So I'm really puzzled, how this is happening. Mind you, I'm specifying the exact same config values for both.
I might push the code, just to provide a reference.
Steveb-p commentedon Jun 10, 2019
Please do, but overall it should work the same. After all, it should be the same KafkaConsumer class. It should be something not really related to enqueue, but PhpRdKafka itself. Gotta do some digging 😄
stale commentedon Jul 18, 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
dirk39 commentedon Sep 12, 2019
This issue should be solved by #947
Steveb-p commentedon Sep 12, 2019
Probably yes actually. Add "Fixes #893" to that PR so this will close automatically once it's merged :)
stale commentedon Oct 12, 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.