Repository navigation
HTTP keep alive management #290
Description
In my application I have random connection errors like this:
ClickHouseStatementImpl - Error during connection to ru.yandex.clickhouse.settings.ClickHouseProperties@785d9e37, reporting failure to data source, message: localhost:8123 failed to respond
org.apache.http.NoHttpResponseException: localhost:8123 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
(I've modified the code to print the stacktrace in ClickHouseStatementImpl)
I think the root cause is that in some rare cases, Apache Http client doesn't detect a connection closed by the server (half-closed socket). I have a Wireshark capture which confirms this.
I've seen that the server returns how long it will keep the connection open but it doesn't seem to be used client side:
POST /?compress=1&extremes=0&database=default HTTP/1.1
Content-Length: 55
Content-Type: text/plain; charset=UTF-8
Host: xxxxxx:8123
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_152)
select timezone() FORMAT TabSeparatedWithNamesAndTypes;HTTP/1.1 200 OK
Date: Mon, 14 Jan 2019 08:04:41 GMT
Connection: Keep-Alive
Content-Type: text/tab-separated-values; charset=UTF-8
X-ClickHouse-Server-Display-Name: xxxxxxxxx
Transfer-Encoding: chunked
Keep-Alive: timeout=3
(...)
and
private ConnectionKeepAliveStrategy createKeepAliveStrategy() {
return new ConnectionKeepAliveStrategy() {
@Override
public long getKeepAliveDuration(HttpResponse httpResponse, HttpContext httpContext) {
// in case of errors keep-alive not always works. close connection just in case
if (httpResponse.getStatusLine().getStatusCode() != HttpURLConnection.HTTP_OK) {
return -1;
}
HeaderElementIterator it = new BasicHeaderElementIterator(
httpResponse.headerIterator(HTTP.CONN_DIRECTIVE));
while (it.hasNext()) {
HeaderElement he = it.nextElement();
String param = he.getName();
//String value = he.getValue();
if (param != null && param.equalsIgnoreCase(HTTP.CONN_KEEP_ALIVE)) {
return properties.getKeepAliveTimeout();
}
}
return -1;
}
};
}
So, shouldn't it be better to parse 'Keep-Alive: timeout=3' to get the keep-alive duration, and if it fails, use properties.getKeepAliveTimeout() ?
Activity
fix 一个偶尔出现的 failed to respond 异常, github issue: ClickHouse/clickhouse…
enqueue commentedon Feb 5, 2019
@bgranvea, @ylltw01 how would you design a test for this?
ylltw01 commentedon Feb 20, 2019
@enqueue i am sorry for that，i made a mistake，please ignore
AlexMAS commentedon Nov 22, 2019
Hi,
Sometimes I get the same exception:
Versions:
Could you explain the reason and how to fix it?
esukanovic commentedon Jan 26, 2020
I'm using latest clickhouse-jdbc version 0.2.3 and have Hikari CP configured to use ClickHouseDataSource with all the properties like socketTimeout stc.
"Failed to respond" exception happens ocassionally and it has nothing to do with timeouts since I'm logging all the query/call durations and can see that it's clearly less than timeout period even in case of failure.
Is it possible that Hikari CP is not aware of connections being closed on server side?
Anyone using it and not having this issue?
UPDATE: After configuring connectionTestQuery & validationTimeout for Hikari the issue seems to be solved.
As part of getting connection from pool Hikari now checks whether connection is alive and closes it if necessary.
I still sometimes see "Failed to respond" error in logs but now it's just for the connection test query during 'isConnectionAlive' check.
zhicwu commentedon Feb 1, 2021
Yes, I was able to reproduce the issue by shutting down WireMockServer before client get any response. Retry can mitigate the issue but it won't work for non-idempotent operation like insertion.
keepAliveTimeoutis deprecated since 0.2.5(see details in #515).lewis-ing commentedon Feb 5, 2021
Hi @esukanovic .do you figure it,today i met same error and using Hikari CP for datasource pool,but i use it for mybatis.
chengbinghan commentedon Jan 11, 2022
i meet this when java insert client server momery not enough