Repository navigation
Large memory usage by gcloud-logging API. #1449
Description
The logging API has a large memory foot print, with approximately 4KB of memory consumed for each log generated.
This was measured with the following program:
package com.google.cloud.runtimes.jetty9;
import java.util.concurrent.TimeUnit;
import java.util.logging.Logger;
public class StackDriverLogging {
static void init() throws Exception {
Logger log = Logger.getLogger(StackDriverLogging.class.getName());
log.info("test info");
log.fine("test fine");
long last = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
// Run 50 times to ensure everything is hot and any queues are max allocated
for (int r=50;r-->0;) {
long total = Runtime.getRuntime().totalMemory();
long free = Runtime.getRuntime().freeMemory();
long used = total - free;
long delta = used - last;
last = used;
System.err.printf("total(%d)-free(%d)=used(%d) delta=%d%n", total, free, used, delta);
for (int i = 1000; i-- > 0;)
log.info("test info");
if (r==10) {
System.err.println("DRAINING...");
// Wait a minute so all queued logs can drain out
TimeUnit.MINUTES.sleep(1);
System.gc();
System.err.println("LAST 10...");
}
Thread.sleep(10);
}
System.err.println("DUMP!");
Thread.sleep(1000000);
}
public static void main(String... args) throws Exception {
init();
}
}Eclipse MAT was used to analyse a head dump taken immediately after 10K log entries were generated, then again once the program had returned to idle and network monitor indicated that all data had been flushed.
After 10L log messages, the heap was 51MB, which reduced to 13.4MB once the logs had been flushed (but no forced GC, so the difference may be larger). Thus at least 38MB appears to be consumed by logging 10K messages, which is 3.9KB per message. The attached reports( 10K Top Consumers.pdf & Idle Top Consumers.pdf ) indicate that 22.9 MB is used by io.grpc.netty.NettyClientTransport$2 and 12.1MB alone is consumed by io.netty.handler.codec.http2.StreamBufferingEncoder.
Ideally, the memory usage per log line could be greatly reduced. Eitherway, it would also be highly desirable to be able to put some limits on the memory consumed by the logging subsystem so that excessively verbose logs are either discarded, summarized, blocked or excepted.
Activity
meltsufin commentedon Dec 6, 2016
Thanks @gregw for a detailed analysis.
@garrettjonesgoogle Can you please take a look at this? 4KB per log message seems excessive.
garrettjonesgoogle commentedon Dec 7, 2016
Unfortunately, google-cloud-java gets its version of Netty indirectly through grpc, so there isn't a super-clean way to update that. Regardless, I would be very interested in seeing what your analysis shows if you update the Netty dependency on your end to 4.1.5 using the exclusions feature of Maven described in #1433 .
gregw commentedon Dec 8, 2016
@garrettjonesgoogle I ran the same test with 4.1.5.Final Netty. It improve the base memory usage, but not the dynamic usage per request. After the 10K log messages total memory usage is now only 45.5MB, but when idle it now reduces to 8.7MB. So there is an approx 5MB saving, but the per message cost is about the same, dominated by 22.8 MB is used by
io.grpc.netty.NettyClientTransport$2and 11.4MB byio.netty.handler.codec.http2.StreamBufferingEncoder.gregw commentedon Jan 19, 2017
As per the discussion in grpc/grpc-java#2554, the issue appears to be caused (or at least aggravated) by the flush size of 1, which results in only an individual log entry being sent per RPC and many log entry lists being created as garbage. The simple flush level will be replaced by a better mechanism called "bundling", to provide timely yet aggregated log entries.
pongad commentedon Jan 23, 2017
@garrettjonesgoogle I see two ways to resolve this
ThresholdBundlingForwarderand friends directly in this classDo you have a preference?
Since this class allows configuring various options on per-handler basis, I think the 2nd option is probably easier to do. Due to the work we're doing on pubsub, the bundling surface feels somewhat unstable. So either way we decide to do it, we might have to rewrite pieces of it. I'll be happy to do the work though.
garrettjonesgoogle commentedon Jan 23, 2017
Bundling was already added to Logging and the GAPIC client was regenerated in d4d494d .
pongad commentedon Jan 24, 2017
@garrettjonesgoogle I somehow missed this. Was the config not in the YAML? I see no reference to bundling in https://github.com/googleapis/googleapis/blob/622cee91552e025d8f2f5733308ee2f633356f98/google/logging/v2/logging_gapic.yaml
garrettjonesgoogle commentedon Jan 24, 2017
Ahh you're right, I haven't submitted it there yet.
pongad commentedon Jan 24, 2017
@gregw After discussing this with Garrett, I think this issue should have been resolved by d4d494d . Could you try again?
meltsufin commentedon Jan 25, 2017
@gregw Can you please take a look?
gregw commentedon Jan 27, 2017
@meltsufin @garrettjonesgoogle
I have rerun the test I did above. The good news is that the network usage appears to be a lot better!
Bad news is that the memory usage appears to be a lot worse:
com.google.api.gax.bundlingcom.google.api.gax.bundlingI'll see if I can attach some detailed reports.
Bad news is that the total image size is a bit bigger at 62MB, of which 54MB is consumed by
com.google.api.gax.bundling. But the good news is that the size remains pretty much constant over the generation of 10K logs. Also good is that the network activity is now hardly noticeable, so the RPC usage appears to be much more efficient.gregw commentedon Jan 27, 2017
Here is a report. Tell me if you need more info.
Export.pdf
31 remaining items
michaelbausor commentedon Mar 17, 2017
This is now released in https://github.com/GoogleCloudPlatform/google-cloud-java/releases/tag/v0.10.0
chore(main): release 2.3.5 (#1449)
chore(deps): update dependency com.google.cloud:google-cloud-logging …
test(deps): update dependency org.graalvm.buildtools:junit-platform-n…
chore(deps): update dependency com.google.cloud:google-cloud-logging …
chore: modify metadata queries to support named schemas (googleapis#1449
chore(main): release 0.127.22-SNAPSHOT (#1449)
chore(main): release 0.127.22-SNAPSHOT (#1449)
deps: update dependency com.google.cloud:google-cloud-core to v2.9.2 (#…
deps: update dependency com.google.cloud:google-cloud-core to v2.9.2 (#…
chore(deps): update dependency com.google.cloud:google-cloud-bigtable…