Solaris follow-up

[mattdowle]

#3954 additional tracing and verbose revealed more detail.
The -2 is a compression error then (not malloc or write, as there was some doubt before), and now we have a zlib version number too. However, the nth=0 and 800MB 1-row buffers output looks suspect, so that should be looked at first. That output is from the verbose=TRUE I added to the test temporarily 'just in case'.

There are some script portability issues to Solaris as well. I've done some research and found these:
https://stackoverflow.com/questions/5725296/difference-between-sh-and-bash
http://manpages.ubuntu.com/manpages/disco/en/man1/checkbashisms.1.html
https://wiki.ubuntu.com/DashAsBinSh
If checkbashisms works then :

• add to CRAN_Release
• suggest it be added to R-exts

Also:

• The "Please retry with verbose=TRUE" part of the error message should not be included if verbose is already TRUE.

---

Reply thanks to Prof Ripley who ran my attempt on Solaris :

It does not even install:

** using staged installation
./configure: !: not found
./configure: !: not found
zlib 1.2.8 is available ok
./configure: test: unknown operator -lz

You seem to be writing bash, not Bourne shell.

And why should pkg-config --libs zlib be just "-lz"? It certainly is
not what the standard zlib package does ...

% pkg-config --libs zlib
-L/opt/csw/lib -lz

If I alter the configure script to run under bash, this gets to

Running the tests in ‘tests/main.R’ failed.
Last 13 lines of output:
omp_get_thread_limit() 1024
omp_get_max_threads() 16
OMP_THREAD_LIMIT unset
OMP_NUM_THREADS unset
RestoreAfterFork true
data.table is using 8 threads. See ?setDTthreads.
No list columns are present. Setting sep2='' otherwise quote='auto'
would quote fields containing sep2.
Column writers: 3 3
args.doRowNames=0 args.rowNames=0 doQuote=-128 args.nrow=200
args.ncol=0 eolLen=2
maxLineLen=51. Found in 0.000s
Writing bom (false), yaml (0 characters) and column names (true) ...
done in 0.000s
Writing 200 rows in 0 batches of 1 rows (each buffer size 200MB,
showProgress=8, nth=0) ... Error in fwrite(DT, file = f1 <-
tempfile(fileext = ".gz"), verbose = TRUE) :
Error -2: compression error. Please retry with verbose=TRUE and
search online for this error message.
Calls: test.data.table -> sys.source -> eval -> eval -> fwrite
Execution halted

I have no idea what it is asking to 'retry with verbose=TRUE', nor what
'this error message' is (that seems to be part of the message).

BDR

[mattdowle]

The args.ncol=0 output baffled me for a while. That's just a %d which should be %lld for nrow in the DTPRINT line 618.

• %d => %lld on line 618
• same on line 775 causing incorrect verbose output "0 batches of 1 rows"
• add output= checks to those tests

So it looks like compressbuff really is returning -2 on line 865. In version 1.2.8 of zlib. Any ideas @philippechataignon or anyone? Seems like some zlib history sleuthing needed. 1.2.8 was 28 Apr 2013 followed by 1.2.9 (31 Dec 2016). The last one (the version I have on Ubuntu) was 1.2.11 (15 Jan 2017).

When our printing problem is fixed for Solaris, this is the correct output (what I see locally) :

DT = data.table(a=rep(1:2,each=100), b=rep(1:4,each=25))
fwrite(DT, file=f1<-tempfile(fileext=".gz"), verbose=TRUE)

No list columns are present. Setting sep2='' otherwise quote='auto' would quote fields containing sep2.
Column writers: 3 3 
args.doRowNames=0 args.rowNames=0 doQuote=-128 args.nrow=200 args.ncol=2 eolLen=1
maxLineLen=51. Found in 0.000s
Writing bom (false), yaml (0 characters) and column names (true) ... done in 0.002s
Writing 200 rows in 1 batches of 200 rows (each buffer size 8MB, showProgress=1, nth=1)

[mattdowle]

Looks like compressbuff() is our own wrapper, which calls deflate() on line 572.

-2 from deflate is Z_STREAM_ERROR. zlib manual for v1.2.11 here contains :

deflate() returns Z_OK if some progress has been made (more input processed or more output produced), Z_STREAM_END if all input has been consumed and all output has been produced (only when flush is set to Z_FINISH), Z_STREAM_ERROR if the stream state was inconsistent (for example if next_in or next_out was Z_NULL or the state was inadvertently written over by the application), or Z_BUF_ERROR if no progress is possible (for example avail_in or avail_out was zero). Note that Z_BUF_ERROR is not fatal, and deflate() can be called again with more input and more output space to continue compressing.

That's from the manual for 1.2.11 though and 1.2.8 on Solaris might be different.

[mattdowle]

That manual for deflateInit2 contains :

This is another version of deflateInit with more compression options. The fields next_in, zalloc, zfree and opaque must be initialized before by the caller.

But on line 554 in our init_stream, we initialize 3 of those but not next_in too.

[mattdowle]

I found the manual for 1.2.7 (the version before 1.2.8 on CRAN's Solaris) but I don't see any differences in this area. deflateInit2 was in 1.2.7 too with the same advice. zlib sets a msg field on error that we're not using. So I'm leaning towards initializing next_in as the manual says, plus including z_stream.msg in the error, and then asking Prof Ripley to rerun to see where we are then.