Segfault when using PyCall with Dates

[mauro3]

Both on 0.6 and 1.0 I get:

julia> using Dates, PyCall                                                                                                                                             
                                                                                                                                                                       
julia> py"$([now()])"                                                                                                                                                  
                                                                                                                                                                       
signal (11): Segmentation fault
in expression starting at no file:0
unknown function (ip: 0x7f63d8dbc26a)                                                                                                                                  
macro expansion at /home/mauro/.julia/packages/PyCall/akNFy/src/exception.jl:81 [inlined]                                                                              
Type at /home/mauro/.julia/packages/PyCall/akNFy/src/pydates.jl:77
array2py at /home/mauro/.julia/packages/PyCall/akNFy/src/exception.jl:82
array2py at /home/mauro/.julia/packages/PyCall/akNFy/src/conversions.jl:325 [inlined]
Type at /home/mauro/.julia/packages/PyCall/akNFy/src/conversions.jl:327
jl_fptr_trampoline at /home/mauro/julia/julia-1.0/src/gf.c:1829                                                                                                        
jl_apply_generic at /home/mauro/julia/julia-1.0/src/gf.c:2182                                                                                                          
do_call at /home/mauro/julia/julia-1.0/src/interpreter.c:324                                                                                                           
eval_value at /home/mauro/julia/julia-1.0/src/interpreter.c:428                                                                                                        
eval_stmt_value at /home/mauro/julia/julia-1.0/src/interpreter.c:363 [inlined]                                                                                         
eval_body at /home/mauro/julia/julia-1.0/src/interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/mauro/julia/julia-1.0/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f63eac667bf)
unknown function (ip: 0x3)
jl_interpret_toplevel_thunk at /home/mauro/julia/julia-1.0/src/interpreter.c:808                                                                                       
jl_toplevel_eval_flex at /home/mauro/julia/julia-1.0/src/toplevel.c:787                                                                                                
jl_toplevel_eval_in at /home/mauro/julia/julia-1.0/src/builtins.c:622                                                                                                  
eval at ./boot.jl:319                                                                                                                                                  
jl_apply_generic at /home/mauro/julia/julia-1.0/src/gf.c:2182
eval_user_input at /home/mauro/julia/julia-1.0/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85                                                                         
macro expansion at /home/mauro/julia/julia-1.0/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:117 [inlined]                                                              
#28 at ./task.jl:259
jl_apply_generic at /home/mauro/julia/julia-1.0/src/gf.c:2182
jl_apply at /home/mauro/julia/julia-1.0/src/julia.h:1536 [inlined]
start_task at /home/mauro/julia/julia-1.0/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 9421455 (Pool: 9419829; Big: 1626); GC: 20
zsh: segmentation fault (core dumped)  julia                                                                                                                           

which means I cannot plot Dates with PyPlot (but I used to be able to do this). Is my installation at fault? I'm on Arch Linux and I use the system python 3.7.0.

[tkf]

It's fixed in master #540 but it's not released yet.

[scottstanie]

Is this still unfixed in the released version? I'm getting segfaults when I convert a Pandas series into Dates:

julia> dts
PyObject 0      2015-01-02
1      2015-01-03
2      2015-01-04
3      2015-01-05
          ...
1092   2017-12-29
1093   2017-12-30
1094   2017-12-31
1095   2018-01-01
Name: dt, Length: 1077, dtype: datetime64[ns]

julia> convert(Array{Dates.Date, 1}, dts)

signal (11): Segmentation fault
in expression starting at REPL[23]:1
PyObject_IsInstance at /usr/lib64/libpython3.4m.so.1.0 (unknown line)
unknown function (ip: 0x7f26fbffc4df)
Allocations: 97739326 (Pool: 97723454; Big: 15872); GC: 115
Segmentation fault (core dumped)

[scottstanie]

Actually it turns out that a few of the dates in dts were PyObject NULL, i think from being a NaN in python... is that already known that trying to convert a series with NaNs will cause a segfault? It took me a awhile to figure that out since the error above was the only thing printed.

EDIT: There is something weirder with this... I can have an array with no NULLs, and convert(Array{Date}, a) still segfaults even when [Date(d) for d in a] works:

Julia Version 1.3.0-rc1.0
Commit 768b25f6a8 (2019-08-18 00:04 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_NUM_THREADS = 8


julia> [Date(d) for d in a]
1077-element Array{Date,1}:
 Date(2015, 1, 2)
 Date(2015, 1, 3)
 Date(2015, 1, 4)
 ⋮
  Date(2017, 12, 31)
 Date(2018, 1, 1)

julia> convert(Array{Date}, a)

signal (11): Segmentation fault
in expression starting at REPL[8]:1
PyObject_IsInstance at /usr/lib64/libpython3.4m.so.1.0 (unknown line)
unknown function (ip: 0x7f4fa43124df)
Allocations: 65796851 (Pool: 65785537; Big: 11314); GC: 73
Segmentation fault (core dumped)

[stevengj]

Converting a series with NaNs to Date should throw an error…

[scottstanie]

^^ Sorry, see the edit- IT seems like NaNs don't have to be in the series.
And just double checking those types:

 unique([typeof(d) for d in a])
1-element Array{DataType,1}:
 DateTime

whereas before the NULL one had the PyObject types:

unique([typeof(d) for d in a])
2-element Array{DataType,1}:
 DateTime
 PyObject

[stevengj]

Can you give a self-contained example in which a NULL PyObject is showing up?

[scottstanie]

I'll try to find one- I'm struggling to get the NULL back into the conversion from the pandas objects... Also the smaller examples seem to not segfault when I have convert(Array{Date}, arr) for an array of datetimes.
I'll post one when I can come up with it