You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tensorflow-core-api:0.2.0-SNAPSHOT is failing on AWS EC2 instances with Tesla V100 GPUs.
all the core-api tests failed with error code:
CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
I have root caused the issue to be CUDA compute 7.0 is not enabled during compilation, running the following command and build from source again fixed the issue. export TF_CUDA_COMPUTE_CAPABILITIES=7.0
Somehow in the release build of tensorflow-core-api, compute 7.0 is not enabled, but compute 3.5 and 7.0 should be the default capability of TF 2.3 according to here. The python packages and main repo built from source works fine without any modification.
Activity
saudet commentedon Sep 10, 2020
BTW, from what I can see, it's compiling PTX code for 3.5 and 5.2, so it should still work.
Is it because you have a need to disable the JIT compiler?