Implementing a more robust caching mechanism

[chudur-budur]

We need to have a more robust function/kernel caching mechanism. An example could be something like that has been implemented in numba.

https://numba.readthedocs.io/en/stable/developer/caching.html

https://github.com/numba/numba/blob/main/numba/core/caching.py

However the numba implementation uses files to save compiled (pickled) kernels. Here are some ideas:

• Instead of using files, we can keep using the memory (dictionary)
• Implement a queuing mechanism that prioritizes the compiled functions/kernels based on cache hit.
• Optimize the file access: we will start saving in file only when the size of the dictionary reaches a certain limit, etc.

The caching will be used here:

https://github.com/IntelPython/numba-dpex/blob/main/numba_dpex/compiler.py