Applying Transform Chain to Geometric Objects

[vikashg]

Is your feature request related to a problem? Please describe.
Sometimes we need to apply transforms to geometric objects (point clouds, lines, curves, and meshes), particularly in radiotherapy planning and object detection.

Describe the solution you'd like
A good solution would be a branch core library that can handle all the geometric transforms while accounting for the image metadata like image spacing and size. A geometric transform chain should be callable in a similar way as we call the regular image transforms. Something like

from monai.transforms.geometric import LoadPoints, RotatePoints

In an ideal case when we have an image annotation pair. We apply the same set of transformation to both objects. But for geometric objects there should be some way of passing messages to the annotations transform chain. For example you are rotating images and annotations. You can apply the rotation to the annotation, but after rotation the size of the image changes in different axes and sometimes it is important that the geometric transform should know about these changes for accurately applying the transforms. So there should be a message passing mechanism between two transform chains. More relevance in discussion #4024

Describe alternatives you've considered
I have written my own transforms to do all these things but it is a bit hacky and not standardized. I have my geometric labels from open-source tools like labelme. My code handles labelme annotations (in a json) but going in I think we will need to have support for STL, DICOM RT annotations.

Additional context
If there are some people working on it, please let me know as I have been working on it and will like to contribute to this branch.

[rijobro]

We really need to make progress on this PR before adding in point transforms (which is definitely on our agenda).

[MMelQin]

Just to add that MONAI Deploy App SDK has added the STL generation from volume segmentation image, closed issue and PR.

For generating DICOM RTStruct instance, for use in radiotherapy planning, the contour(s) of segment(s) on each segmentation image slice need to be calculated, with and without smoothing applied. I had plans to implement this using cv2 (more like porting from a earlier impl), though a utility/transform in MONAI core is better and can be for more general use.

[Nic-Ma]

Hi @vikashg @MMelQin ,

Thanks for the feedback, I agree with the requirements for point transforms, we are working in this direction.
About for your particular question to calculate the contour, we have a similar transform LabelToContour:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/transforms/post/array.py#L508
And you can find some usage example in the tutorial:
https://github.com/Project-MONAI/tutorials/blob/master/modules/postprocessing_transforms.ipynb

Thanks.

[ericspod]

As we consider more and more non-image data the architecture we use to integrate those transforms has to be considered. I'm not sure having a separate section in the transforms for geometric transforms is the way to go, but instead these would be treated at the same level as image transforms. A transform such as RandAffine would be applicable to image and geometry data, so RandAffined would also, rather than having separate transforms which would have to share state somehow. How we represent geometry is another issue since this can be a point cloud in 2D/3D with many other types of associated matrices such as connectivity.