40-validation-early-stop

[madil90]

Fixes #40 fixes #8

Description

Added example for model validation every step with metric logging. Added util functions for early stopping.

Status

Ready

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or new feature that would cause existing functionality to change)
• New tests added to cover the changes
• Docstrings/Documentation updated

with this PR, running the training program gives:

MONAI version: 0.0.1                                                                                                                                                                                                                                          Python version: 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:30:03)  [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
Numpy version: 1.18.1                                                                                                                                                                                                                                         Pytorch version: 1.4.0                                                                                                                                                                                                                                        
Ignite version: 0.3.0                                                                                                        
torch.Size([10, 1, 64, 64, 64]) torch.Size([10, 1, 64, 64, 64])                                                              
INFO:ignite.engine.engine.Engine:Engine run starting with max_epochs=3.                                                      
INFO:ignite.engine.engine.Engine:Epoch[1] Complete. Time taken: 00:00:37                                                       
INFO:ignite.engine.engine.Engine:Epoch[1] Metrics -- Training Dice: 0.1323                                                     
INFO:ignite.engine.engine.Engine:Epoch[1] Loss: 0.9143436551094055                                                             
INFO:ignite.engine.engine.Engine:Engine run starting with max_epochs=1.                                                        
INFO:ignite.engine.engine.Engine:Epoch[1] Complete. Time taken: 00:00:19                                                       
INFO:ignite.engine.engine.Engine:Epoch[1] Metrics -- Mean Dice: 0.1433                                                         
INFO:ignite.engine.engine.Engine:Engine run complete. Time taken 00:00:19                                                      
INFO:ignite.engine.engine.Engine:Epoch: 2/3, Iter: 1/3 -- Training Dice: 0.1323                                                
INFO:ignite.engine.engine.Engine:Epoch: 2/3, Iter: 2/3 -- Training Dice: 0.1323                                                
INFO:ignite.engine.engine.Engine:Epoch: 2/3, Iter: 3/3 -- Training Dice: 0.1323                                                
INFO:ignite.engine.engine.Engine:Epoch[2] Complete. Time taken: 00:00:39                                                       
INFO:ignite.engine.engine.Engine:Epoch[2] Metrics -- Training Dice: 0.1296                                                     
INFO:ignite.engine.engine.Engine:Epoch[2] Loss: 0.9668307900428772                                                             
INFO:ignite.engine.engine.Engine:Engine run starting with max_epochs=1.                                                        
INFO:ignite.engine.engine.Engine:Epoch[1] Complete. Time taken: 00:00:19                                                       
INFO:ignite.engine.engine.Engine:Epoch[1] Metrics -- Mean Dice: 0.1426                                                         
INFO:ignite.engine.engine.Engine:Engine run complete. Time taken 00:00:19                                                      
INFO:ignite.engine.engine.Engine:Epoch: 3/3, Iter: 1/3 -- Training Dice: 0.1296                                                
INFO:ignite.engine.engine.Engine:Epoch: 3/3, Iter: 2/3 -- Training Dice: 0.1296                                                
INFO:ignite.engine.engine.Engine:Epoch: 3/3, Iter: 3/3 -- Training Dice: 0.1296                                                
INFO:ignite.engine.engine.Engine:Epoch[3] Complete. Time taken: 00:00:39                                                       
INFO:ignite.engine.engine.Engine:Epoch[3] Metrics -- Training Dice: 0.1686                                                     
INFO:ignite.engine.engine.Engine:Epoch[3] Loss: 0.825812816619873                                                              
INFO:ignite.engine.engine.Engine:Engine run starting with max_epochs=1.                                                        
INFO:ignite.engine.engine.Engine:Epoch[1] Complete. Time taken: 00:00:19                                                       
INFO:ignite.engine.engine.Engine:Epoch[1] Metrics -- Mean Dice: 0.1430                                                         
INFO:ignite.engine.engine.Engine:Engine run complete. Time taken 00:00:19                                                      
INFO:ignite.engine.engine.Engine:Engine run complete. Time taken 00:02:56

[Nic-Ma]

Hi @madil90 ,

Thanks for your quick PR, I just added 2 comments inline, could you please help take a look?
Thanks.

[wyli]

@madil90 could you update examples/unet_segmentation_3d.py with the same content from the notebook? it's difficult to handle the conflicts if multiple branches modify the notebook

[Nic-Ma]

Looks good to me for current MVP version.
We may need to refactor utils part in later sprints.
And define unified format for model output, then we can use the util function "stopping_fn_from_loss".
Thanks.

[Nic-Ma]

Looks good to me for current MVP version.
We may need to refactor utils part in later sprints.
And define unified format for model output, then we can use the util function "stopping_fn_from_loss".
Thanks.

[Nic-Ma]

looks good to me.