Number of processes started by IPyParallel clusters #693

Activity

1. minrk commented on Apr 21, 2022

   

   minrk

   on Apr 21, 2022

   Member

   More actions

   The connection/process diagram hasn't been updated to include the broadcast scheduler, which is itself multi-process for the tree, and accounts for $2^{depth+1}-1$ processes. The default is a depth of 1, which makes 3 processes. So that's a documentation issue.

   You can run engines without the nanny, but that also loses all the benefits of the nanny (remote engine signalling, prompt crash events, etc.). You get a reduced-functionality version of this with MPI without the nanny, so it may make sense for you.

   The controller can be started with --usethreads. This puts the schedulers all in threads instead of processes. It saves memory, but of course at the expense of parallelism because Python is itself single-threaded. - the Hub will slow way down if you have lots of executions going on, and the broadcast scheduler loses all parallelism so it probably only makes sense to use a broadcast depth of 0 if you do this.

   This config should minimie processes:

   # ipengine_config.py
   c.IPEngine.use_nanny = False
   
   # ipcontroller_config.py
   c.IPController.broadcast_scheduler_depth = 0
   c.IPController.use_threads = True

   It's worth profiling the performance with threads to understand when the memory trade off may be worth it.

   with just one process for the controller and each engine.
2. sahil1105 commented on Apr 23, 2022

   

   sahil1105

   on Apr 23, 2022

   CollaboratorAuthor

   More actions

   Thanks for the detailed explanation @minrk! That makes complete sense.
   I agree, for small setups, e.g. on laptops, where you may only need a few engines, the minimized configuration could make sense. I think just documenting these details explicitly might be sufficient.

   👍React with 👍1Reacted by Min RK
3. sahil1105 commented on May 1, 2022

   

   sahil1105

   on May 1, 2022

   CollaboratorAuthor

   More actions

   # ipengine_config.py
   c.IPEngine.use_nanny = False

   Quick correction: Should be enable_nanny instead of use_nanny.

   👍React with 👍1Reacted by Min RK
4. sahil1105 commented on May 1, 2022

   

   sahil1105

   on May 1, 2022

   Last edited by sahil1105

   CollaboratorAuthor

   More actions

   @minrk I was able to get this working. Thanks again.
   Is there a way to specify these options in ipcluster_config.py or set them in user-code using cluster.config?
5. minrk commented on May 2, 2022

   

   minrk

   on May 2, 2022

   Member

   More actions

   # ipcluster_config.py (or cluster.config)
   c.EngineLauncher.engine_args=["--IPEngine.enable_nanny=False"]
   c.ControllerLauncher.controller_args = ["--usethreads"]
6. sahil1105 commented on May 6, 2022

   

   sahil1105

   on May 6, 2022

   CollaboratorAuthor

   More actions

   Great, tysm @minrk!
   This worked for me:

   c = ipp.Cluster(engines='mpi', n=4)
   c.config.EngineLauncher.engine_args=["--IPEngine.enable_nanny=False"]
   c.config.ControllerLauncher.controller_args = ["--IPController.broadcast_scheduler_depth=0", "--IPController.use_threads=True"]

   Would be great if we can update the docs with the updated diagram and this example.
7. sahil1105 commented on Jun 7, 2022

   

   sahil1105

   on Jun 7, 2022

   CollaboratorAuthor

   More actions

   Hi @minrk, any update on when the documentation can be updated with these details?
   We're planning to add it to our docs as well, but would be good to reference the official docs.
8. minrk commented on Jun 8, 2022

   

   minrk

   on Jun 8, 2022

   Member

   More actions

   I don't have time to work on this right now, but if you wanted to have a stab, I'm happy to review.

   👍React with 👍1Reacted by Sahil Gupta