Skip to content

using future_apply inside a future #65

@comicfans

Description

@comicfans

since fork is unstable in RStudio, I came to another solution, first using multi-session to create a future from main R session, and use multicore to fork in the future expr body(I've modified future to make supportsMulticore return TRUE from none-interactive R), so future_lapply still use fork

plan(multisession)
res = future({
   plan(multicore)
   options(mc.cores = 4)
   options(future.fork.enable=T)
   future_lapply(1:100, function(i){
        return(length(BIG_GLOBAL_VAR))
   })
})

lapply.res = value(res)

I hope such setup should makes big global variable access faster ( should be copied only once) and worker cheaper. with some simple test seems it do complete faster than directly call future_lapply with multisession

my question is : does such setup have some implicit problem ? does future/future.apply expected to work correctly in such setup ? I tried to use such setup with progressr, found that the progressor object can not work, for example

plan(multisession)
with_progress({
  p = progressor(100)
  future({ 
   plan(multicore)
   options(mc.cores = 4)
   options(future.fork.enable=T)
     future_lapply(1:100, function(i){
        p("")
    })
  })
})

is this progressr limitation, or future/future.apply should not be used in such way ? Thanks for advise

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions