Comparison to DynamicGrids.jl

[rafaqz]

For fixed grid models like forest fire, DynamicGrids.jl was around two orders of magnitude faster than Agents.jl the last time I checked.

It also has a good range of visualisation tools - REPL, GTK and web outputs. Also live parameter manipulation. It might be nice to mention that in the docs where you are comparing the forest fire model with mesa.

It wont run a lot of the models you define here, but it's pretty specialised for high-performance gridded models.

[Libbum]

A valid point. We had a comparison before, but removed it once the only cellular automata we had was the game of life. Now that forest fire is back to a static system again, I can re-run some numbers to that end.

[Datseris]

@rafaqz have you seen this: https://juliadynamics.github.io/InteractiveChaos.jl/dev/agents/ ? I don't see something of similar complexity or interactivity in the docs of DynamicGrids.jl. In your readme I did read

[DynamicGrids...] provides better visualization tools.

however. What does this mean, exactly?

Also live parameter manipulation.

Is there a video of this in the docs? I just went through them but I didn't find one.

[Datseris]

DynamicGrids.jl was around two orders of magnitude faster than Agents.jl

Yeap, that is true, I also remember it. Notice however that in version 4.0 we have re-written our GridSpace from scarttch, and it now has a design more similar to DynamicGrids.jl. It is still very much clear to me that DynamicGrids.jl will be faster, but by how much we have to check again.

[rafaqz]

I only mention the visualisation because you mention it as a positive for mesa in the docs. It's good to see you have some live interfaces working too!

But there's a simple REPLOutput built in, and DynamicGridsGtk.jl, DynamicGridsInteract.jl are image-based live interfaces. You can use ColorSchemes.jl to color the simulation, do multi-grid layouts, and set fonts etc for labels/time counter if you need it. A GifOutput is built-in too. See the gifs in the readme https://github.com/cesaraustralia/DynamicGrids.jl for examples of the output. The interfaces just show the same thing, but as the simulation runs. It can do 100fps pretty easily on a simple simulation.

The InteractOutput has control widgets and start/stop etc like your example. There is a video of it in my JuliaCon talk about half way through, but yeah it should be in the DynamicGridsInteract.jl docs. It's in the DynamicGridsInteract.jl readme now, and you can find it here:

https://www.youtube.com/watch?v=cXzYGHw_DaA&feature=youtu.be

One difference is that params are auto-generated for arbitrary custom model structs - the parameters can be nested anywhere in the model object. There is no setup code like in your example to make the output.

I'm currently abstracting out that auto interface generation here: https://github.com/rafaqz/ModelParameters.jl, in the internal InteractModels.jl package (in the process of registration still). I might borrow some of your code from that example for the planned MakieModels subpackage :)

[rafaqz]

[DynamicGrids...] provides better visualization tools.

This was written a long time ago when agents didn't really have much for visualisation, Ill edit.

[rafaqz]

Finally got time to run the GOL benchmarks. These are the life rules in the docs of both packages. DG comes out 2-3 and maybe even 4 orders of magnitude faster for running game of life on varying grid sizes. The last one was looking like taking a whole day for Agents.jl so I killed it.

The comparison is better where there are less agents, and where GPU is less competitive - on small grids.

using Agents, CUDA, DynamicGrids, BenchmarkTools, KernelAbstractions, StaticArrays

rules = (2, 3, 3, 3)

mutable struct Cell <: AbstractAgent
    id::Int
    pos::Dims{2}
    status::Bool
end

function build_model(; rules::Tuple, dims = (100, 100), metric = :chebyshev)
    space = GridSpace(dims; metric)
    properties = Dict(:rules => rules)
    model = ABM(Cell, space; properties)
    idx = 1
    for x in 1:dims[1]
        for y in 1:dims[2]
            add_agent_pos!(Cell(idx, (x, y), false), model)
            idx += 1
        end
    end
    return model
end

function ca_step!(model)
    new_status = fill(false, nagents(model))
    for agent in allagents(model)
        nlive = nlive_neighbors(agent, model)
        if agent.status == true && (nlive ≤ model.rules[4] && nlive ≥ model.rules[1])
            new_status[agent.id] = true
        elseif agent.status == false && (nlive ≥ model.rules[3] && nlive ≤ model.rules[4])
            new_status[agent.id] = true
        end
    end
    for k in keys(model.agents)
        model.agents[k].status = new_status[k]
    end
end

function nlive_neighbors(agent, model)
    neighbor_positions = nearby_positions(agent, model)
    all_neighbors = Iterators.flatten(ids_in_position(np,model) for np in neighbor_positions)
    sum(model[i].status == true for i in all_neighbors)
end

function setup_agents(s)
    model = build_model(rules = rules, dims = (s, s))
    for i in 1:nagents(model)
        if rand() < 0.5
            model.agents[i].status = true
        end
    end
    model
end

# DynamicGrids Life rule states - setting up from scratch for transparancy 
const lifestates = (false, false, false, true, false, false, false, false, false), 
                   (false, false, true, true, false, false, false, false, false)

function setup_dynamicgrids(s, nsteps)
    output = ResultOutput(rand(Bool, s, s); tspan=1:nsteps+1)
    rule = Neighbors(Moore(1)) do hood, state
        @inbounds lifestates[state + 1][sum(hood) + 1]
    end
    output, rule
end

function run_benchmarks(; sizes=(100, 200, 500, 1000, 2000), nsteps=1000)
    for s in sizes
        println("\n\n###################### $s * $s, $nsteps steps #######################")
        output, rule = setup_dynamicgrids(s, nsteps)
        println("\nDynamicGrids SingleCPU: $s * $s")
        @btime sim!($output, $rule, opt=NoOpt(), proc=SingleCPU());
        println("\nDynamicGrids SingleCPU SparseOpt: $s * $s")
        @btime sim!($output, $rule, opt=SparseOpt(), proc=SingleCPU());
        println("\nDynamicGrids Threaded (5): $s * $s")
        @btime sim!($output, $rule, opt=NoOpt(), proc=ThreadedCPU());
        println( "\nDynamicGrids Threaded SparseOpt (5): $s * $s")
        @btime sim!($output, $rule, opt=SparseOpt(), proc=ThreadedCPU());
        println( "\nDynamicGrids CUDA GPU (5): $s * $s")
        @btime CUDA.@sync sim!($output, $rule, opt=SparseOpt(), proc=CuGPU());
        println("\nAgents: $s * $s")
        model = setup_agents(s)
        @btime for i in 1:$nsteps step!($model, $dummystep, $ca_step!, 1) end
    end
end

julia> run_benchmarks()


###################### 100 * 100, 1000 steps #######################

DynamicGrids SingleCPU: 100 * 100
  44.933 ms (5591 allocations: 548.94 KiB)

DynamicGrids SingleCPU SparseOpt: 100 * 100
  37.041 ms (6596 allocations: 1.09 MiB)

DynamicGrids Threaded (5): 100 * 100
  22.456 ms (75627 allocations: 10.46 MiB)

DynamicGrids Threaded SparseOpt (5): 100 * 100
  25.964 ms (78354 allocations: 11.05 MiB)

DynamicGrids CUDA GPU (5): 100 * 100
  82.766 ms (542711 allocations: 47.80 MiB)

Agents: 100 * 100
  4.773 s (10001000 allocations: 1.80 GiB)


###################### 200 * 200, 1000 steps #######################

DynamicGrids SingleCPU: 200 * 200
  185.801 ms (5593 allocations: 608.59 KiB)

DynamicGrids SingleCPU SparseOpt: 200 * 200
  147.550 ms (6598 allocations: 1.16 MiB)

DynamicGrids Threaded (5): 200 * 200
  81.241 ms (75705 allocations: 10.52 MiB)

DynamicGrids Threaded SparseOpt (5): 200 * 200
  64.855 ms (77560 allocations: 11.10 MiB)

DynamicGrids CUDA GPU (5): 200 * 200
  46.048 ms (183169 allocations: 11.71 MiB)

Agents: 200 * 200
  24.371 s (40002000 allocations: 7.19 GiB)


###################### 500 * 500, 1000 steps #######################

DynamicGrids SingleCPU: 500 * 500
  1.438 s (5593 allocations: 1022.34 KiB)

DynamicGrids SingleCPU SparseOpt: 500 * 500
  1.022 s (6600 allocations: 1.67 MiB)

DynamicGrids Threaded (5): 500 * 500
  425.345 ms (75671 allocations: 10.92 MiB)

DynamicGrids Threaded SparseOpt (5): 500 * 500
  307.628 ms (78796 allocations: 11.64 MiB)

DynamicGrids CUDA GPU (5): 500 * 500
  165.871 ms (544455 allocations: 17.93 MiB)

Agents: 500 * 500
  322.102 s (250002000 allocations: 44.94 GiB)


###################### 1000 * 1000, 1000 steps #######################

DynamicGrids SingleCPU: 1000 * 1000
  6.092 s (5597 allocations: 2.43 MiB)

DynamicGrids SingleCPU SparseOpt: 1000 * 1000
  4.545 s (6604 allocations: 3.47 MiB)

DynamicGrids Threaded (5): 1000 * 1000
  1.192 s (78236 allocations: 12.44 MiB)

DynamicGrids Threaded SparseOpt (5): 1000 * 1000
  1.318 s (80120 allocations: 13.48 MiB)

DynamicGrids CUDA GPU (5): 1000 * 1000
  567.078 ms (1769255 allocations: 39.14 MiB)

Agents: 1000 * 1000
  1637.079 s (1000002000 allocations: 179.75 GiB)


###################### 2000 * 2000, 1000 steps #######################

DynamicGrids SingleCPU: 2000 * 2000
  29.593 s (5597 allocations: 8.17 MiB)

DynamicGrids SingleCPU SparseOpt: 2000 * 2000
  22.071 s (6604 allocations: 10.63 MiB)

DynamicGrids Threaded (5): 2000 * 2000
  5.810 s (76190 allocations: 18.11 MiB)

DynamicGrids Threaded SparseOpt (5): 2000 * 2000
  5.591 s (79195 allocations: 20.62 MiB)

DynamicGrids CUDA GPU (5): 2000 * 2000
  2.193 s (6546754 allocations: 122.07 MiB)

Agents: 2000 * 2000
^CERROR: InterruptException:

[Libbum]

Wow, nice!

Snowed under with other engagements at the moment, but will certainly take a look at these and add something about this in the docs soon.

[Libbum]

@rafaqz sorry for the delay on this one. We're going to do some reshuffling of our comparisons - this dedicated repo is where we'll ultimately put your comparison code. Documentation of the benchmarks will stay in the Agents.jl docs once all this shuffle is complete.

[Tortar]

Re-ran the GOL benchmarks with the latest version of agents and dynamicgrids, and now agents is only 2 times slower in the single-cpu case, which is a win in my opinion for agents given the previous benchmark :D

benchmarks

using Agents, Random, DynamicGrids, BenchmarkTools

function setup_dynamicgrids(s, nsteps)
    output = ResultOutput(rand(Bool, s, s); tspan=1:nsteps+1)
    life = Neighbors(Moore(1)) do data, hood, state, I
        born_survive = (false, false, false, true, false, false, false, false, false), 
                       (false, false, true, true,  false, false, false, false, false)
        born_survive[state + 1][sum(hood) + 1]
    end
    return output, life
end

@agent Automaton GridAgent{2} begin end

function build_model(alive_probability, dims, metric = :chebyshev)
    space = GridSpaceSingle(dims; metric, periodic=false)
    status = zeros(Bool, dims)
    new_status = zeros(Bool, dims)
    rules = (2, 3, 3, 3)
    properties = (; rules, status, new_status)
    model = UnremovableABM(Automaton, space; model_step!, properties, rng = Xoshiro())
    for pos in Agents.positions(model)
        if rand(abmrng(model)) < alive_probability
            status[pos...] = true
        end
    end
    return model
end

function model_step!(model)
    new_status = model.new_status
    status = model.status
    @inbounds for pos in Agents.positions(model)
        n = alive_neighbors(pos, model)
        if status[pos...] == true && (n ≤ model.rules[4] && n ≥ model.rules[1])
            new_status[pos...] = true
        elseif status[pos...] == false && (n ≥ model.rules[3] && n ≤ model.rules[4])
            new_status[pos...] = true
        else
            new_status[pos...] = false
        end
    end
    status .= new_status
    return
end

function alive_neighbors(pos, model)
    c = 0
    @inbounds for near_pos in nearby_positions(pos, model)
        if model.status[near_pos...] == true
            c += 1
        end
    end
    return c
end

function run_model_agents()
    model = build_model(0.5, (100, 100))
    Agents.step!(model, 1000)
end

function run_model_dynamicgrids()
    output, life = setup_dynamicgrids(100, 1000)
    sim!(output, life, opt=NoOpt(), proc=SingleCPU())
end

@benchmark run_model_dynamicgrids()
@benchmark run_model_agents()

julia> @benchmark run_model_dynamicgrids()
BenchmarkTools.Trial: 73 samples with 1 evaluation.
 Range (min … max):  67.655 ms … 83.744 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     68.477 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   68.645 ms ±  1.846 ms  ┊ GC (mean ± σ):  0.00% ± 0.00%

     ▁         ▁      ▃▃  █▃
  ▄▇▄█▇▄▄▁▇▁▁▁▁█▁▇▄▄▄▇██▄▁██▆▆▆▆▁▇▄▄▄▁▁▁▁▁▄▁▁▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▄ ▁
  67.7 ms         Histogram: frequency by time        69.9 ms <

 Memory estimate: 269.67 KiB, allocs estimate: 2541.

julia> @benchmark run_model_agents()
BenchmarkTools.Trial: 36 samples with 1 evaluation.
 Range (min … max):  137.426 ms … 158.551 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     138.684 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   140.068 ms ±   4.688 ms  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ▂▄█▆  ▂
  █████▆█▁▁▄▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▄ ▁
  137 ms           Histogram: frequency by time          159 ms <

 Memory estimate: 1.31 MiB, allocs estimate: 34889.

But I'm not sure how to handle the matter here since we don't have this model in the benchmarks, probably something more generic about cellular automata efficiency of dynamicgrids should be put in the docs of Agents.jl itself

[Datseris]

I am closing this issue, because now there are instructions online of how to contribute new frameworks here. DynamicsGrids.jl is not an agent based modelling framework which means that from the examples we have in this repo, it can simulate forest fire and Schelling. Or at least, I don't see how it could do wolf sheep grass. In any case, @rafaqz if you want to add your framework to our comparison for only the forest fire or also the Schelling feel free to follow the README instructions and contribute a PR.

[Datseris]

Actually, I'll keep this open until you put this PR in. Just making it clear that we won't be doing this PR for you (and the same goes for any other framework; if someone wants to be part of the comparison they have to do the PR).

[rafaqz]

The code is in the readme, and I have personally written a lot of code at your request, @Datseris ;)

But sure, I will put a PR together at some stage, thanks for keeping this open and reminding me. I honestly just have too many other packages to get time to do this.

One question: are GPUs allowed?