Progress for OSPP project `210370190` application

[peterchen96]

Hi, Mr. Tian Jun @findmyway. Sorry for the late response to your latest email. Here I want to report my recent progress and problems.

recent progress

Recently, I implement Neural Fictitious Self-play (NFSP) based on ReinforcementLearning.jl and the main structure as the following:

• rl_agent:

Agent(
    policy = QBasedPolicy(
        learner = DQNLearner,
        explorer = EpsilonGreedyExplorer,
        ),
    trajectory = CircularArraySARTTrajectory,
)

• sl_agent:

Agent(
    policy = QBasedPolicy(
        learner = AverageLearner,
        explorer = GreedyExplorer,
        ),
    trajectory = CircularArraySARTTrajectory,
)

where AverageLearner is an AbstractLearner which I imitated the structure from DQNLearner and rewrite its network updating process.

The code can run on GPU successfully, although its computation utility is inefficient. And the result is shown in README.md.

recent problems

• use nash_conv for evaluation:

  In the paper, they use exploitability as evaluation metric and I found the function nash_conv.jl in ./ReinforcementLearningZoo/src/algorithms/cfr/ which is the double of the exploitability in the kuhn_poker game.

  However, I meet the following error when using nash_conv:

  julia> RLZoo.nash_conv(my_policy, env)
  ERROR: AssertionError: The copy method doesn't seem to be implemented for the environment: KuhnPokerEnv(Symbol[], Symbol[])

  Also, how to pass my nfsp into nash_conv is also a big problem. Or if I rewrite nash_conv function as the plus of the best_response_value(a::NFSPAgent, env) to each player, how can I get best response policy?

• low utitily rate of cpu and gpu:

  I guess that frequent data transfermation between cpu and gpu is the main reason. For example, when call action = policy(env), it will run the following code:

  env |>
  state |>
  x -> Flux.unsqueeze(x, ndims(x) + 1) |>
  x -> send_to_device(device(learner), x) |>
  learner.approximator |>
  send_to_host |> 
  vec

  which will generate a transformation between CPU and GPU in each step.

  Maybe I can copy the networks to CPU when no need to update parameters?

[findmyway]

I'll take a close look into your implementation later today after work.

Some quick notes:

In the paper, they use exploitability as evaluation metric and I found the function nash_conv.jl in ./ReinforcementLearningZoo/src/algorithms/cfr/ which is the double of the exploitability in the kuhn_poker game.

Yes.

However, I meet the following error when using nash_conv:

A PR is welcomed to add a Base.copy implementation into RLEnvs. When calculating nash_conv, in the following line:

https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/blob/ebd95191776c19b9b3da71d6f51cd0183e6d66f2/src/ReinforcementLearningZoo/src/algorithms/cfr/nash_conv.jl#L22

child(env, a) will try to create a new environment instance and apply the action a. If the copy is not implemented, then the original game state will be mutated, which will not get the correct nash_conv.

Also, how to pass my nfsp into nash_conv is also a big problem.

It should work out of the box as far as I can see. Make sure some interfaces are implemented for your policy, like prob(policy, env) and prob(policy, env, a).

I guess that frequent data transfermation between cpu and gpu is the main reason.

No, that's required when you want to predict the action at each time step.

Like I said before, your model hidden_layers_sizes = (128, 128, 128, 128) is way too large for such a small game. I think a small network with CPU only would be good enough to test your implementation.

[findmyway]

By the way, it's always better to continue our discussion on the original issue. JuliaReinforcementLearning/ReinforcementLearning.jl#324

[peterchen96]

Thank you for your reading and kindly advice.

A PR is welcomed to add a Base.copy implementation into RLEnvs. When calculating nash_conv, in the following line:

I will try it in the next few days. For now, I define the new struct NFSPAgent in NFSPAgent.jl which substitutes the original NFSPAgent::Dict.

Like I said before, your model hidden_layers_sizes = (128, 128, 128, 128) is way too large for such a small game. I think a small network with CPU only would be good enough to test your implementation.

I reset hidden_layers_sizes as (64, 64) . New results and parameter settings have been updated.

And set used_device=gpu is just test whether my implementation can run on GPU, although the current code runs less efficiently on GPU.

By the way, it's always better to continue our discussion on the original issue. JuliaReinforcementLearning/ReinforcementLearning.jl#324

I will sort out my progress and update it on JuliaReinforcementLearning/ReinforcementLearning.jl#324 later.