Reinforcement Learning

Agent(g::GraphSystem; t_block=missing, tspan=(0.0, 1.0), u0=[], p=[],
      action_selection=missing, kwargs...)

A reinforcement learning agent, used to interact with an AbstractEnvironment to simulate a learning task.

Arguments :

• g : A GraphSystem containing the model that the agent is using to make choices and update its connections during reinforcement learning.

Keyword arguments :

• u0 : Initial conditions for the model in g. If not provided then default values will be used.
• p : Parameter values for the model in g. If not provided then default values will be used.
• t_block : The time period of a PeriodicCallback which will reset the cumulative spike counter of neurons in the model. This is optional and can be useful when the plasticity rules require number of spikes within specific time windows to update connection weights.
• tspan : Timespan of a single trial. run_experiment! overrides this with the trial duration of the environment it is given, so it only matters when driving trials directly with run_trial!.
• actionselection : The policy the agent uses. missing (the default) takes the unique AbstractActionSelection node in g, warning if there is none. Pass nothing to declare that this agent deliberately makes no scored choice — a conditioning protocol, say, where feedback is identically 1 — which suppresses that warning. Any other value is used as given, but must also be a node in g, or `connectactionselection!will never run and itscompetitorstates` will stay empty.
• kwargs... : All other keyword arguments are passed to the ODEProblem that is constructed inside the agent and solved during runexperiment! and runtrial! .

AbstractImageEnvironment <: AbstractEnvironment

An environment whose per-trial stimulus is one column of an ImageStimulus held in its source field. Subtypes get get_trial_stimulus and update_trial_stimulus! for free; they must provide source, N_trials, t_trial and a mutable current_trial.

ClassificationEnvironment(stim::ImageStimulus, N_trials; name, namespace, t_stimulus, t_pause)

Create an environment for reinforcement learning. A set of images is presented to the agent to be classified. This struct stores the correct class for each image, and the current trial of the experiment.

Arguments:

• stim: The ImageStimulus, created from a set of images
• N_trials: Number of trials. The agent performs one classification each trial.
• t_stimulus: The length of time the stimulus is on (ms)
• t_pause: The length of time the stimulus is off (ms)

ConditioningEnvironment(stim::ImageStimulus[, N_trials]; trial_groups = nothing)
ConditioningEnvironment(data::Union{StructArray, String}[, N_trials];
                        t_stimulus, t_pause, trial_groups = nothing)

Create an environment for a conditioning protocol: a stimulus is presented each trial, the circuit makes no classification, and nothing is scored. Unlike ClassificationEnvironment there is no category field — though ImageStimulus still requires a category column in data, and it stays readable as env.source.param_vals.category.

feedback is identically 1 on every trial. In the usual setup — no AbstractActionSelection node in the graph, so Agent(g; action_selection = nothing) — run_trial! sets that directly and never calls env(action); the callable method exists so that a protocol which does attach a policy (for a read-out, say) still scores every action as correct and so leaves the weight updates alone.

trial_groups labels each trial with the conditioning pathway it belongs to, and learning rules gate themselves to a label with their own group keyword (see HebbianPresynapticPlasticity, current_trial_group). It is the single place the protocol design lives — which trial is the fear trial, which is the reward trial — rather than being re-derived as trial indices at every rule construction. run_trial! advances current_trial only after a trial's weight updates have been applied, which is what lines the gate up with the trial that just ran.

trial_groups defaults to nothing, for a protocol with no gating at all. A gated rule paired with such an environment is rejected by validate_rule_groups, since it could never fire.

Arguments:

• stim / data: the ImageStimulus, or the data (or CSV filepath) to build one from
• N_trials: Number of trials. Defaults to the number of images in the stimulus.
• t_stimulus: The length of time the stimulus is on (ms). Only when building from data.
• t_pause: The length of time the stimulus is off (ms). Only when building from data.
• trialgroups: Vector{Symbol} of length `Ntrials, ornothing` for no gating.

env = ConditioningEnvironment(images, 2; t_stimulus = 1000.0, t_pause = 0.0,
                              trial_groups = [:fear, :reward])

fear_rule   = HebbianPresynapticPlasticity(; K = 0.03,  group = :fear)
reward_rule = HebbianPresynapticPlasticity(; K = 0.055, group = :reward)

GreedyPolicy(; name, t_decision, namespace, competitor_states = Symbol[], metadata = Dict(), tags = Set{Symbol}())

A policy that makes a choice by picking the state with the highest value among competitor_states which represent each available choice. t_decision is the time of the decision.

HebbianPlasticity(; K, W_lim,
                  state_pre = nothing,
                  state_post = nothing,
                  t_pre = nothing,
                  t_post = nothing,
                  group = nothing)

Hebbian plasticity rule. The connection weight is updated according to :

\[ w_{j+1} = w_j + \text{feedback} × K x_\text{pre} x_\text{post} (W_\text{lim} - w)\]

where feedback is a binary indicator of the correctness of the model's action, and x indicates the activity of the pre- and post-synaptic neuron states state_pre and state_post at timepoints t_pre and t_post respectively.

Arguments: - K : the learning rate of the connection - Wlim : the maximum weight for the connection - statepre : state of the presynaptic neuron that is used in the plasticity rule (by default this is state V in neurons). - statepost : statepre : state of the postsynaptic neuron that is used in the plasticity rule (by default this is state V in neurons). - tpre : timepoint at which `statepreis evaluated to be used in the plasticity rule. - t_post : t_pre : timepoint at whichstatepostis evaluated to be used in the plasticity rule. - group : trial-group label to gate the update to.nothing(the default) updates on every trial; otherwise the update is applied only on trials the environment labelsgroup(see [currenttrial_group](@ref)), and on any other trial the weight is left bit-for-bit unchanged. See [ConditioningEnvironment`](@ref) for the conditioning protocols this is for.

See also HebbianModulationPlasticity.

HebbianPresynapticPlasticity(; K, state_pre = nothing, t_pre = nothing,
                             group = nothing)

Purely presynaptic Hebbian plasticity rule. The connection weight is updated in proportion to the presynaptic activity alone, with no dependence on the postsynaptic state:

\[ w_{j+1} = w_j + \text{feedback} × K x_\text{pre}\]

where feedback is a binary indicator of the correctness of the model's action (identically 1 when the graph has no action selection blox) and $x_\text{pre}$ is the activity of the presynaptic neuron state state_pre at time t_pre.

With the default state_pre — the presynaptic neuron's SpikeMonitor spikes_cumulative, bound automatically per connection while wiring — and the default t_pre = nothing (evaluate at the end of the trial), the update is w += K × (number of presynaptic spikes during the trial) — every synapse of a projection is potentiated according to how strongly its source neuron fired. This is the update used to model reward conditioning in the anxiety-circuit experiments.

group restricts the update to one stage of a conditioning protocol, so that a single run_experiment! can pair each trial's stimulus with its own designated plastic pathway: attach a group=:fear rule to one route and a group=:reward rule to another, and each is conditioned only on its own trial. See ConditioningEnvironment.

Unlike HebbianPlasticity there is no W_lim, so the weight is unbounded: K sets the scale of a single conditioning round and repeated trials accumulate linearly.

Arguments: - K : the learning rate of the connection - statepre : state of the presynaptic neuron that is used in the plasticity rule (by default the presynaptic SpikeMonitor's `spikescumulative). - t_pre : timepoint at whichstatepreis evaluated.nothing(the default) evaluates it at the final timepoint of the trial's solution. - group : trial-group label to gate the update to.nothing(the default) updates on every trial; otherwise the update is applied only on trials the environment labelsgroup(see [currenttrial_group`](@ref)), and on any other trial the weight is left bit-for-bit unchanged.

See also HebbianPlasticity, HebbianModulationPlasticity.

HebbianModulationPlasticity(; K, decay, α, θₘ,
                            state_pre = nothing,
                            state_post = nothing,
                            t_pre = nothing,
                            t_post = nothing,
                            t_mod = nothing,
                            modulator = nothing,
                            group = nothing)

Hebbian plasticity rule, modulated by the dopamine reward prediction error. The weight update is largest when the reward prediction error is far from the modulation threshold θₘ.

\[ ϵ = \text{feedback} - (\text{DA}_b - \text{DA}) w_{j+1} = w_j + \max(\times K x_\text{pre} x_\text{post} ϵ(ϵ + θₘ) dσ(α(ϵ + θₘ)) - \text{decay} × w, -w)\]

where feedback is a binary indicator of the correctness of the model's action, DA_b is the baseline dopamine level, DA is the modulator's dopamine release, dσ is the derivative of the logistic function, and x indicates the activity of the pre- and post-synaptic neuron states state_pre and state_post at timepoints t_pre and t_post respectively. The decay prevents the weights from diverging.

Arguments: - K: the learning rate of the connection - decay: Decay of the weight update - α: the selectivity of the derivative of the logistic function - θₘ: the modulation threshold for the reward prediction error - statepre : state of the presynaptic neuron that is used in the plasticity rule (by default this is state V in neurons). - statepost : statepre : state of the postsynaptic neuron that is used in the plasticity rule (by default this is state V in neurons). - tpre : timepoint at which state_pre is evaluated to be used in the plasticity rule. - tpost : tpre : timepoint at which state_post is evaluated to be used in the plasticity rule. - group : trial-group label to gate the update to. nothing (the default) updates on every trial; otherwise the update is applied only on trials the environment labels group (see current_trial_group), and on any other trial the weight is left bit-for-bit unchanged. Note this is the only way to skip this rule on a trial: unlike the other two rules its gradient is not zero at feedback == 0.

See also HebbianPlasticity.

weight_gradient(lr::AbstractLearningRule, sol, w, feedback, group)

Calculate the way that the weight w should change based on the solution of the reinforcement learning experiment.

group is the trial-group label the environment reports for the trial that just ran (see current_trial_group), or nothing for an environment without trial groups. A rule that is gated to a group of its own should open with

trial_gate_active(lr.group, group) || return zero(w)

and define rule_group so that the gate can also be validated up front.

rule_group(lr::AbstractLearningRule)

The trial-group label lr is gated to, or nothing to update on every trial. Defaults to nothing; override it for a rule that carries a group, so that validate_rule_groups can check the gate before an experiment runs.

trial_gate_active(group, current_group)

Whether a rule gated to group should update on a trial labeled current_group. A group of nothing means every trial.

run_warmup(agent::AbstractAgent, env::AbstractEnvironment, t_warmup; kwargs...)

Run the initial solve of the RL experiment for t_warmup.

run_trial!(agent::AbstractAgent, env::AbstractEnvironment, weights, u0; kwargs...)

Run a single trial of a RL experiment. Update the connection weights according to the learning rules.

run_experiment!(agent::AbstractAgent, env::AbstractEnvironment; verbose=false, t_warmup=0, kwargs...)

Perform a full RL experiment with agent in the environment env. Will run until the maximum number of trials in env is reached.

trial_group_labels(env::AbstractEnvironment)

One trial-group label per trial of env, in trial order, or nothing when env has no trial groups at all (the default).

This is the single place an environment declares its groups: both current_trial_group and possible_trial_groups derive from it, so they cannot disagree. Override this rather than either of them — e.g. trial_group_labels(env::ConditioningEnvironment) = env.trial_groups.

An environment whose label is computed rather than tabulated may override current_trial_group directly instead, but then owes possible_trial_groups as well, or validate_rule_groups will reject every gated rule paired with it.

current_trial_group(env::AbstractEnvironment)

The trial-group label of env's current trial, or nothing. Learning rules gated to a group (see rule_group) only update on trials whose label matches theirs.

Derived from trial_group_labels, which is what an environment should override.

possible_trial_groups(env::AbstractEnvironment)

Every label env can return from current_trial_group over a full experiment, used by validate_rule_groups to reject a rule gated to a label the environment can never produce.

Derived from trial_group_labels, which is what an environment should override. (nothing,) for an environment with no trial groups.

validate_rule_groups(learning_rules, env)

Throw if any rule in learning_rules is gated (see rule_group) to a label that env can never produce (see possible_trial_groups).

Without this check such a rule simply never fires: no error, no warning, the weights never move, and the trace looks entirely normal.