LeanMachineLearning exposition

Bandits.ArrayModel.hasCondDistrib_reward_hist_action_pullCount๐Ÿ”—

This page has the declaration's own card below, then its dependency graph, then a card for each dependency (type dependencies first, then the rest of the transitive closure). For a theorem, the graph and the dependency cards only follow its statement's dependencies (its proof is replaced by sorry, so what it proves doesn't depend on how); for everything else, both the type and the body/value are followed, since their content is part of what later declarations build on.

Minimal Lean file

hasCondDistrib_reward_hist_action_pullCount๐Ÿ”—

LemmaBandits.ArrayModel.hasCondDistrib_reward_hist_action_pullCount
Details

The conditional distribution of the reward at time n + 1, given the history up to time n, the action at time n + 1, and the number of times that action has been pulled before time n + 1, is equal to the kernel ฮฝ.

๐Ÿ”—theorem
Bandits.ArrayModel.hasCondDistrib_reward_hist_action_pullCount.{u_1,
    u_2}
  {๐“ : Type u_1} {R : Type u_2} {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“] [StandardBorelSpace ๐“]
  [DecidableEq ๐“] [Countable ๐“] [StandardBorelSpace R] [Nonempty R]
  (alg : Learning.Algorithm ๐“ R) (ฮฝ : ProbabilityTheory.Kernel ๐“ R)
  [ProbabilityTheory.IsMarkovKernel ฮฝ] (n : โ„•) :
  ProbabilityTheory.HasCondDistrib (reward alg (n + 1))
    (fun ฯ‰ =>
      (hist alg ฯ‰ n, action alg (n + 1) ฯ‰,
        Learning.pullCount (action alg) (action alg (n + 1) ฯ‰) (n + 1)
          ฯ‰))
    (ProbabilityTheory.Kernel.prodMkLeft (โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R)
      (ProbabilityTheory.Kernel.prodMkRight โ„• ฮฝ))
    (arrayMeasure ฮฝ)
Bandits.ArrayModel.hasCondDistrib_reward_hist_action_pullCount.{u_1,
    u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“]
  [Countable ๐“] [StandardBorelSpace R]
  [Nonempty R]
  (alg : Learning.Algorithm ๐“ R)
  (ฮฝ : ProbabilityTheory.Kernel ๐“ R)
  [ProbabilityTheory.IsMarkovKernel ฮฝ]
  (n : โ„•) :
  ProbabilityTheory.HasCondDistrib
    (reward alg (n + 1))
    (fun ฯ‰ =>
      (hist alg ฯ‰ n, action alg (n + 1) ฯ‰,
        Learning.pullCount (action alg)
          (action alg (n + 1) ฯ‰) (n + 1)
          ฯ‰))
    (ProbabilityTheory.Kernel.prodMkLeft
      (โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R)
      (ProbabilityTheory.Kernel.prodMkRight
        โ„• ฮฝ))
    (arrayMeasure ฮฝ)

Code

lemma hasCondDistrib_reward_hist_action_pullCount
    (alg : Algorithm ๐“ R) (ฮฝ : Kernel ๐“ R) [IsMarkovKernel ฮฝ] (n : โ„•) :
    HasCondDistrib (reward alg (n + 1))
      (fun ฯ‰ โ†ฆ (hist alg ฯ‰ n, action alg (n + 1) ฯ‰,
        pullCount (action alg) (action alg (n + 1) ฯ‰) (n + 1) ฯ‰))
      ((ฮฝ.prodMkRight _).prodMkLeft _) (arrayMeasure ฮฝ)
Type uses (8)
Body uses (17)
Used by (1)

Actions: Source ยท Open Issue

Proof
by
  have h_meas : Measurable fun ฯ‰ โ†ฆ pullCount (action alg) (action alg (n + 1) ฯ‰) (n + 1) ฯ‰ := by
    change Measurable ((fun p : (probSpace ๐“ R) ร— ๐“ โ†ฆ pullCount (action alg) p.2 (n + 1) p.1) โˆ˜
      (fun ฯ‰ : probSpace ๐“ R โ†ฆ (ฯ‰, action alg (n + 1) ฯ‰)))
    exact (measurable_uncurry_pullCount (by fun_prop) _).comp (by fun_prop)
  refine hasCondDistrib_of_condDistrib_eq (by fun_prop) (by fun_prop) ?_
  refine condDistrib_prod_of_forall_condDistrib_cond (by fun_prop) (by fun_prop) (by fun_prop) _ ?_
  intro (a, m) ham
  have h_eq : ((ฮฝ.prodMkRight _).prodMkLeft _).comap (fun ฯ‰ : (Iic n โ†’ ๐“ ร— R) โ†ฆ (ฯ‰, a, m))
        (by fun_prop) =
      Kernel.const _ (ฮฝ a) := by ext; simp
  rw [h_eq, condDistrib_congr_left (reward_ae_eq_cond alg ฮฝ a n m)]
  refine (condDistrib_of_indepFun ?_ (by fun_prop) (by fun_prop)).trans (ae_of_all _ fun ฯ‰ โ†ฆ ?_)
  ยท exact (indepFun_snd_hist_cond alg ฮฝ a n m).symm
  ยท simp only [Kernel.const_apply]
    have : (fun ฯ‰ โ†ฆ (action alg (n + 1) ฯ‰,
          pullCount (action alg) (action alg (n + 1) ฯ‰) (n + 1) ฯ‰)) โปยน' {(a, m)} =
        ({ฯ‰ | action alg (n + 1) ฯ‰ = a โˆง
          pullCount (action alg) a (n + 1) ฯ‰ = m}).indicator 1 โปยน' {1} := by
      ext ฯ‰
      simp [Set.indicator_apply]
      grind
    rw [this, cond_of_indepFun, map_snd_apply_arrayMeasure m a]
    ยท exact (indepFun_snd_apply_pullCount_action alg ฮฝ a m n).symm
    ยท refine Measurable.indicator (by fun_prop) ?_
      exact MeasurableSet.inter ((measurableSet_singleton _).preimage (by fun_prop))
        ((measurableSet_singleton _).preimage (by fun_prop))
    ยท fun_prop
    ยท simp
    ยท convert ham
      ext ฯ‰
      simp only [Set.mem_preimage, Set.indicator_apply, Set.mem_setOf_eq, Pi.one_apply,
        Set.mem_singleton_iff, ite_eq_left_iff, not_and, zero_ne_one, imp_false, Classical.not_imp,
        Decidable.not_not, Prod.mk.injEq, and_congr_right_iff]
      intro ha
      simp [ha]

Dependency graph

Type dependencies (8)

Algorithm๐Ÿ”—

StructureLearning.Algorithm
Details

A stochastic, sequential algorithm.

๐Ÿ”—structure
Learning.Algorithm.{u_4, u_5} (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] : Type (max u_4 u_5)
Learning.Algorithm.{u_4, u_5}
  (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“]
  [MeasurableSpace ๐“จ] : Type (max u_4 u_5)

Code

structure Algorithm (๐“ ๐“จ : Type*) [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] where
  /-- Policy or sampling rule: distribution of the next action. -/
  policy : (n : โ„•) โ†’ Kernel (Iic n โ†’ ๐“ ร— ๐“จ) ๐“
  /-- The policy is a Markov kernel. -/
  [h_policy : โˆ€ n, IsMarkovKernel (policy n)]
  /-- Distribution of the first action. -/
  p0 : Measure ๐“
  /-- The first action distribution is a probability measure. -/
  [hp0 : IsProbabilityMeasure p0]
Used by (216)

Actions: Source ยท Open Issue

probSpace๐Ÿ”—

DefinitionBandits.ArrayModel.probSpace
Details

Probability space for the array model of stochastic bandits.

๐Ÿ”—def
Bandits.ArrayModel.probSpace.{u_1, u_2} (๐“ : Type u_1) (R : Type u_2) :
  Type (max u_1 u_2)
Bandits.ArrayModel.probSpace.{u_1, u_2}
  (๐“ : Type u_1) (R : Type u_2) :
  Type (max u_1 u_2)

Code

def probSpace : Type _ := (โ„• โ†’ I) ร— (โ„• โ†’ ๐“ โ†’ R)
Used by (64)

Actions: Source ยท Open Issue

instMeasurableSpaceProbSpace๐Ÿ”—

InstanceBandits.ArrayModel.instMeasurableSpaceProbSpace
Details

No docstring.

๐Ÿ”—def
Bandits.ArrayModel.instMeasurableSpaceProbSpace.{u_3, u_4}
  {๐“ : Type u_3} {R : Type u_4} [MeasurableSpace R] :
  MeasurableSpace (probSpace ๐“ R)
Bandits.ArrayModel.instMeasurableSpaceProbSpace.{u_3,
    u_4}
  {๐“ : Type u_3} {R : Type u_4}
  [MeasurableSpace R] :
  MeasurableSpace (probSpace ๐“ R)

Code

instance {๐“ R : Type*} [MeasurableSpace R] : MeasurableSpace (probSpace ๐“ R)
Type uses (1)
Used by (41)

Actions: Source ยท Open Issue

Proof
inferInstanceAs (MeasurableSpace ((โ„• โ†’ I) ร— (โ„• โ†’ ๐“ โ†’ R)))

reward๐Ÿ”—

DefinitionBandits.ArrayModel.reward
Details

Reward received at time n in the array model.

๐Ÿ”—def
Bandits.ArrayModel.reward.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“] (alg : Learning.Algorithm ๐“ R)
  (n : โ„•) (ฯ‰ : probSpace ๐“ R) : R
Bandits.ArrayModel.reward.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“]
  (alg : Learning.Algorithm ๐“ R) (n : โ„•)
  (ฯ‰ : probSpace ๐“ R) : R

Code

noncomputable
def reward [DecidableEq ๐“] (alg : Algorithm ๐“ R) (n : โ„•) (ฯ‰ : probSpace ๐“ R) : R :=
  (hist alg ฯ‰ n โŸจn, by simpโŸฉ).2
Type uses (2)
Body uses (1)
Used by (24)

Actions: Source ยท Open Issue

hist๐Ÿ”—

DefinitionBandits.ArrayModel.hist
Details

History of actions and rewards up to time n in the array model.

๐Ÿ”—def
Bandits.ArrayModel.hist.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“] (alg : Learning.Algorithm ๐“ R)
  (ฯ‰ : probSpace ๐“ R) (n : โ„•) : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R
Bandits.ArrayModel.hist.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“]
  (alg : Learning.Algorithm ๐“ R)
  (ฯ‰ : probSpace ๐“ R) (n : โ„•) :
  โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R

Code

noncomputable
def hist [DecidableEq ๐“] (alg : Algorithm ๐“ R) (ฯ‰ : probSpace ๐“ R) : (n : โ„•) โ†’ Iic n โ†’ ๐“ ร— R
| 0 => fun _ โ†ฆ (initAlgFunction alg (ฯ‰.1 0), ฯ‰.2 0 (initAlgFunction alg (ฯ‰.1 0)))
| n + 1 =>
  let hn : Iic n โ†’ ๐“ ร— R := hist alg ฯ‰ n
  let a : ๐“ := algFunction alg n hn (ฯ‰.1 (n + 1))
  fun i โ†ฆ if hin : i โ‰ค n then hn โŸจi, by simp [hin]โŸฉ else (a, ฯ‰.2 (pullCount' n hn a) a)
Type uses (2)
Body uses (3)
Used by (30)

Actions: Source ยท Open Issue

action๐Ÿ”—

DefinitionBandits.ArrayModel.action
Details

Action taken at time n in the array model.

๐Ÿ”—def
Bandits.ArrayModel.action.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“] (alg : Learning.Algorithm ๐“ R)
  (n : โ„•) (ฯ‰ : probSpace ๐“ R) : ๐“
Bandits.ArrayModel.action.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] [DecidableEq ๐“]
  (alg : Learning.Algorithm ๐“ R) (n : โ„•)
  (ฯ‰ : probSpace ๐“ R) : ๐“

Code

noncomputable
def action [DecidableEq ๐“] (alg : Algorithm ๐“ R) (n : โ„•) (ฯ‰ : probSpace ๐“ R) : ๐“ :=
  (hist alg ฯ‰ n โŸจn, by simpโŸฉ).1
Type uses (2)
Body uses (1)
Used by (43)

Actions: Source ยท Open Issue

pullCount๐Ÿ”—

DefinitionLearning.pullCount
Details

Number of times action a was chosen up to time t (excluding t).

๐Ÿ”—def
Learning.pullCount.{u_1, u_3} {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„•
Learning.pullCount.{u_1, u_3}
  {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (a : ๐“)
  (t : โ„•) (ฯ‰ : ฮฉ) : โ„•

Code

noncomputable
def pullCount (A : โ„• โ†’ ฮฉ โ†’ ๐“) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„• :=
  #(filter (fun s โ†ฆ A s ฯ‰ = a) (range t))
Used by (146)

Actions: Source ยท Open Issue

arrayMeasure๐Ÿ”—

DefinitionBandits.ArrayModel.arrayMeasure
Details

Probability measure for the array model of stochastic bandits.

๐Ÿ”—def
Bandits.ArrayModel.arrayMeasure.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R}
  (ฮฝ : ProbabilityTheory.Kernel ๐“ R) :
  MeasureTheory.Measure (probSpace ๐“ R)
Bandits.ArrayModel.arrayMeasure.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R}
  (ฮฝ : ProbabilityTheory.Kernel ๐“ R) :
  MeasureTheory.Measure (probSpace ๐“ R)

Code

noncomputable
def arrayMeasure (ฮฝ : Kernel ๐“ R) : Measure (probSpace ๐“ R) :=
  (Measure.infinitePi fun _ โ†ฆ volume).prod (streamMeasure ฮฝ)
Type uses (2)
Body uses (1)
Used by (29)

Actions: Source ยท Open Issue

All dependencies, transitively (6)

instIsProbabilityMeasureP0๐Ÿ”—

InstanceLearning.instIsProbabilityMeasureP0
Details

No docstring.

๐Ÿ”—theorem
Learning.instIsProbabilityMeasureP0.{u_1, u_2} {๐“ : Type u_1}
  {๐“จ : Type u_2} {m๐“ : MeasurableSpace ๐“} {m๐“จ : MeasurableSpace ๐“จ}
  (alg : Algorithm ๐“ ๐“จ) :
  MeasureTheory.IsProbabilityMeasure (Algorithm.p0 alg)
Learning.instIsProbabilityMeasureP0.{u_1,
    u_2}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  (alg : Algorithm ๐“ ๐“จ) :
  MeasureTheory.IsProbabilityMeasure
    (Algorithm.p0 alg)

Code

instance (alg : Algorithm ๐“ ๐“จ) : IsProbabilityMeasure alg.p0
Type uses (1)
Used by (13)

Actions: Source ยท Open Issue

Proof
alg.hp0

initAlgFunction๐Ÿ”—

DefinitionBandits.ArrayModel.initAlgFunction
Details

The initial action is the image of a uniform random variable by this function.

๐Ÿ”—def
Bandits.ArrayModel.initAlgFunction.{u_1, u_2} {๐“ : Type u_1}
  {R : Type u_2} {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R}
  [Nonempty ๐“] [StandardBorelSpace ๐“] (alg : Learning.Algorithm ๐“ R) :
  โ†‘unitInterval โ†’ ๐“
Bandits.ArrayModel.initAlgFunction.{u_1,
    u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“]
  (alg : Learning.Algorithm ๐“ R) :
  โ†‘unitInterval โ†’ ๐“

Code

noncomputable
def initAlgFunction (alg : Algorithm ๐“ R) : I โ†’ ๐“ :=
  (Measure.exists_measurable_map_eq alg.p0).choose
Type uses (1)
Body uses (1)
Used by (12)

Actions: Source ยท Open Issue

instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy๐Ÿ”—

InstanceLearning.instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy
Details

No docstring.

๐Ÿ”—theorem
Learning.instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy.{u_1,
    u_2}
  {๐“ : Type u_1} {๐“จ : Type u_2} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ} (alg : Algorithm ๐“ ๐“จ) (n : โ„•) :
  ProbabilityTheory.IsMarkovKernel (Algorithm.policy alg n)
Learning.instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy.{u_1,
    u_2}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  (alg : Algorithm ๐“ ๐“จ) (n : โ„•) :
  ProbabilityTheory.IsMarkovKernel
    (Algorithm.policy alg n)

Code

instance (alg : Algorithm ๐“ ๐“จ) (n : โ„•) : IsMarkovKernel (alg.policy n)
Type uses (1)
Used by (14)

Actions: Source ยท Open Issue

Proof
alg.h_policy n

algFunction๐Ÿ”—

DefinitionBandits.ArrayModel.algFunction
Details

The next action is the image of the history and a uniform random variable by this function.

๐Ÿ”—def
Bandits.ArrayModel.algFunction.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“] (alg : Learning.Algorithm ๐“ R) (n : โ„•) :
  (โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R) โ†’ โ†‘unitInterval โ†’ ๐“
Bandits.ArrayModel.algFunction.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R} [Nonempty ๐“]
  [StandardBorelSpace ๐“]
  (alg : Learning.Algorithm ๐“ R) (n : โ„•) :
  (โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R) โ†’
    โ†‘unitInterval โ†’ ๐“

Code

noncomputable
def algFunction (alg : Algorithm ๐“ R) (n : โ„•) :
    (Iic n โ†’ ๐“ ร— R) โ†’ I โ†’ ๐“ :=
  (Kernel.exists_measurable_map_eq_unitInterval (alg.policy n)).choose
Type uses (1)
Body uses (1)
Used by (17)

Actions: Source ยท Open Issue

pullCount'๐Ÿ”—

DefinitionLearning.pullCount'
Details

Number of pulls of arm a up to (and including) time n. This is the number of entries in h in which the arm is a.

๐Ÿ”—def
Learning.pullCount'.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  [DecidableEq ๐“] (n : โ„•) (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R) (a : ๐“) : โ„•
Learning.pullCount'.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  [DecidableEq ๐“] (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R) (a : ๐“) :
  โ„•

Code

noncomputable
def pullCount' (n : โ„•) (h : Iic n โ†’ ๐“ ร— R) (a : ๐“) := #{s | (h s).1 = a}
Used by (29)

Actions: Source ยท Open Issue

streamMeasure๐Ÿ”—

DefinitionBandits.streamMeasure
Details

Measure of an infinite stream of rewards from each action.

๐Ÿ”—def
Bandits.streamMeasure.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {mR : MeasurableSpace R}
  (ฮฝ : ProbabilityTheory.Kernel ๐“ R) : MeasureTheory.Measure (โ„• โ†’ ๐“ โ†’ R)
Bandits.streamMeasure.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {mR : MeasurableSpace R}
  (ฮฝ : ProbabilityTheory.Kernel ๐“ R) :
  MeasureTheory.Measure (โ„• โ†’ ๐“ โ†’ R)

Code

noncomputable
def streamMeasure (ฮฝ : Kernel ๐“ R) : Measure (โ„• โ†’ ๐“ โ†’ R) :=
  Measure.infinitePi fun _ โ†ฆ Measure.infinitePi ฮฝ
Used by (56)

Actions: Source ยท Open Issue