LeanMachineLearning exposition

Bandits.ArrayModel.hasCondDistrib_reward_pullCount_actionπŸ”—

This page has the declaration's own card below, then its dependency graph, then a card for each dependency (type dependencies first, then the rest of the transitive closure). For a theorem, the graph and the dependency cards only follow its statement's dependencies (its proof is replaced by sorry, so what it proves doesn't depend on how); for everything else, both the type and the body/value are followed, since their content is part of what later declarations build on.

Minimal Lean file

hasCondDistrib_reward_pullCount_actionπŸ”—

LemmaBandits.ArrayModel.hasCondDistrib_reward_pullCount_action
Details

The conditional distribution of the reward at time n + 1, given the action at time n + 1 and the number of times that action has been pulled before time n + 1, is equal to the kernel Ξ½.

πŸ”—theorem
Bandits.ArrayModel.hasCondDistrib_reward_pullCount_action.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2} {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐] [StandardBorelSpace 𝓐]
  [DecidableEq 𝓐] [Countable 𝓐] [StandardBorelSpace R] [Nonempty R]
  (alg : Learning.Algorithm 𝓐 R) (Ξ½ : ProbabilityTheory.Kernel 𝓐 R)
  [ProbabilityTheory.IsMarkovKernel Ξ½] (n : β„•) :
  ProbabilityTheory.HasCondDistrib (reward alg (n + 1))
    (fun Ο‰ =>
      (action alg (n + 1) Ο‰,
        Learning.pullCount (action alg) (action alg (n + 1) Ο‰) (n + 1)
          Ο‰))
    (ProbabilityTheory.Kernel.prodMkRight β„• Ξ½) (arrayMeasure Ξ½)
Bandits.ArrayModel.hasCondDistrib_reward_pullCount_action.{u_1,
    u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐]
  [Countable 𝓐] [StandardBorelSpace R]
  [Nonempty R]
  (alg : Learning.Algorithm 𝓐 R)
  (Ξ½ : ProbabilityTheory.Kernel 𝓐 R)
  [ProbabilityTheory.IsMarkovKernel Ξ½]
  (n : β„•) :
  ProbabilityTheory.HasCondDistrib
    (reward alg (n + 1))
    (fun Ο‰ =>
      (action alg (n + 1) Ο‰,
        Learning.pullCount (action alg)
          (action alg (n + 1) Ο‰) (n + 1)
          Ο‰))
    (ProbabilityTheory.Kernel.prodMkRight
      β„• Ξ½)
    (arrayMeasure Ξ½)

Code

lemma hasCondDistrib_reward_pullCount_action
    (alg : Algorithm 𝓐 R) (Ξ½ : Kernel 𝓐 R) [IsMarkovKernel Ξ½] (n : β„•) :
    HasCondDistrib (reward alg (n + 1))
      (fun Ο‰ ↦ (action alg (n + 1) Ο‰, pullCount (action alg) (action alg (n + 1) Ο‰) (n + 1) Ο‰))
      (Ξ½.prodMkRight _) (arrayMeasure Ξ½)
Type uses (7)
Body uses (14)
Used by (1)

Actions: Source Β· Open Issue

Proof
by
  have h_meas : Measurable fun Ο‰ ↦ pullCount (action alg) (action alg (n + 1) Ο‰) (n + 1) Ο‰ := by
    change Measurable ((fun p : (probSpace 𝓐 R) Γ— 𝓐 ↦ pullCount (action alg) p.2 (n + 1) p.1) ∘
      (fun Ο‰ : probSpace 𝓐 R ↦ (Ο‰, action alg (n + 1) Ο‰)))
    exact (measurable_uncurry_pullCount (by fun_prop) _).comp (by fun_prop)
  refine hasCondDistrib_of_condDistrib_eq (by fun_prop) (by fun_prop) ?_
  refine (condDistrib_ae_eq_cond
    (Measurable.prodMk (by fun_prop) (by fun_prop)) (by fun_prop)).trans ?_
  rw [Filter.EventuallyEq, ae_iff_of_countable]
  intro ⟨a, m⟩ ham
  simp only [Kernel.prodMkRight_apply]
  calc
    Measure.map (reward alg (n + 1))
      (arrayMeasure Ξ½)[|(fun Ο‰ ↦ (action alg (n + 1) Ο‰,
        pullCount (action alg) (action alg (n + 1) Ο‰) (n + 1) Ο‰)) ⁻¹' {(a, m)}]
  _ = Measure.map (fun Ο‰ ↦ Ο‰.2 m a)
      (arrayMeasure Ξ½)[|(fun Ο‰ ↦ (action alg (n + 1) Ο‰,
        pullCount (action alg) (action alg (n + 1) Ο‰) (n + 1) Ο‰)) ⁻¹' {(a, m)}] := by
    rw [reward_eq]
    refine Measure.map_congr
      (ae_cond_of_forall_mem ((measurableSet_singleton _).preimage (by fun_prop)) (fun x hx ↦ ?_))
    simp only [Set.mem_preimage, Set.mem_singleton_iff, Prod.mk.injEq] at hx
    simp only [hx.1] at hx ⊒
    simp [hx.2]
  _ = Measure.map (fun Ο‰ ↦ Ο‰.2 m a)
      (arrayMeasure Ξ½)[|({Ο‰ | action alg (n + 1) Ο‰ = a ∧
        pullCount (action alg) a (n + 1) Ο‰ = m}).indicator 1 ⁻¹' {1}] := by
    congr with Ο‰
    simp only [Set.mem_preimage, Set.mem_singleton_iff, Prod.mk.injEq, Set.indicator_apply,
      Set.mem_setOf_eq, Pi.one_apply, ite_eq_left_iff, not_and, zero_ne_one, imp_false,
      Classical.not_imp, Decidable.not_not, and_congr_right_iff]
    intro ha
    simp [ha]
  _ = Ξ½ a := by
    rw [cond_of_indepFun, map_snd_apply_arrayMeasure m a]
    Β· exact (indepFun_snd_apply_pullCount_action alg Ξ½ a m n).symm
    Β· refine Measurable.indicator (by fun_prop) ?_
      exact MeasurableSet.inter ((measurableSet_singleton _).preimage (by fun_prop))
        ((measurableSet_singleton _).preimage (by fun_prop))
    Β· fun_prop
    Β· simp
    Β· rw [Measure.map_apply (by fun_prop) (by simp)] at ham
      convert ham
      ext Ο‰
      simp only [Set.mem_preimage, Set.indicator_apply, Set.mem_setOf_eq, Pi.one_apply,
        Set.mem_singleton_iff, ite_eq_left_iff, not_and, zero_ne_one, imp_false, Classical.not_imp,
        Decidable.not_not, Prod.mk.injEq, and_congr_right_iff]
      intro ha
      simp [ha]

Dependency graph

Type dependencies (7)

AlgorithmπŸ”—

StructureLearning.Algorithm
Details

A stochastic, sequential algorithm.

πŸ”—structure
Learning.Algorithm.{u_4, u_5} (𝓐 : Type u_4) (𝓨 : Type u_5)
  [MeasurableSpace 𝓐] [MeasurableSpace 𝓨] : Type (max u_4 u_5)
Learning.Algorithm.{u_4, u_5}
  (𝓐 : Type u_4) (𝓨 : Type u_5)
  [MeasurableSpace 𝓐]
  [MeasurableSpace 𝓨] : Type (max u_4 u_5)

Code

structure Algorithm (𝓐 𝓨 : Type*) [MeasurableSpace 𝓐] [MeasurableSpace 𝓨] where
  /-- Policy or sampling rule: distribution of the next action. -/
  policy : (n : β„•) β†’ Kernel (Iic n β†’ 𝓐 Γ— 𝓨) 𝓐
  /-- The policy is a Markov kernel. -/
  [h_policy : βˆ€ n, IsMarkovKernel (policy n)]
  /-- Distribution of the first action. -/
  p0 : Measure 𝓐
  /-- The first action distribution is a probability measure. -/
  [hp0 : IsProbabilityMeasure p0]
Used by (216)

Actions: Source Β· Open Issue

probSpaceπŸ”—

DefinitionBandits.ArrayModel.probSpace
Details

Probability space for the array model of stochastic bandits.

πŸ”—def
Bandits.ArrayModel.probSpace.{u_1, u_2} (𝓐 : Type u_1) (R : Type u_2) :
  Type (max u_1 u_2)
Bandits.ArrayModel.probSpace.{u_1, u_2}
  (𝓐 : Type u_1) (R : Type u_2) :
  Type (max u_1 u_2)

Code

def probSpace : Type _ := (β„• β†’ I) Γ— (β„• β†’ 𝓐 β†’ R)
Used by (64)

Actions: Source Β· Open Issue

instMeasurableSpaceProbSpaceπŸ”—

InstanceBandits.ArrayModel.instMeasurableSpaceProbSpace
Details

No docstring.

πŸ”—def
Bandits.ArrayModel.instMeasurableSpaceProbSpace.{u_3, u_4}
  {𝓐 : Type u_3} {R : Type u_4} [MeasurableSpace R] :
  MeasurableSpace (probSpace 𝓐 R)
Bandits.ArrayModel.instMeasurableSpaceProbSpace.{u_3,
    u_4}
  {𝓐 : Type u_3} {R : Type u_4}
  [MeasurableSpace R] :
  MeasurableSpace (probSpace 𝓐 R)

Code

instance {𝓐 R : Type*} [MeasurableSpace R] : MeasurableSpace (probSpace 𝓐 R)
Type uses (1)
Used by (41)

Actions: Source Β· Open Issue

Proof
inferInstanceAs (MeasurableSpace ((β„• β†’ I) Γ— (β„• β†’ 𝓐 β†’ R)))

rewardπŸ”—

DefinitionBandits.ArrayModel.reward
Details

Reward received at time n in the array model.

πŸ”—def
Bandits.ArrayModel.reward.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐] (alg : Learning.Algorithm 𝓐 R)
  (n : β„•) (Ο‰ : probSpace 𝓐 R) : R
Bandits.ArrayModel.reward.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐]
  (alg : Learning.Algorithm 𝓐 R) (n : β„•)
  (Ο‰ : probSpace 𝓐 R) : R

Code

noncomputable
def reward [DecidableEq 𝓐] (alg : Algorithm 𝓐 R) (n : β„•) (Ο‰ : probSpace 𝓐 R) : R :=
  (hist alg Ο‰ n ⟨n, by simp⟩).2
Type uses (2)
Body uses (1)
Used by (24)

Actions: Source Β· Open Issue

actionπŸ”—

DefinitionBandits.ArrayModel.action
Details

Action taken at time n in the array model.

πŸ”—def
Bandits.ArrayModel.action.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐] (alg : Learning.Algorithm 𝓐 R)
  (n : β„•) (Ο‰ : probSpace 𝓐 R) : 𝓐
Bandits.ArrayModel.action.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐]
  (alg : Learning.Algorithm 𝓐 R) (n : β„•)
  (Ο‰ : probSpace 𝓐 R) : 𝓐

Code

noncomputable
def action [DecidableEq 𝓐] (alg : Algorithm 𝓐 R) (n : β„•) (Ο‰ : probSpace 𝓐 R) : 𝓐 :=
  (hist alg Ο‰ n ⟨n, by simp⟩).1
Type uses (2)
Body uses (1)
Used by (43)

Actions: Source Β· Open Issue

pullCountπŸ”—

DefinitionLearning.pullCount
Details

Number of times action a was chosen up to time t (excluding t).

πŸ”—def
Learning.pullCount.{u_1, u_3} {𝓐 : Type u_1} {Ξ© : Type u_3}
  [DecidableEq 𝓐] (A : β„• β†’ Ξ© β†’ 𝓐) (a : 𝓐) (t : β„•) (Ο‰ : Ξ©) : β„•
Learning.pullCount.{u_1, u_3}
  {𝓐 : Type u_1} {Ξ© : Type u_3}
  [DecidableEq 𝓐] (A : β„• β†’ Ξ© β†’ 𝓐) (a : 𝓐)
  (t : β„•) (Ο‰ : Ξ©) : β„•

Code

noncomputable
def pullCount (A : β„• β†’ Ξ© β†’ 𝓐) (a : 𝓐) (t : β„•) (Ο‰ : Ξ©) : β„• :=
  #(filter (fun s ↦ A s Ο‰ = a) (range t))
Used by (146)

Actions: Source Β· Open Issue

arrayMeasureπŸ”—

DefinitionBandits.ArrayModel.arrayMeasure
Details

Probability measure for the array model of stochastic bandits.

πŸ”—def
Bandits.ArrayModel.arrayMeasure.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R}
  (Ξ½ : ProbabilityTheory.Kernel 𝓐 R) :
  MeasureTheory.Measure (probSpace 𝓐 R)
Bandits.ArrayModel.arrayMeasure.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R}
  (Ξ½ : ProbabilityTheory.Kernel 𝓐 R) :
  MeasureTheory.Measure (probSpace 𝓐 R)

Code

noncomputable
def arrayMeasure (Ξ½ : Kernel 𝓐 R) : Measure (probSpace 𝓐 R) :=
  (Measure.infinitePi fun _ ↦ volume).prod (streamMeasure Ξ½)
Type uses (2)
Body uses (1)
Used by (29)

Actions: Source Β· Open Issue

All dependencies, transitively (7)

instIsProbabilityMeasureP0πŸ”—

InstanceLearning.instIsProbabilityMeasureP0
Details

No docstring.

πŸ”—theorem
Learning.instIsProbabilityMeasureP0.{u_1, u_2} {𝓐 : Type u_1}
  {𝓨 : Type u_2} {m𝓐 : MeasurableSpace 𝓐} {m𝓨 : MeasurableSpace 𝓨}
  (alg : Algorithm 𝓐 𝓨) :
  MeasureTheory.IsProbabilityMeasure (Algorithm.p0 alg)
Learning.instIsProbabilityMeasureP0.{u_1,
    u_2}
  {𝓐 : Type u_1} {𝓨 : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {m𝓨 : MeasurableSpace 𝓨}
  (alg : Algorithm 𝓐 𝓨) :
  MeasureTheory.IsProbabilityMeasure
    (Algorithm.p0 alg)

Code

instance (alg : Algorithm 𝓐 𝓨) : IsProbabilityMeasure alg.p0
Type uses (1)
Used by (13)

Actions: Source Β· Open Issue

Proof
alg.hp0

initAlgFunctionπŸ”—

DefinitionBandits.ArrayModel.initAlgFunction
Details

The initial action is the image of a uniform random variable by this function.

πŸ”—def
Bandits.ArrayModel.initAlgFunction.{u_1, u_2} {𝓐 : Type u_1}
  {R : Type u_2} {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R}
  [Nonempty 𝓐] [StandardBorelSpace 𝓐] (alg : Learning.Algorithm 𝓐 R) :
  ↑unitInterval β†’ 𝓐
Bandits.ArrayModel.initAlgFunction.{u_1,
    u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐]
  (alg : Learning.Algorithm 𝓐 R) :
  ↑unitInterval β†’ 𝓐

Code

noncomputable
def initAlgFunction (alg : Algorithm 𝓐 R) : I β†’ 𝓐 :=
  (Measure.exists_measurable_map_eq alg.p0).choose
Type uses (1)
Body uses (1)
Used by (12)

Actions: Source Β· Open Issue

instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicyπŸ”—

InstanceLearning.instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy
Details

No docstring.

πŸ”—theorem
Learning.instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy.{u_1,
    u_2}
  {𝓐 : Type u_1} {𝓨 : Type u_2} {m𝓐 : MeasurableSpace 𝓐}
  {m𝓨 : MeasurableSpace 𝓨} (alg : Algorithm 𝓐 𝓨) (n : β„•) :
  ProbabilityTheory.IsMarkovKernel (Algorithm.policy alg n)
Learning.instIsMarkovKernelForallSubtypeNatMemFinsetIicProdPolicy.{u_1,
    u_2}
  {𝓐 : Type u_1} {𝓨 : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {m𝓨 : MeasurableSpace 𝓨}
  (alg : Algorithm 𝓐 𝓨) (n : β„•) :
  ProbabilityTheory.IsMarkovKernel
    (Algorithm.policy alg n)

Code

instance (alg : Algorithm 𝓐 𝓨) (n : β„•) : IsMarkovKernel (alg.policy n)
Type uses (1)
Used by (14)

Actions: Source Β· Open Issue

Proof
alg.h_policy n

algFunctionπŸ”—

DefinitionBandits.ArrayModel.algFunction
Details

The next action is the image of the history and a uniform random variable by this function.

πŸ”—def
Bandits.ArrayModel.algFunction.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] (alg : Learning.Algorithm 𝓐 R) (n : β„•) :
  (β†₯(Finset.Iic n) β†’ 𝓐 Γ— R) β†’ ↑unitInterval β†’ 𝓐
Bandits.ArrayModel.algFunction.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐]
  (alg : Learning.Algorithm 𝓐 R) (n : β„•) :
  (β†₯(Finset.Iic n) β†’ 𝓐 Γ— R) β†’
    ↑unitInterval β†’ 𝓐

Code

noncomputable
def algFunction (alg : Algorithm 𝓐 R) (n : β„•) :
    (Iic n β†’ 𝓐 Γ— R) β†’ I β†’ 𝓐 :=
  (Kernel.exists_measurable_map_eq_unitInterval (alg.policy n)).choose
Type uses (1)
Body uses (1)
Used by (17)

Actions: Source Β· Open Issue

pullCount'πŸ”—

DefinitionLearning.pullCount'
Details

Number of pulls of arm a up to (and including) time n. This is the number of entries in h in which the arm is a.

πŸ”—def
Learning.pullCount'.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  [DecidableEq 𝓐] (n : β„•) (h : β†₯(Finset.Iic n) β†’ 𝓐 Γ— R) (a : 𝓐) : β„•
Learning.pullCount'.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  [DecidableEq 𝓐] (n : β„•)
  (h : β†₯(Finset.Iic n) β†’ 𝓐 Γ— R) (a : 𝓐) :
  β„•

Code

noncomputable
def pullCount' (n : β„•) (h : Iic n β†’ 𝓐 Γ— R) (a : 𝓐) := #{s | (h s).1 = a}
Used by (29)

Actions: Source Β· Open Issue

histπŸ”—

DefinitionBandits.ArrayModel.hist
Details

History of actions and rewards up to time n in the array model.

πŸ”—def
Bandits.ArrayModel.hist.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐] (alg : Learning.Algorithm 𝓐 R)
  (Ο‰ : probSpace 𝓐 R) (n : β„•) : β†₯(Finset.Iic n) β†’ 𝓐 Γ— R
Bandits.ArrayModel.hist.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R} [Nonempty 𝓐]
  [StandardBorelSpace 𝓐] [DecidableEq 𝓐]
  (alg : Learning.Algorithm 𝓐 R)
  (Ο‰ : probSpace 𝓐 R) (n : β„•) :
  β†₯(Finset.Iic n) β†’ 𝓐 Γ— R

Code

noncomputable
def hist [DecidableEq 𝓐] (alg : Algorithm 𝓐 R) (Ο‰ : probSpace 𝓐 R) : (n : β„•) β†’ Iic n β†’ 𝓐 Γ— R
| 0 => fun _ ↦ (initAlgFunction alg (Ο‰.1 0), Ο‰.2 0 (initAlgFunction alg (Ο‰.1 0)))
| n + 1 =>
  let hn : Iic n β†’ 𝓐 Γ— R := hist alg Ο‰ n
  let a : 𝓐 := algFunction alg n hn (Ο‰.1 (n + 1))
  fun i ↦ if hin : i ≀ n then hn ⟨i, by simp [hin]⟩ else (a, Ο‰.2 (pullCount' n hn a) a)
Type uses (2)
Body uses (3)
Used by (30)

Actions: Source Β· Open Issue

streamMeasureπŸ”—

DefinitionBandits.streamMeasure
Details

Measure of an infinite stream of rewards from each action.

πŸ”—def
Bandits.streamMeasure.{u_1, u_2} {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐} {mR : MeasurableSpace R}
  (Ξ½ : ProbabilityTheory.Kernel 𝓐 R) : MeasureTheory.Measure (β„• β†’ 𝓐 β†’ R)
Bandits.streamMeasure.{u_1, u_2}
  {𝓐 : Type u_1} {R : Type u_2}
  {m𝓐 : MeasurableSpace 𝓐}
  {mR : MeasurableSpace R}
  (Ξ½ : ProbabilityTheory.Kernel 𝓐 R) :
  MeasureTheory.Measure (β„• β†’ 𝓐 β†’ R)

Code

noncomputable
def streamMeasure (Ξ½ : Kernel 𝓐 R) : Measure (β„• β†’ 𝓐 β†’ R) :=
  Measure.infinitePi fun _ ↦ Measure.infinitePi Ξ½
Used by (56)

Actions: Source Β· Open Issue