LeanMachineLearning exposition

Bandits.UCB.ucbIndex_le_ucbIndex_arm๐Ÿ”—

This page has the declaration's own card below, then its dependency graph, then a card for each dependency (type dependencies first, then the rest of the transitive closure). For a theorem, the graph and the dependency cards only follow its statement's dependencies (its proof is replaced by sorry, so what it proves doesn't depend on how); for everything else, both the type and the body/value are followed, since their content is part of what later declarations build on.

Minimal Lean file

ucbIndex_le_ucbIndex_arm๐Ÿ”—

LemmaBandits.UCB.ucbIndex_le_ucbIndex_arm
Details

No docstring.

๐Ÿ”—theorem
Bandits.UCB.ucbIndex_le_ucbIndex_arm.{u_1} {K : โ„•} {hK : 0 < K} {c : โ„}
  {ฮฝ : ProbabilityTheory.Kernel (Fin K) โ„}
  [ProbabilityTheory.IsMarkovKernel ฮฝ] {ฮฉ : Type u_1}
  {mฮฉ : MeasurableSpace ฮฉ} {P : MeasureTheory.Measure ฮฉ}
  [MeasureTheory.IsProbabilityMeasure P] {A : โ„• โ†’ ฮฉ โ†’ Fin K}
  {R : โ„• โ†’ ฮฉ โ†’ โ„} {n : โ„•}
  (h :
    Learning.IsAlgEnvSeq A R (ucbAlgorithm hK c)
      (Learning.stationaryEnv ฮฝ) P)
  (a : Fin K) (hn : K โ‰ค n) :
  โˆ€แต (h : ฮฉ) โˆ‚P,
    Learning.empMean A R a n h + ucbWidth A c a n h โ‰ค
      Learning.empMean A R (A n h) n h + ucbWidth A c (A n h) n h
Bandits.UCB.ucbIndex_le_ucbIndex_arm.{u_1}
  {K : โ„•} {hK : 0 < K} {c : โ„}
  {ฮฝ : ProbabilityTheory.Kernel (Fin K) โ„}
  [ProbabilityTheory.IsMarkovKernel ฮฝ]
  {ฮฉ : Type u_1} {mฮฉ : MeasurableSpace ฮฉ}
  {P : MeasureTheory.Measure ฮฉ}
  [MeasureTheory.IsProbabilityMeasure P]
  {A : โ„• โ†’ ฮฉ โ†’ Fin K} {R : โ„• โ†’ ฮฉ โ†’ โ„}
  {n : โ„•}
  (h :
    Learning.IsAlgEnvSeq A R
      (ucbAlgorithm hK c)
      (Learning.stationaryEnv ฮฝ) P)
  (a : Fin K) (hn : K โ‰ค n) :
  โˆ€แต (h : ฮฉ) โˆ‚P,
    Learning.empMean A R a n h +
        ucbWidth A c a n h โ‰ค
      Learning.empMean A R (A n h) n h +
        ucbWidth A c (A n h) n h

Code

lemma ucbIndex_le_ucbIndex_arm
    (h : IsAlgEnvSeq A R (ucbAlgorithm hK c) (stationaryEnv ฮฝ) P) (a : Fin K) (hn : K โ‰ค n) :
    โˆ€แต h โˆ‚P, empMean A R a n h + ucbWidth A c a n h โ‰ค
      empMean A R (A n h) n h + ucbWidth A c (A n h) n h
Type uses (5)
Body uses (10)
Used by (1)

Actions: Source ยท Open Issue

Proof
by
  filter_upwards [arm_ae_eq_ucbNextArm h (n - 1)] with h h_arm
  have : n - 1 + 1 = n := by grind
  have h_not_lt : ยฌ n - 1 < K - 1 := by grind
  simp only [this, nextArm, h_not_lt, โ†“reduceIte] at h_arm
  have : Nonempty (Fin K) := Fin.pos_iff_nonempty.mp hK
  simp_rw [h_arm, empMean_eq_empMean' (by grind : n โ‰  0),
    ucbWidth_eq_ucbWidth' (A := A) (R := R) _ _ _ _ (by grind : n โ‰  0)]
  exact isMaxOn_argmax (fun a โ†ฆ empMean' (n - 1) (history A R (n - 1) h) a
    + ucbWidth' c (n - 1) (history A R (n - 1) h) a) _

Dependency graph

Type dependencies (5)

IsAlgEnvSeq๐Ÿ”—

StructureLearning.IsAlgEnvSeq
Details

An algorithm-environment sequence: a sequence of actions and feedbacks generated by an algorithm interacting with an environment.

๐Ÿ”—structure
Learning.IsAlgEnvSeq.{u_1, u_2, u_3} {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“} {m๐“จ : MeasurableSpace ๐“จ}
  {mฮฉ : MeasurableSpace ฮฉ} (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ)
  (alg : Algorithm ๐“ ๐“จ) (env : Environment ๐“ ๐“จ)
  (P : MeasureTheory.Measure ฮฉ) [MeasureTheory.IsFiniteMeasure P] : Prop
Learning.IsAlgEnvSeq.{u_1, u_2, u_3}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  {mฮฉ : MeasurableSpace ฮฉ} (A : โ„• โ†’ ฮฉ โ†’ ๐“)
  (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (alg : Algorithm ๐“ ๐“จ)
  (env : Environment ๐“ ๐“จ)
  (P : MeasureTheory.Measure ฮฉ)
  [MeasureTheory.IsFiniteMeasure P] : Prop

Code

structure IsAlgEnvSeq
    (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (alg : Algorithm ๐“ ๐“จ) (env : Environment ๐“ ๐“จ)
    (P : Measure ฮฉ) [IsFiniteMeasure P] : Prop where
  /-- The action sequence is measurable. -/
  measurable_action n : Measurable (A n) := by fun_prop
  /-- The feedback sequence is measurable. -/
  measurable_feedback n : Measurable (Y n) := by fun_prop
  /-- The first action has the correct law. -/
  hasLaw_action_zero : HasLaw (fun ฯ‰ โ†ฆ (A 0 ฯ‰)) alg.p0 P
  /-- The first feedback has the correct conditional distribution. -/
  hasCondDistrib_feedback_zero : HasCondDistrib (Y 0) (A 0) env.ฮฝ0 P
  /-- The next action has the correct conditional distribution given the history. -/
  hasCondDistrib_action n :
    HasCondDistrib (A (n + 1)) (history A Y n) (alg.policy n) P
  /-- The next feedback has the correct conditional distribution given the history and
  next action. -/
  hasCondDistrib_feedback n :
    HasCondDistrib (Y (n + 1)) (fun ฯ‰ โ†ฆ (history A Y n ฯ‰, A (n + 1) ฯ‰))
      (env.feedback n) P
Type uses (3)
Used by (111)

Actions: Source ยท Open Issue

ucbAlgorithm๐Ÿ”—

DefinitionBandits.ucbAlgorithm
Details

The UCB algorithm.

๐Ÿ”—def
Bandits.ucbAlgorithm {K : โ„•} (hK : 0 < K) (c : โ„) :
  Learning.Algorithm (Fin K) โ„
Bandits.ucbAlgorithm {K : โ„•} (hK : 0 < K)
  (c : โ„) : Learning.Algorithm (Fin K) โ„

Code

noncomputable
def ucbAlgorithm (hK : 0 < K) (c : โ„) : Algorithm (Fin K) โ„ :=
  detAlgorithm (UCB.nextArm hK c) (by fun_prop) โŸจ0, hKโŸฉ
Type uses (1)
Body uses (3)
Used by (16)

Actions: Source ยท Open Issue

stationaryEnv๐Ÿ”—

DefinitionLearning.stationaryEnv
Details

A stationary environment, in which the distribution of the next feedback depends only on the last action.

๐Ÿ”—def
Learning.stationaryEnv.{u_1, u_2} {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {m๐“จ : MeasurableSpace ๐“จ}
  (ฮฝ : ProbabilityTheory.Kernel ๐“ ๐“จ)
  [ProbabilityTheory.IsMarkovKernel ฮฝ] : Environment ๐“ ๐“จ
Learning.stationaryEnv.{u_1, u_2}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  (ฮฝ : ProbabilityTheory.Kernel ๐“ ๐“จ)
  [ProbabilityTheory.IsMarkovKernel ฮฝ] :
  Environment ๐“ ๐“จ

Code

def stationaryEnv (ฮฝ : Kernel ๐“ ๐“จ) [IsMarkovKernel ฮฝ] : Environment ๐“ ๐“จ := obliviousEnv fun _ โ†ฆ ฮฝ
Type uses (1)
Body uses (1)
Used by (81)

Actions: Source ยท Open Issue

empMean๐Ÿ”—

DefinitionLearning.empMean
Details

Empirical mean reward obtained when pulling action a up to time t (exclusive).

๐Ÿ”—def
Learning.empMean.{u_1, u_3} {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•)
  (ฯ‰ : ฮฉ) : โ„
Learning.empMean.{u_1, u_3} {๐“ : Type u_1}
  {ฮฉ : Type u_3} [DecidableEq ๐“]
  (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“)
  (t : โ„•) (ฯ‰ : ฮฉ) : โ„

Code

noncomputable
def empMean (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„ :=
  sumRewards A R' a t ฯ‰ / pullCount A a t ฯ‰
Body uses (2)
Used by (34)

Actions: Source ยท Open Issue

ucbWidth๐Ÿ”—

DefinitionBandits.UCB.ucbWidth
Details

The exploration bonus of the UCB algorithm, which corresponds to the width of a confidence interval.

๐Ÿ”—def
Bandits.UCB.ucbWidth.{u_1} {K : โ„•} {ฮฉ : Type u_1} (A : โ„• โ†’ ฮฉ โ†’ Fin K)
  (c : โ„) (a : Fin K) (n : โ„•) (ฯ‰ : ฮฉ) : โ„
Bandits.UCB.ucbWidth.{u_1} {K : โ„•}
  {ฮฉ : Type u_1} (A : โ„• โ†’ ฮฉ โ†’ Fin K)
  (c : โ„) (a : Fin K) (n : โ„•) (ฯ‰ : ฮฉ) : โ„

Code

noncomputable def ucbWidth (A : โ„• โ†’ ฮฉ โ†’ Fin K) (c : โ„) (a : Fin K) (n : โ„•) (ฯ‰ : ฮฉ) : โ„ :=
  โˆš(2 * c * log (n + 1) / pullCount A a n ฯ‰)
Body uses (1)
Used by (16)

Actions: Source ยท Open Issue

All dependencies, transitively (17)

Algorithm๐Ÿ”—

StructureLearning.Algorithm
Details

A stochastic, sequential algorithm.

๐Ÿ”—structure
Learning.Algorithm.{u_4, u_5} (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] : Type (max u_4 u_5)
Learning.Algorithm.{u_4, u_5}
  (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“]
  [MeasurableSpace ๐“จ] : Type (max u_4 u_5)

Code

structure Algorithm (๐“ ๐“จ : Type*) [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] where
  /-- Policy or sampling rule: distribution of the next action. -/
  policy : (n : โ„•) โ†’ Kernel (Iic n โ†’ ๐“ ร— ๐“จ) ๐“
  /-- The policy is a Markov kernel. -/
  [h_policy : โˆ€ n, IsMarkovKernel (policy n)]
  /-- Distribution of the first action. -/
  p0 : Measure ๐“
  /-- The first action distribution is a probability measure. -/
  [hp0 : IsProbabilityMeasure p0]
Used by (216)

Actions: Source ยท Open Issue

Environment๐Ÿ”—

StructureLearning.Environment
Details

A stochastic environment.

๐Ÿ”—structure
Learning.Environment.{u_4, u_5} (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] : Type (max u_4 u_5)
Learning.Environment.{u_4, u_5}
  (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“]
  [MeasurableSpace ๐“จ] : Type (max u_4 u_5)

Code

structure Environment (๐“ ๐“จ : Type*) [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] where
  /-- Distribution of the next observation as function of the past history. -/
  feedback : (n : โ„•) โ†’ Kernel ((Iic n โ†’ ๐“ ร— ๐“จ) ร— ๐“) ๐“จ
  /-- The feedback kernels are Markov kernels. -/
  [h_feedback : โˆ€ n, IsMarkovKernel (feedback n)]
  /-- Distribution of the first observation given the first action. -/
  ฮฝ0 : Kernel ๐“ ๐“จ
  /-- The initial observation kernel is a Markov kernel. -/
  [hp0 : IsMarkovKernel ฮฝ0]
Used by (128)

Actions: Source ยท Open Issue

history๐Ÿ”—

DefinitionLearning.history
Details

History of the algorithm-environment sequence up to time n.

๐Ÿ”—def
Learning.history.{u_1, u_2, u_3} {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (n : โ„•) (ฯ‰ : ฮฉ) :
  โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— ๐“จ
Learning.history.{u_1, u_2, u_3}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} (A : โ„• โ†’ ฮฉ โ†’ ๐“)
  (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (n : โ„•) (ฯ‰ : ฮฉ) :
  โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— ๐“จ

Code

def history (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (n : โ„•) (ฯ‰ : ฮฉ) : Iic n โ†’ ๐“ ร— ๐“จ :=
  fun i โ†ฆ (A i ฯ‰, Y i ฯ‰)
Used by (72)

Actions: Source ยท Open Issue

detAlgorithm๐Ÿ”—

DefinitionLearning.detAlgorithm
Details

A deterministic algorithm, which chooses the action given by the function nextAction.

๐Ÿ”—def
Learning.detAlgorithm.{u_1, u_2} {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {m๐“จ : MeasurableSpace ๐“จ}
  (nextA : (n : โ„•) โ†’ (โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— ๐“จ) โ†’ ๐“)
  (h_next : โˆ€ (n : โ„•), Measurable (nextA n)) (action0 : ๐“) :
  Algorithm ๐“ ๐“จ
Learning.detAlgorithm.{u_1, u_2}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  (nextA :
    (n : โ„•) โ†’
      (โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— ๐“จ) โ†’ ๐“)
  (h_next :
    โˆ€ (n : โ„•), Measurable (nextA n))
  (action0 : ๐“) : Algorithm ๐“ ๐“จ

Code

noncomputable
def detAlgorithm (nextA : (n : โ„•) โ†’ (Iic n โ†’ ๐“ ร— ๐“จ) โ†’ ๐“)
    (h_next : โˆ€ n, Measurable (nextA n)) (action0 : ๐“) :
    Algorithm ๐“ ๐“จ where
  policy n := Kernel.deterministic (nextA n) (h_next n)
  p0 := Measure.dirac action0
Type uses (1)
Used by (15)

Actions: Source ยท Open Issue

nextAction๐Ÿ”—

DefinitionLearning.RoundRobin.nextAction
Details

Action chosen by the Round-Robin algorithm at time n + 1. This is action (n + 1) % K.

๐Ÿ”—def
Learning.RoundRobin.nextAction {K : โ„•} (hK : 0 < K) (n : โ„•) : Fin K
Learning.RoundRobin.nextAction {K : โ„•}
  (hK : 0 < K) (n : โ„•) : Fin K

Code

noncomputable
def RoundRobin.nextAction (hK : 0 < K) (n : โ„•) : Fin K := โŸจ(n + 1) % K, Nat.mod_lt _ hKโŸฉ
Used by (14)

Actions: Source ยท Open Issue

max๐Ÿ”—

DefinitionFunction.max
Details

The maximum value of a tuple.

๐Ÿ”—def
Function.max.{u_1, u_2} {ฮน : Type u_1} {ฮฑ : Type u_2} [LinearOrder ฮฑ]
  [Fintype ฮน] [Nonempty ฮน] (f : ฮน โ†’ ฮฑ) : ฮฑ
Function.max.{u_1, u_2} {ฮน : Type u_1}
  {ฮฑ : Type u_2} [LinearOrder ฮฑ]
  [Fintype ฮน] [Nonempty ฮน] (f : ฮน โ†’ ฮฑ) : ฮฑ

Code

abbrev max : ฮฑ := univ.sup' univ_nonempty f
Used by (8)

Actions: Source ยท Open Issue

exists_argmax๐Ÿ”—

Lemmaexists_argmax
Details

No docstring.

๐Ÿ”—theorem
exists_argmax.{u_1, u_2} {ฮน : Type u_1} {ฮฑ : Type u_2} [LinearOrder ฮฑ]
  [Fintype ฮน] [Nonempty ฮน] (f : ฮน โ†’ ฮฑ) : โˆƒ i, f i = Function.max f
exists_argmax.{u_1, u_2} {ฮน : Type u_1}
  {ฮฑ : Type u_2} [LinearOrder ฮฑ]
  [Fintype ฮน] [Nonempty ฮน] (f : ฮน โ†’ ฮฑ) :
  โˆƒ i, f i = Function.max f

Code

lemma exists_argmax : โˆƒ i, f i = f.max
Type uses (1)
Used by (3)

Actions: Source ยท Open Issue

Proof
by
  obtain โŸจi, -, hiโŸฉ := Finset.exists_mem_eq_sup' (by simp : Finset.univ.Nonempty) f
  exact โŸจi, hi.symmโŸฉ

argmax๐Ÿ”—

Definitionargmax
Details

The index of the maximum value of a tuple.

๐Ÿ”—def
argmax.{u_1, u_2} {ฮน : Type u_1} {ฮฑ : Type u_2} [LinearOrder ฮฑ]
  [Fintype ฮน] [Nonempty ฮน] (f : ฮน โ†’ ฮฑ) : ฮน
argmax.{u_1, u_2} {ฮน : Type u_1}
  {ฮฑ : Type u_2} [LinearOrder ฮฑ]
  [Fintype ฮน] [Nonempty ฮน] (f : ฮน โ†’ ฮฑ) : ฮน

Code

noncomputable def argmax := (exists_argmax f).choose
Body uses (2)
Used by (17)

Actions: Source ยท Open Issue

sumRewards'๐Ÿ”—

DefinitionLearning.sumRewards'
Details

Sum of rewards of arm a up to (and including) time n.

๐Ÿ”—def
Learning.sumRewards'.{u_1} {๐“ : Type u_1} [DecidableEq ๐“] (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— โ„) (a : ๐“) : โ„
Learning.sumRewards'.{u_1} {๐“ : Type u_1}
  [DecidableEq ๐“] (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— โ„) (a : ๐“) :
  โ„

Code

noncomputable
def sumRewards' (n : โ„•) (h : Iic n โ†’ ๐“ ร— โ„) (a : ๐“) :=
  โˆ‘ s, if (h s).1 = a then (h s).2 else 0
Used by (9)

Actions: Source ยท Open Issue

pullCount'๐Ÿ”—

DefinitionLearning.pullCount'
Details

Number of pulls of arm a up to (and including) time n. This is the number of entries in h in which the arm is a.

๐Ÿ”—def
Learning.pullCount'.{u_1, u_2} {๐“ : Type u_1} {R : Type u_2}
  [DecidableEq ๐“] (n : โ„•) (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R) (a : ๐“) : โ„•
Learning.pullCount'.{u_1, u_2}
  {๐“ : Type u_1} {R : Type u_2}
  [DecidableEq ๐“] (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— R) (a : ๐“) :
  โ„•

Code

noncomputable
def pullCount' (n : โ„•) (h : Iic n โ†’ ๐“ ร— R) (a : ๐“) := #{s | (h s).1 = a}
Used by (29)

Actions: Source ยท Open Issue

empMean'๐Ÿ”—

DefinitionLearning.empMean'
Details

Empirical mean of arm a at time n.

๐Ÿ”—def
Learning.empMean'.{u_1} {๐“ : Type u_1} [DecidableEq ๐“] (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— โ„) (a : ๐“) : โ„
Learning.empMean'.{u_1} {๐“ : Type u_1}
  [DecidableEq ๐“] (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— โ„) (a : ๐“) :
  โ„

Code

noncomputable
def empMean' (n : โ„•) (h : Iic n โ†’ ๐“ ร— โ„) (a : ๐“) :=
  (sumRewards' n h a) / (pullCount' n h a)
Body uses (2)
Used by (18)

Actions: Source ยท Open Issue

ucbWidth'๐Ÿ”—

DefinitionBandits.ucbWidth'
Details

The exploration bonus of the UCB algorithm, which corresponds to the width of a confidence interval.

๐Ÿ”—def
Bandits.ucbWidth' {K : โ„•} (c : โ„) (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ Fin K ร— โ„) (a : Fin K) : โ„
Bandits.ucbWidth' {K : โ„•} (c : โ„) (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ Fin K ร— โ„)
  (a : Fin K) : โ„

Code

noncomputable def ucbWidth' (c : โ„) (n : โ„•) (h : Iic n โ†’ Fin K ร— โ„) (a : Fin K) : โ„ :=
  โˆš(2 * c * log (n + 2) / pullCount' n h a)
Body uses (1)
Used by (6)

Actions: Source ยท Open Issue

nextArm๐Ÿ”—

DefinitionBandits.UCB.nextArm
Details

Arm pulled by the UCB algorithm at time n + 1.

๐Ÿ”—def
Bandits.UCB.nextArm {K : โ„•} (hK : 0 < K) (c : โ„) (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ Fin K ร— โ„) : Fin K
Bandits.UCB.nextArm {K : โ„•} (hK : 0 < K)
  (c : โ„) (n : โ„•)
  (h : โ†ฅ(Finset.Iic n) โ†’ Fin K ร— โ„) :
  Fin K

Code

noncomputable
def UCB.nextArm (hK : 0 < K) (c : โ„) (n : โ„•) (h : Iic n โ†’ Fin K ร— โ„) : Fin K :=
  have : Nonempty (Fin K) := Fin.pos_iff_nonempty.mp hK
  if n < K - 1 then RoundRobin.nextAction hK n else
  argmax (fun a โ†ฆ empMean' n h a + ucbWidth' c n h a)
Body uses (4)
Used by (7)

Actions: Source ยท Open Issue

measurable_nextArm๐Ÿ”—

LemmaBandits.UCB.measurable_nextArm
Details

No docstring.

๐Ÿ”—theorem
Bandits.UCB.measurable_nextArm {K : โ„•} (hK : 0 < K) (c : โ„) (n : โ„•) :
  Measurable (nextArm hK c n)
Bandits.UCB.measurable_nextArm {K : โ„•}
  (hK : 0 < K) (c : โ„) (n : โ„•) :
  Measurable (nextArm hK c n)

Code

lemma UCB.measurable_nextArm (hK : 0 < K) (c : โ„) (n : โ„•) : Measurable (nextArm hK c n)
Type uses (1)
Body uses (9)
Used by (3)

Actions: Source ยท Open Issue

Proof
by
  refine Measurable.ite (by simp) (by fun_prop) ?_
  have : Nonempty (Fin K) := Fin.pos_iff_nonempty.mp hK
  unfold ucbWidth'
  fun_prop

obliviousEnv๐Ÿ”—

DefinitionLearning.obliviousEnv
Details

An oblivious environment, in which the distribution of the next feedback depends only on the last action, but in a possibly time-dependent manner.

๐Ÿ”—def
Learning.obliviousEnv.{u_1, u_2} {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“} {m๐“จ : MeasurableSpace ๐“จ}
  (ฮฝ : โ„• โ†’ ProbabilityTheory.Kernel ๐“ ๐“จ)
  [โˆ€ (n : โ„•), ProbabilityTheory.IsMarkovKernel (ฮฝ n)] : Environment ๐“ ๐“จ
Learning.obliviousEnv.{u_1, u_2}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  (ฮฝ : โ„• โ†’ ProbabilityTheory.Kernel ๐“ ๐“จ)
  [โˆ€ (n : โ„•),
      ProbabilityTheory.IsMarkovKernel
        (ฮฝ n)] :
  Environment ๐“ ๐“จ

Code

def obliviousEnv (ฮฝ : โ„• โ†’ Kernel ๐“ ๐“จ) [โˆ€ n, IsMarkovKernel (ฮฝ n)] : Environment ๐“ ๐“จ where
  feedback n := (ฮฝ (n + 1)).prodMkLeft _
  ฮฝ0 := ฮฝ 0
Type uses (1)
Used by (10)

Actions: Source ยท Open Issue

sumRewards๐Ÿ”—

DefinitionLearning.sumRewards
Details

Sum of rewards obtained when pulling action a up to time t (exclusive).

๐Ÿ”—def
Learning.sumRewards.{u_1, u_3} {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•)
  (ฯ‰ : ฮฉ) : โ„
Learning.sumRewards.{u_1, u_3}
  {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“)
  (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•)
  (ฯ‰ : ฮฉ) : โ„

Code

def sumRewards (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„ :=
  โˆ‘ s โˆˆ range t, if A s ฯ‰ = a then R' s ฯ‰ else 0
Used by (44)

Actions: Source ยท Open Issue

pullCount๐Ÿ”—

DefinitionLearning.pullCount
Details

Number of times action a was chosen up to time t (excluding t).

๐Ÿ”—def
Learning.pullCount.{u_1, u_3} {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„•
Learning.pullCount.{u_1, u_3}
  {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (a : ๐“)
  (t : โ„•) (ฯ‰ : ฮฉ) : โ„•

Code

noncomputable
def pullCount (A : โ„• โ†’ ฮฉ โ†’ ๐“) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„• :=
  #(filter (fun s โ†ฆ A s ฯ‰ = a) (range t))
Used by (146)

Actions: Source ยท Open Issue