LeanMachineLearning exposition

Learning.IsAlgEnvSeq.adapted_sumRewards_add_one๐Ÿ”—

This page has the declaration's own card below, then its dependency graph, then a card for each dependency (type dependencies first, then the rest of the transitive closure). For a theorem, the graph and the dependency cards only follow its statement's dependencies (its proof is replaced by sorry, so what it proves doesn't depend on how); for everything else, both the type and the body/value are followed, since their content is part of what later declarations build on.

Minimal Lean file

adapted_sumRewards_add_one๐Ÿ”—

LemmaLearning.IsAlgEnvSeq.adapted_sumRewards_add_one
Details

No docstring.

๐Ÿ”—theorem
Learning.IsAlgEnvSeq.adapted_sumRewards_add_one.{u_1, u_3}
  {๐“ : Type u_1} {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {mฮฉ : MeasurableSpace ฮฉ} [DecidableEq ๐“] {P : MeasureTheory.Measure ฮฉ}
  [MeasureTheory.IsProbabilityMeasure P] {A : โ„• โ†’ ฮฉ โ†’ ๐“}
  [StandardBorelSpace ๐“] {R' : โ„• โ†’ ฮฉ โ†’ โ„} {alg : Algorithm ๐“ โ„}
  {env : Environment ๐“ โ„} (h : IsAlgEnvSeq A R' alg env P) (a : ๐“) :
  MeasureTheory.Adapted (filtration โ‹ฏ โ‹ฏ) fun n =>
    sumRewards A R' a (n + 1)
Learning.IsAlgEnvSeq.adapted_sumRewards_add_one.{u_1,
    u_3}
  {๐“ : Type u_1} {ฮฉ : Type u_3}
  {m๐“ : MeasurableSpace ๐“}
  {mฮฉ : MeasurableSpace ฮฉ} [DecidableEq ๐“]
  {P : MeasureTheory.Measure ฮฉ}
  [MeasureTheory.IsProbabilityMeasure P]
  {A : โ„• โ†’ ฮฉ โ†’ ๐“} [StandardBorelSpace ๐“]
  {R' : โ„• โ†’ ฮฉ โ†’ โ„} {alg : Algorithm ๐“ โ„}
  {env : Environment ๐“ โ„}
  (h : IsAlgEnvSeq A R' alg env P)
  (a : ๐“) :
  MeasureTheory.Adapted (filtration โ‹ฏ โ‹ฏ)
    fun n => sumRewards A R' a (n + 1)

Code

lemma IsAlgEnvSeq.adapted_sumRewards_add_one [StandardBorelSpace ๐“] {R' : โ„• โ†’ ฮฉ โ†’ โ„}
    {alg : Algorithm ๐“ โ„} {env : Environment ๐“ โ„}
    (h : IsAlgEnvSeq A R' alg env P) (a : ๐“) :
    Adapted (IsAlgEnvSeq.filtration h.measurable_action h.measurable_feedback)
      (fun n โ†ฆ sumRewards A R' a (n + 1))
Type uses (5)
Body uses (1)

Actions: Source ยท Open Issue

Proof
(h.stronglyAdapted_sumRewards_add_one a).adapted

Dependency graph

Type dependencies (5)

Algorithm๐Ÿ”—

StructureLearning.Algorithm
Details

A stochastic, sequential algorithm.

๐Ÿ”—structure
Learning.Algorithm.{u_4, u_5} (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] : Type (max u_4 u_5)
Learning.Algorithm.{u_4, u_5}
  (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“]
  [MeasurableSpace ๐“จ] : Type (max u_4 u_5)

Code

structure Algorithm (๐“ ๐“จ : Type*) [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] where
  /-- Policy or sampling rule: distribution of the next action. -/
  policy : (n : โ„•) โ†’ Kernel (Iic n โ†’ ๐“ ร— ๐“จ) ๐“
  /-- The policy is a Markov kernel. -/
  [h_policy : โˆ€ n, IsMarkovKernel (policy n)]
  /-- Distribution of the first action. -/
  p0 : Measure ๐“
  /-- The first action distribution is a probability measure. -/
  [hp0 : IsProbabilityMeasure p0]
Used by (216)

Actions: Source ยท Open Issue

Environment๐Ÿ”—

StructureLearning.Environment
Details

A stochastic environment.

๐Ÿ”—structure
Learning.Environment.{u_4, u_5} (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] : Type (max u_4 u_5)
Learning.Environment.{u_4, u_5}
  (๐“ : Type u_4) (๐“จ : Type u_5)
  [MeasurableSpace ๐“]
  [MeasurableSpace ๐“จ] : Type (max u_4 u_5)

Code

structure Environment (๐“ ๐“จ : Type*) [MeasurableSpace ๐“] [MeasurableSpace ๐“จ] where
  /-- Distribution of the next observation as function of the past history. -/
  feedback : (n : โ„•) โ†’ Kernel ((Iic n โ†’ ๐“ ร— ๐“จ) ร— ๐“) ๐“จ
  /-- The feedback kernels are Markov kernels. -/
  [h_feedback : โˆ€ n, IsMarkovKernel (feedback n)]
  /-- Distribution of the first observation given the first action. -/
  ฮฝ0 : Kernel ๐“ ๐“จ
  /-- The initial observation kernel is a Markov kernel. -/
  [hp0 : IsMarkovKernel ฮฝ0]
Used by (128)

Actions: Source ยท Open Issue

IsAlgEnvSeq๐Ÿ”—

StructureLearning.IsAlgEnvSeq
Details

An algorithm-environment sequence: a sequence of actions and feedbacks generated by an algorithm interacting with an environment.

๐Ÿ”—structure
Learning.IsAlgEnvSeq.{u_1, u_2, u_3} {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“} {m๐“จ : MeasurableSpace ๐“จ}
  {mฮฉ : MeasurableSpace ฮฉ} (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ)
  (alg : Algorithm ๐“ ๐“จ) (env : Environment ๐“ ๐“จ)
  (P : MeasureTheory.Measure ฮฉ) [MeasureTheory.IsFiniteMeasure P] : Prop
Learning.IsAlgEnvSeq.{u_1, u_2, u_3}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  {mฮฉ : MeasurableSpace ฮฉ} (A : โ„• โ†’ ฮฉ โ†’ ๐“)
  (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (alg : Algorithm ๐“ ๐“จ)
  (env : Environment ๐“ ๐“จ)
  (P : MeasureTheory.Measure ฮฉ)
  [MeasureTheory.IsFiniteMeasure P] : Prop

Code

structure IsAlgEnvSeq
    (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (alg : Algorithm ๐“ ๐“จ) (env : Environment ๐“ ๐“จ)
    (P : Measure ฮฉ) [IsFiniteMeasure P] : Prop where
  /-- The action sequence is measurable. -/
  measurable_action n : Measurable (A n) := by fun_prop
  /-- The feedback sequence is measurable. -/
  measurable_feedback n : Measurable (Y n) := by fun_prop
  /-- The first action has the correct law. -/
  hasLaw_action_zero : HasLaw (fun ฯ‰ โ†ฆ (A 0 ฯ‰)) alg.p0 P
  /-- The first feedback has the correct conditional distribution. -/
  hasCondDistrib_feedback_zero : HasCondDistrib (Y 0) (A 0) env.ฮฝ0 P
  /-- The next action has the correct conditional distribution given the history. -/
  hasCondDistrib_action n :
    HasCondDistrib (A (n + 1)) (history A Y n) (alg.policy n) P
  /-- The next feedback has the correct conditional distribution given the history and
  next action. -/
  hasCondDistrib_feedback n :
    HasCondDistrib (Y (n + 1)) (fun ฯ‰ โ†ฆ (history A Y n ฯ‰, A (n + 1) ฯ‰))
      (env.feedback n) P
Type uses (3)
Used by (111)

Actions: Source ยท Open Issue

filtration๐Ÿ”—

DefinitionLearning.IsAlgEnvSeq.filtration
Details

Filtration generated by the history up to time n.

๐Ÿ”—def
Learning.IsAlgEnvSeq.filtration.{u_1, u_2, u_3} {๐“ : Type u_1}
  {๐“จ : Type u_2} {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ} {mฮฉ : MeasurableSpace ฮฉ} {A : โ„• โ†’ ฮฉ โ†’ ๐“}
  {Y : โ„• โ†’ ฮฉ โ†’ ๐“จ} (hA : โˆ€ (n : โ„•), Measurable (A n))
  (hY : โˆ€ (n : โ„•), Measurable (Y n)) : MeasureTheory.Filtration โ„• mฮฉ
Learning.IsAlgEnvSeq.filtration.{u_1, u_2,
    u_3}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  {mฮฉ : MeasurableSpace ฮฉ} {A : โ„• โ†’ ฮฉ โ†’ ๐“}
  {Y : โ„• โ†’ ฮฉ โ†’ ๐“จ}
  (hA : โˆ€ (n : โ„•), Measurable (A n))
  (hY : โˆ€ (n : โ„•), Measurable (Y n)) :
  MeasureTheory.Filtration โ„• mฮฉ

Code

def IsAlgEnvSeq.filtration (hA : โˆ€ n, Measurable (A n)) (hY : โˆ€ n, Measurable (Y n)) :
    Filtration โ„• mฮฉ where
  seq i := MeasurableSpace.comap (history A Y i) inferInstance
  mono' i j hij := by
    simp only
    rw [โ† measurable_iff_comap_le]
    have : history A Y i = (fun h k โ†ฆ h โŸจk.1, by grindโŸฉ) โˆ˜ history A Y j := rfl
    rw [this]
    exact measurable_comp_comap _ (by fun_prop)
  le' i := by
    rw [โ† measurable_iff_comap_le]
    exact Learning.measurable_history hA hY i
Body uses (3)
Used by (18)

Actions: Source ยท Open Issue

sumRewards๐Ÿ”—

DefinitionLearning.sumRewards
Details

Sum of rewards obtained when pulling action a up to time t (exclusive).

๐Ÿ”—def
Learning.sumRewards.{u_1, u_3} {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•)
  (ฯ‰ : ฮฉ) : โ„
Learning.sumRewards.{u_1, u_3}
  {๐“ : Type u_1} {ฮฉ : Type u_3}
  [DecidableEq ๐“] (A : โ„• โ†’ ฮฉ โ†’ ๐“)
  (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•)
  (ฯ‰ : ฮฉ) : โ„

Code

def sumRewards (A : โ„• โ†’ ฮฉ โ†’ ๐“) (R' : โ„• โ†’ ฮฉ โ†’ โ„) (a : ๐“) (t : โ„•) (ฯ‰ : ฮฉ) : โ„ :=
  โˆ‘ s โˆˆ range t, if A s ฯ‰ = a then R' s ฯ‰ else 0
Used by (44)

Actions: Source ยท Open Issue

All dependencies, transitively (3)

history๐Ÿ”—

DefinitionLearning.history
Details

History of the algorithm-environment sequence up to time n.

๐Ÿ”—def
Learning.history.{u_1, u_2, u_3} {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (n : โ„•) (ฯ‰ : ฮฉ) :
  โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— ๐“จ
Learning.history.{u_1, u_2, u_3}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} (A : โ„• โ†’ ฮฉ โ†’ ๐“)
  (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (n : โ„•) (ฯ‰ : ฮฉ) :
  โ†ฅ(Finset.Iic n) โ†’ ๐“ ร— ๐“จ

Code

def history (A : โ„• โ†’ ฮฉ โ†’ ๐“) (Y : โ„• โ†’ ฮฉ โ†’ ๐“จ) (n : โ„•) (ฯ‰ : ฮฉ) : Iic n โ†’ ๐“ ร— ๐“จ :=
  fun i โ†ฆ (A i ฯ‰, Y i ฯ‰)
Used by (72)

Actions: Source ยท Open Issue

measurable_comp_comap๐Ÿ”—

LemmaMeasureTheory.measurable_comp_comap
Details

No docstring.

๐Ÿ”—theorem
MeasureTheory.measurable_comp_comap.{u_1, u_2, u_3} {ฮฑ : Type u_1}
  {ฮฒ : Type u_2} {ฮณ : Type u_3} {mฮฒ : MeasurableSpace ฮฒ}
  {mฮณ : MeasurableSpace ฮณ} (f : ฮฑ โ†’ ฮฒ) {g : ฮฒ โ†’ ฮณ} (hg : Measurable g) :
  Measurable (g โˆ˜ f)
MeasureTheory.measurable_comp_comap.{u_1,
    u_2, u_3}
  {ฮฑ : Type u_1} {ฮฒ : Type u_2}
  {ฮณ : Type u_3} {mฮฒ : MeasurableSpace ฮฒ}
  {mฮณ : MeasurableSpace ฮณ} (f : ฮฑ โ†’ ฮฒ)
  {g : ฮฒ โ†’ ฮณ} (hg : Measurable g) :
  Measurable (g โˆ˜ f)

Code

lemma measurable_comp_comap (f : ฮฑ โ†’ ฮฒ) {g : ฮฒ โ†’ ฮณ} (hg : Measurable g) :
    Measurable[mฮฒ.comap f] (g โˆ˜ f)
Used by (10)

Actions: Source ยท Open Issue

Proof
by
  rw [measurable_iff_comap_le, โ† MeasurableSpace.comap_comp]
  exact MeasurableSpace.comap_mono hg.comap_le

measurable_history๐Ÿ”—

LemmaLearning.measurable_history
Details

No docstring.

๐Ÿ”—theorem
Learning.measurable_history.{u_1, u_2, u_3} {๐“ : Type u_1}
  {๐“จ : Type u_2} {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ} {mฮฉ : MeasurableSpace ฮฉ} {A : โ„• โ†’ ฮฉ โ†’ ๐“}
  {Y : โ„• โ†’ ฮฉ โ†’ ๐“จ} (hA : โˆ€ (n : โ„•), Measurable (A n))
  (hY : โˆ€ (n : โ„•), Measurable (Y n)) (n : โ„•) :
  Measurable (history A Y n)
Learning.measurable_history.{u_1, u_2,
    u_3}
  {๐“ : Type u_1} {๐“จ : Type u_2}
  {ฮฉ : Type u_3} {m๐“ : MeasurableSpace ๐“}
  {m๐“จ : MeasurableSpace ๐“จ}
  {mฮฉ : MeasurableSpace ฮฉ} {A : โ„• โ†’ ฮฉ โ†’ ๐“}
  {Y : โ„• โ†’ ฮฉ โ†’ ๐“จ}
  (hA : โˆ€ (n : โ„•), Measurable (A n))
  (hY : โˆ€ (n : โ„•), Measurable (Y n))
  (n : โ„•) : Measurable (history A Y n)

Code

lemma measurable_history (hA : โˆ€ n, Measurable (A n))
    (hY : โˆ€ n, Measurable (Y n)) (n : โ„•) :
    Measurable (history A Y n)
Type uses (1)
Used by (10)

Actions: Source ยท Open Issue

Proof
by
  unfold history
  fun_prop