Discrete Dynamic Programming

Markov Chains

A worker’s employment dynamics is described by the stochastic matrix

\[P = \begin{bmatrix} 1-\alpha & \alpha \\ \beta & 1-\beta \end{bmatrix}\]

with \(\alpha\in(0,1)\) and \(\beta\in (0,1)\). First line corresponds to employment, second line to unemployment.

Which is the stationary equilibrium? (choose any value for \(\alpha\) and \(\beta\))

α = 0.3
β = 0.5
γ = 0.2
P = [
    (1-α) α/2 α/2;
    β/2  (1-β) β/2;
    γ/2 γ/2 (1-γ);
]

3×3 Matrix{Float64}:
 0.7   0.15  0.15
 0.25  0.5   0.25
 0.1   0.1   0.8

μ0 = [1.0, 1.0, 1.0]/3
μ0' * (P^10)

1×3 adjoint(::Vector{Float64}) with eltype Float64:
 0.322581  0.193548  0.483871

function solve_steady_state(P; T=100)
    n = size(P,1)
    μ0 = (ones(n)/n)'
    for t in 1:T
        μ1 = μ0*P
        η = maximum(abs, μ1 - μ0)
        if η<1e-10
            return μ1'
        end
        μ0 = μ1
    end
    error("No convergence")
end

solve_steady_state (generic function with 1 method)

solve_steady_state(P)

3-element Vector{Float64}:
 0.3225806452587981
 0.19354838711776282
 0.48387096762343945

# using linear algrebra

using LinearAlgebra: I
# I is the identity operator

M = P' - I

# modify last line of M

M[end,:] .= 1.0
M

# define right hand side
r = zeros(size(M,1))

r[end] = 1.0

M \ r

3-element Vector{Float64}:
 0.32258064516129037
 0.19354838709677416
 0.48387096774193555

M = P' - I

# modify last line of M

M1 = [
    M ;  # concatenate along first dimension
    ones(size(M,1))'  # ' to turn the vector a 1x3 matrix
]
M1

# # define right hand side
r = [zeros(size(M,1)) ; 1]

M1 \ r

3-element Vector{Float64}:
 0.32258064516129054
 0.19354838709677397
 0.4838709677419355

In the long run, what will the the fraction \(p\) of time spent unemployed? (Denote by \(X_m\) the fraction of dates were one is unemployed)

Illustrate this convergence by generating a simulated series of length 10000 starting at \(X_0=1\). Plot \(X_m-p\) against \(m\). (Take \(\alpha=\beta=0.1\)).

Job-Search Model

We want to solve the following model, adapted from McCall.

• When unemployed in date, a job-seeker
  • consumes unemployment benefit \(c_t = \underline{c}\)
  • receives in every date \(t\) a job offer \(w_t\)
    • \(w_t\) is i.i.d.,
    • takes values \(w_1, w_2, w_3\) with probabilities \(p_1, p_2, p_3\)
  • if job-seeker accepts, becomes employed at rate \(w_t\) in the next period
  • else he stays unemployed
• When employed at rate \(w\)
  • worker consumes salary \(c_t = w\)
  • with small probability \(\lambda>0\) looses his job:
    • starts next period unemployed
  • otherwise stays employed at same rate
• Objective: \(\max E_0 \left\{ \sum \beta^t \log(c_t) \right\}\)

What are the states, the controls, the reward of this problem ? Write down the Bellman equation.

Define a parameter structure for the model.

Define a function value_update(V_U::Vector{Float64}, V_E::Vector{Float64}, x::Vector{Bool}, p::Parameters)::Tuple{Vector, Vector}, which takes in value functions tomorrow and a policy vector and return updated values for today.

Define a function policy_eval(x::Vector{Bool}, p::Parameter)::Tuple{Vector, Vector} which takes in a policy vector and returns the value(s) of following this policies forever. You can add relevant arguments to the function.

Define a function bellman_step(V_E::Vector, V_U::Vector, p::Parameters)::Tuple{Vector, Vector, Vector} which returns updated values, together with improved policy rules.

Implement Value Function

Implement Policy Iteration and compare rates of convergence.

Discuss the Effects of the Parameters