“Heckmeck am Bratwurmeck” or How to grill the maximum number of worms Roland C. Seydel 24/05/2012 (1) “Heckmeck am Bratwurmeck” 24/05/2012 1 / 29 Overview 1 Introducing the dice game The basic rules Understanding the strategy 2 Computing the optimal strategy Value function and state space Computing the value function 3 The results (2) “Heckmeck am Bratwurmeck” 24/05/2012 2 / 29 Introducing the dice game Overview 1 Introducing the dice game The basic rules Understanding the strategy 2 Computing the optimal strategy Value function and state space Computing the value function 3 The results (3) “Heckmeck am Bratwurmeck” 24/05/2012 3 / 29 Introducing the dice game The basic rules The rules (I) Goal of the game: Get the most worms by throwing high (spot) numbers. 1 There are eight dice with six sides each, going from 1 to 6, where instead of a 6 a worm is shown. 2 You can throw the dice as often as you want; after each throw you choose a number which you haven’t chosen yet and put aside all the dice showing this number. 3 The numbers of the dice put aside are added, where a worm (6) counts as a 5. No 6 put aside ⇒ 0 points! You can take a piece from the shelf with number ≤ your points, and put it on top of your pile. (4) “Heckmeck am Bratwurmeck” 24/05/2012 4 / 29 Introducing the dice game The basic rules The rules (II) The winner is the player with the most worms in his pile. Several additional rules which matter to us: Not able to put aside a new number? ⇒ 0 points! You can also take a piece from the top of your coplayers’ piles, if you match the number exactly. If your points are not sufficient to pick a piece, you lose the piece on top of your own pile. Rules we ignore for now: Lost pieces are put back on the shelf; the piece with the largest available number is removed from the shelf. Your coplayers can also take pieces from you when it’s their turn! The game is over when the shelf is empty. (5) “Heckmeck am Bratwurmeck” 24/05/2012 5 / 29 Introducing the dice game The basic rules Example game (6) “Heckmeck am Bratwurmeck” 24/05/2012 6 / 29 Introducing the dice game Understanding the strategy Optimal strategy: Easy to tell Is selecting four 1 a good idea? Are two 5 better, or two 6? Are two 4 better, or two 5? If I could lose four worms, should I bet or rather risk nothing? (7) “Heckmeck am Bratwurmeck” 24/05/2012 7 / 29 Introducing the dice game Understanding the strategy Optimal strategy: Difficult to tell Are three 5 better, or two 6? And what about four 5? Could it be optimal to select one or more 3 in the first throw? Should I take the five 5, or is it too dangerous? What is the expected / most likely outcome in terms of worms? Should I stop with three 5 and two 6, or continue? (8) “Heckmeck am Bratwurmeck” 24/05/2012 8 / 29 Introducing the dice game Understanding the strategy Parallels to option pricing Early exercise: The player can decide when to stop and exercise. ; American option Optimal control: At each point in time, a decision has to be taken. ; Swing option Knock out: If there is no 6, or you are not able to put aside a new number, our points are 0. ; Barrier option Conclusion Compute the optimal solution by option pricing methods? (9) “Heckmeck am Bratwurmeck” 24/05/2012 9 / 29 Computing the optimal strategy Overview 1 Introducing the dice game The basic rules Understanding the strategy 2 Computing the optimal strategy Value function and state space Computing the value function 3 The results (10) “Heckmeck am Bratwurmeck” 24/05/2012 10 / 29 Computing the optimal strategy Differences to option pricing Option pricing Continuous state space Continuous time Typically 1d state space Same state space in time Knockout happens at barrier line Random noise is added to process No decision needed Heckmeck Discrete state space Only up to 8 times Up to 8d state space Changing state space No clear line for knockout Random is not added to state Each throw needs a decision11 (control) 00 0011 11 0011 00 00 11 0011 11 0011 11 00 11 11 00 00 11 00 00 11 00 00 11 00 00 00 11 0011 11 0011 11 00 11 00 11 00 11 00 11 00 11 00 11 0011 11 0011 11 00 11 11 00 00 11 00 11 0000 11 0000 11 00 11 11 00 00 11 00 00 00 11 00 11 00 00 00 11 0011 11 0011 11 00 11 00 11 0011 11 0011 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 (11) “Heckmeck am Bratwurmeck” 24/05/2012 11 / 29 Computing the optimal strategy Assumptions 1 The player wants to maximize the expected number of worms in his turn 2 Reactions of other players are not anticipated (single-player optimization) 3 Not the total outcome of the game is optimized, but one single turn of up to 6 throws! (12) “Heckmeck am Bratwurmeck” 24/05/2012 12 / 29 Computing the optimal strategy Value function and state space Value function Situation An array of pieces available on the shelf, on other players’ piles and on your own pile is called a situation. State space A vector x ∈ {1, . . . , 6}t of already selected dice at time t ∈ {0, . . . , 8} is called the state at time t. The state space for time t is the ensemble of all possible selections with x ∈ {1, . . . , 6}t . We consider only sorted states with notation x = {. . .}. Value function For a particular situation, the value function v (x) is the expected number of worms (on pieces), assuming that starting from a state of x the expectation-optimal decisions will be taken. Caveat: From now on, worms are only worms on pieces, not on dice! (13) “Heckmeck am Bratwurmeck” 24/05/2012 13 / 29 Computing the optimal strategy Value function and state space Value function: payoff (Worm) payoff For a particular situation let w (n) be the number of worms you would get or lose for a sum of points n. Then the (worm) payoff p(x) of a state x is defined by the number of worms you would get or lose upon termination. In formulas, ! (P i min(xi , 5) 6 ∈ x . p(x) = w (n(x)) = w 0 else Each situation has a worst payoff w (0), equal to the negative number of worms on top of your pile. Examples: p({5, 5, 5, 5, 5, 5, 5, 5}) = w (0) ←− {. . .} sorted vector! p({1, 2, 3, 4, 5, 6, 6, 6}) = 3 p({1, 5, 5, 5, 6}) = 1 (14) “Heckmeck am Bratwurmeck” 24/05/2012 14 / 29 Computing the optimal strategy Value function and state space Value function on intermediate states Intermediate state For a state x ∈ {1, . . . , 6}t and a throw y ∈ {1, . . . , 6}8−t , the tuple (x, y ) is called intermediate state, i.e., a state that still needs a decision. We can also define the value function of intermediate states (x, y ): Either there is a valid best choice y˜ ⊂ y (in particular y˜ ∩ x = ∅) such that v ((x, y )) = v ({x, y˜ }), or there is no valid choice, in which case v ((x, y )) = w (0) (worst payoff). Conclusion: It is sufficient to compute only the value function on normal states! (15) “Heckmeck am Bratwurmeck” 24/05/2012 15 / 29 Computing the optimal strategy Value function and state space Value function: optimal exercise For states in {1, . . . , 6}8 , the value function is equal to the payoff (assuming full shelf), e.g.: v ({5, 5, 5, 5, 5, 5, 5, 5}) = p({5, 5, 5, 5, 5, 5, 5, 5}) = w (0) v ({1, 2, 3, 4, 5, 6, 6, 6}) = p({1, 2, 3, 4, 5, 6, 6, 6}) = 3 We call this the terminal condition. There are other types of states for which the value function is determined by the payoff function: Optimal exercise It is optimal to exercise in a state x ∈ {1, . . . , 6}t if v (x) = p(x), i.e., the value function equals the payoff function. In this case the expected optimal number of worms can be obtained immediately. Yet we do not know the value function yet . . . (16) “Heckmeck am Bratwurmeck” 24/05/2012 16 / 29 Computing the optimal strategy Value function and state space State space in “time” t: example Throw dice (random) Decide x = {4, 5, 5, 5} x = {3, 3, 5, 5, 5} y = {1, 3, 3, 4, 5} x = {1, 5, 5, 5} x = {5, 5, 5, 6, 6} x = {5, 5, 5} y = {5, 5, 5, 6, 6} Exercise? y = {5, 5, 5, 5, 5} Knock out! t=3 (17) t=4 “Heckmeck am Bratwurmeck” t=5 24/05/2012 17 / 29 Computing the optimal strategy Computing the value function Finding the value function: Possible approaches Monte Carlo simulation? This is the approach implicitly chosen by experienced players: Simulate dice throws on the computer Try different strategies in different (simulated) games. Problem: Causality difficult to establish because of multitude of possible strategies ⇒ Need many simulations! Use recursion programming principle? Solution fastest to implement: Uses ( v (x) = max {p(x), E[maxy ⊂Y v ((x, y ))]} p(x) |x| < 8 |x| = 8 (1) where Y follows a multinominal (dice) distribution One command v ({}) starts the whole calculation recursively Problem: Takes an eternity, because most states are computed multiple times Backward induction! Start at terminal time and compute p(x), then go backwards in time using (1). Compute each state only once (18) “Heckmeck am Bratwurmeck” 24/05/2012 18 / 29 Computing the optimal strategy Computing the value function Needed: The multinomial distribution Q: If you toss a coin 5 times, what is the probability of getting 5 − 4 heads and 4 tails? 5! 5−4 (1 − 0.5)4 A: Binomial distribution! (5−4)! 4! 0.5 Multinomial (dice) distribution If you throw a k-dimensional dice n times, then the probability of getting xi times the spot number i for i = 1, . . . , k is fM (x) = where Pk i=1 pi (19) = 1 and Pn n! p x1 · . . . · pkxk x1 ! · . . . · xk ! 1 i=1 xi = n. “Heckmeck am Bratwurmeck” 24/05/2012 19 / 29 Computing the optimal strategy Computing the value function Backward induction The backward induction makes the implicit equation (1) explicit by making sure that the values on the right hand side are already computed: Algorithm: Backward induction 1 2 At time t = 8, determine terminal payoff p(x) for all possible states x ∈ {1, . . . , 6}8 Go backwards in time t → t − 1, for each t-state x do: 1 2 3 4 Compute distribution of possible dice scenarios Check for each scenario whether knocked out (no new spot numbers), or else to which future state the scenario could lead Take the expectation EY [maxy ⊂Y v ((x, y ))] with v from future times Take the maximum with p(x) (exercise instead of throwing dice) MATLAB implementation runs in just a few seconds on a normal PC! (20) “Heckmeck am Bratwurmeck” 24/05/2012 20 / 29 The results Overview 1 Introducing the dice game The basic rules Understanding the strategy 2 Computing the optimal strategy Value function and state space Computing the value function 3 The results (21) “Heckmeck am Bratwurmeck” 24/05/2012 21 / 29 The results Contour lines of value function, full shelf, initial throw Heckmeck initial throw 7 1.5 2 0.5 4 0.5 3 8 3.5 2.5 1 1 1. 0. 5 5 5 2 3 4 2.5 3 0.5 2 1.5 1 #dice selected 6 2 1.5 1 1 1 1.5 2 2.5 3 3.5 4 spot number 4.5 5 5.5 6 Figure: Contour lines of value function in initial throw, if shelf starts at 21; dependent on the number of selected dice (vertical axis) of a particular spot (22) “Heckmeck am Bratwurmeck” 24/05/2012 22 / 29 The results Distribution of points Dice sum distribution for shelf starting at 30 0.5 0.45 0.45 0.4 0.4 0.35 0.35 0.3 Probability Probability Dice sum distribution for full shelf 0.5 0.25 0.3 0.25 0.2 0.2 0.15 0.15 0.1 0.1 0.05 0.05 0 0 5 10 15 20 25 Dice sum 30 35 40 0 0 5 10 15 20 25 Dice sum 30 35 40 Figure: Distribution of dice sum under optimal strategy, from a Monte Carlo simulation with 1000 paths. Left: Full shelf, right: shelf starting at 30 Explanation: Left almost no values in [5, 20] because for these dice sums knockout is very likely. (23) “Heckmeck am Bratwurmeck” 24/05/2012 23 / 29 The results Most likely selections What is the most likely initial selection of dice, assuming we act optimally? Answer: two 6 Number of dice Spots 1 2 3 4 5 6 1 0.0003 0.0019 0.0113 0.0724 0.0037 0.0150 2 0.0000 0.0000 0.0507 0.1013 0.1666 0.2347 3 0.0000 0.0000 0.0009 0.0531 0.0982 0.1035 4 0.0000 0.0000 0.0003 0.0206 0.0260 0.0260 5 0.0000 0.0000 0.0001 0.0037 0.0042 0.0042 6 0.0000 0.0000 0.0000 0.0004 0.0004 0.0004 7 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0 0.0 0.0 0.0 0.0 0.0 Table: Probability that a number of dice (horizontal axis) carrying a certain number of spots (vertical axis) is selected under the optimal strategy. (24) “Heckmeck am Bratwurmeck” 24/05/2012 24 / 29 The results Optimal exercise decisions: Example Original question: Stop with three 5 and two 6? (25) “Heckmeck am Bratwurmeck” 24/05/2012 25 / 29 The results Optimal exercise decisions: Example Original question: Stop with three 5 and two 6? >> [v,A,A_inv] = heckmeck_v4(21:36, [], []); >> heckmeck_value([5,5,5,6,6],A_inv,v) 2.6277 (25) “Heckmeck am Bratwurmeck” 24/05/2012 25 / 29 The results Optimal exercise decisions: Example Original question: Stop with three 5 and two 6? >> [v,A,A_inv] = heckmeck_v4(21:36, [], []); >> heckmeck_value([5,5,5,6,6],A_inv,v) 2.6277 >> [v,A,A_inv] = heckmeck_v4([21:31,33:36], [], [32]); >> heckmeck_value([5,5,5,6,6],A_inv,v) 2.4352 (25) “Heckmeck am Bratwurmeck” 24/05/2012 25 / 29 The results Optimal exercise decisions: Example Original question: Stop with three 5 and two 6? >> [v,A,A_inv] = heckmeck_v4(21:36, [], []); >> heckmeck_value([5,5,5,6,6],A_inv,v) 2.6277 >> [v,A,A_inv] = heckmeck_v4([21:31,33:36], [], [32]); >> heckmeck_value([5,5,5,6,6],A_inv,v) 2.4352 >> [v,A,A_inv] = heckmeck_v4([24,33,34], [21, 25,29], [32]); >> heckmeck_value([5,5,5,6,6],A_inv,v) 2 (25) “Heckmeck am Bratwurmeck” 24/05/2012 25 / 29 The results Optimal strategy in Monte Carlo simulation MC simulation vs. backward induction for initial throw, full shelf 1.9 Mean of MC sim Value function (backward induction) Value fct +/− stdev 1.85 Number of worms 1.8 1.75 1.7 1.65 1.6 1.55 1.5 0 1000 2000 3000 4000 5000 6000 Number of paths 7000 8000 9000 10000 Figure: Convergence of mean of Monte Carlo simulation depending on number of paths (blue line) in initial throw, if shelf starts at 21. Green: backward induction (26) “Heckmeck am Bratwurmeck” 24/05/2012 26 / 29 The results Optimal strategy in Monte Carlo simulation (2) In the whole game, what is our probability of winning if others pursue “average” strategies? Test: Optimal strategy vs. fuzzy strategy We test our results in a two-player game: Player 1 follows the optimal strategy (derived from the value function v) Player 2 derives his strategy from a “value function” misestimated by ±0.1 worms (by randomly perturbing v with stdev of 0.1) Result: Player 1 wins 17 out of 20 games! Conclusion: Even a slight difference in optimality makes us win most of the games (law of large numbers because of many rounds per game!) (27) “Heckmeck am Bratwurmeck” 24/05/2012 27 / 29 The results Extensions / References Possible extensions: Incorporate risk in pricing? → Utility functions Compute optimal strategies for the whole game Reference: Reiner Knizia: Heckmeck am Bratwurmeck (Pickomino), Zoch-Verlag 2005 (28) “Heckmeck am Bratwurmeck” 24/05/2012 28 / 29 Value function dependent on shelf, initial throw Value depending on shelf 3.5 die=4 #selected=1 die=4 #selected=2 die=4 #selected=3 die=4 #selected=4 die=4 #selected=5 die=5 #selected=1 die=5 #selected=2 die=5 #selected=3 die=5 #selected=4 die=5 #selected=5 die=6 #selected=1 die=6 #selected=2 die=6 #selected=3 die=6 #selected=4 die=6 #selected=5 Expected optimal worms 3 2.5 2 1.5 1 0.5 0 20 22 24 26 28 30 Shelf starts at... 32 34 36 Figure: Expected number of worms under optimal strategy for different initial dice choices, dependent on the minimal number available on the shelf. (29) “Heckmeck am Bratwurmeck” 24/05/2012 29 / 29

© Copyright 2020