These are the answers for Assignment 4, the information theory problems. --------------------------------------------------------------------- First problem: 1. H, T 2. h = -( 2/3 log 2/3 + 1/3 log 1/3) 3. sym code H 0 T 1 4. sym prob code TT 1/9 111 TH 2/9 110 HT 2/9 10 HH 4/9 0 5. Yes, you could have lots of tails 6. Assume 18 flips (or 9 of the "big" experiments). Then you'd expect 1 TT for 3 bit, 2 TH for 6 bits, 2 HT for 4 bits, and 4 HH for 4 bits. That's a total of 17 bits, vs 18 bits for the naive encoding. ---------------------------------------------------------------------- Second problem: 1. 1, 2, 3, 4, 5, 6 2. h = -( 2/7 log 1/7 + 1/7 log 1/14 + 4/7 log 2/7 ) 3. 1: 000 2: 001 3: 010 4: 011 5: 100 6: 101 4. sym prob code 5 2/7 0 6 2/7 10 1 1/7 110 2 1/7 1110 3 1/14 11110 4 1/14 11111 5. Yes, could have many 3's and 4's 6. Assume 14 rolls. Then you'd expect 2 1's for 6 bits, 2 2's for 8 bits, 1 3 for 5 bits, 1 4 for 5 bits, 4 5's for 4 bits, and 4 6's for 8 bits. This gives a total of 36 bits, vs 42 bits for the naive encoding. ---------------------------------------------------------------------- Third Question: 1. SH, SC, SJ, UH, UC, UJ (each being one symbol) 2. for sweetness: prob S is 2/3 and prob U is 1/3 for temp: prob H is 1/4, prob C is 1/4, prob J is 1/2 since sweetness and temp are independent, each symbol is just the product of its component probabilities SH 1/6 SC 1/6 SJ 1/3 UH 1/12 UC 1/12 UJ 1/6 h = -( 1/2 log 1/6 + 1/3 log 1/3 + 1/6 log 1/12 ) 3. SH: 000 SC: 001 SJ: 010 UH: 011 UC: 100 UJ: 101 4. sym prob code SJ 1/3 0 SH 1/6 10 SC 1/6 110 UJ 1/6 1110 UH 1/12 11110 UC 1/12 11111 5. Yes, could get many UH and UC's. 6. Assume 12 individuals were polled at random. You'd expect 4 SJ for 4 bits, 2 SH for 4 bits, 2 SC for 6 bits, 2 UJ for 8 bits, 1 UH for 5 bits, and 1 UC for 5 bits. This gives a total of 32 bits, versus 36 bits for the naive encoding.