1

236601 - Coding and Algorithms for

MemoriesLecture 5

2

Write-Once Memories (WOM)• Introduced by Rivest and Shamir, “How to

reuse a write-once memory”, 1982• The memory elements represent

bits (2 levels) and are irreversibly programmed from ‘0’ to ‘1’

Q: How many cells are required to write 100 bits twice? P1: Is it possible to do better…?P2: How many cells to write k bits twice?P3: How many cells to write k bits t times?P3’: What is the total number of bits that is possible to write in n cells in t writes?

1st Write

2nd Write

3

Definition: WOM Codes • Definition: An [n,t;M1,…,Mt] t-write WOM code is a

coding scheme which consists of n cells and guarantees any t writes of alphabet size M1,…,Mt by programming cells from zero to one– A WOM code consists of t encoding and decoding maps Ei, Di, 1

≤i≤ t– E1: {1,…,M1} {0,1}n

– For 2 ≤i≤ t, Ei: {1,…,Mi}×{0,1}n {0,1}n

such that for all (m,c)∊{1,…,Mi}×{0,1}n, Ei(m,c) ≥ c– For 1 ≤i≤ t, Di: {0,1}n {1,…,Mi}

such that for Di(Ei(m,c)) =m for all (m,c)∊{1,…,Mi}×{0,1}n

• The sum-rate of the WOM code is R = (Σ1

t logMi)/n Rivest Shamir: [3,2;4,4], R = (log4+log4)/3=4/3

4

The Capacity of WOM Codes• The Capacity Region for two writes

C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}h(p) – the entropy function h(p) = -plog(p)-(1-p)log(1-p)– p – the prob to program a cell on the 1st write, thus R1 ≤

h(p)– After the first write, 1-p out of the cells aren’t

programmed, thus R2 ≤ 1-p• The maximum achievable sum-rate is

maxp∊[0,0.5]{h(p)+(1-p)} = log3 achieved for p=1/3:

R1 = h(1/3) = log(3)-2/3R2 = 1-1/3 = 2/3

5

The Capacity of WOM Codes• The Capacity Region for two writes

C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}h(p) – the entropy function h(p) = -plog(p)-(1-p)log(1-p)

• The Capacity Region for t writes:Ct-WOM={(R1,…,Rt)| ∃p1,p2,…pt-1∊[0,0.5], R1 ≤ h(p1), R2 ≤ (1–p1)h(p2),…, Rt-1≤ (1–p1)(1–pt–2)h(pt–1) Rt ≤ (1–p1)(1–pt–2)(1–pt–1)}• p1 - prob to prog. a cell on the 1st write: R1 ≤ h(p)• p2 - prob to prog. a cell on the 2nd write (from the remainder): R2≤(1-p1)h(p2)• pt-1 - prob to prog. a cell on the (t-1)th write (from the remainder):

Rt-1 ≤ (1–p1)(1–pt–2)h(pt–1)• Rt ≤ (1–p1)(1–pt–2)(1–pt–1) because (1–p1)(1–pt–2)(1–pt–1) cells weren’t programmed

• The maximum achievable sum-rate is log(t+1)

6

James Saxe’s WOM Code • [n,n/2-1; n/2,n/2-1,n/2-2,…,2] WOM Code

– Partition the memory into two parts of n/2 cells each– First write:

• input symbol m∊{1,…,n/2} • program the ith cell of the 1st group

– The ith write, i≥2:• input symbol m∊{1,…,n/2-i+1} • copy the first group to the second group• program the ith available cell in the 1st group

– Decoding:• There is always one cell that is programmed in the 1st and not in the

2nd group• Its location, among the non-programmed cells, is the message value

– Sum-rate: (log(n/2)+log(n/2-1)+ … +log2)/n=log((n/2)!)/n≈ (n/2log(n/2))/n ≈ (log n)/2

7

The Coset Coding Scheme• Cohen, Godlewski, and Merkx ‘86 – The coset coding

scheme– Use Error Correcting Codes (ECC) in order to construct WOM-

codes– Let C[n,n-r] be an ECC with parity check matrix H of size r×n– Write r bits: Given a syndrome s of r bits, find a length-n vector

e such that H⋅eT = s– Use ECC’s that guarantee on successive writes to find vectors

that do not overlap with the previously programmed cells– The goal is to find a vector e of minimum weight such that only

0s flip to 1s

8

The Coset Coding Scheme• C[n,n-r] is an ECC with an r×n parity check matrix H• Write r bits: Given a syndrome s of r bits, find a

length-n vector e such that H⋅eT = s• Example: H is the parity check matrix of a Hamming

code– s=100, v1 = 0000100: c = 0000100 – s=000, v2 = 1001000: c = 1001100 – s=111, v3 = 0100010: c = 1101110– s=010, … can’t write!

• This matrix gives a [7,3:8,8,8] WOM code

9

Binary Two-Write WOM-Codes• First Write: program only vectors v such that rank(Hv)

= r VC = { v ∊ {0,1}n | rank(Hv) = r}– For H we get |VC| = 92 - we can write 92 messages– Assume we write v1 = 0 1 0 1 1 0 0

v1 = (0 1 0 1 1 0 0)

10

Binary Two-Write WOM-Codes• First Write: program only vectors v such that rank(Hv)

= r, VC = { v ∊ {0,1}n | rank(Hv) = r}• Second Write Encoding:

• Second Write Decoding: Multiply the received word by H:H⋅(v1 + v2) = H⋅v1 + H⋅v2 = s1+ (s1 + s2) = s2

v1 = (0 1 0 1 1 0 0)

1. Write a vector s2 of r bits2. Calculate s1 = H⋅v13. Find v2 such that Hv1⋅v2 = s1+s2

a

4. v2 exists since rank(Hv1) = ra

5. Write v1+v2 to memory

1. s2 = 0012. s1 = H⋅v1 = 0103. Hv1⋅v2 = s1+s2 = 011

a

4. v2 = 0 0 0 0 0 1 1 5. v1+v2 = 0 1 0 1 1 1 1

11

Sum-rate Results• The construction works for any linear code C• For any C[n,n-r] with parity check matrix H,

VC = { v ∊ {0,1}n | rank(Hv) = r}• The rate of the first write is:

R1(C) = (log2|VC|)/n• The rate of the second write is: R2(C) = r/n• Thus, the sum-rate is: R(C) = (log2|VC| + r)/n• In the last example:

– R1= log(92)/7=6.52/7=0.93, R2=3/7=0.42, R=1.35• Goal: Choose a code C with parity check matrix

H that maximizes the sum-rate

12

Capacity Achieving Results• The Capacity region

C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}

• Theorem: For any (R1, R2)∊C2-WOM and ε>0, there exists a linear code C satisfying

R1(C) ≥ R1-ε and R2(C) ≥ R2–ε• By computer search– Best unrestricted sum-rate 1.4928 (upper bound

1.58)– Best fixed sum-rate 1.4546 (upper bound 1.54)

13

Capacity Region and Achievable Rates of Two-Write WOM codes

14

Non-Binary WOM Codes • Definition: An [n,t; M1,…,Mt]q t-write WOM

code is a coding scheme that consists of n q-ary cells and guarantees any t writes of alphabet size M1,…,Mt only by increasing the cell levels

• The sum-rate of the WOM-code is R = (Σi=1

tlogMi)/n

15

WOM Capacity• The capacity of non-binary WOM-codes was given by

Fu and Han Vinck, ‘99

• The maximal sum-rate using t writes and q-ary cells is

• There is no tight upper bound on the sum-rate in case equal amount of information is written on each write

11

logqqt

C

16

Basic Constructions• Construction 1: Assume t=q–1 and there are n q-

ary cells– On each write, n binary bits are written– Write to the cells level by level– The sum-rate is n(q–1)/n = q-1 (upper bound ≈ 2(q-1))

• Construction 2: Assume t=2 and q is odd– First write: use only levels 0,1,…,(q-1)/2– Second write: use only levels (q-1)/2,…,q-1– The sum-rate is 2log((q+1)/2)– The difference between the upper bound is

11

2log2)1(log2

2)1(log

qqqqq

17

Construction 3• Assume q is a power of 2, q=2m

• Every cell is converted into m bits• For 1 ≤ i ≤ m, the ith bits from each cell

comprise a t-write binary WOM code• The m WOM codes write into their

corresponding bits in the cells independently

• Since every bit can only change from 0 to 1, the level of each cell cannot decrease– Use the binary expansion ϕ:{0,1}m→{0,…,2m-

1}x=(x0,…,xm-1), ϕ(x) = ∑j 2m-j xj

18

Example – Construction 3

• Let q = 8 = 23 and n = 3• Every cell corresponds to 3 bits• Use Rivest-Shamir WOM-code• The ith bits from each cell

comprise a WOM-code• The sum-rate is 6∙2/3 = 4 (upper bound is

5.17)

Write Number Data bits Encoding by the RS base-code

Encoded values in the 8-ary cells

1 (11,01,10) (100,001,010) (4,1,2)

2 (00,11,01) (101,111,110) (7,3,6)

19

Construction 3• Theorem: If q=2m and there exists a t-write

binary WOM-code of sum-rate R, then there exists a t-write q-ary WOM-code of sum-rate mR

20

Binary Two-Write WOM-Codes• First Write: program only vectors v such that rank(Hv)

= r, VC = { v ∊ {0,1}n | rank(Hv) = r}• Second Write Encoding:

• Second Write Decoding: Multiply the received word by H:H⋅(v1 + v2) = H⋅v1 + H⋅v2 = s1+ (s1 + s2) = s2

v1 = (0 1 0 1 1 0 0)

1. Write a vector s2 of r bits2. Calculate s1 = H⋅v13. Find v2 such that Hv1⋅v2 = s1+s2

a

4. v2 exists since rank(Hv1) = ra

5. Write v1+v2 to memory

1. s2 = 0012. s1 = H⋅v1 = 0103. Hv1⋅v2 = s1+s2 = 011

a

4. v2 = 0 0 0 0 0 1 1 5. v1+v2 = 0 1 0 1 1 1 1

21

Non-Binary Two-Write WOM-Codes

• Let C[n,n-r] be a linear code w/ p.c.m H of size r×n over GF(q)

• For a vector v ∊ GF(q)n, let Hv be the matrix H with 0’s in the columns that correspond to the positions of the nonzeros’s in v

• First Write: program only vectors v such that rank(Hv)=r, VC = { v ∊ GF(q)n | rank(Hv) = r}

• Second Write Encoding: Write a vector s2 of r symbols over GF(q)– Let v1 be the programmed vector on the first write, and let s1 = H⋅v1– Remember that rank(Hv1) = r

a

– Find v2 such that Hv1⋅v2 = -s1+s2F

– A solution v2 exists since rank(Hv1) = ra

– Write v1+v2 to memory• Second Write Decoding: Multiply the received word by H:

H⋅(v1 + v2) = H⋅v1 + H⋅v2 = s1+ (-s1 + s2) = s2• This construction works for any linear code over GF(q)

22

Code Rate• The construction works for any linear code C over GF(q)• For any C[n,n-r] with parity check matrix H,

VC = { v ∊ GF(q)n | rank(Hv) = r}• The rate of the first write is: R1(C) = (log2|VC|)/n• The rate of the second write is: R2(C) = r⋅log2q/n• Thus, the total rate is: R(C) = (log2|VC| + r⋅log2q)/n• By choosing the parity check matrix H uniformly at random, it is

possible to achieve the following capacity region: {(R1,R2)|∃p∊[0,1-1/q], R1≤h(p)+p⋅log2(q-1), R2≤(1-

p)⋅log2q}– The maximum achievable rate log2(2q-1) is achieved for p=(q-1)/(2q-1)

• Remarks: – This is not an optimal non-binary two-write WOM code construction– Cells cannot be programmed twice– However, these codes are useful to construct multiple-write WOM codes

23

Three-Write WOM Codes• Let C3 be an [n,2; 2nR1,2nR2] two-write WOM code over GF(3)• Construct a binary [2n,3; 2nR1,2nR2,2n] 3-write WOM code as follows:

– The code has 2n cells, which are broken into n 2-cell blocks– First write: use the first write of the code C3

• Write a message m1 from {1,…, 2nR1}• This message is supposed to be written as a ternary vector v1 ∊ GF(3)n

• Write the vector v1 into the n 2-cell blocks, using the mapping Φ :GF(3)→ (GF(2),GF(2)), where Φ(0)=(0,0), Φ(1)=(1,0), Φ(2)=(0,1)

– Second write: use the second write of the code C3• Write a message m2 from {1,…, 2nR2}• The ternary vector v2 is written into the n 2-cell blocks using the mapping Φ• Each 2-cell block is written at most once and at most one cell is programmed

C3 Encoder

m1∊{1,…, 2nR1}

v1∊GF(3)n

Φ u1∊{0,1}2n

C3 Encoder

m2∊{1,…, 2nR2}

v2∊GF(3)n

Φ u2∊{0,1}2n

24

Three-Write WOM Codes• Let C3 be an [n,2; 2nR1,2nR2] two-write WOM code over GF(3)• Construct a binary [2n,3; 2nR1,2nR2,2n] 3-write WOM code as follows:

– The code has 2n cells, which are broken into n 2-cell blocks– First write: use the first write of the code C3

• Write a message m1 from {1,…, 2nR1}• This message is supposed to be written as a ternary vector v1 ∊ GF(3)n

• Write the vector v1 into the n 2-cell blocks, using the mapping Φ :GF(3)→ (GF(2),GF(2)), where Φ(0)=(0,0), Φ(1)=(1,0), Φ(2)=(0,1)

– Second write: use the second write of the code C3• Write a message m2 from {1,…, 2nR2}• The ternary vector v2 is written into the n 2-cell blocks using the mapping Φ• Each 2-cell block is written at most once and at most one cell is

programmed– Third Write: write n bits

• Each bit is stored in a 2-cell block: the bit value is 1 iff both cells are programmed

• It is possible to write a bit in each block since at most one cell in each block was programmed

25

Three-Write WOM Code - Example

• Ex: C3 is a 2-write WOM code over GF(3) with n=7 cells• First Write

– Write a vector v1 over GF(3) according to the first write of C3

– Assume the vector is v1 = 2 0 1 0 2 1 0– Using the mapping Φ, v1 is written into the 2n = 14 binary cells

2 0 1 0 2 1 0 →Φ 01.00.10.00.01.10.00• Second Write

– Write a vector v2 over GF(3) according to the second write of C3

– Assume the vector is v2 = 2 1 1 0 2 1 2– Using the mapping Φ, v2 is written into the 2n = 14 binary cells

2 1 1 0 2 1 2 →Φ 01.10.10.00.01.10.01• Third Write: Write 7 bits

– Assume the message is v3 = 0 1 1 0 1 0 1 , then write to memory01.11.11.00.11.10.11

26

Code Rate• Theorem: Let C3 be an [n,2; 2nR1,2nR2] 2-write WOM

code over GF(3) of sum-rate R1+R2, then there exists a binary [2n,3; 2nR1,2nR2,2n] 3-write WOM code of sum-rate (R1+R2+1)/2

• The best sum-rate for the code C3 is log5 = 2.32• Conclusion: It is possible to find a binary 3-write

WOM code of sum-rate (log5 +1)/2 = 1.66• Using a computer search, the best sum-rate found

for WOM codes over GF(3) is 2.22 and thus there exists a 3-write WOM code of sum-rate 1.61

27

Four-Write WOM Codes• Let C3 be an [n,2; 2nR3,1,2nR3,2] two-write WOM code over GF(3) and

let C2 be an [n,2; 2nR2,1,2nR2,2] binary two-write WOM code• Construct a [2n,4; 2nR1,2nR2, 2nR2,1,2nR2,2] 4-write WOM code as

follows:– The code has 2n cells, which are broken into n 2-cell blocks– First write: use the first write of the code C3

• Write a message from {1,…,2nR3,1}• This message is supposed to be written as a ternary vector v ∊ GF(3)n

• Write the vector v into the n 2-cell blocks, using the mapping Φ :GF(3)→ (GF(2),GF(2)), where Φ(0)=(0,0), Φ(1)=(1,0), Φ(2)=(0,1)

– Second write: use the second write of the code C3• Write a message from {1,…,2nR3,2}• The ternary vector is written into the n 2-cell blocks using the mapping Φ• Note that each 2-cell block is written only once

– Third and fourth writes: use the first and second writes of C2• Each bit is stored in a 2-cell block: the bit value is 1 iff both cells are

programmed• It is possible to write a bit in each block since at most one cell in each

block was programmed

28

Code Rate• Theorem: Let C3 be an [n,2; 2nR3,1,2nR3,2] 2-write WOM

code over GF(3) (found before) of sum-rate R1+R2, and let C2 be an [n,2; 2nR2,1,2nR2,2] binary two-write WOM code then there exists a binary [2n,4; 2nR3,1,2nR3,2,2nR2,1,2nR2,2] 4-write WOM code of sum-rate (R3,1+R3,2+R2,1+R2,2)/2a

• The best sum-rate for the code C3 is log5 = 2.32 and for the code C2 is log3 = 1.58

• Conclusion: It is possible to find a binary 3-write WOM code of sum-rate (log5 + log3)/2 = 1.95

• Using a computer search, the best sum-rate found for codes over GF(3) is 2.22 and for codes over GF(2) is 1.49. Thus there exists a 4-write WOM code of sum-rate 1.855

29

Five-Write WOM-Codes• Use a 2-write WOM code C3 over GF(3) of sum-rates R3,1, R3,2

and a binary 3-write WOM code C2 of sum-rates R2,1, R2,2, R2,3a

• The first and second writes are implemented as before using the first and second writes of the code C3

• The third, fourth, and fifth writes are also implemented as before by using the first, second, and third writes of C2

• It is possible to achieve a five-write WOM code of rate (log5+(log5+1)/2)/2=1.99 and there exists a code of rate 1.915

• For six writes, we will change the code C2 to be a 4-write WOM code such that it is possible to achieve WOM-rate (log5+1.95)/2 = 2.14 and there exists a code of rate 2.0375