Upload
sinshaw-bekele
View
17
Download
0
Tags:
Embed Size (px)
DESCRIPTION
capacity
Citation preview
NFORMATION AND COMPUTATlON 86, 1-13 ( 1990)
On Numerical Methods of Calculating the Capacity of Continuous-Input Discrete-Output
Memoryless Channels
CHEIN-I CHANC AND SIMON C. FAN
Department of Electrical Engineering, Uniaersiry of Maryland, Baltimore Counfy Cumpus,
Baltimore Mawland 21228 3
AND
LEE D. DAVISSON
Electrical Engineering Department. Unioersiry of Maryland.
College Park, Maryland 20742
A computational scheme for calculating the capacity of continuous-input discrete-output memoryless channels is presented. By adopting relative entropy as a performance measure between two channel transition probabilities the method suggests an algorithm to discretize continuous channel inputs into a set of finite desired channel inputs so that the discrete version of the well-known Arimoto- Blahut algorithm is readily applied. Compared to recent algorithms developed by Chang and Davisson, the algorithm has a simple structure for numerical implemen- tations. To support this justification a numerical example is studied and the relative performance is compared based on computing time. ? I990 Academx Press. Inc
I. INTRODUCTION
The problem of computation of channel capacity for discrete memoryless channels was solved by Arimoto (1972) and Blahut (1972). Although the algorithm developed by them indeed offers a very efficient computational method, a continuous version of this algorithm is not practical in the sense that at each iteration cycle it needs to repeatedly compute integrals over the entire channel input space which is generally uncountable. In order to circumvent the difficulty of evaluation of integrals, Chang and Davisson (1988) devised two algorithms (to be called Algorithm I and Algorithm II) for discretization of channel inputs so that the elegant Arimoto-Blahut algorithm is readily applied. The idea is to use a succes-
0890-5401/90 $3.00 Copyright (’ 1990 by Academx Press. Inc
All rights of reproducuon in any form reserved
2 CHANG, FAN, AND DAVISON
sion of finite approximations to achieve the channel capacity within any desired accuracy. The technique involved in their algorithms is to find a set of local maxima for a nonlinear function which usually requires a large amount of computing time,
In this paper, we further propose a simple algorithm which is also a dis- cretization algorithm but has a much simpler structure than Chang and Davisson’s algorithms. In particular, the suggested algorithm does not necessarily search for local maxima; instead, it partitions the channel input space into a finite class of subspaces according to the criterion of relative entropy whose concept was widely used in source coding. Since the channel capacity is calculated on the basis of mutual information, the relative entropy is adopted for measuring how close the mutual information conveyed by two channel transition probabilities are. If channel transition probabilities yield nearly the same mutual information, we group them into a class and select one of the members of this class to be a repre- sentative for channel capacity computation. With these candidates chosen from each of such classes, a continuous channel input space can be discretized into a finite set of channel inputs each of which represents one class whose members have very close mutual information and so, a new discrete memoryless channel is introduced to be a test channel to approximate the original channel. Obviously, the more relined the groups, the more accurate the approximation.
The proposed algorithm (to be called Algorithm III) involves a two- stage implementation which requires generating an adequate discrete test channel by means of a sequence of discretization procedures; it then utilizes the Arimoto-Blahut algorithm to find an approximation to the original channel capacity. The dicretization process is devised based on a slight modification of a lemma proven for noiseless universal source codes in Davisson et al. (1981).
This approach is very similar to a quantization technique which was developed by Finamore and Pearlman (1980) for discretization of a con- tinuous memoryless source to calculate the rate-distortion function with the Blahut rate-distortion function algorithm. The main difference is that the distortion measure is not relative entropy which results in different approaches.
The paper is organized as follows. In Section II the lemma (i.e., Lemma 1) derived in Davisson et al. (1981) is modified and reproven for a memoryless channel. In the following section, a simple procedure to discretize continuous channel inputs is presented. By coupling the Arimoto-Blahut algorithm, the capacity of a continuous-input discrete- output memoryless channel can be calculated and approximated to any desired accuracy. To compare the relative performance of Chang and Davisson’s algorithms and Algorithm 111 a numerical example is studied. It
THECAPACITYOFMEMORYLESSCHANNELS 3
is shown that in general, Algorithm III does not perform as well as Chang and Davisson’s algorithms (Algorithms I and II); however, the payoff is its easy computer implementation. More importantly, numerical results also show that when the number of channel outputs is large, Algorithms I, II, and III achieve nearly the same performance. In this case, computing time becomes a significant issue for numerical computations; thus it will be a chief advantage of Algorithm III and can make Algorithm III more attrac- tive than Algorithms I and II.
II. PRELIMINARY
In this section we interpret Lemma 1 in Davisson et af. ( 1981) and prove it for a memoryless channel.
Suppose that a continuous-input discrete-output memoryless channel is specified by input space X, output space Y, = {v,, . . . . ~~1, and channel transition probabilities {P( yr. 1 x} rE .y,Jk E y,w. Let p(y), q(y) be two probability vectors defined on the channel output space, Y,. The relative entropy between p(.r) and q(y) is defined by
ff(p(!lh q(y))= f P(?‘k)log- P(Yk)
k= 1 d?‘k)’
Then Lemma 1 in Davisson et al. ( 1981) can be modified and proven for a memoryless channel described above as follows.
THEOREM 1. Given a memoryless channel with input space X, output space Y, , channel transition probabilities ( P( yk / x) ) .‘; E ,y. ?* t y,b,, and an arbitrarily small number E > 0, there exist a finite positive integer J, a finite set of probability vectors on Y,, , F = { qJ,( y) } ,!= , , and a corresponding finite partition of the input space X, (S, ) := , such that
log Jd -M[log S], @- 1
where 6 = - j42e"3
s,= (-~EXIH(P(yl.~Y),ql,(o)<&}, and
x= i, s,. .j= 1
(1)
Note that the probabilit~~ vectors q.,, t do not have to be the channel transition probabilities.
4 CHANG, FAN, AND DAVISSON
Proof. Let Q( Y,) be the set of all probability vectors defined on Y, and I/ (1 o; be the uniform norm defined by
II Pll, =-y& I P(Y)l.
Let m be a positive number which will be specified later on, and
6= l W(m + 1)’ (2)
Let Qs be the set of all probability vectors q(y) defined on Y, such that for any y in Y,, q(y) = it3 for some integer i>O. Then according to the definition of Q6, the set Q, is finite. If we let J be the cardinality of Qd, it is easy to show that log J< -M[log S] and for each p(y) in Q( Y,) there is at least one q(y) in Q, so that II p - q 11 E 6 (A4 - 1) 6.
For any channel transition probability P(y / x) given with x E X, we want to construct a probability vector q:(y) defined on Y, such that
H(P(Y I XI, q:LJ4 <EL
Assuming that P(y, 1 x) B P(y, I x) > . . . > P(y, I x), it is clear that P(y, I x) > l/M. Now for each 2 6 k d M we introduce a new set of probability vectors as follows:
qt,(.vk)=L1+6-‘P(y,Ix)J6,
where La J is the largest integer 6 a. As a consequence, we obtain the inequalities
kt2 P(Y,lX)< E d(JhK(M- l)b+ : P(YklX). k=2 k=2
If we define
then
d(.h ) = 1 - .f q:(h), k=2
O<P(y, Ix)-qt,(y,)<M- 1)s.
Furthermore, by the ( Ielinition of qt, q:(y) E Q, and
H(P(Y
(3)
k=l
bp(Y,Ix)log PfY,lX) 4.1-b’lI-~)’
THE CAPACITY OF MEMORYLESS CHANNELS 3
The above inequality holds because for any 2 6 k < M, P( yn- 1 x) < q’,( yk), and thus log(P(y, 1 x)/4:( yk)) < 0.
However, from Eq. (3) we have
>+wl)J 4.0’1) \ p(?-, I +x1 PO’, I .u)
al-M(M-1)6.
Plugging the 6 defined by (2) it yields that
This implies that
Therefore, if we choose nr such that (m + 1)/m = e’ and substitute the chosen m = l/(e’ - 1) into (2), we obtain the desired 6 which is given by (e’- 1)/M’s”. This shows that for every input .Y E X and its associated channel transition probability P( J’ 1 X) we can find a probability vector qi corresponding to it such that H(P(JJ/ x), qj;(y)) <E, where qi E Q,. Since Q, is finite with cardinality J and every member q(y) of Qs is of the form i6 for some integer i, we can arrange Q, in a lexicographic order and let { qz,},!= , be such an ordering of Q,. As a result, the matrix I + +L,(Yk)3,J2& I induced by QB is a discretized channel transition matrix of the original channel transition probabilities { P(y, 1 x)) which satisfies the desired properties and thus the proof follows.
As shown in Theorem 1, if we use the relative entropy as a criterion for discretization, a continuous-input discrete-output memoryless channel can be actually discretized into a discrete memoryless channel where the size of inputs, J, is the cardinality of the set Q, bounded by C” log’. The parameter 6 is determined by the assigned error tolerance E and the size of the channel outputs, M, as well. Moreover, for any member .Y in dass S,, the relative entropy between P( y 1 X) and q:(y) is always less than E. This implies that as far as mutual information is concerned, the element X, is sufficient enough to represent all members in class S,. It also notes that in the proof of the theorem, the compactness of the channel input space X is not required. However, in practice, we assume that X is a compact (i.e., bounded and closed) set in the real line.
In the following section, a constructive procedure to generate the desired sets F= {.x~},!=, and is,}:=, will be presented. As we will see from an example considered in Section IV, the number of desired channel inputs, J.
6 CHANG, FAN, AND DAVISSON
which we actually need is generally much less than the upper bound given by (1). The scheme is simple and is based on the assumption that X is com- pact. Nevertheless, this assumption is not essential because we can always make it compact by including its limit points if X is not closed.
III. A SIMPLE PROCEDURE FOR THEOREM 1
The main purpose of this section is to describe a simple implementable computer algorithm for Theorem 1.
Let X be a closed interval [a, h]. It has been shown in Nelson (1966) that the set (H( ., q(y))1 q(y) E Q( Y,)} is equicontinuous on Q( Y,), i.e., given any E >O and a probability vector on Y,, I, there exists an 4 such that if for any p(p’) E Q( Y,), // p -@ 11 71 < 4, then it implies that I H(P(Y), q(y)) - fJ(d(y), qOJ))l < 6 for all q(y) E Q( Y,). Furthermore, it was also shown in Davisson et al. ( 1980) that the relative entropy H(p(g), q(y)) is convex in p(y). Based on these properties we propose a simple algorithm analogous to a procedure in Davisson et al. (1981) as follows. Since the relative entropy H(P(y 1 x), P(y ( z)) between two transition probabilities P( ~1 IX), P( ~11~) is specified by the inputs x and Z, we will use the notation H(x, 2) to denote H(P(v1.u) P(JJ)z)) for simplicity.’
Algorithm CD (Channel Discretization)
1. Initialization: Set s0 = an assigned error tolerance and r = 0.
2. Set x0 = a, E, = ~~2--~, and J= 0.
3. Set J=J+ 1. Find rlJ >-yJ-r such that H(x~-,,~,)=a,.
4. If zJ > b, go to step (7). Otherwise, find xJ > zJ such that H(x,, zJ) = cr.
5. If .YJ+ * <b, go to step 3. Otherwise, continue.
6. If J> M, let xJ= b, output the sets (.~~j:= I and {z,>f=,, and stop. Otherwise, let r = r + 1 and go to step (2).
7. IfJ>M, let zJ= b, output the sets {x~}::: and {zj)f= r, and stop. Otherwise, let r = r + 1 and go to step (2).
Algorithm III
1. Initialization: Let E = an assigned error tolerance for the Arimoto-Blahut algo- rithm.
THECAPACITYOFMEMORYLESSCHANNELS 7
2. Apply Algorithm CD to produce the desired sets (x,) and (z,).
3. Apply the Arimoto-Blahut algorithm to the discretized channel determined by the input set F= {z,}:= , generated by Algorithm CD, the output set Y,, and the channel transition matrix CfYYkl,-.I).? J /E k LkE Y,W’ I
4. Stop and output the channel capacity found in step (3) which is supposed to be an approximate of the original channel capacity.
It is worth noting that in step (6) of Algorithm CD, the condition that J> M is examined because we must produce a sufficient number of inputs ; before applying the Arimoto-Blahut algorithm. This fact is justified by kallager (1968, Corollary 3, p. 96). If J < M, it means that the prescribed error tolerance is not small enough, i.e., the relative entropy between two channel transition probabilities is still too large. Therefore, the whole procedure must repeat again with a smaller error threshold until condition (6) is met. Moreover, the set {x,} partitions [a, h] into a finite number of classes { Si}, where S, = [x, , , x,] .
IV. A NUMERICAL EXAMPLE
As discussed previously, the problem of calculating the capacity of con- tinuous-input discrete-output channels can be solved by three numerical methods which are Chang and Davisson’s algorithms (Algorithms I and II) developed in Chang and Davisson (1988) and Algorithm III proposed in this paper. To compare their relative performance, we consider the following example which was studied by Chang and Davisson (1988, 1990).
Let a generalized binary-like memoryless channel be specified by the input space X= [0, 11, the output space Y, = (0, 1, . . . . M j, and the channel transition probabilities { P(k 1 x)} given by
P(klx)= ; i >
Xk(l-.X)M?
Then the channel capacity is defined by
C, = max p kEo ;P(x)P(kIX)log~i2.r], [ J P
where q,(k) = SAP(X) P(k 1 x) dx. Notice that in this example, the numver of channel outputs is M+ 1.
Moreover, the property that the channel transition probabilities are sym-
TABL
E I
A Co
mpa
rison
of
th
e Pe
rform
ance
s be
twee
n AL
GORI
THM
S I,
II,
and
III
on
VAXj
VMS
8600
(E
=
6.0
x 10
e4)
8 Al
gorit
hm
IA
Algo
rithm
IB
Al
gorit
hm
II Al
gorit
hm
III
5 (s
ubst
itutio
n by
pr
obab
ility)
(S
ubst
itutio
n by
m
utua
l inf
o)
(Add
ition)
P 2
Sing
le
Mul
tiple
Si
ngle
M
ultip
le
Sing
le
Mul
tiple
7 5
CH
CPU
CH
CPU
CH
CPU
CH
CPU
CH
CPU
CH
CPU
CH
CPU
M Ca
pacit
y (m
s)
Capa
city
(ms)
Ca
pacit
y (m
s)
Capa
city
(ms)
Ca
pacit
y (m
s)
Capa
city
(ms)
Ca
pacit
y (m
s)
P 5 8 1
1.MH)
o 2
1.00
0000
00
2 1.
0000
0000
3
1.00
0000
00
1 l.O
OOOO
OOO
2 1.
0000
0000
4
0.64
7360
38
4 2
3 1.
0874
6283
5
1.08
7462
83
5 1.
0874
6283
6
1.08
7462
83
6 1.
0874
6283
6
1.08
7462
83
5 0.
8994
7792
6
4 1.
2479
0640
30
1.
2479
0640
29
1.
2479
0640
26
1.
2479
0640
33
1.
2479
0661
40
1.
2479
0661
39
0.
9845
8114
13
5 1.
3722
7190
28
1.
3722
7190
26
1.
3722
7190
27
1.
3722
7190
31
1.
3722
7190
30
1.
3722
7190
27
1.
1628
5289
52
6 1.
4579
7721
48
1.
4579
7721
44
1.
4579
7721
43
1.
4579
7721
56
1.
4579
7721
46
1.
4579
7721
45
1.
2560
0123
36
7 1.
5358
5166
32
31
1.53
5870
65
1048
1.
5358
5166
32
23
1.53
5870
65
1142
1.
5358
5162
30
23
1.53
5869
57
3058
1.
3981
6191
13
4
8 1.
6079
2294
27
87
1.60
7954
59
1323
1.
6079
2294
27
61
1.60
7954
59
1390
1.
6079
1719
26
33
1.60
7951
16
3108
1.
4610
3310
14
60
9 1.
6714
8638
56
13
1.67
1491
99
3190
1.
6714
8638
55
76
1.67
1491
99
3500
1.
6714
8664
54
30
1.67
1492
04
4228
1.
5185
3639
13
74
IO
1.72
6803
18
1131
1 1.
7268
2231
58
19
1.72
6803
11
1113
9 1.
7268
2231
63
79
1.72
6800
62
1153
5 1.
7268
2026
88
62
1.62
2913
52
433
I1
1.77
8012
72
1368
4 1.
7780
2079
78
25
1.77
8012
48
1357
5 1.
7780
2079
86
47
1.77
8012
70
1396
5 1.
7780
2110
10
299
1.67
2787
84
902
12
1.82
5831
07
1667
9 1.
8258
4637
97
29
1.82
5831
12
1638
0 1.
8258
1869
14
603
1.82
5831
21
1635
3 1.
8258
4434
11
950
1.71
8704
18
3035
13
1.87
0267
51
1147
4 1.
8702
8438
86
48
1.87
0267
61
1136
7 1.
8702
8444
87
21
1.87
0266
67
1170
3 1.
8702
8312
11
844
1.79
5168
66
3353
14
1.91
1314
71
7293
1.
9113
9536
20
663
1.91
1314
71
7160
1.
9113
9536
22
318
1.91
1314
71
6980
1.
9113
8040
16
465
1.84
5785
61
3654
15
1.94
9629
98
2151
4 1.
9496
4308
11
577
1.94
9629
98
2123
7 1.
9496
4308
12
577
1.94
9630
24
2021
8 1.
9496
3458
11
955
1.87
3736
24
6598
16
1.98
5842
64
3143
3 1.
9858
8584
16
857
1.98
5842
64
3102
3 1.
9858
8403
15
430
1.98
5842
85
3046
5 1.
9858
7671
25
749
1.90
9171
09
8041
17
2.02
0193
51
2451
8 2.
0202
1339
19
216
2.02
0193
51
2394
4 2.
0202
1339
21
027
2.02
0193
26
2448
0 2.
0202
2505
30
369
1.94
2583
40
6954
18
2.05
2747
39
3566
8 2.
0528
0604
30
221
2.05
2747
39
3480
1 2.
0528
1205
33
695
2.05
2747
35
3358
2 2.
0528
0645
35
591
1.99
8996
42
9579
19
2.08
3675
46
2770
2 2.
0837
2756
27
180
2.08
3675
46
2720
1 2.
0837
2756
29
262
2.08
3675
50
2624
8 2.
0837
2758
30
257
2.02
9507
21
1125
1
20
2.11
3043
47
2390
1 2.
1130
5547
15
396
2.11
3043
47
2370
9 2.
1130
5547
16
559
2.11
3043
47
2337
7 2.
1130
7271
24
230
2.05
8512
24
1384
7
21
2.14
1137
32
6279
7 2.
1411
6808
43
758
2.14
1137
32
6237
7 2.
1411
6808
47
212
2.14
1137
35
5646
0 2.
1411
6790
46
847
2.08
6209
48
4639
22
2.16
8054
08
4187
9 2.
1681
0322
37
509
2.16
8054
12
4185
6 2.
1681
0322
40
519
2.16
8054
21
4052
2 2.
1681
0326
43
307
2.11
2721
24
2127
6
23
2.19
3923
88
6257
1 2.
1939
7183
49
322
2.19
3923
88
6234
7 2.
1939
7183
53
157
2.19
3923
95
5732
3 2.
1939
7166
52
097
2.13
7984
55
2392
2
24
2.21
8769
16
5176
4 2.
2188
1660
45
270
2.21
8769
16
5171
6 2.
2188
1660
49
166
2.21
8769
16
4853
5 2.
2188
1651
49
163
2.16
2343
27
2333
8
25
2.24
2676
07
5785
0 2.
2426
9341
38
892
2.24
2676
07
5794
2 2.
2426
9341
41
845
2.24
2676
15
5449
0 2.
2426
9314
40
121
2.20
4381
28
2636
9
26
2.26
5676
89
5526
3 2.
2656
8557
32
967
2.26
5676
89
5512
3 2.
2656
8671
76
388
2.26
5676
99
5232
0 2.
2656
8556
35
794
2.22
7094
87
2288
7
27
2.28
7897
88
1078
03
2.28
7917
04
5796
3 2.
2878
9788
10
7992
2.
2879
153
1 57
558
2.28
7898
23
9949
4 2.
2879
1477
63
940
2.24
9071
24
3005
7
28
2.30
9381
79
9873
4 2.
3094
0827
61
971
2.30
9381
79
9906
8 2.
3094
0827
65
747
2.30
9381
90
9296
7 2.
3094
0754
71
944
2.27
0391
70
3597
8
29
2.33
0172
46
7966
9 2.
3302
0603
54
385
2.33
0172
46
7985
1 2.
3302
0603
58
030
2.33
0172
46
7595
1 2.
3302
0748
65
990
2.29
0967
04
2909
4
30
2.35
0336
65
1020
67
2.35
0358
23
6514
0 2.
3503
3665
10
2020
2.
3503
5711
68
576
2.35
0336
68
9725
2 2.
3503
5611
68
935
2.31
0831
14
2633
9
31
2.36
9870
41
9062
9 2.
3698
8413
54
949
2.36
9870
41
9089
0 2.
3698
8708
76
403
2.36
9870
44
8682
1 2.
3698
8209
60
335
2.33
0098
12
2123
5
32
2.38
8821
74
8678
7 2.
3888
2862
50
008
2.38
8821
74
8685
1 2.
3888
3406
73
635
2.38
8821
74
8332
1 2.
3888
2774
55
844
2.34
8863
29
2812
4
10 CHANG, FAN, AND DAVISSON
metric with respect to i further eases the computation of Algorithm III, where steps (4)-(7) in Algorithm CD can be simplified as follows.
1. If zJ> f, set L=2J- 1 and go to step (7). Otherwise, find .yJ > zJ such that H(.u,, zJ) = si and continue.
2. If .yJ < +, go to step (3). Otherwise, let L = 2J and continue.
3. f L>M, let sJ=& zLCIPj=z, for 1 <j<J, .xL-,=x, for 0 <j d J- 1, and output { x~}.,“=, and {q ;=,. } Then stop. Otherwise, let r = r + 1 and go to step (2).
4. f La&f, let I z~=~,z~+,~~=z, for l<jdJ-1, s~-~=.x, for O<j<J- 1, and output (x;>:=, and {z,>f=,. Then stop. Otherwise, let r = r + 1 and go to step (2).
The numerical results in Table I are obtained by Algorithm I, Algo- rithm II, and the above modified version of Algorithm III. Although Algorithms I and II were given in Chang and Davisson (1988), in order to compare Algorithm III, here we discuss briefly their ideas, in particular, a slightly diffeent approach (approach B) which was not in their paper will be described below.
In general, Algorithms I and II are designed based on a sequence of iterations by trial and error proceses. In other words, both algorithms are executed by first guessing a finite set of channel inputs to form a discrete test channel, then computing its capacity, say Cf. In order to see whether or not Ct is desirable, the algorithms further find the maximum of mutual information yielded by every single channel input averaged over the chan- nel outputs and compare it to Ct. If the difference meets a prescribed error tolerance, the guessed test channel is good, which means that the Ct is the desired channel capacity and the algorithms terminate. Otherwise, a new test channel must be regenerated by either dumping those channel inputs which are not important in channel capacity computation or replacing them with some other promising channel inputs. Algorithm II is basically devised to take care of the former situation; in the meantime, add certain prospective channel inputs. On the other hand, Algorithm I is developed to handle the latter case. The process of how to replace points for Algo- rithms I is made according to either a single-point replacement or a multi- ple-point replacement in two different approaches, which are (A) those channel inputs of the test channel with small probabilities will be replaced and (B) those channel inputs whose mutual information averaged over the channel outputs are small will be replaced. (Notice that all the channel out- puts remain unchanged through executions of Algorithms I, II, and III.) To distinguish Algorithm I implemented in four different methods we denote Algorithm I using approach A with a single-point replacement by Algo-
THE CAPACITY OF MEMORYLESS CHANNELS 11
rithm IAS, Algorithm I using approach A with a multiple-point replace- ment by Algorithm IAM, Algorithm I with approach B and a single replacement by Algorithm IBS, and Algorithm I with approach B and a multiple-point replacement by Algorithm IBM. As noticed in Chang and Davisson ( 1988, 1990) Algorithm I was implemented only based on approached A (i.e., Algorithm IAS and Algorithm IAM), where those chan- nel inputs with zero probability or small probabilities are replaced. However, the criterion of using mutual information of channel inputs for replacement was not considered in Chang and Davisson (1988, 1990) and so, the results produced by approach B are new. A more detailed study on this example can be found in Fan ( 1989).
Figures 1 and 2 are plotted on the basis of CPU time of the three algo- rithms (Algorithm I, Algorithm II, and Algorithm III) run on a VAX 8600 computer with four different types of Algorithm I. Table I is also provided with details. All the results in the table and figures show that Algorithm III yields a moderate performance when the number of channel outputs, Y,, is small; but when Y,v gets large, Algorithm III becomes more efficient. What is most important in this case is it produces a satisfactory performance and essentially achieves the same performances as do Algorithm I and Algorithm II. This substantiates the assertion made earlier and justifies that Algorithm III is indeed a very efficient algorithm compared to Algorithm I and Algorithm II.
90 -
80 -
70 1 !r
2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303~32
M+l = NUMBER OF IYPUTS . IAS - IBS 11s : III
FKXJKE I
12 CHANG, FAN, AND DAVISSON
60
2 3 4 5 6 7 8 9 1011121314151617161920212223242526272829303132
M+l = NUMBER OF INPUTS . IAM + IBM 0 IIM n III
FIGURE 2
V. CONCLUSION
Three numerical methods of calculating the capacity of continuous-input discrete-output memoryless channels are considered. Of particular interest is a simple computational method (Algorithm III). Numerical results show that Algorithm III has an advantage of simple implementations on com- puters; but it is compensated for approximations with less accuracy. Nevertheless, when the channel output space is very large, Algorithm III does offer comparable performance to Algorithms I and II. Thus, in this case, Algorithm III is more attractive than Algorithms I and II because Algorithms I and II need more iterations and also tremendous computing time to find local maxima. Moreover, as reported in Chang et al. (1988), Algorithm III can also be used as an alternative method to find minimax codes for source matching problems (Davisson et al., 1980).
RECEIVED September 3, 1987; ACCEPTED April 27, 1989
REFERENCES
ARIMOTO, S. (1972), An algorithm for computing the capacity of arbitrary discrete memoryless channels, IEEE Trans. Inform. Theory IT-B, No. 1, 14-20.
BLAHUT, R. E. (1972), Computation of channel capacity and rate-distortion function, IEEE Trans. Inform. Themy IT-H, No. 4. 460-473.
THE CAPACITY OF MEMORYLESS CHANNELS 13
CHANG. C.-I, ANU DAVISON, L. D. (1990) Two iterative algorithms for finding minimax solutions, IEEE Trans. Inform. Theory, in press.
CHANG, C.-l, AND DAVISSON, L. D. (1988), On calculating the capacity of an infinite-input finite (infinite)-output channel, IEEE Trans. Itzform. Theory IT-34, No. 5, 10041010.
CHANG, C.-I, FAN, S. C.. AND DAVISSON, L. D. (1988), A simple method of calculating channel capacity and finding minimax codes for source matching problems. in “Proceedings, 1988 Conference on Information Science and Systems, Princeton University, Princeton, NJ.” pp. 365-369.
DAVISSON, L. D., AND LEON-GARCIA, A. (1980). A source matching approach to linding minimax codes, fEEE Trans. kform. Theory IT-X, No. 2, 166-174.
DAVISSON, L. D., MCELIETE. R. J., PURSLEY, M. B., AND WALLACE, M. S. (1981). Efficient universal noiseless source codes, IEEE Trans. InJorm. Theory IT-27. No. 3, 269-279.
FAN. S. C. (1989), “A Numerical Study of Computation of the Capacity of Continuous-input Discrete-Output Memoryless Channels,” Master’s thesis, Department of Electrical Engineering, University of Maryland, Baltimore County campus, Baltimore, MD.
FINAMORE, W. A., AND PEARLMAN. W. A. (1980), Optimal encoding of discrete-time con- tinuous-amplitude memoryless sources with finite output alphabets. IEEE Trans. Inform. Theory IT-26, No. 2, 144-155.
GALLAGER, R. G. (1968). “Information Theory and Reliable Communication.” Wiley, New York.
NELSON, W. (1966). Minimax solution of statistical decision problems by iteration, Ann. Math. Statist. 31. 1643-1657.