EE317: Computation and Simulation - DCUee317/Course_notes/handout7-1up.pdf · EE317: Computation and Simulation Handout Seven - Random Number Generation Dr. Conor Brennan School of

EE317: Computation and SimulationHandout Seven - Random Number Generation

Dr. Conor BrennanSchool of Electronic Engineering

January 16, 2014

Dr. Conor Brennan School of Electronic Engineering EE317: Computation and Simulation

Randomness is a fact of life.... In many ways it enriches it, or atleast makes it more interesting.


Probability and statistics and troublesome concepts as humansoften have a poor intuition for them.


Let’s examine the so-called birthdayparadox. N people are in a room.How large must N be before thechances that at least two people inthe roomshare a birthday?

Lets work out the complementary probability - that no-one sharesa birthday. This is given by (assuming independence)

PN =365

365

364

365

363

365· · · 365− N + 1

365

and the probability of a shared birthday is thus

P = 1− PN


For engineers randomness can be beneficial or problematic. Forexample radio engineers must contend with random fading of thepower received by a mobile. However recently engineers have usedthis randomness for security communications.

Ray tracing – Image theory

Ground

TransmiYer

Receiver

TransmiYer image

€

ˆ n

€

θr

€

θi

Ray path

Reflec/on point


Fading is caused by destructive interference of the EM waves andit causes large fluctuations in the power received. It is random innature and radio engineers use many models to characterise it(Rayleigh, Ricean etc).

(a) Fading along a straight line causedby two rays interfering

(b) More general random fading intwo-dimensional urban region


The Monte Carlo method is a numerical method for statisticalsimulation which utilises sequences of random numbers to performthe simulation.

The name “Monte Carlo” waschosen by John Von Neumann.The name refers to the MonteCarlo casino in Monaco, wherethe uncle of a colleague usedto gamble....


The Monte Carlo method has a wide range of engineeringapplications, from the simulation of bit error rates intelecommunications to tolerance analysis in VLSI designs.

A “random number generator”is one of the primary compo-nents of any Monte Carlo sim-ulation.


We would like to create a se-quence of independent randomnumbers with a specified distri-bution, e.g. uniform, Gaussian.

This means, loosely, that each number is obtained purely bychance, having nothing to do with other numbers of the sequence,and that each number generated has a specified probability offalling in any given range of values.

A uniform distribution on a finite interval is one in which eachpossible number within the interval is equally probable. In thesenotes a distribution is generally understood to be uniform unlesssome other distribution is specifically mentioned.


Each of the ten digits 0 through 9 will occur about 1/10 of thetime in a uniform sequence of random digits.

Each pair of digits (01,35,64 etc) should occur about 1/100 of thetime, etc.

Yet if we take a truly random sequence of a million digits, it willnot always have exactly 100,000 zeros, 100,000 ones, etc. In fact,chances of this are quite slim; a sequence of such sequences willonly have this character on the average.


Any specified sequence of a million digits is equally as probable asthe seemingly special sequence consisting of a million zeros.Equally the choice 1, 2, 3, 4, 5, 6 is as likely as any else to win theLotto (and you may have to share it with fewer people).

Also - truly independent random processes are memoryless. If weare choosing a million digits at random and if the first 999,999 ofthem happen to come out to be zero, the chance that the finaldigital is zero is still exactly 1/10, in a truly random situation.


“Anyone who considers arithmetical methods of producing randomdigits is, of course, in a state of sin.” - John von Neumann (1951)

All random number generators are based upon specificmathematical algorithms which are repeatable and sequential. Assuch, the numbers are just pseudo random, but they are oftencalled random numbers for simplicity, with the understanding thatthey only appear to be random, but actually are deterministic.


Desirable properties include:

• Uncorrelated Sequences - The sequences of random numbersshould be serially uncorrelated. This means that any subsequenceof random numbers should not be correlated with any othersubsequence of random numbers. Most especially, n-tuples ofrandom numbers should be independent of one another.

• Long Period - The generator should be of long period. Ideally,the generator should not repeat; practically, the repetition shouldoccur only after the generation of a very large set of randomnumbers.


Desirable properties continued:

• Uniformity - The random numbers should be uniformlydistributed within a specified region. For example, if randomnumbers on [0,1) are to be generated, it would be poor practicewere more than half to fall into [0, 0.1), assuming the sample sizeis sufficiently large. The properties of uniformity and uncorrelatedsequences are loosely related. Often, when there is a lack ofuniformity, there are n-tuples of random numbers which arecorrelated. In this case, the space might be filled in a definite,easily observable pattern.

• Efficiency - As literally millions or even billions of randomnumbers are required in a large-scale simulation, the generatorshould be efficient.


Almost all random number generators have as their basis asequence of pseudorandom integers (there are exceptions).

The integers or fixed point numbers can be manipulatedarithmetically to yield floating point or real numbers.

The random number cycle can be presented in terms of eitherintegers or real numbers. Here, for clarity, we confine ourselves tointegers.


Start Finish Numbers traversed

in sequence


The above figure illustrates a random number cycle representing,in this case, a sequence of 12 integers. Each red dot represents adistinct integer.

The nature of the cycle illustrates that:

(1) the sequence has a finite number of integers,

(2) the sequence gets traversed in a particular order, and

(3) the sequence repeats if the period of the generator is exceeded


Linear Congruential Generators are by far the most commonlyused generators. They generate uniform random numbers, basedon the following formula.

Xn+1 = (aXn + c) mod m

where

m is the modulus satisfying m > 0

a is the multiplier satisfying 0 ≤ a ≤ m

c is the increment satisfying 0 ≤ c ≤ m and

X0 the initial seed satisfying 0 ≤ X0 ≤ m.

mod m means take the remainder left over after integer divisionby m. Note that in practical situations we would choose a largevalue of m. In these notes however we shall use smaller values ofm so that we can explicitly explore concepts such as maximumperiod, randomness etc.


In the case of c = 0, it is called a multiplicative congruentialrandom number generator.

The case c 6= 0 is termed a mixed congruential random numbergenerator.

Multiplicative LCGs are faster than mixed LCGs because of oneless addition operation.

Multiplicative LCGs are adequate to good for many applications.


The modulo operation provides the appearance of randomness.

Note that Xn+1 depends only on the previous integer Xn. Thischaracteristic of LCG minimises storage requirements.

At the same it imposes restrictions on the period: the longestperiod possible is m.

For convenience, we refer to the sequence generated in this fashionas LCG (a, c , m, X0).


The method described above can generate integer values from 0 tom. How can we use it to generate real-valued numbers?

With Xn determined, the corresponding real number can beobtained with

Rn =Xn

m

or

Rn =Xn

m − 1

When dividing by m the Rn values are distributed in [0, 1).Dividing by m − 1, Rn would be distributed in [0,1].


It is desired that Rn values are uniformly distributed between zeroand one, where any particular Rn is just as likely to appear as anyother Rn, and the average of the Rn sequence is approximately 0.5.

The choice of the four numbers a, c , m, X0 are of vital importanceto the quality of the generator. If they are not chosen properly, thegenerator may not have the longest possible period, or generatednumbers may not exhibit good randomness, or the generator maynot be efficiently implementable.


Example

The choice of multiplier a has effects on both the period lengthand the randomness of the sequence generated. This is shown withthe following example LCG (a, 0, 13, 1). Consider the case a = 4

Xn+1 = (4Xn + 0) mod 13

= 4Xn mod 13

SoX1 = 4 · 1 mod 13 = 4

X2 = 4 · 4 mod 13 = 3

X3 = 4 · 3 mod 13 = 12

X4 = 4 · 12 mod 13 = 9

X5 = 4 · 9 mod 13 = 10

X6 = 4 · 10 mod 13 = 1 = X0

after which the sequence repeats.Dr. Conor Brennan School of Electronic Engineering EE317: Computation and Simulation

Example contd

a Sequence Period

0 0, 0 . . . 1

1 1, 1 . . . 1

2 1, 2, 4, 8, 3, 6, 12, 11, 9, 5, 10, 7, 1 . . . 12

3 1, 3, 9, 1 . . . 3

4 1, 4, 3, 12, 9, 10, 1 . . . 6

5 1, 5, 12, 8, 1 . . . 4

6 1, 6, 10, 8, 9, 2, 12, 7, 3, 5, 4, 11, 1 . . . 12

7 1, 7, 10, 5, 9, 11, 12, 6, 3, 8, 4, 2, 1 . . . 12

8 1, 8, 12, 5, 1 . . . 4

9 1, 9, 3, 1 . . . 3

10 1, 10, 9, 12, 3, 4, 1 . . . 6

11 1, 11, 4, 5, 3, 7, 12, 2, 9, 8, 10, 6, 1 . . . 12

12 1, 12, 1 . . . 2


Let’s do a Matlab example. We will make a code to shufflerandomly through some audio tracks.


For a full length period LCG, choosing a different initial seed justshifts the starting point in the sequence which is completelydetermined by the values a, c and m.

Example

LCG (6, 0, 13, X0) is a full length generator. If the initial seedX0 = 1, the resulting sequence is

1, 6, 10, 8, 9, 2, 12, 7, 3, 5, 4, 11, 1 . . .

Any initial seed between 1 and 12 will not affect the apparentrandomness of the sequence, just a shift of the starting point ofthe period. However if a LCG is not a full length period generator,then different initial seed may result in different sequence withpossibly different period.


Example

If a LCG is not a full length period generator, then a differentinitial seed X0 may result in a different sequence and a differentlength of period. This can be shown with LCG (3, 0, 16, X0). Thereis no good reason to use a generator with a period less than themaximum period.

X0 Sequence period

1, 3, 9 or 11 . . . 1, 3, 9, 11, 1 . . . 4

5, 7, 13 or 15 . . . 5, 15, 13, 7, 5 . . . 4

2 or 6 . . . 2, 6, 2 . . . 2

4 or 12 . . . 4, 12, 4 . . . 2

10 or 14 . . . 10, 14 . . . 2

8 . . . 8, 8 . . . 1

0 . . . 0, 0 . . . 1


The choice of a, c , m, X0

1) When m = 2n and c > 0, the maximum period m can beachieved if and only if a mod 4 = 1 and c is odd.

Example

For generator LCG (a, 1, 16, X0), the maximum period 16 isobtained if and only if a = 1, 5, 9 or 13. We have

a Sequence

1 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1 . . .

5 1, 6, 15, 12, 13, 2, 11, 8, 9, 14, 7, 4, 5, 10, 3, 0, 1 . . .

9 1, 10, 11, 4, 5, 14, 15, 8, 9, 2, 3, 12, 13, 6, 7, 0, 1 . . .

13 1, 14, 7, 12, 13, 10, 3, 8, 9, 6, 15, 4, 5, 2, 11, 0, 1 . . .

Can you see a problem with this generator? All maximum periodsequences generated using m = 2n have the feature of alternatingodd and even numbers. So the low-order bits of this kind of LCGare not random. This choice is not recommended.


Example continued:

When a = 3 or a = 11, the period is 8, when a = 7 the period is 4,when a = 15, the period is 2. Worse still, when a = 2, 4, 6, 8, 10, 12or 14, the LCG will eventually generate a constant sequence (aperiod of only 1) after less than 4 other numbers which depend onthe initial seed.


The choice of a, c , m, X0 continued

2) When m = 2n and c = 0, the maximum period is m4 (a quarter

of the modulus), and is obtained if and only if a mod 8 = 3 or amod 8 = 5 (5 is preferred) and the initial seed is odd. Thelow-order bits of this kind of LCGs are not random. Notrecommended.

Example

For generator LCG (a, 0, 16, X0), the maximum period 4 is obtainedif and only if a = 3, 5, 11 or 13.

a Sequence

3 1, 3, 9, 11, 1 . . .

5 1, 5, 9, 13, 1 . . .

11 1, 11, 9, 3, 1 . . .

13 1, 13, 9, 5, 1 . . .


The choice of a, c , m, X0 continued

3) When the modulus m is a prime number and a > 1, themaximum period is m − 1, no matter whether c = 0 or not. Themaximum period m − 1 is obtained if and only if a is a primitiveelement modulo m.

If m is prime then a is a primitive element modulo m (or primitiveroot of m) if and only if an mod m 6= 1 for n = 1, 2, 3, ,m − 2.The low-order bits of this kind of LCGs may or may not berandom.

Recommended. (Prime moduli are best in terms of sequencerandomness.)


Note that there is always one integer which maps only to itself, (inthe case of c = 0, X0 = 0 maps to itself.).

Even when c is non-zero, the maximum period of the generator isstill one less than the modulus.

Thus, for linear congruential generators with a prime modulus,using a non-zero c does not increase the period except whena = 1.

Normally, a should be greater than 1, in order that the generatedsequence has some randomness.


Example

ForLCG (6, c , 13, 1), find out the integer z which maps to itself fordifferent increment c . Note that 6 is a primitive root of 13 (Youcan verify this by writing a simple Matlab code).

c 0 1 2 3 4 5 6 7 8 9 10 11 12

z 0 5 10 2 7 12 4 9 1 6 11 3 8

To understand the above table consider the specific case of c = 2.If Xn = 10 we get

Xn+1 = (aXn + c) mod m

= (6 · 10 + 2) mod 13

= (62) mod 13

= 10


Multiplicative LCGs with Prime moduli are recommended. Parkand Miller1 recommend LCG (16807, 0, 2147483647, X0), in which

a = 75 = 16807

c = 0

m = 231 − 1 = 2147483647

It has a full period of m − 1 and good randomness.

1Park, S.K. and K.W.Miller, Random Number Generators: Good ones arehard to find, Comm. A.C.M., vol. 32, n. 10, Oct. 1988, pg. 1192-1201Dr. Conor Brennan School of Electronic Engineering EE317: Computation and Simulation

Note that choosing the same initial seed will result in the samesequence of numbers being produced each time.

A random initial seed can also be selected based on the currentvalue of the clock in the computer. For example we could use

temp = clock ; % produces the date in the format year month dayhour minute second.X = round(temp(6)) ;

to give us an initial number between 0 and 60. We can scale thisto be between 0 and m − 1 as required.


How to avoid overflow

If we directly implement LCG (16807, 0, 2147483647, X0) on a 32bit machine, overflow will happen, because, aXn can be as large as16807 ∗ 2147483646 ' 1.03× 245.

So direct implementation using integer operations will be wrong,unless the maxint is greater than or equal to 246 − 1 . Whenoverflow happens, the host system will produce an error, and theLCG will not be able to generate the next random number.


How to avoid overflow

One way to avoid the overflow problem is to do the integercalculations in real arithmetic, because most systems have largerreal number range than integer range. Direct implementation usingreal arithmetic on any system where real numbers are representedwith a 47-bit or larger mantissa (including the sign bit) will workcorrectly.

For example, direct implementation using real arithmetic will becorrect on all systems that support the IEEE 64-bit real arithmeticstandard since the mantissa in that case is 53-bits. There are otherways that can enable a 32 bit machine to generate a sequence ofrandom integers with a period of 231 − 1 without causing overflow.


Average and Variance of Random Numbers

The expected value and variance of a random variable X withdensity function f (x) is given by

E [X ] =

∫ ∞−∞

xf (x)dx

Var [X ] =

∫ ∞−∞

(x − E [X ])2 f (x)dx

For a uniform random variable we have

f (x) =

{1 0 < x < 10 otherwise

and so

E [X ] =

∫ 1

0xdx =

x2

2

∣∣∣∣10

=1

2

Var [X ] =

∫ 1

0(x − 0.5)2 dx =

1

12


Hence a sample of numbers drawn from a good uniform randomnumber generator should have a sample average close to 0.5 and asample variance close to 1

12 = 0.08333...

These conditions of average and variance approaching thetheoretical values are necessary but not sufficient conditions for agood random number generator.

For example, the average and variance for LCG (1, 1, 4096, 1) is0.4999 and 0.0833 respectively, very close to the ideal figures, butLCG (1, 1, 4096, 1) is definitely not a good random numbergenerators, because it generates the sequence. . . 0, 1, 2, 3, , 4093, 4094, 4095, 0 . . . .

For LCG (6, 3, 13, 1) the average is 0.4872 and the variance is0.0812. The departure of these values from the ideal ones of 1/2and 1/12 is due to the short period of this sequence and the rathercoarse resolution of the generated real numbers.


Documents

EE317: Computation and Simulation - DCUee317/Course_notes/handout7-1up.pdf · EE317: Computation and Simulation Handout Seven - Random Number Generation Dr. Conor Brennan School of