Upload
sandra-cunningham
View
217
Download
2
Tags:
Embed Size (px)
Citation preview
InstructorNeelima Gupta
Introduction to some tools to designing algorithms through Sorting
• Iterative• Divide and Conquer
Iterative Algorithms: Insertion Sort – an example
x1,x2,........., xi-1,xi,.......…,xn
For I = 2 to nInsert the ith element xi in the partially sorted list x1,x2,........., xi-1.
(at rth position)
An Example: Insertion Sort
15 8 7 101 2 3 4
12 5 5 6
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}
At Iteration 1: key = 8
Thanks Brijesh Kumar (08) : MCA -12
An Example: Insertion Sort
8 15 7 101 2 3 4
12 55 6
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}
At Iteration 2: key = 7
Thanks Brijesh Kumar (08) : MCA -12
An Example: Insertion Sort
7 8 15 101 2 3 4
12 55 6
At Iteration 3: key = 10
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}Thanks Brijesh Kumar (08) : MCA -12
An Example: Insertion Sort
7 8 10 151 2 3 4
12 55 6
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}
At Iteration 4: key = 12
Thanks Brijesh Kumar (08) : MCA -12
An Example: Insertion Sort
7 8 10 121 2 3 4
15 55 6
At Iteration 5: key = 5
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}Thanks Brijesh Kumar (08) : MCA -12
An Example: Insertion Sort
5 7 8 101 2 3 4
12 155 6
Final Output
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}Thanks Brijesh Kumar (08) : MCA -12
Analysis: Insertion Sort
Thanks : MCA 2012 Dharam Deo Prasad
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key)
{A[j+1] = A[j]j = j - 1
}A[j+1] = key
} }
Statement C N
InsertionSort(A, n) {for i = 2 to n { c1 n
key = A[i] c2 (n-1)j = i - 1; c3 (n-1)while (j > 0) and (A[j] > key) c4 Σ(Ti + 1) {
A[j+1] = A[j] c5 Σ Ti j = j - 1 c6 Σ Ti
}A[j+1] = key c7 (n-1)
}} where Ti is number of while expression evaluations for the ith for loop iterationCi is the constant time required for 1 execution of the statementN is the number of times the statement is executed
Running Time Analysis
Thanks : MCA 2012 Dharam Deo Prasad
Total time
Thanks : MCA 2012 Dharam Deo Prasad
• T(n) = (c1 + c2 + c3 + c7 )n – (c2 + c3 + c7) + [(c4 + c5 + c6) Ti + c4 ]
n
i=2
∑
Worst Case
Thanks : MCA 2012 Dharam Deo Prasad
Worst case: Ti = i – 1
i.e. Ti = (i – 1)
= n(n-1)/2hence, T(n) = (c1 + c2 + c3 + c4 + c7 )n – (c2 +c3 + c4 + c7) +
(c4 + c5 + c6)n(n-1/2)= an2 + bn + c
where,a = 1/2 (c4 + c5 + c6)
b = -1/2 (c4 + c5 + c6) + (c1+
c2 + c3 + c4 + c7 )
c = -(c2 + c3 + c4 + c7)
n
i=2
∑n
i=2
∑
Best Case
Thanks : MCA 2012 Dharam Deo Prasad
Best case: Ti = 1
i.e. Ti = 1
=(n – 1)hence, T(n) = (c1 + c2 + c3 + c4 + c7 )n – (c2 +c3 + c4 + c7) +
(c4 + c5 + c6)(n-1)= an + b
where,a = c1 + c2 + c3 + 2c4
+ c5 + c6 + c7
b = -(c2 + c3 + 2c4 + c5
+ c6 + c7 )
n
i=2
∑n
i=2
∑
Analysis of AlgorithmsBefore we move ahead, let us define the
notion of analysis of algorithms more formally
Input SizeTime and space complexity
This is generally a function of the input sizeHow we characterize input size depends:
Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges Etc
Lower BoundsPlease understand the following
statements carefully.Any algorithm that sorts by removing at
most one inversion per comparison does at least n(n-1)/2 comparisons in the worst case.
Hence Insertion Sort is optimal in this category of algorithms.
Optimal?What do you mean by the term “Optimal”?
Answer: If an algorithm runs as fast in the worst case as it is possible (in the best case), we say that the algorithm is optimal. i.e if the worst case performance of an algorithm matches the lower bound, the algorithm is said to be “optimal”
Inversion :-
Example : 4, 2, 3
No. of pairs = 3C2 , out of which (4,2) is out of order i.e. inversion(2,3) is in order(4,3) inversion
Thanks to:Dileep Jaiswal (11) :MCA 2012
In ‘n’ elements there will be n(n-1)/2 inversions in worst case.
Thus, if an algorithm sorts by removing at most one inversion per comparison then it must do at least n(n-1)/2 comparisons in the worst case.
Thanks to:Dileep Jaiswal (11) :MCA 2012
Insertion Sort
x1,x2,…...,xk-1, xk,...........…..,xi-1,xi
Let xi is inserted after xk-1
No. of comparisons = (i-1) – (k – 1) +1 = i – k + 1 No. of inversions removed = (i-1) – (k – 1) = i – k No. of inversions removed/comparison
= (1-k+1)/(i-k)
< = 1
Thanks to:Dileep Jaiswal (11) :MCA 2012
Thus Insertion sort falls in the category of comparison algorithms that remove at most one inversion per comparison.
Insertion sort is optimal in this category of algorithms .
Thanks to:Dileep Jaiswal (11) :MCA 2012
SELECTION SORTThe algorithm works as follows:Find the maximum value in the array.Swap it with the value in the last positionRepeat the steps above for the remainder of
the array.
Thanks to: MCA 2012 Chhaya Nathani(9)
Selection Sort : An Example For i = n to 2
a) Select the maximum in a[1].......a[i]. b) Swap it with a[i].
Thanks to: MCA 2012 Chhaya Nathani(9)
Selection Sort
x1,x2,…, xk,...........……, xn
Let the maximum is located at xk
Swap it with xn
ContinueThanks to:Dileep Jaiswal (11) :MCA 2012
Selection Sort
x1,x2,…, xk,...........……, xn-i+1,……xn
Suppose we are at the ith iteration
Let the maximum is located at xk
Swap it with xn-i+1
Thanks to:Dileep Jaiswal (11) :MCA 2012
5 20 6 4 15 3 10 2
Thanks to: MCA 2012 Chhaya Nathani(9)
An Example: Selection Sort
5 2 6 4 15 3 10 20
An Example: Selection Sort
Thanks to: MCA 2012 Chhaya Nathani(9)
5 2 6 4 10 3 15 20
An Example: Selection Sort
Thanks to: MCA 2012 Chhaya Nathani(9)
Thanks to: MCA 2012 Chhaya Nathani(9)
5 2 6 4 3 10 15 20
An Example: Selection Sort
Thanks to: MCA 2012 Chhaya Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)
5 2 3 4 6 10 15 20
An Example: Selection Sort
Thanks to: MCA 2012 Chhaya Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)
4 2 3 5 6 10 15 20
An Example: Selection Sort
Thanks to: MCA 2012 Chhaya Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)
3 2 4 5 6 10 15 20
An Example: Selection Sort
Thanks to: MCA 2012 Chhaya Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)Thanks to: MCA 2012 Chhaya
Nathani(9)
2 3 4 5 6 10 15 20
DONE!!!!
An Example: Selection Sort
SelectionSort(A, n) { for i = n to 2 {
max=i for j=i-1 to 1
{If a[j]>a[max]Max=j }
If(max!=i){Swap a[i]……a[max]}
Thanks to: MCA 2012 Chhaya Nathani(9)
ANALYSIS: SELECTION SORTSelecting the largest element requires
scanning all n elements (this takes n − 1 comparisons)
and then swapping it into the last position. Finding the next largest element requires scanning the remaining n − 1 elements and so on...
(n − 1) + (n − 2) + ... + 2 + 1 = n(n − 1) / 2 i.e Θ(n2) comparisonsT(n)= Θ(n2)
Thanks to: MCA 2012 Chhaya Nathani(9)
Selection Sort
x1,x2,…, xk,...........……, xn-i+1,……xn
Suppose we are at the ith iteration
Let the maximum is located at xk No. of comparisons = (n – i + 1) - 1 = n - i No. of inversions removed = (n – i + 1) – 1 – (k-1) = n – i
– k + 1No. of inversions removed/comparison
= ( n – i – k +1)/ (n – i) < = 1 (since k >= 1)
Thanks to:Dileep Jaiswal (11) :MCA 2012
Thus Selection sort also falls in category of comparison algorithms that remove at most one inversion per comparison.
Selection sort is also optimal in this category of algorithms .
Thanks to:Dileep Jaiswal (11) :MCA 2012
Merge Sort(Divide and Conquer
Technique)
Divide the list into nearly equal halves Sort each half recursively Merge these lists
Thanks to:Gaurav Gulzar (MCA -11)
18 9 15 13 20 10 7 5
18 9 15 13 20 10 7 5
18 9
18
15 13
9 15 13 20 10 7 5
20 10 7 5
Divide into 2
Subsequence
Thanks to Himanshu (MCA 2012, Roll No 14)
Next Sort the 2 subsequences and merge them.
Thanks to Himanshu (MCA 2012, Roll No 14)
5 7 9 10 13 15 18 20
9 13 15 18 5 7 10 20
9 18
18
13 15
9 15 13 20 10 7 5
10 20 5 7
Sort & merge 2 Subsequence
Initial Subsequence
Sorted Sequence
Thanks to Himanshu (MCA 2012, Roll No 14)
MergingLet we have two sorted lists :
A:
B:
Compare a1 with b1 and put smaller one in new array C and increase the index of array having smaller element.
Thanks to:Gaurav Gulzar (MCA -11)
a1 a2 a3 ……………. an
b1 b2 b3 ……………. bm
Let a1 < b1
A:
B:
Sorted Array C
C:
Thanks to:Gaurav Gulzar (MCA -11)
a1 a2 a3 ……………. an
b1 b2 b3 ……………. bm
a1
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
50
52
Thanks to:Gaurav Gulzar (MCA -11)
60
80
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
30
35
50
52
Thanks to:Gaurav Gulzar (MCA -11)
60
80
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
30
35
50
52
38
Thanks to:Gaurav Gulzar (MCA -11)
60
80
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
30
35
50
52
38 40 45
Thanks to:Gaurav Gulzar (MCA -11)
60
80
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
30
35
50
52
38 40 45 50
Thanks to:Gaurav Gulzar (MCA -11)
60
80
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
30
35
50
52
38 40 45 50 52
Thanks to:Gaurav Gulzar (MCA -11)
60
80
Example 1:
A:
B:
Sorted Array C
C:
20
40
10
30
35
3845
10
20
30
35
50
52
38 40 45 50 52
Thanks to:Gaurav Gulzar (MCA -11)
60
80
60 80
Worst Case Analysis
If no. of elements in first list is ‘n’ & in second list is ‘m’ then:
Total No. of comparisons = n-1+m
i.e. O(m + n) in worst case and
Thanks to:Gaurav Gulzar (MCA -11)
What is the best case?
Number of Comparisons in the best case:
min(n , m), i.e. Ω(min(n , m))
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 8
50
52
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 810
30
50
52
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 810
30
50
52
35
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 810
30
50
52
35
38
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 810
30
50
52
35
38
45
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 810
30
50
52
35
38
45
50
Thanks to:Gaurav Gulzar (MCA -11)
Example 2:
A:
B:
Sorted Array C
C:
5 8
10
30
35
3845
5 810
30
50
52
35
38
45
50
52
Thanks to:Gaurav Gulzar (MCA -11)
What is the best case?Arrays
Total number of steps: Ω(m+n)…copying the rest of the elements of the bigger array.
Linked List :Total number of steps:
min(n , m), i.e. Ω(min(n , m)) in best case.
Analysis of Merge Sort Since size of both the lists to be merged is n/2, in either case (arrays or linked list), time to merge the two lists is Θ(n).
If T(n) = no. of comparisons performed on an Input of size ‘n’ then :
T(n) = T(n/2) + T(n/2) + Θ(n) = 2T(n/2) +cn ∀ n>= n0
∴ T(n) = O(nlogn)
Thanks to:Gaurav Gulzar (MCA -11)
If T(n) = no. of comparisons performed on an Input of size ‘n’ then :
T(n) = T(n/2) + T(n/2) + Θ(n) = 2T(n/2) +cn ∀ n>= n0
∴ T(n) = O(nlogn)
Thanks to:Gaurav Gulzar (MCA -11)
Final PointsMerging is Θ(m + n) in case of an ArrayIf we use link list then it is Ω(min(m , n))Time for merging is O(n) and Ω(n/2) i.e.
Θ(n)
Thanks to:Gaurav Gulzar (MCA -11)
Conclusion Merge Sort = Θ(nlogn)Have we beaten the lower bound?
No,It just means that merge sort does not fall in the previous category of algorithms. It removes > 1 inversions per comparisons.
Thanks to:Gaurav Gulzar (MCA -11)
What is the Worst case
Best Case
for Merge Sort?
Merge Sort Vs Insertion Sort
What is the advantage of merge sort?
What is the advantage of insertion sort?
Merge Sort Vs Insertion Sort contd..Merge Sort is faster but does not take
advantage if the list is already partially sorted.
Insertion Sort takes advantage if the list is already partially sorted.
Lower BoundAny algorithm that sorts by comparison only
does at least (n lg n) comparisons in the worst case.
Decision Trees• Provides an abstraction of comparison sorts. In a decision tree, each node represents a comparison. Insertion sort applied on x1, x2, x3
x1:x2
x2:x3
x1:x3
x3:x1
x1<x2<x3
x3>x1>x2
x2:x3
x2>x1>x3
x2>x3>x1
x1>x2>x3
x1>x3>x2
x2>x
3
x1<x2
x3>x2
x3<x
1
x3>x1
x1>x2
x3>x
1
x1>x3
x2>x3
x3>x2
•What is the minimum number of leaves in a decision tree?
•Longest path of the tree gives us the height of the tree, and actually represents the worst case scenario for the algorithm.
•height of a decision tree = Ω (n log n)i.e. any comparison sort will perform at least (n logn) comparisons in the worst case.
Decision Trees
Proof : •Let h denotes the height of the tree. •What’s the maximum # of leaves of a binary tree of height h? : 2h
•2h >= number of leaves >= n! (where n = no. of elements and n! is a lower bound on the no. of leaves in the decision tree)
=> h>= log(n!) > n log n ( By Stirling’s Approximation)
( SA: n!= √ (2.π.n) .(n/e)n
> (n/e)n
Thus, log(n!) > n log n – n log e > n logn. )
Merge Sort is OptimalThus the time to comparison sort n elements is
(n lg n)Corollary: Merge-sort is asymptotically optimal
comparison sorts.Later we’ll see another sorting algorithm in
this category namely heap-sort that is also optimal.
We’ll also see some algorithms which beat this bound. Needless to say those algorithms are not purely based on comparisons. They do something extra.
Quick Sort(Divide and Conquer
Technique) Pick a pivot element x Partition the array into two subarrays around
the pivot x such that elements in left subarray is less than equal to x and element in right subarray is greater than x
Recursively sort left subarray and right subarray
Thanks to:Krishn Kant Kundan (MCA -19)
x ≤ x >x
Quick Sort (Algorithm)
QUICKSORT(A, p, q)if p < q
k=PARTITION(A, p, q) QUICKSORT(A, p, k-1) QUICKSORT(A, k+1, q)
Thanks to:Krishn Kant Kundan (MCA -19)
Quick Sort (Example)Let we have following elements in our array :-
i j
i j
i j
Thanks to:Krishn Kant Kundan (MCA -19)
27 14 9 22 8 41 56 31 15 53 99 11 30 24
14 27 9 22 8 41 56 31 15 53 99 11 30 24
14 9 27 22 8 41 56 31 15 53 99 11 30 24
Quick Sort (Example)
i j
i j
i j
Thanks to:Krishn Kant Kundan (MCA -19)
14 9 22 27 8 41 56 31 15 53 99 11 30 24
14 9 22 8 27 41 56 31 15 53 99 11 30 24
14 9 22 8 27 24 56 31 15 53 99 11 30 41
Quick Sort (Example)
i j
i j
i j
Thanks to:Krishn Kant Kundan (MCA -19)
14 9 22 8 24 27 56 31 15 53 99 11 30 41
14 9 22 8 24 27 30 31 15 53 99 11 56 41
14 9 22 8 24 27 11 31 15 53 99 30 56 41
Quick Sort (Example)
i j
i j
i j
Thanks to:Krishn Kant Kundan (MCA -19)
14 9 22 8 24 11 27 31 15 53 99 30 56 41
14 9 22 8 24 11 27 99 15 53 31 30 56 41
14 9 22 8 24 11 27 53 15 99 31 30 56 41
Quick Sort (Example)
i j
i j (stop)
(recursive call on left array) (recursive call on right array)
Thanks to:Krishn Kant Kundan (MCA -19)
14 9 22 8 24 11 27 15 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
i j i j
i j i j
i j i j
i j i j
i j i j
i j i j (stop)
i j (st op)
14 9 22 8 24 11 15 53 99 31 30 56 4127
9 14 22 8 24 11 15 27 53 41 31 30 56 99
9 14 15 8 24 11 22 27 41 53 31 30 56 99
9 14 11 8 24 15 22 27 41 31 53 30 56 99
9 11 14 8 24 15 22 27 41 31 30 53 56 99
9 11 8 14 24 15 22 27 41 31 30 53 56 99
9 11 8 14 24 15 22 27 41 31 30 53 56 99
9 11 8 14 24 15 22 27 41 31 30 53 56 99
i j i j i j i j
9 8 11 14 15 24 22 27 31 41 30 53 56 99
i j i j i j i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
(stop) i j i j (stop) (stop) i j i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
Sorted Array
Thanks to:Krishn Kant Kundan (MCA -19)
Analyzing QuicksortWorst Case?
Partition is always unbalancedWorst Case input?
Already-sorted input, if the first element is always picked as the pivot
Best case?Partition is perfectly balanced
Best Case input??
Worst Case and Best Case input when the middle element is always picked as the pivot?
Worst Case of QuicksortIn the worst case:
T(1) = (1)T(n) = T(n - 1) + (n)
Does the recursion look familiar?T(n) = (n2)
Best Case of QuicksortIn the best case:
T(n) = 2T(n/2) + (n)
Does the recursion familiar?T(n) = (n lg n)
Why does Qsort works well in practice?
Suppose that partition() always produces a 9-to-1 split. This looks quite unbalanced!
The recurrence is:T(n) = T(9n/10) + T(n/10) + n
T(n) = θ (n log n)Such an imbalanced partition and θ(n log n)
time?
Why does Qsort works well in practice?Intuitively, a real-life run of quicksort will
produce a mix of “bad” and “good” splitsPretend for intuition that they alternate
between best-case (n/2 : n/2) and worst-case (n-1 : 1)
What happens if we bad-split root node, then good-split the resulting size (n-1) node?
Why does Qsort works well in practice?Intuitively, a real-life run of quicksort will
produce a mix of “bad” and “good” splitsPretend for intuition that they alternate
between best-case (n/2 : n/2) and worst-case (n-1 : 1)
What happens if we bad-split root node, then good-split the resulting size (n-1) node? We end up with three subarrays, size 1, (n-1)/2, (n-
1)/2 Combined cost of splits = n + n -1 = 2n -1 = O(n) No worse than if we had good-split the root node!
Why does Qsort works well in practice?
Intuitively, the O(n) cost of a bad split (or 2 or 3 bad splits) can be absorbed into the O(n) cost of each good split
Thus running time of alternating bad and good splits is still O(n lg n), with slightly higher constants
How can we be more rigorous? : we’ll do average analysis of Qsort later while doing randomized algorithms.
Quicksort Vs Merge Sort
Merge Sort takes O(n lg n) in the worst caseQuick Sort takes O(n2) in the worst caseSo why would anybody use Qsort instead of
merge sort?Because in practice, Qsort is quick as the worst
case doesn’t happen often.
Up Next
Linear-Time Sorting Algorithms
The End