Upload
felicity-cameron
View
222
Download
0
Tags:
Embed Size (px)
Citation preview
Precise News Video Text Detection and
Text Extraction Based on
Multiple Frames Integration
Advisor: Dr. Shwu-Huey Yen
Student: Hsiao-Wei Chang
1
2
Outline
1 Introduction
2 Related work
3 Edge detection
4 Proposed Method
5 Experimental Results
6 Conclusion
1 Introduction
Video text can be scene text or caption text. Caption text, static or scrolling, is superimposed in a later stage of videos producing. Most static texts provide concise and direct description of the content presented in news video.
3
The procedure of video textual information extraction can be divided into two categories; detection and extraction.
4
2 Related work
Video text detection methods can be classified into three classes.
The first class is texture-based.
The second class presumes that a text string contains a uniform color.
The third class is edge-based.
5
Text extraction methods can be classified into two classes.
The first class is global .
The second class is local.
6
Niblack proposed calculating the threshold value by shifting a window across the image. The threshold is determined by the following formula:
T(x,y) = m(x,y) + k * s(x,y),
where m(x,y) and s(x,y) are local mean and standard deviation values, respectively. The value of k is a parameter.
7
Sauvola established on Niblack’s algorithm. The threshold is determined by the following formula:
T(x,y) = m(x,y) * (1 + k * (s(x,y)/R - 1))
where m(x,y) and s(x,y) are as in Niblack's formula. R is the dynamic range of standard deviation, and a parameter k.
8
3 Edge detection
3.1 Sobel Edge detection
The Sobel masks are
the horizontal mask the vertical mask
9
-1 0 1-2 0 2-1 0 1
-1 -2 -10 0 01 2 1
3.2 Canny edge detection
The Canny algorithm contains three parameters.
The size of the Gaussian filter.
Thresholds: the use of two thresholds with hysteresis allows more flexibility than in a single-threshold approach.
11
(a) original image (b) Sobel edge (c) Canny edge
(a) original image (b) Sobel edge (c) Canny edge
12
4 The Proposed Method
4.1 Text Detection
People need 2 seconds or more to process a complex scene. Thus, if videos are played f frames per second, we are interested in video texts focusing on a fixed location for at least 2f consecutive frames. Let k be the nearest integer that is not less than f.
13
On the mth round, four reference frames on frames (m-1)k+i, i= 1, k/3, 2k/3, 3k/3 are chosen. We set h and w are the height and width of the character size.
14
Fig. 1. The flowchart of the proposed approach.
15
Start
A video sequence
J = 1
k frames of round J
Four grayscale reference images
Four Canny edge maps
AND - edge - map
Text location masking
Exact text labeling
Save for text extraction
>=k frames left
J = J + 1
End
Yes
No
Step 1: Get four reference frames from the given one round of video frames and transform them into grayscale images. We use Eq.(1) to accomplish this.
(1)
where Y is the intensity value.
16
BGRY 114.0587.0299.0
Step 2: Execute the edge detection. The Canny edge detector is applied on each grayscale image yielding an edge map. A simple line deletion (horizontal and vertical) is followed if a line is too long.
19
Step 3: Do logical AND on Canny edge maps. Note that after AND operation, a position (i, j) is true (an edge pixel) if all four Canny edge maps are true at (i, j).
21
22
Fig. 5. The AND-edge-map: the result after taking AND operation on four Canny edge maps of Fig. 4.
Step 4: Mask text location. A three-stage technique is designed to find the text mask. (a) A window the size of w h (presumed character size) slides from left to right (per column) and top to bottom (per row) on the AND-edge-map.
23
The value of BWT represents the transitions from black to white or from white to black.
where b()=1 if it is black and 0 otherwise. If BWT is larger than the threshold TBWT , this window is masked.
24
)|),1(),(|()|)1,(),(|(1
0
1
1
1
0
1
1
w
j
h
i
h
i
w
j
jibjibjibjibBWT
The threshold TBWT depends on the character size. “I” is the one with the least black-and-white transitions in English letters. Figure 6 shows each letter has dimension approximately 1018 and the BWT in Fig. 6(b) is 72 which is 0.4 of 1018. The experimental results on many multilingual video texts are satisfying when is 0.35, TBWT = (wh).
26
(b) The rough mask is examined. A horizontal line segment of length w comprising points on (i, j), …, (i+w-1, j) will be eliminated if neither of these points is an edge; and a vertical line segment of length h comprising points on (i, j), …, (i, j+h-1) will be eliminated if neither of them is an edge point.
27
(c) Considering the different background and various contrast in reference frames, the results of the Canny edge detector of the same text on different frames may differ in a few pixels. This usually causes characters to lose some pixels in the AND-operation.
28
To alleviate this problem, a morphological compensated image M is obtained by first applying a closing. Isolated blobs of small size are removed followed by a dilation to extend the text region.
30
(a) The obtained rough mask (b) Non-text pixels removal
(c) Isolated noises removal (d) Morphological compensation
Step 5: Exact text location. To do the refinement, we examine every white pixel. If a white pixel is located on (i, j) position, then points located on top, bottom, left, and right of (i, j) position of the edge map will be examined.
31
In checking the upward direction, on the binary edge map, if (i, j-1), (i, j-2), …, (i, j-k+1) points are all white and (i, j-k) is the first edge pixel (black point), then every point on (i, j-1), (i, j-2), …, (i, j-k+1) in the rough text (grayscale) image must be non-text accordingly. Thus, any non-white pixels will be converted into white in the rough text image.
32
33
(a) The rough text image (b) The result after refinement
Effectiveness of text refinement before and after.
Finally, to label the detected texts, we do a simple binarization on the refined text image followed by a morphological operation to connect texts that are close to each other.
37
Start
Get a text detected box
Extend boundary
Transform Color image into grayscale image
Create Canny edge map
End
Get B from line-traversing
Does B cover entire range of histogram
modify the background pixel
intensity
Get the threshold T
No
Yes
Fig. 12. The flowchart of the proposed approach
38
First, for video text which is captured in a tightly bounded, we extend both the upper and lower boundary of the box at least 1 pixel to ensure that sufficient pixels are included for the application of Canny edge detector.
We compare the pixel intensity of each column both from top to bottom and from bottom to top until we reach the pixel position identified as Canny edge or the end of the boundary if there is no Canny edge. When the comparison is complete, we get the range of the background intensity.
39
Let bmin and bmax be the least and the largest intensity values for those encountered pixels. The range of the background intensity is thus defined as B = [bmin, bmax]. It is possible that B covers the entire range of the intensity. According to B, we will discuss the binarization in two cases.
40
Case (1): B does not cover the entire range of the histogram.
We use the interval B to determine the range of foreground intensity F. Since text polarity is initially unknown, the midpoint of B can be used to estimate the foreground intensity.
41
If intensity of the midpoint of B is bright (>128), then the text is positive polarity, and negative polarity otherwise.
42
,otherwise]255,1[
,128)(]1,0[
max
maxmin21
min
b
bbbF
44
[0, ] if it is positive polarity (dark text),
[ , 255] otherwise,
p kT
p k
where p and are the peak and the standard deviation of F, and k is a constant. In our experiments, k is set as 1.80.
we define the text intensity T as
45
(a) (b) (c)
(d) (e) (a) original image, (b) extended image, (c) grayscale image, (d) Canny edge map, (e) final image for OCR.
Case (2): B covers the entire range of the histogram.
Fig. 15. original image
Otsu Lin Niblack Sovola
Fig. 16. binarization results
46
To solve this problem, we reverse the refined image to recover the original text polarity. Repeat the steps, i.e., producing histogram, Canny edge map, and the background range B on this reversed image. Now, the background intensity range B is [0, 97].
47
For each background pixel P, if its intensity is within the range of text intensity T, then its intensity value will be replaced by
where I(P) is the intensity of P.
49
255 if it is positive polarity,( )
0 otherwise,I P
(a) (b)
Fig. 20. (a) modify background intensities of Fig. 15;
(b) proposed algorithm binarization result
50
5 The Experimental Results
5.1 Text Detection Results
The proposed text detection algorithm was evaluated on multilingual videos clips for a total of approximately 30 minutes. All of these videos have a resolution of 400300 and a frame rate 29.97 per second. The presumed largest character size wh was set to be 2020 and in TBWT was set to be 0.35.
51
52
(a) (b) (c) Fig. 22.Three bounding boxes for the same text from large to small.
(a) (b) (c)
(d) (e) Fig. 23. Detected boxes (in yellow) for a ground truth text (in blue) that (a)
and (b) fail to detect, (c)~(e) truly detect it but only (e) accurately matched.
In Fig. 22. the dimensions of bounding boxes are 8820, 8618, and 8416 for (a), (b), (c) respectively. Any of them are perfect, but the area in (c) is only 76.4% of that in (a). To accommodate these cases, we take the medium bounding box, the one in (b), as the ground truth.
53
A detected box truly detects the texts if the area ratio r, defined in (3), is at least 50%.
54
)__(
)__(
BOXGBOXDArea
BOXGBOXDArear
(3)
where D_BOX is a detected box, like yellow boxes in Fig. 23, G_BOX is the ground truth text box, like blue boxes in Fig. 23.
To focus the bounding preciseness, if the area ratio r is at least 80% we say the text is accurately detected. We use recall (R), precision (P), and quality of bounding preciseness (Q) to measure the efficacy of algorithms as in (14), (15), (16).
55
(14)
(15)
(16)
56
)__(#)__(#
)__(#
)_(#
)__(#
FBOXDTBOXD
TBOXD
BOXD
TBOXDP
)__(#
)___(#
TBOXD
TBOXDAcuQ
)_(#
)__(#
BOXG
TBOXDR
57
(a) (b)
(c) (d) Fig. 24. The ambiguities in defining a ground truth for “TALK & VIRGINIA SHOWDOWN”.
Take characters “TALK & VIRGINIA SHOWDOWN” in Fig. 24 as an example. We accept all (a)~(d) cases to be ground truths. However, cases (a) and (b) are better because detected regions should be as tight as possible.
58
60
(e) (f)
Fig. 25. Some results of our method(size 400x300). (a) and (b) are CNN videos, (c) ESPN video, (d) NHK video from Japan, (e) and (f) are two different news videos from Taiwan.
Table 1. Results on Different Video sources
Video Sources
Type/LanguageLengthmin sec
# of G. boxes
D_BOX_F D_BOX_T
# of False Alarms
# of boxes w.r < 50%
# of boxes w.50% r < 80%
# of boxes w.r 80%
CNN News/English 9’16” 404 0 4 9 391
ESPN Sport/English 5’07” 111 2 11 29 71
NHK News/Japanese 5’35” 310 0 0 0 310
ETTV News/Chinese 5’29” 939 0 0 27 912
TVBS News/Chinese 4’51” 1340 0 64 57 1219
Total --/-- 30’18” 3104 2 79 122 2903
61
Table 2. Results on R, P, and Q
Channel Recall Precision Quality
CNN 400/404= 99.01% 400/404= 99.01% 391/400= 97.75%
ESPN 100/111= 90.09% 100/113= 88.50% 71/100= 71.00%
NHK 310/310=100.00% 310/310=100.00% 310/310=100.00%
ETTV 939/939=100.00% 939/939=100.00% 912/939= 97.12%
TVBS 1276/1340= 95.22% 1276/1340= 95.22% 1219/1276= 95.53%
Average 3025/3104= 97.45% 3025/3106= 97.39% 2903/3025= 95.97%
62
We summarize the contributions of the proposed algorithm in the following:
Precise boxes. Positive and negative text polarities. The alignment and the length of text strings. Sizes, fonts, and multi-color characters.
The worst experimental result is the ESPN video due to the small, isolated, and sparsely located characters. We repeated the experiment with a presumed character size of wh to be halved (1010).
64
65
Fig. 28. Some results of other’s method. (a) Hua et al., 2001, (b) Wang et al., 2004, (c) Anthimopoulos et al., 2008, (d) Huang et al., 2008.
(a) (b)
(c) (d)
5.2 Text Extraction Results
We evaluated the performance of our proposed method comparing with Otsu, Lin, Niblack, and Sauvola methods.
71
Niblack’s method (cited from [30]) [25] approach (cited from [30])
our method
original image (cited from [30]) [30] approach (cited from [30])
72
original image (cited from [29]) Otsu’s method
Lin’s method Niblack’s method (cited from [29])
Sauvola’s method (cited from [29]) adaptive Niblack’s (cited from [29])
adaptive Sauvola’s (cited from [29]) our method
73
original image (cited from [29]) Otsu’s mehod
Sauvola’s method (cited from [29]) adaptive Niblack’s (cited from [29])
adaptive Sauvola’s (cited from [29]) our method
Lin’s method Niblack’s method (cited from [29])
6 Conclusion
We proposed a general text detection algorithm that is applicable to multilingual news videos without any limitations on text colors, fonts or sizes, alignments, or length of text strings. This algorithm has excellent performances in recall (R), precision (P), and quality of bounding preciseness (Q).
74
For text extraction, our proposed method has the merits both from global and adaptive methods. It is parameter-free, computation efficient and robust to various video text including color, font size, alignment.