Computer Visioncs-courses.mines.edu/csci507/schedule/27/Ransac.pdfAlgorithm (RANSAC) 1) Randomly select a minimal set of points and solve for the transformation 2) Count the number

Computer VisionColorado School of Mines

Colorado School of Mines

Computer Vision

Professor William HoffDept of Electrical Engineering &Computer Science

http://inside.mines.edu/~whoff/ 1


RANSAC

2


Identifying Correct Matches

• Let’s say that we want to match points from one image to another• We detect feature points in each image (e.g., SIFT or SURF points) and

then match their descriptors• Now we have a set of matching pairs, but some are incorrect• How do we identify the correct matches?

3

Graffiti image dataset from http://kahlan.eps.surrey.ac.uk/featurespace/web/data.htm


SIFT Points from each image

4

Number of features detected: 1741 Number of features detected: 2052


Tentative matches

5

1

2

3

456

78

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

2526

27

2829

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

5859

60

6162

63

64

65

66

67

68

6970

7172

73

74

75

76

77

78

79

80

81

828384

85

86

87

88

89

90

91

92

93

94

9596

97

9899

100

101102

103104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144145146

147

148

149

150

151

152

153

154155

156

157

158

159

160

161

162

163

164

165

166

167

168

169170

171

172

173174175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196197

198

199

200

201202

203

204

205

206

207

208

209

210211

212

213

214

215216

217

218

219

220

221

222

223

224

225226

227

228

229

230

231

232

233

234

235

236

237

238239

240

241242

243

244

245

246

247

248

249250251

252

253254

255

256257

258

259

260261262

263

264

265

266

267

268

269270

271

272

273

274

275

276277

278

279280

281

282

283284

285

286287

288

289

290

291

292

293

294

295

296297

298

299

300301

302

303

304

305

306

307308

309

310

311

312

313

314

315

316 317

318319

320

321

322

323

324

325

326

327328

329330

331

332

333

334335

336

337

338

339

340341

342

343

344

345

346

347

348

349

350

351

352

353

354

355356357

358

359

360

361

362

363

364

365

366367

368

369370

371

372

373 374

375

376

377

378

379

380

381382383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398399

400

401

402

403

404

405

406 407

408

409

410

411

412

413414

415

416

417

418

419420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437438439

440

441

442

1

2

34

5

6

78

9

10

11

1213

14

15

16

17

18

19

20

21

22

23

24

2526

27

2829

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46 47

48

49

50

51

52

53

54

55

56

57

5859

60

6162

63

64

65

66

67

68

6970

7172

73

74

75

76

77

78

79

80

81

828384

85

86

87

88

89

90

91

92

93

94

9596

97

9899

100

101102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120121

122

123

124125

126

127

128

129

130

131

132

133

134

135

136

137

138139

140

141

142

143

144145146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169170

171

172

173174175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196197

198

199

200

201202

203

204

205

206

207

208

209

210211

212

213

214

215216

217

218

219

220

221

222

223

224

225226

227

228

229

230

231

232

233

234

235

236

237

238239

240

241242

243

244

245

246

247

248

249250251

252

253254

255

256257258

259

260261262

263

264

265

266

267

268

269270

271

272

273

274

275

276277

278

279280

281

282

283284

285

286287

288

289

290

291

292

293

294

295

296297

298

299

300301

302

303

304

305

306

307308

309

310

311

312

313

314

315

316317

318319

320

321

322

323

324

325

326

327328

329330

331

332

333

334335

336

337

338

339

340341

342

343

344

345

346

347

348349

350

351

352

353

354

355356357

358

359

360

361

362

363

364

365

366367

368

369370

371

372

373374

375

376377

378

379

380

381382383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398399

400

401

402

403

404

405

406

407

408

409410

411

412

413414

415

416

417

418

419420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437438

439

440

441

442

Tentative matches (b

ased

on feature de

scrip

tors): 44

2

Computer VisionColorado School of Mines6

% Demonstrate the matching of a plane, from one image to another.clear variablesclose all

%rng(0); % Reset random number generator

% Read in test images.I1 = imread('img1.png');I2 = imread('img3.png');

% If color, convert them to grayscaleif size(I1,3)>1 I1 = rgb2gray(I1); endif size(I2,3)>1 I2 = rgb2gray(I2); end

% Extract SIFT features.

% First make sure the vl_sift code is in the pathif exist('vl_sift', 'file')==0

run('C:\Users\William\Documents\Research\vlfeat-0.9.20\toolbox\vl_setup');end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% First imageI1 = single(I1); % Convert to single precision floating pointfigure(1), imshow(I1,[]);

% These parameters limit the number of features detectedpeak_thresh = 0; % increase to limit; default is 0edge_thresh = 10; % decrease to limit; default is 10[f1,d1] = vl_sift(I1, ...

'PeakThresh', peak_thresh, ...'edgethresh', edge_thresh );

fprintf('Number of frames (features) detected: %d\n', size(f1,2));

% Show all SIFT features detectedh = vl_plotframe(f1) ;set(h,'color','y','linewidth',1.5) ;

Find SIFT points in images (1 of 3)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Second imageI2 = single(I2);figure(2), imshow(I2,[]);

[f2,d2] = vl_sift(I2, ...'PeakThresh', peak_thresh, ...'edgethresh', edge_thresh );

fprintf('Number of frames (features) detected: %d\n', size(f2,2));

% Show all SIFT features detectedh = vl_plotframe(f2) ;set(h,'color','y','linewidth',1.5) ;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Match features% The index of the original match and the closest descriptor is stored in% each column of matches and the distance between the pair is stored in% scores.

% Define threshold for matching. Descriptor D1 is matched to a descriptor% D2 only if the distance d(D1,D2) multiplied by THRESH is not greater than% the distance of D1 to all other descriptorsthresh = 1.5; % default = 1.5; increase to limit matches[matches, scores] = vl_ubcmatch(d1, d2, thresh);fprintf('Number of matching frames (features): %d\n', size(matches,2));

indices1 = matches(1,:); % Get matching featuresf1match = f1(:,indices1); d1match = d1(:,indices1);

indices2 = matches(2,:);f2match = f2(:,indices2);d2match = d2(:,indices2);



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Show matchesfigure(3), imshow([I1,I2],[]);o = size(I1,2) ;line([f1match(1,:);f2match(1,:)+o], ...

[f1match(2,:);f2match(2,:)]) ;for i=1:size(f1match,2)

x = f1match(1,i);y = f1match(2,i);text(x,y,sprintf('%d',i), 'Color', 'r');

endfor i=1:size(f2match,2)

x = f2match(1,i);y = f2match(2,i);text(x+o,y,sprintf('%d',i), 'Color', 'r');

end



Fitting a Transformation

• We want to find a transformation between the two images

• What transformation to use?– Depends on your application ... for a planar object, or a rotating camera, you could use a homography (projective transform)

• We can calculate this transformation from corresponding points– To fit a homography, this requires at least 4 corresponding image points

– However, incorrect correspondences will corrupt our estimate

9


Robust Regression

• Ordinary least‐squares fitting minimizes an objective function, which is the sum of squared errors

• If there are outliers, they can have very large errors, so they “skew” the fitting

• One solution is to use a modified objective function (see next slide), which down‐weights the influence of large errors– Least median of squares (LMS) can also be used

• This works well if there aren’t that many outliers (e.g., 10% or 20% or so)

10


Also, see MATLAB’s “robustdemo”

From: http://web.as.uky.edu/statistics/users/pbreheny/764‐F11/notes/12‐1.pdf

These are called “M‐estimators” because they are (M)inimizing a loss function


Large Numbers of Outliers

• If the outliers are a large fraction of the data, then M‐estimators don’t work well

• Instead we need a type of clustering, or voting method– The Hough transform method in Lowe’s original SIFT paper is an example of this

• Another method is called RANSAC (Random Sample Consensus)– RANSAC has been widely used in many engineering applications

12


Algorithm (RANSAC)

1) Randomly select a minimal set of points and solve for the transformation

2) Count the number of points that agree with this transformation

3) If this transformation has the highest number of inliers so far, save it

4) Repeat for N trials and return the best transformation

13


Example: line fitting

We will randomly select subsets of 2 points (since 2 is the minimum number of points to fit a line)


Example: line fitting

n=2


Model fitting


Measure distances


Count inliers

c=3


Another trial

c=3


The best model

c=15


RANSAC Algorithm

• “Random Sample Consensus”– Can be used to find inliers for any type of model fitting

• Widely used in computer vision

• Requires two parameters:– The number of trials N (how many do we need?)– The agreement threshold (how close does an inlier have to be?)

Fischler, M. A., & Bolles, R. C. 1981. Random sampling consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the Associations for Computing Machinery, 24(26), 381–395.

21


Number of samples (trials) required• Assume we know , the probability that a point is an outlier

– So (1‐ ) is the probability that any point is an inlier

• The probability that a sample of size s is all inliers is (1‐ )s– So 1‐(1‐ )s is the probability of getting at least one outlier in that sample

• With N samples (trials), the probability that all samples have at least one outlier is palloutliers = (1‐(1‐ )s )N

• We want at least one sample to have no outliers. The probability is pnotalloutliers = 1 ‐ palloutliers

• We’ll pick pnotalloutliers to be some high value, like 0.99.

• If we know , we can solve for N

22

))1(1log()1log(

siersnotalloutlpN

N is the number of trials needed to get at least one sample of all inliers, with probability pnotalloutliers

Try example: = 0.5p = 0.99s = 2N = 10


Number of samples required

23


Better estimate for required number of trials

• Again, assume we know , the probability that a point is an outlier– So (1‐ ) is the probability that any point is an inlier

• Assume we have M total points. The estimated number of inliers is 1

• The number of ways we can pick a sample of s points out of M points is

• The number of ways we can choose s points that are all inliers is

• The probability that any sample of s points is all inliers is p=n/m

24

)1log()1log(

ppN iersnotalloutl

)!(!!

knkn

kn

“n choose k” is the number of ways to choose k items from a set of n items

N is the number of trials needed to get at least one sample of all inliers, with probability pnotalloutliers


Estimating outlier probability

• If you don’t know the outlier probability in advance, you can estimate it from the data:– Start with a very conservative guess for (say, 0.9)– Compute the number of trials required ... it will be very large

– Each time you get a fit, count the number of inliers– If the number of inliers is the best so far, revise your estimate of : 1 /

– Re‐compute the number of trials required using the new estimate of and continue

25


Deciding if a point is an inlier

• You can just pick a threshold for a point’s residual error, for it to be counted as an inlier (e.g., you could measure this empirically)

• Another strategy:– Compute the conditional probability of the residual assuming that it is

an inlier, and compare it to the conditional probability of the residual assuming that it is an outlier

– Namely, test if |– Typically, we assume inliers have Gaussian distributed noise (with

standard deviation ), and outliers have uniform distributed noise – This approach is called MLESAC (“maximum likelihood estimator

sample consensus”)

26

P. H. S. Torr , A. Zisserman, "MLESAC: A new robust estimator with application to estimating image geometry" , Computer Vision and Image Understanding, vol 78 Issue 1, April 2000, pp 138 ‐ 156.http://www.robots.ox.ac.uk/~vgg/publications/2000/Torr00/torr00.pdf


Demo Program

• This program matches two images using a homography– So the two images should be images of a plane– Or, the two images should be taken with a rotating (but not

translating) camera

• Files– main.m

• Reads the images, detects SIFT features, finds tentative matches

– fitHomographyRansac.m• A function that takes tentative matching features, and returns the homography (if possible)

– imfuseWarped.m• A function that fuses two images by blending

27


Demo Program

• (Go through Matlab code)

• Demonstrate on sample images– img1.png, img3.png (from the

graffiti dataset)

– floor1.jpg, floor2.jpg

– table1.jpg, table2.jpg

28


1

1

2

2

3

3

4

4

5

5

6

6

7

7

8

8

9

9

10

10

11

11

12

12

13

13

14

14

15

15

16

16

17

17

18

18

19

19

20

20

21

2122

22

23

23

24

24

25

25

26

26

2727

28

2829

29

30 30

31

31

3232

3333

34 34

35

35

36

36

3737

38

38

3939

40 40

41 41

42

42

43 43

44

44

45

45

46

46

47

47

48

4849

49

50

50

51

51

52

5253

53

54

54

55

55

56

56

57

57

58

58

59

59

60

60

61

61

62

62

63

6364

64

65

65

66

66

67

67

68

68

69

69

70

70

71

71

72

7273

73

74

74

75

75

76

76

77

77

78

78

79

7980

80

81

81

82

82

83

83

84

84

8585

86

86

87

87

88

8889

8990

90

91 91

9292

93 93

9494

95

95

96

96

97

97

98 9899 99

100100

101 101102 102

103103

104

104

105

105

106

106

107 107

108 108

109109

110 110

111 111

112112

113 113

114114

115 115

116 116

117

117

118 118

119

119

120

120121 121

122 122

123

123

124

124

125

125

126

126

127 127

128

128

129

129130

130

131

131132

132

133

133

134

134

135

135

136

136

137

137

138

138

139

139

140

140

141

141

142

142

143

143

144

144

145

145

146

146

147

147

148

148149

149

150

150

151

151152

152

153

153

154

154

155

155156

156

157

157

158

158

159

159

160

160

161161

162

162

163163

164

164

165 165

166 166

167167

168

168

169169

170170

171171

172 172

173 173174 174175 175

176 176

177177

178 178

179179

180 180

181181

182

182

183183

184 184

185 185

186 186

187 187

188 188

189189

190 190

191

191

192 192

193193

194194

195195

196

196197

197

198 198

199

199200

200

201

201

202

202

203

203

204

204

205 205

206

206

207

207

208

208

209209

210210211211

212

212

213

213

214

214

215

215

216

216

217

217

218

218

219

219

220

220

221

221

222

222

223223

224224

225

225

226

226227

227

228

228

229

229

230

230

231

231

232

232

233

233

234

234

235

235

236

236

237237

238

238

239

239

240240

241241

242242

243243

244244

245245

246

246

247 247

248248

249

249

250

250

251

251

252252

253

253254254

255

255

256256

257257

258258

259259

260260261261

262262

263

263

264 264

265 265

266266

267

267

268268

269

269

270

270

271

271

272 272

273

273

274

274

275

275

276

276

277

277

278

278

279279

280280

281

281282

282

283

283284

284

285

285286

286

287

287

288

288

289

289

290

290

291

291

292

292

293

293

294

294295

295

296

296

297

297

298

298

299

299

300

300

301

301

302

302

303

303

304

304

305

305

306

306

307

307

308

308

309

309

310

310

311

311

312

312

313313

314314

315

315316

316317

317

318318

319319

320

320321321

322 322

323 323

324 324

325

325

326 326

327327328 328

329

329

330

330

331331

332

332

333

333

334

334

335

335

336 336

337

337

338

338

339

339

340

340

341

341

342

342343

343

344

344

345

345

346

346

347

347348

348

349

349350

350

351

351

352

352

353

353

354

354

355

355

356

356

357

357

358358

359

359

360360

361 361

362 362

363363

364364

365365

366 366367 367

368368

369369

370370

371

371

372

372

373

373

374

374

375

375

376

376

377

377378

378

379

379

380

380

381

381

382

382

383

383

384

384

385 385

386386

387

387

388388

389389

390390

391391

392 392

393 393

394

394

395

395396

396

397

397

398

398

399

399

400400

401 401

402402

403403

404

404

405 405

406 406407

407

408408

409

409

410

410

411

411

412

412

413

413

414

414

415

415

416

416

417

417

418

418

419419

420420

421 421

422 422

423423

424424

425425

426

426427

427428

428

429

429430

430

431431

432432

433 433

434

434

435

435

436

436

437437

438438

439

439

440 440

441441

442442372

inliers

difference image fused image