IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
Hunting segfaultsfor beginners
Uwe Volker
XING AG
August 2012
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
6 The End
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
1 IntroductionWhat is a segfault?Examples - CExamples - Perl
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
6 The EndUwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
if you try to access an invalid address
(or write to a protected address)
BOOOM!
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
if you try to access an invalid address
(or write to a protected address)
BOOOM!
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
if you try to access an invalid address
(or write to a protected address)
BOOOM!
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
What is a segfault?
segfault = segmentation fault
every process has memory pages
these pages are mapped to physical memory
if you try to access an invalid address
(or write to a protected address)
BOOOM!
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
Examples - C
using uninitialized pointers
dereferencing NULL pointers
using ”freed” pointers
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
Examples - Perl
bug in a XS extension
bug in Perl itself (rare)
Perl 5.6.1:
perl -e ’undef a’
perl -e ’*::=%::=0’
Perlmonks thread: (Golf) Segfault Perl
http://perlmonks.org/?node_id=156461
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
Examples - Perl
bug in a XS extension
bug in Perl itself (rare)
Perl 5.6.1:
perl -e ’undef a’
perl -e ’*::=%::=0’
Perlmonks thread: (Golf) Segfault Perl
http://perlmonks.org/?node_id=156461
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
What is a segfault?Examples - CExamples - Perl
Examples - Perl
bug in a XS extension
bug in Perl itself (rare)
Perl 5.6.1:
perl -e ’undef a’
perl -e ’*::=%::=0’
Perlmonks thread: (Golf) Segfault Perl
http://perlmonks.org/?node_id=156461
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
On the shellCore dump fileCGI script
1 Introduction
2 Detecting segfaultsOn the shellCore dump fileCGI script
3 Devel::Trace
4 gdb
5 Devel::bt
6 The EndUwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
On the shellCore dump fileCGI script
On the shell
p e r l s e g f a u l t . p lSegmentat ion f a u l t ( c o r e dumped )
#!/ u s r / b i n / p e r luse Debug : : DumpCore ;Debug : : DumpCore : : s e g v ;
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
On the shellCore dump fileCGI script
On the shell
p e r l s e g f a u l t . p lSegmentat ion f a u l t ( c o r e dumped )
#!/ u s r / b i n / p e r luse Debug : : DumpCore ;Debug : : DumpCore : : s e g v ;
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
On the shellCore dump fileCGI script
Core dump file
$ u l i m i t −c u n l i m i t e d$ p e r l s e g f a u l t . p lSegmentat ion f a u l t ( c o r e dumped )$ l l c o r e−rw−r−−−−− 1 uwe uwe 1695744 J u l 26 1 4 : 0 8 c o r e
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
On the shellCore dump fileCGI script
CGI script
personal story: CGI script in Apache
no output, no entry in logfiles (access.log and error.log)
but when I wrote to some file, the content was there
so the script was getting executed...
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
On the shellCore dump fileCGI script
CGI script
personal story: CGI script in Apache
no output, no entry in logfiles (access.log and error.log)
but when I wrote to some file, the content was there
so the script was getting executed...
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
1 Introduction
2 Detecting segfaults
3 Devel::TraceUsageHow do I spot a segfault?Other uses for Devel::Trace
4 gdb
5 Devel::bt
6 The EndUwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
Usage
”Print out each line before it is executed (like sh -x)”
perl -d:Trace program
for CGI: put it in your shebang line
>> . / t e s t : 4 : p r i n t ” Statement 1 at l i n e 4\n” ;>> . / t e s t : 5 : p r i n t ” Statement 2 at l i n e 5\n” ;>> . / t e s t : 6 : p r i n t ” C a l l to sub x r e t u r n s ” , &x ( ) , ” at l i n e 6 .\ n” ;>> . / t e s t : 1 2 : p r i n t ” I n sub x at l i n e 1 2 .\ n” ;>> . / t e s t : 1 3 : return 1 3 ;>> . / t e s t : 8 : e x i t 0 ;
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
Usage
”Print out each line before it is executed (like sh -x)”
perl -d:Trace program
for CGI: put it in your shebang line
>> . / t e s t : 4 : p r i n t ” Statement 1 at l i n e 4\n” ;>> . / t e s t : 5 : p r i n t ” Statement 2 at l i n e 5\n” ;>> . / t e s t : 6 : p r i n t ” C a l l to sub x r e t u r n s ” , &x ( ) , ” at l i n e 6 .\ n” ;>> . / t e s t : 1 2 : p r i n t ” I n sub x at l i n e 1 2 .\ n” ;>> . / t e s t : 1 3 : return 1 3 ;>> . / t e s t : 8 : e x i t 0 ;
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
Usage
”Print out each line before it is executed (like sh -x)”
perl -d:Trace program
for CGI: put it in your shebang line
>> . / t e s t : 4 : p r i n t ” Statement 1 at l i n e 4\n” ;>> . / t e s t : 5 : p r i n t ” Statement 2 at l i n e 5\n” ;>> . / t e s t : 6 : p r i n t ” C a l l to sub x r e t u r n s ” , &x ( ) , ” at l i n e 6 .\ n” ;>> . / t e s t : 1 2 : p r i n t ” I n sub x at l i n e 1 2 .\ n” ;>> . / t e s t : 1 3 : return 1 3 ;>> . / t e s t : 8 : e x i t 0 ;
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
How do I spot a segfault?
look at the last few lines
if it stops immediately, it might be a segfault
grep for your script name
output can be very large, with long lines
grep -v site perl
in my case: buggy MSSQL driver (easysoft)
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
How do I spot a segfault?
look at the last few lines
if it stops immediately, it might be a segfault
grep for your script name
output can be very large, with long lines
grep -v site perl
in my case: buggy MSSQL driver (easysoft)
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
How do I spot a segfault?
look at the last few lines
if it stops immediately, it might be a segfault
grep for your script name
output can be very large, with long lines
grep -v site perl
in my case: buggy MSSQL driver (easysoft)
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
Other uses for Devel::Trace
your program is behaving strange and you have no debuggerat hand
(use grep and grep -v to filter the output)
Does this code get executed?
Which part of the conditional was taken?
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow do I spot a segfault?Other uses for Devel::Trace
Other uses for Devel::Trace
your program is behaving strange and you have no debuggerat hand
(use grep and grep -v to filter the output)
Does this code get executed?
Which part of the conditional was taken?
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdbIntroductionUsageCore dump file - reloaded
5 Devel::bt
6 The EndUwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
Introduction
GNU debugger
command line debugger
we use it to extract the stacktrace from the core dump file
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
Usage
$ gdb p e r l c o r eCore was g e n e r a t e d by ‘ p e r l p e r l / s e g f a u l t . p l ’ .Program t e r m i n a t e d w i t h s i g n a l 11 , Segmentat ion f a u l t .#0 0 x00007f2 f5d086754 i n c r a s h n o w f o r r e a l ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ”) at DumpCore . xs : 1 010 p r i n t f (”%d ” , ∗p ) ; /∗ c a u s e a s e g f a u l t ∗/( gdb )
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
Usage
$ gdb p e r l c o r eCore was g e n e r a t e d by ‘ p e r l p e r l / s e g f a u l t . p l ’ .Program t e r m i n a t e d w i t h s i g n a l 11 , Segmentat ion f a u l t .#0 0 x00007f2 f5d086754 i n c r a s h n o w f o r r e a l ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ”) at DumpCore . xs : 1 010 p r i n t f (”%d ” , ∗p ) ; /∗ c a u s e a s e g f a u l t ∗/( gdb ) where#0 0 x00007f2 f5d086754 i n c r a s h n o w f o r r e a l ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ”) at DumpCore . xs : 1 0#1 0 x00007f2 f5d086789 i n crash now ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ” , attempt num =42) at DumpCore . xs : 1 7#2 0 x00007f2 f5d086820 i n XS Debug DumpCore segv ( cv=0x1087c10 ) at DumpCore . xs : 2 6#3 0 x0000000000488db3 i n P e r l p p e n t e r s u b ( )#4 0 x0000000000480a7d i n P e r l r u n o p s s t a n d a r d ( )#5 0 x00000000004336b4 i n p e r l r u n ( )#6 0 x000000000041bddc i n main ( )( gdb )
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
Core dump file - reloaded
ulimit -c unlimited
current directory has to be writable
(can be tricky with Apache)
ps auxww|grep apache
ls -l /proc/1234/cwd
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
Core dump file - reloaded
ulimit -c unlimited
current directory has to be writable
(can be tricky with Apache)
ps auxww|grep apache
ls -l /proc/1234/cwd
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
IntroductionUsageCore dump file - reloaded
Core dump file - reloaded
ulimit -c unlimited
current directory has to be writable
(can be tricky with Apache)
ps auxww|grep apache
ls -l /proc/1234/cwd
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow does it work?
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::btUsageHow does it work?
6 The End
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow does it work?
Usage
”Automatic gdb backtraces on errors”
just use the module
it registers signal handlers for SIGSEGV (and a few more)
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow does it work?
Usage
”Automatic gdb backtraces on errors”
just use the module
it registers signal handlers for SIGSEGV (and a few more)
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
UsageHow does it work?
How does it work?
the signal handler forks off a process which runs gdb
gdb attaches to the parent and outputs the stacktrace
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
SourcesQuestions?
1 Introduction
2 Detecting segfaults
3 Devel::Trace
4 gdb
5 Devel::bt
6 The EndSourcesQuestions?
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
SourcesQuestions?
Sources
http://en.wikipedia.org/wiki/Segmentation_fault
http://modperlbook.org/html/
21-6-Analyzing-Dumped-core-Files.html
http://www.linux-magazin.de/Heft-Abo/Ausgaben/
2007/01/Getriebeschaden
Uwe Volker Hunting segfaults
IntroductionDetecting segfaults
Devel::Tracegdb
Devel::btThe End
SourcesQuestions?
Questions?
Uwe Volker Hunting segfaults