Debug Hacks, Table of Contents

Hi,

I've translated Debug Hacks table of contents into English.
Please feel free to ask me if you have any questions.

DEBUG HACKS,
english translation by Hiro Yoshioka
6/25/09

Table of Contents

Chapter 1 Introduction
1 what is a debug
2 road maps of debug hacks
3 hints of debug

Chapter 2 The Things you have to know before debugging
4 how to get core dump of a process
5 introduction of gdb
6 introduction of gdb, part 2
7 introduction of gdb, part 3
8 basic of Intel architecture
9 basic of stack
10 basic of parameter passing of function call (x86_64)
11 basic of parameter passing of function call (i386)
12 basic of parameter passing of function call (C++)
13 how to learn assembly language
14 mapping between source code and assembly code

Chapter 3 Getting Started Kernel Debug
15 how to read Oops messages
16 connecting serial console using minicom
17 getting kernel messages thru network
18 debug using SysRq key
19 getting kernel crash dumps using diskdump
20 getting kernel crash dumps using Kdump
21 how to use crash
22 getting kernel crash dumps using IPMI watchdog timer when a kernel freeze
23 getting kernel crash dumps using NMI watchdog timer when a kernel freeze
24 assembly language idiom in the kernel source code
25 assembly language idiom in the kernel source code, part 2

Chapter 4 Practice of application debug
26 application program aborts SIGSEGV
27 corruption of a back trace information
28 destroying memory by invalid access of an array
29 detecting invalid access using watch point of gdb
30 defects and faults of malloc() and free()
31 application stall (deadlock)
32 application stall (infinite loop)

Chapter 5 Practice of kernel debug
33 kernel panic (NULL pointer)
34 kernel panic (destruction of list data)
35 kernel panic (race condition)
36 kernel stall (infinite loop)
37 kernel stall (spin lock)
38 kernel stall (spin lock, part 2)
39 kernel stall (semaphore)
40 stall of real time process
41 slow down
42 high cpu load

Chapter 6 Debug Techniques
43 using strace to detect the cause of error
44 useful options of objdump
45 how to use Valgrind (basic)
46 how to use Valgrind (advance)
47 getting kernel internal status using kprobes
48 getting kernel internal status using jprobes
49 getting kernel internal status using kprobes (inserting probes)
50 getting kernel internal status using kprobes (inspecting variables)
51 getting value of a variable optimized out by a compiler using KAHO
52 debugging running linux kernel using systemtap
53 debugging running linux kernel using systemtap, part2
54 reading /proc/meminfo
55 /proc/PID/mem to read contents of memory
56 behavior and implementation of OOM Killer
57 fault injection
58 finding hidden bug in the linux kernel using a fault injection
59 init section of linux kernel
60 solving performance issues
61 getting information using VMware Vprobe
62 getting memory dump by XEN
63 understanding an implementation of a function call thru GOT/PLT
64 debugging an initramfs image
65 detecting a stall of a realtime process by RT Watchdog
66 check 64bit capability of intel x86 processor

The editor of Debug Hacks gave me the following description of this book in English. Thanks Ryoko Akaike

HIGHLIGHT

This explains the debug techniques fully under the long term development of Linux Kernel and application development process. This book covers the necessary preparation, its knowledge, quick finding tools of problems and how to solve them quickly by using the most useful techniques and tools. There are many examples to be used in the book. This book helps the Linux development engineers.

Mr. Yukihiro Matsumoto, creator of Ruby strongly recommends this book for the process of programming development.


DESCRIPTION

This book provides the feature of the debugging techniques of Linux development
Engineers.
The readers can :
- Know how to find the problems.
- Prepare for the knowledge to find the problems.
- Find the bugs of the system.
- Amend the problems.
- Get the useful techniques and tools.
- Get the most effective tools in the development stage.

Master the problem solving techniques by the detailed examples.

ABOUT THE KEY AUTHORS
Mr. Hirotaka Yoshioka
Dr. Kazuhiro Yamato
Mr. Naohiro Ooiwa
Mr. Toyo Abe
Mr. Shunsuke Yoshida

All engineers belong to Miracle Linux Corporation, which provides the Linux
System packages and its global consulting services of Linux to the various
Corporations in Japan.

Debug Hacks

Debug Hacks is a book to introduce tips and tools for removing bugs. Debugging is one of time consuming process in the software development. However there is few books to describe debug.

Debugging is a kind of black art. We have never been taught in a class room. We need to have debug skill. Each programmer has own debug techniques but almost nobody try to describe their tips.

This is our challenge to describe the art of software debugging. The authors are linux kernel engineers of Miracle Linux Corporation (http://www.miraclelinux.com). I am one of the authors :-)

It covers from C/C++ application debugging to the Linux kernel debugging. We show the real world examples which we have fixed in our services.

Chapter 1 Warmingup, Chapter 2 Preparation for user land programs, Chapter 3 Preparation for kernel debugging, Chapter 4 Debugging for Application programs, Chapter 5 Debugging for Kernel Programs, Chapter 6 Various Tips and Tools.

This book covers GDB, strace, objdump, Valgrind, kprobes, jprobes, KAHO, systemtap, proc file system, oprofile, VMware vprove, fault injection, Xenand so on.

You will learn many tips to debug your programs.

Bad news is that it is written in Japanese. You may ask O'reilly Japan to translate this book to other languages, that is, English, Chinese, Korean, and so on.

YLUG (Yokohama Linux Users Group) and Kernel Code Reading Party

This memo is inspired by the blog written by Ted.
http://thunk.org/tytso/blog/2009/02/04/how-active-are-your-local-linux-users-groups/

Let me describe Yokohama Linux Users Group (YLUG).
http://ylug.jp (Many pages are written in Japanese)

More than 10 years ago, I was living in the Silicon Valley and worked for an RDBMS company as a RDBMS kernel engineer. My job was nothing related with Linux. It was 1998. Netscape opened the source code aka the Mozilla Project. Silicon Valley Linux Users Group had an offline meeting which hosted Marc Andreessen of Netscape who made this crazy idea on April, 1st of 1998. Marc said, "It is April, 1st, it is not April Fool but real." I know nothing about Open Source nor Bazaar model at that time.

I came back to Japan on the end of 1998. There were Linux bubble and many LUG (Linux Users Group) in Japan were established.

YLUG (Yokohama Linux Users Group) is one of LUG which was established in January of 1999. I joined YLUG. Mailing list was very active in those days. I asked an easy question to the mailing list. How does the linux kernel implement a system call? The answer was "int 0x80" and "read the source code." Hmm, I find/grep "int 0x80" and browse the code. I happened to think that it might be fun to read the code with YLUG members. It was a little bit bother to read the code by myself, but reading the code with colleagues may give us other insight. It might be fun.

So I set up an offline meeting to read and have a party. It was held April 28th, 1999 and more than 30 people joined it. It was *fun*. Many people said, let's have another one. Many who could not attend the meeting asked us to have another one. We have more than 90 times and almost once a month since then.

I named it "Kernel Code Reading Party."

Although, we have not read the code these days, Kernel Code Reading Party has two parts, 1) a speaker makes a informal technical seminar, 2) we have a beer party. We will have 10th anniversary and 100th Kernel Code Reading Party in this October or November time frame.

From Junior high school students to 50 something like me, from linux beginners to kernel hackers, audience are very wide.

YLUG's Kernel Code Reading Party is very active and we really enjoy it.

Someday Join Us.

Let's Get Startted

Hi,

okdt recommended me to write a blog in English. so here it is. Although, I have two blogs written in Japanese, I wish this page gave you something happened Open Source Communities in Japan.

My diary Someday in the Future (MIRAI NO ITSUKA　未来のいつか) http://d.hatena.ne.jp/hyoshiok/
My blog The Power of Dreams (YUME NO CHIKARA ユメのチカラ) http://blog.miraclelinux.com/yume/

Enjoy,