Debugging a Challenge Binary

This walk-through will guide you through debugging a challenge binary.

During the creation of of Challenge Binaries (CBs), an author may need to use tools to inspect and debug various built binaries.

Inspecting a CB

The unix file command can be used as a quick, high-level test that a file in question is a CGC binary.

$ file LUNGE_00001
LUNGE_00001: CGC 32-bit LSB executable, (CGC/Linux)

CGC tool-chain items intended to be specifically used with CBs can be found in standard cross-build locations for i386-linux-cgc. Specifically:

/usr/i386-linux-cgc/bin/
/usr/bin/i386-linux-cgc-*

When wishing to use these tools, you must specify the full path to the tool, use the 'long name' for the specific tool in /usr/bin, or prepend the /usr/i386-linux-cgc/bin directory to your path. For instance, objdump can be used in these ways.

The full path for either location of objdump:

$ /usr/bin/i386-linux-cgc-objdump -f LUNGE_00001
$ /usr/i386-linux-cgc/bin/objdump -f LUNGE_00001

Using the 'long name' which, because it is in /usr/bin, is already in PATH:

$ i386-linux-cgc-objdump -f LUNGE_00001

Prepending the location of the i386-linux-cgc tools to PATH:

$ which objdump
/usr/bin/objdump
$ PATH=/usr/i386-linux-cgc/bin:$PATH
$ which objdump
/usr/i386-linux-cgc/bin/objdump
$ objdump -f LUNGE_00001

In any of these cases, the output of objdump -f should be similar to:

$ objdump -f LUNGE_00001

LUNGE_00001:     file format cgc32-i386
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x08048074

The CGC version of objdump can be used to perform many inspection functions, such as observing section headers, or observing symbols:

$ objdump -h LUNGE_00001

LUNGE_00001:     file format cgc32-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000255  08048074  08048074  00000074  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .rodata       00000027  080482cc  080482cc  000002cc  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .eh_frame     00000054  080482f4  080482f4  000002f4  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .comment      0000001c  00000000  00000000  00000348  2**0
                  CONTENTS, READONLY
  4 .debug_aranges 00000040  00000000  00000000  00000364  2**0
                  CONTENTS, READONLY, DEBUGGING
  5 .debug_info   000001a2  00000000  00000000  000003a4  2**0
                  CONTENTS, READONLY, DEBUGGING
  6 .debug_abbrev 00000134  00000000  00000000  00000546  2**0
                  CONTENTS, READONLY, DEBUGGING
  7 .debug_line   000000da  00000000  00000000  0000067a  2**0
                  CONTENTS, READONLY, DEBUGGING
  8 .debug_str    000000ba  00000000  00000000  00000754  2**0
                  CONTENTS, READONLY, DEBUGGING
  9 .debug_loc    00000064  00000000  00000000  0000080e  2**0
                  CONTENTS, READONLY, DEBUGGING

$ objdump -t LUNGE_00001

LUNGE_00001:     file format cgc32-i386

SYMBOL TABLE:
08048074 l    d  .text	00000000 .text
080482cc l    d  .rodata	00000000 .rodata
080482f4 l    d  .eh_frame	00000000 .eh_frame
00000000 l    d  .comment	00000000 .comment
00000000 l    d  .debug_aranges	00000000 .debug_aranges
00000000 l    d  .debug_info	00000000 .debug_info
00000000 l    d  .debug_abbrev	00000000 .debug_abbrev
00000000 l    d  .debug_line	00000000 .debug_line
00000000 l    d  .debug_str	00000000 .debug_str
00000000 l    d  .debug_loc	00000000 .debug_loc
00000000 l    df *ABS*	00000000 service.c
00000000 l    df *ABS*	00000000 libc.c
00000000 l    df *ABS*	00000000
08048121 g       .text	00000000 random
0804807f g       .text	00000000 _terminate
0804810d g       .text	00000000 deallocate
08048074 g       .text	00000000 _start
08049348 g       .eh_frame	00000000 __bss_start
0804813b g     F .text	0000010f main
0804808d g       .text	00000000 transmit
080480cd g       .text	00000000 fdwait
080480ad g       .text	00000000 receive
08049348 g       .eh_frame	00000000 _edata
08049348 g       .eh_frame	00000000 _end
0804824a g     F .text	0000007f transmit_all
080480f3 g       .text	00000000 allocate

A stripped CB would exhibit less information in the case of symbols:

$ strip LUNGE_00001
$ objdump -t LUNGE_00001

LUNGE_00001:     file format cgc32-i386

SYMBOL TABLE:
no symbols

Using STDERR to debug a CB

The age-old method of inserting additional output into a program in order to track flow can certainly be used to debug a CB. For example, preprocessor directives (specifically #ifdef PATCHED_N) are used to indicated what should or should not be included in the PATCHED version of a CB. Suppose, you want to test that code associated with such a preprocessor is reachable, and therefore a vulnerable condition is reachable. A simple way to do this is by adding additional transmit calls in order to create output to STDERR.

Consider inserting the following lines into the example service.c beginning with the ifdef on line 24:

#ifdef PATCHED_1
    transmit(STDERR, "Reached point 4\n", 16, 0);
    _terminate(4);
#else
    transmit(STDERR, "Reached point 5\n", 16, 0);
    _terminate(5);

    if (buf[0] == 0x41 && buf[1] == 0x42) {
        char *p = 0;
        p[0] = 10;
    }
#endif

By inserting additional calls to transmit the output of the CB will clearly indicate if compilation process included the #ifdef PATCHED_<n> block or the else block. Similarly, by inserting additional _terminate calls, the CB will exit when this debugging error condition is met, and the CB return code can be tested to determine which of the _terminate calls was reached. In fact, _terminate can be used in this fashion to debug without using transmit at all.

After building the CB with the two additional transmit and _terminate calls, we can observe the effects via output and via return code:

$ ./bin/LUNGE_00001 > /dev/null
Reached point 5
$ echo $?
5
$ ./bin/LUNGE_00001_patched > /dev/null
Reached point 4
$ echo $?
4

In the unpatched version of the CB, we reach the transmit indicating we 'Reached point 5' and the error code stemming from the _terminate is also 5 (as reported by $?). Similarly, we see point 4 and code 4 for the patched version of the CB.

Note: If you already have pollers or PoVs in place, altering the output of a CB might cause these to no longer work. Since the debugging output is temporary code that will later be removed, it is probably not worth altering pollers and PoVs to match the additional debugging output. For this reason, instead of issuing a generic make CB authors might consider using the build make target: make build. The build target will compile the unpatched and patched CBs, but will not perform additional steps required prior to submission of the final CB.

Using GDB to debug a CB

GDB has been modified to work with CBs. Simply load a CB into GDB by starting GDB with the CB as a command line argument. The code listing below demonstrates loading a CB, setting a breakpoint by name, executing the CB and then single-stepping after GDB encounters the breakpoint. Since the CB has not been stripped, symbols are loaded from the CB and the breakpoint can be set by name (main) and line source information is provided while single stepping.

$ gdb LUNGE_00001
GNU gdb (GDB) 7.4.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from bin/LUNGE_00001...done.
(gdb) break main
Breakpoint 1 at 0x8048147: file src/service.c, line 11.
(gdb) r
Starting program: bin/LUNGE_00001

Breakpoint 1, main () at src/service.c:11
11      ret = transmit_all(STDOUT, STR1, sizeof(STR1) - 1);
(gdb) s
transmit_all (fd=1, buf=0x80482cc "This implements a simple echo service\n", size=38) at lib/libc.c:5
5       size_t sent = 0;
(gdb) s
6       size_t sent_now = 0;
(gdb) s
9       if (!buf)

This sample service has a conditional, intentional bug at line 26:

$ head -n 29 src/service.c | tail -n 4
  if (buf[0] == 0x41 && buf[1] == 0x42) {
      char *p = 0;
      p[0] = 10;
  }

The CB can be executed from GDB, and the write of AB to reach the bug can be provided interactively. Here we set a breakpoint on the condition around the basic block containing the bug (by line number - 'line 25'). First we provide input that fails the condition (afd), then we provide input that passes the condition (AB). When GDB encounters the null pointer assignment, GDB catches SIGSEGV and prompts the user about the signal. When debugging a CB during development, a binary with debugging symbols is available in the directory build/release/bin/. If this binary is used, GDB can show the source code surrounding the line that caused the SIGSEGV.

$ gdb LUNGE_00001
GNU gdb (GDB) 7.4.1
...
Reading symbols from bin/LUNGE_00001...done.
(gdb) break 25
Breakpoint 1 at 0x80481e1: file src/service.c, line 25.

(gdb) run
Starting program: bin/LUNGE_00001
This implements a simple echo service
afd

Breakpoint 1, main () at src/service.c:26
26          if (buf[0] == 0x41 && buf[1] == 0x42) {
(gdb) disp buf
1: buf = "afd\n", '\000' <repeats 1019 times>
(gdb) c
Continuing.
afd
AB

Breakpoint 1, main () at src/service.c:26
26          if (buf[0] == 0x41 && buf[1] == 0x42) {
1: buf = "AB\n", '\000' <repeats 1020 times>
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x08048205 in main () at src/service.c:28
28              p[0] = 10;
1: buf = "AB\n", '\000' <repeats 1020 times>

(gdb) list src/service.c:28
23
24	#ifdef PATCHED_1
25	#else
26          if (buf[0] == 0x41 && buf[1] == 0x42) {
27              char *p = 0;
28              p[0] = 10;
29          }
30	#endif
31
32          ret = transmit_all(STDOUT, buf, size);
(gdb)

The sample service is purposefully simple, but we can still observe a simple backtrace by pausing execution at transmit which will be called via the library function transmit_all:

$ gdb LUNGE_00001
GNU gdb (GDB) 7.4.1
...
Reading symbols from bin/LUNGE_00001...done.
(gdb) break transmit
Breakpoint 1 at 0x804808d
(gdb) run
Starting program: bin/LUNGE_00001

Breakpoint 1, 0x0804808d in transmit ()
(gdb) bt
#0  0x0804808d in transmit ()
#1  0x080482a4 in transmit_all (fd=1, buf=0x80482cc "This implements a simple echo service\n", size=38) at lib/libc.c:16
#2  0x08048163 in main () at src/service.c:11

GDB can also show a disassembly of the function:

(gdb) disassemble transmit_all
Dump of assembler code for function transmit_all:
   0x0804824a <+0>:	push   %ebp
   0x0804824b <+1>:	mov    %esp,%ebp
   0x0804824d <+3>:	sub    $0x28,%esp
=> 0x08048250 <+6>:	movl   $0x0,-0xc(%ebp)
   0x08048257 <+13>:	movl   $0x0,-0x14(%ebp)
   0x0804825e <+20>:	cmpl   $0x0,0xc(%ebp)
   0x08048262 <+24>:	jne    0x804826b <transmit_all+33>
   0x08048264 <+26>:	mov    $0x1,%eax
   0x08048269 <+31>:	jmp    0x80482c7 <transmit_all+125>
   ...

GDB may also be used to view the contents of the secret flag page

$ gdb -q bin/CADET_00003
Reading symbols from bin/CADET_00003...(no debugging symbols found)...done.
(gdb) info files
Symbols from "/home/vagrant/CADET_00003/bin/CADET_00003".
Local exec file:
       `/home/vagrant/CADET_00003/bin/CADET_00003', file type elf32-i386.
       Entry point: 0x80485fc
       0x080480a0 - 0x08048700 is .text
       0x08048700 - 0x08048796 is .rodata
       0x08049798 - 0x0805d98c is .data
(gdb) b *0x80485fc
Breakpoint 1 at 0x80485fc
(gdb) run
Starting program: /home/vagrant/CADET_00003/bin/CADET_00003

Breakpoint 1, 0x080485fc in ?? ()
(gdb) info proc map
process 5714
Mapped address spaces:

       Start Addr   End Addr       Size     Offset objfile
        0x8048000  0x8049000     0x1000        0x0 /home/vagrant/CADET_00003/bin/CADET_00003
        0x8049000  0x805e000    0x15000        0x0 /home/vagrant/CADET_00003/bin/CADET_00003
       0x4347c000 0x4347d000     0x1000        0x0  <== This is the secret page (only anonymous page at start)
       0xbaa8b000 0xbaaab000    0x20000        0x0 [stack]
(gdb) x /16xb 0x4347c000
0x4347c000:     0x4c    0xf5    0x57    0x33    0xa1    0xb0    0xd3    0x5f
0x4347c008:     0x5f    0x0c    0xf2    0xca    0x5e    0x93    0x29    0x0c
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/vagrant/CADET_00003/bin/CADET_00003

Breakpoint 1, 0x080485fc in ?? ()
(gdb) info proc map
process 5718
Mapped address spaces:

       Start Addr   End Addr       Size     Offset objfile
        0x8048000  0x8049000     0x1000        0x0 /home/vagrant/CADET_00003/bin/CADET_00003
        0x8049000  0x805e000    0x15000        0x0 /home/vagrant/CADET_00003/bin/CADET_00003
       0x4347c000 0x4347d000     0x1000        0x0  <== This is the secret page (only anonymous page at start)
       0xbaa8b000 0xbaaab000    0x20000        0x0 [stack]
(gdb) x /16xb 0x4347c000
0x4347c000:     0xa1    0x94    0x0e    0x63    0xf1    0xf5    0xf9    0x62
0x4347c008:     0xcd    0x5c    0x8c    0x87    0x06    0xc9    0xec    0xfa
(gdb)

Using core files with GDB

The CGC kernel is capable of creating CGC core files when a CB crashes. ulimit for core files is typically set to 0, to enable core dumps this ulimit must be non-zero (recommend: unlimited). Valid core dumps cannot be created on folders shared with the host using VirtualBox. The core file will be created, but will have a zero length (use -d option to cb-server as below).

$ ulimit -c
0
$ ulimit -c unlimited
$ ulimit
unlimited

If a CB crashes (and the ulimit for core files is non-zero), a file named core will be created (e.g. core will be dumped).

$ ./LUNGE_00001
This implements a simple echo service
AB
Segmentation fault (core dumped)
$ file core
core: CGC 32-bit LSB core file (CGC/Linux)

Such a CGC core file can be supplied to GDB along with the binary (as a second argument or via the -c option).

$ gdb LUNGE_00001 core
GNU gdb (GDB) 7.4.1
...
Reading symbols from /vagrant/cgc/trunk/challenge-sets/cqe/examples/service-template/bin/LUNGE_00001...done.
[New LWP 7512]
Core was generated by ''.
Program terminated with signal 11, Segmentation fault.
#0  0x08048205 in main () at src/service.c:28
28              p[0] = 10;

Using GDB to attach to a running process

Locate the PID of the already running process.

$ ps -ef | grep LUNGE_00001
root      7621  7586  0 03:53 pts/0    00:00:00 [LUNGE_00001]

Then use GDB to attach to the process

$ gdb
GNU gdb (GDB) 7.4.1
...
(gdb) attach 7621
Attaching to process 7621
Reading symbols from bin/LUNGE_00001...done.
0x080480c6 in receive ()

Similarly, GDB can attach to processes launched via the cb-server as long as --debug is added to the command line as to disable the internal ptracing of CBs. In this example, port 5555 is specified. Note, the CB timeout should be set to an exceedingly long time for debugging purposes. In this example, a full year has been specified. netstat can be used to verify the CB is listening.

console 1:

$ cb-server -t 31536000 -p 5555 --insecure -d bin --debug LUNGE_00001

console 2:

$ netstat -ano | grep 5555
tcp        0      0 0.0.0.0:5555            0.0.0.0:*               LISTEN      off (0.00/0/0)
$ nc localhost 5555
This implements a simple echo service

console 3:

$ ps -ef | grep LUNGE_00001
vagrant   7653  3356  0 04:02 pts/0    00:00:00 cb-server -t 5000 -p 5555 --insecure -d /tmp/test LUNGE_00001
vagrant   7657  7653  0 04:04 pts/0    00:00:00 [LUNGE_00001]
$ gdb
GNU gdb (GDB) 7.4.1
...
(gdb) attach 7657
Attaching to process 7657
Reading symbols from /tmp/test/LUNGE_00001...done.
0x080480c6 in receive ()

Debugging a CFE POV

Launch the challenge binary without timing out the CB:

$ cb-server -d bin CADET_00003 --insecure -p 5535 -t 0

In another terminal, launch the POV with the

$ cb-replay-pov --host 127.0.0.1 --port 5535 pov/pov_1.pov --attach_port 2601

In another terminal, launch gdb and issue the command target :remote 2601

$ gdb -q pov/pov_1.pov
(gdb) target remote :2601
Remote debugging using :2601
Remote debugging using :2601
Reading symbols from /lib/i386-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libutil.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/python2.7/lib-dynload/_multiprocessing.so...(no debugging symbols found)...done.
Reading symbols from /usr/lib/python2.7/lib-dynload/_hashlib.so...(no debugging symbols found)...done.
Reading symbols from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/python2.7/lib-dynload/_ssl.so...(no debugging symbols found)...done.
0xb768267e in open () from /lib/i386-linux-gnu/libc.so.6
(gdb)

At this point, cb-replay-pov was waiting for gdb to attach to the process. Use c to continue once.

(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x080487a3 in _start ()
(gdb)

At this point, the POV has been loaded, but execution has not begun. You can debug the POV from here.

SEE ALSO

For information regarding the building a CB, see the building-a-cb.md walk-through

For information regarding using the cb-server, see man cb-server(1)