Segfault¶
In case you experience a segfault, you will see something like this:
This is the end. This software just had a segmentation fault. The bug you encountered may even be exploitable. If you want to assist in fixing the bug, please send the backtrace below to nepenthesdev@gmail.com. You can create better backtraces with gdb, for more information visit http://dionaea.carnivore.it/#segfault Once you read this message, your tty may be broken, simply type reset, so it will come to life again:
/opt/dionaea/bin/dionaea(sigsegv_backtrace_cb+0x20)[0x805c11e]
[0x70d420]
/opt/dionaea/lib/libemu/libemu.so.2(emu_env_w32_eip_check+0x94)[0x186974]
/opt/dionaea/lib/dionaea/emu.so(run+0x39)[0x89cced]
/opt/dionaea/lib/dionaea/emu.so(profile+0xbb)[0x89db88]
/opt/dionaea/lib/dionaea/emu.so(proc_emu_on_io_in+0x1e1)[0x89bfc5]
/opt/dionaea/bin/dionaea(recurse_io_process+0x31)[0x805df4a]
/opt/dionaea/bin/dionaea(processors_io_in_thread+0x85)[0x805e08d]
/opt/dionaea/bin/dionaea(threadpool_wrapper+0x2e)[0x805c99a]
/opt/dionaea/lib/libglib-2.0.so.0[0xaa9498]
/opt/dionaea/lib/libglib-2.0.so.0[0xaa7a2f]
/lib/libpthread.so.0[0xd8973b]
/lib/libc.so.6(clone+0x5e)[0x2b3cfe]
While the backtrace itself gives an idea what might be wrong, it does not fix the problem. To fix the problem, the logfiles usually help, as dionaea is very verbose by default. Below are some hints how to get started with debugging, click here <#support> for assistance.
debugging
Valgrind¶
Valgrind does a great job, here is how I use it:
valgrind -v --leak-check=full --leak-resolution=high --show-reachable=yes \
--log-file=dionaea-debug.log /opt/dionaea/bin/dionaea --my-dionaea-options
gdb
logfile assisted
For the above example, I was able to scrape the shellcode from the logfile, and run it in libemu, without involving dionaea at all, reducing the problem:
gdb /opt/dionaea/bin/sctest
(gdb) run -S -s 10000000 -g < sc.bin
Starting program: /media/sda4/opt64/dionaea/bin/sctest -S -s 10000000 -g < sc.bin
Once it crashed, I retrieved a full backtrace:
Program received signal SIGSEGV, Segmentation fault.
env_w32_hook_GetProcAddress (env=0x629a30, hook=<value optimized out>) at environment/win32/env_w32_dll_export_kernel32_hooks.c:545
545 struct emu_env_hook *hook = (struct emu_env_hook *)ehi->value;
(gdb) bt full
#0 env_w32_hook_GetProcAddress (env=0x629a30, hook=<value optimized out>) at environment/win32/env_w32_dll_export_kernel32_hooks.c:545
dll = 0x6366f0
ehi = <value optimized out>
hook = <value optimized out>
c = 0x611180
mem = <value optimized out>
eip_save = <value optimized out>
module = 2088763392
p_procname = 4289925
procname = <value optimized out>
#1 0x00007ffff7b884fb in emu_env_w32_eip_check (env=0x629a30) at environment/win32/emu_env_w32.c:306
dll = <value optimized out>
ehi = <value optimized out>
hook = 0x64c5b0
eip = <value optimized out>
#2 0x0000000000403995 in test (e=0x60f0e0) at sctestmain.c:277
hook = 0xe2
ev = 0x0
iv = <value optimized out>
cpu = 0x611180
mem = <value optimized out>
env = 0x629a30
na = <value optimized out>
j = 7169
last_vertex = 0x0
graph = 0x0
eh = 0x0
ehi = 0x0
ret = <value optimized out>
eipsave = 2088807840
#3 0x00000000004044e4 in main (argc=5, argv=0x7fffffffe388) at sctestmain.c:971
e = <value optimized out>
In this case, the problem was a bug in libemu.
gdb dump memory
Once again, it broke, and we got a backtrace:
#0 0xb70b0b57 in emu_queue_enqueue (eq=0xb3da0918, data=0x4724ab) at emu_queue.c:63
eqi = (struct emu_queue_item *) 0x0
#1 0xb70b15d1 in emu_shellcode_run_and_track (e=0xb4109cd0, data=0xb411c698 "", datasize=<value optimized out>, eipoffset=<value optimized out>,
steps=256, etas=0xb410cd60, known_positions=0xb3d7a810, stats_tested_positions_list=0xb3da3bf0, brute_force=true) at emu_shellcode.c:408
current_pos_ti_diff = (struct emu_tracking_info *) 0x88c3c88
current_pos_ht = <value optimized out>
current_pos_v = <value optimized out>
current_pos_satii = (struct emu_source_and_track_instr_info *) 0xb407e7f8
bfs_queue = (struct emu_queue *) 0xb3e17668
ret = 4662443
eipsave = <value optimized out>
hook = <value optimized out>
j = 4
es = <value optimized out>
eli = (struct emu_list_item *) 0xb3e17658
cpu = (struct emu_cpu *) 0xb4109ab0
mem = (struct emu_memory *) 0xb410c3a0
eq = (struct emu_queue *) 0xb3da0918
env = (struct emu_env *) 0xb3e10208
eli = (struct emu_list_item *) 0x4724ab
#2 0xb70b1a2a in emu_shellcode_test (e=0xb4109cd0, data=0xb411c698 "", size=<value optimized out>) at emu_shellcode.c:546
es = (struct emu_stats *) 0xb3d92b28
new_results = (struct emu_list_root *) 0xb3da3bf0
offset = <value optimized out>
el = (struct emu_list_root *) 0xb4100510
etas = (struct emu_track_and_source *) 0xb410cd60
eh = (struct emu_hashtable *) 0xb3d7a810
eli = (struct emu_list_item *) 0xb3d92b40
results = (struct emu_list_root *) 0xb3d82850
es = <value optimized out>
__PRETTY_FUNCTION__ = "emu_shellcode_test"
#3 0xb712140c in proc_emu_on_io_in (con=0x8864b58, pd=0x87dc388) at detect.c:145
e = (struct emu *) 0xb4109cd0
ctx = (struct emu_ctx *) 0x87a2400
offset = 14356
streamdata = (void *) 0xb411c698
size = 8196
ret = 0
__PRETTY_FUNCTION__ = "proc_emu_on_io_in"
#4 0x0805e8be in recurse_io_process (pd=0x87dc388, con=0x8864b58, dir=bistream_in) at processor.c:167
No locals.
#5 0x0805ea01 in processors_io_in_thread (data=0x8864b58, userdata=0x87dc388) at processor.c:197
con = (struct connection *) 0x8864b58
pd = (struct processor_data *) 0x87dc388
__PRETTY_FUNCTION__ = "processors_io_in_thread"
#6 0x0805d2da in threadpool_wrapper (data=0x87d7bd0, user_data=0x0) at threads.c:49
t = (struct thread *) 0x87d7bd0
timer = (GTimer *) 0xb4108540
#7 0xb77441f6 in g_thread_pool_thread_proxy (data=0x83db460) at gthreadpool.c:265
task = (gpointer) 0x87d7bd0
pool = (GRealThreadPool *) 0x83db460
#8 0xb7742b8f in g_thread_create_proxy (data=0x83dc7d0) at gthread.c:635
__PRETTY_FUNCTION__ = "g_thread_create_proxy"
#9 0xb76744c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
No symbol table info available.
#10 0xb75f36de in clone () from /lib/i686/cmov/libc.so.6
No symbol table info available.
Again, it was a bug in libemu, an unbreakable loop consuming all memory. To reproduce, we have to dump the tested buffer, therefore we need the buffers address and size. Luckily the size is noted in frame #2 as 8196 and and the data address is a parameter which got not optimized out for frame #2:
dump binary memory /tmp/sc.bin 0xb411c698 0xb411e89c
Afterwards, debugging libemu by feeding the data into sctest is easy.
I’ve had fun with objgraph and gdb debugging reference count leaks in python too, here <http://carnivore.it/2009/12/23/arcane_bugs> is the writeup:
gdb python3 embedded
Sometimes, there is something wrong with the python scripts, but gdb does not provide any useful output:
bt full
#12 0xb765f12d in PyEval_EvalFrameEx (f=0x825998c, throwflag=0) at Python/ceval.c:2267
stack_pointer = (PyObject **) 0x8259af0
next_instr = (unsigned char *) 0x812fabf "m'"
opcode = 100
oparg = <value optimized out>
why = 3071731824
err = 1
x = (PyObject *) 0xb7244aac
v = <value optimized out>
w = (PyObject *) 0xadb5e4dc
u = (PyObject *) 0xb775ccb0
freevars = (PyObject **) 0x8259af0
retval = (PyObject *) 0x0
tstate = (PyThreadState *) 0x809aab0
co = (PyCodeObject *) 0xb717b800
instr_ub = -1
instr_lb = 0
instr_prev = -1
first_instr = (unsigned char *) 0x812f918 "t"
names = (PyObject *) 0xb723f50c
consts = (PyObject *) 0xb71c9f7c
opcode_targets = {0xb765d202, 0xb765f60a, 0xb766133a, 0xb76612db, 0xb7661285, 0xb7661222, 0xb765d202, 0xb765d202, 0xb765d202, 0xb76611dd,
0xb766114b, 0xb76610b9, 0xb766100f, 0xb765d202, 0xb765d202, 0xb7660f7d, 0xb765d202, 0xb765d202, 0xb765d202, 0xb7660eb7, 0xb7660dfb, 0xb765d202,
0xb7660d30, 0xb7660c65, 0xb7660ba9, 0xb7660aed, 0xb7660a31, 0xb7660975, 0xb76608b9, 0xb76607fd, 0xb765d202 <repeats 24 times>, 0xb7660736, 0xb766066b,
0xb76605af, 0xb76604f3, 0xb765d202, 0xb7660437, 0xb766035d, 0xb76602ad, 0xb7661aba, 0xb76619fe, 0xb7661942, 0xb7661886, 0xb7661b76, 0xb76614a8,
0xb7661413, 0xb766138e, 0xb766171f, 0xb76616e6, 0xb765d202, 0xb765d202, 0xb765d202, 0xb766162a, 0xb766156e, 0xb76601f1, 0xb7660135, 0xb76617ca,
0xb7660120, 0xb765fff7, 0xb765d202, 0xb765fd72, 0xb765fc6e, 0xb765d202, 0xb765fc1d, 0xb765fe17, 0xb765fd90, 0xb765fec0, 0xb765fb41, 0xb765fadc,
0xb765f9ed, 0xb765f94d, 0xb765f8be, 0xb765f7e3, 0xb765f779, 0xb765f6bd, 0xb765f66c, 0xb765ef1d, 0xb765eea2, 0xb765ede1, 0xb765ed1a, 0xb765ec35,
0xb765ebc3, 0xb765eb30, 0xb765ea69, 0xb765f1c7, 0xb765f027, 0xb765f560, 0xb765efc1, 0xb76630e3, 0xb766310c, 0xb765e64c, 0xb765e592, 0xb765f49a,
0xb765f3de, 0xb765d202, 0xb765d202, 0xb765f39e, 0xb7663135, 0xb766315f, 0xb765e9cb, 0xb765d202, 0xb765e948, 0xb765e8bb, 0xb765e817, 0xb765d202,
0xb765d202, 0xb765d202, 0xb765d2ae, 0xb765e3e0, 0xb7663275, 0xb765e1a2, 0xb766324e, 0xb765e0ba, 0xb765e01e, 0xb765df74, 0xb765d202, 0xb765d202,
0xb7663189, 0xb76631d3, 0xb7663220, 0xb765e149, 0xb765d202, 0xb765de09, 0xb765dec0, 0xb765f2c0, 0xb765d202 <repeats 108 times>}
#13 0xb7664ac0 in PyEval_EvalCodeEx (co=0xb717b800, globals=0xb7160b54, locals=0x0, args=0x84babb8, argcount=9, kws=0x0, kwcount=0, defs=0xb719e978,
defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3198
f = (PyFrameObject *) 0x825998c
retval = <value optimized out>
freevars = (PyObject **) 0x8259af0
tstate = (PyThreadState *) 0x809aab0
x = <value optimized out>
u = <value optimized out>
Luckily python3 ships with some gdb macros, which assist in dealing with this mess. You can grab them over here <http://svn.python.org/view/python/tags/r311/Misc/gdbinit?view=markup>, place them to ~/.gdbinit, where ~ is the homedirectory of the user dionaea runs as. If you get /warning: not using untrusted file “/home/user/.gdbinit”/ you are running gdb via sudo, and the file /home/user/.gdbinit has to be owned by root. If you are running as root, and you get /Program received signal SIGTTOU, Stopped (tty output)./, run stty -nostop before running gdb, reattach the process with fg, close gdb properly, and start over.
Once you got the macros loaded properly at gdb startup, set a breakpoint on PyEval_EvalFrameEx after dionaea loaded everything:
break PyEval_EvalFrameEx
Then we have some useful macros for gdb:
up
pyframev
pyframev combines the output of pyframe and pylocals.
Be aware you can segfault dionaea now from within gdb, going up, out of the python call stack and calling some of the macros can and in most cases will segfault dionaea, therefore use backtrace to make sure you are still within valid frames. We can’t use pystack or pystackv as they rely on Py_Main, which is an invalid assumption for embedded python.