SegfaultΒΆ

In case you experience a segfault, you will see something like this:

This is the end. This software just had a segmentation fault. The bug you encountered may even be exploitable. If you want to assist in fixing the bug, please send the backtrace below to nepenthesdev@gmail.com. You can create better backtraces with gdb, for more information visit http://dionaea.carnivore.it/#segfault Once you read this message, your tty may be broken, simply type reset, so it will come to life again

/opt/dionaea/bin/dionaea(sigsegv_backtrace_cb+0x20)[0x805c11e] [0x70d420] /opt/dionaea/lib/libemu/libemu.so.2(emu_env_w32_eip_check+0x94)[0x186974] /opt/dionaea/lib/dionaea/emu.so(run+0x39)[0x89cced] /opt/dionaea/lib/dionaea/emu.so(profile+0xbb)[0x89db88] /opt/dionaea/lib/dionaea/emu.so(proc_emu_on_io_in+0x1e1)[0x89bfc5] /opt/dionaea/bin/dionaea(recurse_io_process+0x31)[0x805df4a] /opt/dionaea/bin/dionaea(processors_io_in_thread+0x85)[0x805e08d] /opt/dionaea/bin/dionaea(threadpool_wrapper+0x2e)[0x805c99a] /opt/dionaea/lib/libglib-2.0.so.0[0xaa9498] /opt/dionaea/lib/libglib-2.0.so.0[0xaa7a2f] /lib/libpthread.so.0[0xd8973b] /lib/libc.so.6(clone+0x5e)[0x2b3cfe]

While the backtrace itself gives an idea what might be wrong, it does not fix the problem. To fix the problem, the logfiles usually help, as dionaea is very verbose by default. Below are some hints how to get started with debugging, click here <#support> for assistance.

debugging

Valgrind

Valgrind does a great job, here is how I use it:

valgrind -v –leak-check=full –leak-resolution=high –show-reachable=yes –log-file=dionaea-debug.log /opt/dionaea/bin/dionaea –my-dionaea-options

gdb

logfile assisted

For the above example, I was able to scrape the shellcode from the logfile, and run it in libemu, without involving dionaea at all, reducing the problem.

gdb /opt/dionaea/bin/sctest (gdb) run -S -s 10000000 -g < sc.bin Starting program: /media/sda4/opt64/dionaea/bin/sctest -S -s 10000000 -g < sc.bin

Once it crashed, I retrieved a full backtrace:

Program received signal SIGSEGV, Segmentation fault. env_w32_hook_GetProcAddress (env=0x629a30, hook=<value optimized out>) at environment/win32/env_w32_dll_export_kernel32_hooks.c:545 545 struct emu_env_hook *hook = (struct emu_env_hook *)ehi->value;

(gdb) bt full #0 env_w32_hook_GetProcAddress (env=0x629a30, hook=<value optimized out>) at environment/win32/env_w32_dll_export_kernel32_hooks.c:545

dll = 0x6366f0 ehi = <value optimized out> hook = <value optimized out> c = 0x611180 mem = <value optimized out> eip_save = <value optimized out> module = 2088763392 p_procname = 4289925 procname = <value optimized out>
#1 0x00007ffff7b884fb in emu_env_w32_eip_check (env=0x629a30) at environment/win32/emu_env_w32.c:306
dll = <value optimized out> ehi = <value optimized out> hook = 0x64c5b0 eip = <value optimized out>
#2 0x0000000000403995 in test (e=0x60f0e0) at sctestmain.c:277
hook = 0xe2 ev = 0x0 iv = <value optimized out> cpu = 0x611180 mem = <value optimized out> env = 0x629a30 na = <value optimized out> j = 7169 last_vertex = 0x0 graph = 0x0 eh = 0x0 ehi = 0x0 ret = <value optimized out> eipsave = 2088807840
#3 0x00000000004044e4 in main (argc=5, argv=0x7fffffffe388) at sctestmain.c:971
e = <value optimized out>

In this case, the problem was a bug in libemu.

gdb dump memory

Once again, it broke, and we got a backtrace:

#0 0xb70b0b57 in emu_queue_enqueue (eq=0xb3da0918, data=0x4724ab) at emu_queue.c:63
eqi = (struct emu_queue_item *) 0x0
#1 0xb70b15d1 in emu_shellcode_run_and_track (e=0xb4109cd0, data=0xb411c698 “”, datasize=<value optimized out>, eipoffset=<value optimized out>,
steps=256, etas=0xb410cd60, known_positions=0xb3d7a810, stats_tested_positions_list=0xb3da3bf0, brute_force=true) at emu_shellcode.c:408
current_pos_ti_diff = (struct emu_tracking_info *) 0x88c3c88 current_pos_ht = <value optimized out> current_pos_v = <value optimized out> current_pos_satii = (struct emu_source_and_track_instr_info *) 0xb407e7f8 bfs_queue = (struct emu_queue *) 0xb3e17668 ret = 4662443 eipsave = <value optimized out> hook = <value optimized out> j = 4 es = <value optimized out> eli = (struct emu_list_item *) 0xb3e17658 cpu = (struct emu_cpu *) 0xb4109ab0 mem = (struct emu_memory *) 0xb410c3a0 eq = (struct emu_queue *) 0xb3da0918 env = (struct emu_env *) 0xb3e10208 eli = (struct emu_list_item *) 0x4724ab
#2 0xb70b1a2a in emu_shellcode_test (e=0xb4109cd0, data=0xb411c698 “”, size=<value optimized out>) at emu_shellcode.c:546
es = (struct emu_stats *) 0xb3d92b28 new_results = (struct emu_list_root *) 0xb3da3bf0 offset = <value optimized out> el = (struct emu_list_root *) 0xb4100510 etas = (struct emu_track_and_source *) 0xb410cd60 eh = (struct emu_hashtable *) 0xb3d7a810 eli = (struct emu_list_item *) 0xb3d92b40 results = (struct emu_list_root *) 0xb3d82850 es = <value optimized out> __PRETTY_FUNCTION__ = “emu_shellcode_test”
#3 0xb712140c in proc_emu_on_io_in (con=0x8864b58, pd=0x87dc388) at detect.c:145
e = (struct emu *) 0xb4109cd0 ctx = (struct emu_ctx *) 0x87a2400 offset = 14356 streamdata = (void *) 0xb411c698 size = 8196 ret = 0 __PRETTY_FUNCTION__ = “proc_emu_on_io_in”

#4 0x0805e8be in recurse_io_process (pd=0x87dc388, con=0x8864b58, dir=bistream_in) at processor.c:167 No locals. #5 0x0805ea01 in processors_io_in_thread (data=0x8864b58, userdata=0x87dc388) at processor.c:197

con = (struct connection *) 0x8864b58 pd = (struct processor_data *) 0x87dc388 __PRETTY_FUNCTION__ = “processors_io_in_thread”
#6 0x0805d2da in threadpool_wrapper (data=0x87d7bd0, user_data=0x0) at threads.c:49
t = (struct thread *) 0x87d7bd0 timer = (GTimer *) 0xb4108540
#7 0xb77441f6 in g_thread_pool_thread_proxy (data=0x83db460) at gthreadpool.c:265
task = (gpointer) 0x87d7bd0 pool = (GRealThreadPool *) 0x83db460
#8 0xb7742b8f in g_thread_create_proxy (data=0x83dc7d0) at gthread.c:635
__PRETTY_FUNCTION__ = “g_thread_create_proxy”

#9 0xb76744c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 No symbol table info available. #10 0xb75f36de in clone () from /lib/i686/cmov/libc.so.6 No symbol table info available.

Again, it was a bug in libemu, an unbreakable loop consuming all memory. To reproduce, we have to dump the tested buffer, therefore we need the buffers address and size. Luckily the size is noted in frame #2 as 8196 and and the data address is a parameter which got not optimized out for frame #2.

dump binary memory /tmp/sc.bin 0xb411c698 0xb411e89c

Afterwards, debugging libemu by feeding the data into sctest is easy.

I’ve had fun with objgraph and gdb debugging reference count leaks in python too, here <http://carnivore.it/2009/12/23/arcane_bugs> is the writeup.

gdb python3 embedded

Sometimes, there is something wrong with the python scripts, but gdb does not provide any useful output:

bt full #12 0xb765f12d in PyEval_EvalFrameEx (f=0x825998c, throwflag=0) at Python/ceval.c:2267

stack_pointer = (PyObject **) 0x8259af0 next_instr = (unsigned char ) 0x812fabf “m’” opcode = 100 oparg = <value optimized out> why = 3071731824 err = 1 x = (PyObject *) 0xb7244aac v = <value optimized out> w = (PyObject *) 0xadb5e4dc u = (PyObject *) 0xb775ccb0 freevars = (PyObject *) 0x8259af0 retval = (PyObject *) 0x0 tstate = (PyThreadState *) 0x809aab0 co = (PyCodeObject *) 0xb717b800 instr_ub = -1 instr_lb = 0 instr_prev = -1 first_instr = (unsigned char *) 0x812f918 “t” names = (PyObject *) 0xb723f50c consts = (PyObject *) 0xb71c9f7c opcode_targets = {0xb765d202, 0xb765f60a, 0xb766133a, 0xb76612db, 0xb7661285, 0xb7661222, 0xb765d202, 0xb765d202, 0xb765d202, 0xb76611dd,

0xb766114b, 0xb76610b9, 0xb766100f, 0xb765d202, 0xb765d202, 0xb7660f7d, 0xb765d202, 0xb765d202, 0xb765d202, 0xb7660eb7, 0xb7660dfb, 0xb765d202, 0xb7660d30, 0xb7660c65, 0xb7660ba9, 0xb7660aed, 0xb7660a31, 0xb7660975, 0xb76608b9, 0xb76607fd, 0xb765d202 <repeats 24 times>, 0xb7660736, 0xb766066b, 0xb76605af, 0xb76604f3, 0xb765d202, 0xb7660437, 0xb766035d, 0xb76602ad, 0xb7661aba, 0xb76619fe, 0xb7661942, 0xb7661886, 0xb7661b76, 0xb76614a8, 0xb7661413, 0xb766138e, 0xb766171f, 0xb76616e6, 0xb765d202, 0xb765d202, 0xb765d202, 0xb766162a, 0xb766156e, 0xb76601f1, 0xb7660135, 0xb76617ca, 0xb7660120, 0xb765fff7, 0xb765d202, 0xb765fd72, 0xb765fc6e, 0xb765d202, 0xb765fc1d, 0xb765fe17, 0xb765fd90, 0xb765fec0, 0xb765fb41, 0xb765fadc, 0xb765f9ed, 0xb765f94d, 0xb765f8be, 0xb765f7e3, 0xb765f779, 0xb765f6bd, 0xb765f66c, 0xb765ef1d, 0xb765eea2, 0xb765ede1, 0xb765ed1a, 0xb765ec35, 0xb765ebc3, 0xb765eb30, 0xb765ea69, 0xb765f1c7, 0xb765f027, 0xb765f560, 0xb765efc1, 0xb76630e3, 0xb766310c, 0xb765e64c, 0xb765e592, 0xb765f49a, 0xb765f3de, 0xb765d202, 0xb765d202, 0xb765f39e, 0xb7663135, 0xb766315f, 0xb765e9cb, 0xb765d202, 0xb765e948, 0xb765e8bb, 0xb765e817, 0xb765d202, 0xb765d202, 0xb765d202, 0xb765d2ae, 0xb765e3e0, 0xb7663275, 0xb765e1a2, 0xb766324e, 0xb765e0ba, 0xb765e01e, 0xb765df74, 0xb765d202, 0xb765d202, 0xb7663189, 0xb76631d3, 0xb7663220, 0xb765e149, 0xb765d202, 0xb765de09, 0xb765dec0, 0xb765f2c0, 0xb765d202 <repeats 108 times>}

#13 0xb7664ac0 in PyEval_EvalCodeEx (co=0xb717b800, globals=0xb7160b54, locals=0x0, args=0x84babb8, argcount=9, kws=0x0, kwcount=0, defs=0xb719e978,
defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3198
f = (PyFrameObject ) 0x825998c retval = <value optimized out> freevars = (PyObject *) 0x8259af0 tstate = (PyThreadState *) 0x809aab0 x = <value optimized out> u = <value optimized out>

Luckily python3 ships with some gdb macros, which assist in dealing with this mess. You can grab them over here <http://svn.python.org/view/python/tags/r311/Misc/gdbinit?view=markup>, place them to ~/.gdbinit, where ~ is the homedirectory of the user dionaea runs as. If you get /warning: not using untrusted file “/home/user/.gdbinit”/ you are running gdb via sudo, and the file /home/user/.gdbinit has to be owned by root. If you are running as root, and you get /Program received signal SIGTTOU, Stopped (tty output)./, run stty -nostop before running gdb, reattach the process with fg, close gdb properly, and start over.

Once you got the macros loaded properly at gdb startup, set a breakpoint on PyEval_EvalFrameEx after dionaea loaded everything:

break PyEval_EvalFrameEx

Then we have some useful macros for gdb:

up pyframev

pyframev combines the output of pyframe and pylocals.

Be aware you can segfault dionaea now from within gdb, going up, out of the python call stack and calling some of the macros can and in most cases will segfault dionaea, therefore use backtrace to make sure you are still within valid frames. We can’t use pystack or pystackv as they rely on Py_Main, which is an invalid assumption for embedded python.