Debugger (Ludwig Hähne) .
Introduction
RosAsm comes with an integrated debugger which is built on top of the Win32 Debug API. When you Run / F5 your application (the debuggee) from inside RosAsm, it is automatically debugged. If you try to run a DLL, the debugger asks for a host process, which is expected to load the library.
The debugger will point out eventual exceptions in your source with the faulty instruction highlighted and a detailed exception description.
Furthermore you can set breakpoints in your source either at design time or at run time. When the debuggee encounters such a breakpoint, the OS transfers control to the debugger and halts all threads of the debuggee.
With the debug dialog you can view the current flag states, register & data label values and view the contents of the whole address space of your application. The flags are embedded in the toolbar and can be shown and hidden through its context menu (right-click on toolbar).
Exceptions
When the exception box pops up, something went wrong in your application. The debug dialog title shows 'EXCEPTION' and the exception dialog title tells about the code module in which the crash occurred. A detailed exception description is given in the text window. Furthermore the instruction that caused the exception and its address are provided. In case of access violations additional information about the address which was tried to access and the access mode is shown below.
In general, the debuggee must be terminated when an exception occurs. Take care that the debug dialog is closed when you press Terminate. Therefore be sure to analyze the cause which may have led to the crash before you exit.
When you make use of structured exception handlers (SEH) the exception is still reported but you have the chance to forward it to the handler by Pass to SEH. If the exception is handled, the debuggee continues, otherwise the exception will be reported again without the possibility to pass it to the handler.
If the exception dialog caption doesn't show the name of your application but some other module like user32.dll, the exception happened outside of your application and (hopefully) a call to a external routine is highlighted. This does not mean, however, that it isn't your fault :) Most of the time missing or wrong parameters are the reason for these crashes. Check the call stack if in doubt which parameters have been passed to the routine(s).
Another possibility is an Exception in the Non-Code Section. The instruction pointer (EIP) was corrupted and triggered an access violation when the CPU tried to execute code at an inaccessible address. Instructions that may corrupt EIP are stray jumps or a ret when a wrong number of arguments have been passed. A look at the call stack might give a hint.
Registers
The register tab gives insight to the contents of the CPU registers. The debugger checks whether MMX, SSE is supported on your machine and shows additional pages in the tab if appropriate. Segment selectors and debug register (+EIP) pages can viewed / hidden in the debug dialog menu settings. The combo-box offers various representations of the register contents, particularly useful to debug MMX/SSE code with a vector representation of mm0-7 / xmm0-7.
The general purpose registers page differs from the other pages in that it contains buttons for each entry: If EBX contains a valid 32-bit virtual address in the process address space, clicking on the EBX-button takes you to the address referenced by EBX in the memory inspector.
The register contents are all shown zeroed until an exception occurred or a breakpoint is reached.
Setting Breakpoints
You can insert Breakpoints into your source, in two ways:
Write int 3 (or int3) at the desired location. These are static breakpoints, represented by a 0CC Byte really inserted inside your PE Code Section, like any other Instruction. You cannot deactivate static breakpoints once the debuggee is running but you can switch off 'Hold on breakpoints' in the debug dialog menu settings to switch off all breakpoints.
Insert one or more dynamic breakpoint(s). These are breakpoints that are not represented inside your real Code. Instead, the Debugger inserts (and removes) them, on the fly, while your Application is being debugged. You can define such dynamic breakpoints by a simple mouse double-click, on the left margin of the source editor. When you double-click, a float menu offers you options for inserting/removing a breakpoint. If you use a small font, clicking exactly upon the very first left empty space (the margin), may be difficult. So, another option is available, for the same action: F4, that also runs the dynamic breakpoints float menu, and proposes the insertion at the beginning of the caret line. Note: When the debugger is running your application, you can add/delete dynamic breakpoints (whereas you cannot edit your source, at that time).
Flow control / Tracing
When a breakpoint is encountered the next, not yet executed instruction is highlighted. To continue you can use the Continue menu items, the toolbar buttons or the corresponding shortcuts.
Run / F6 lets the debuggee continue to run without interruption through the debugger.
Step Into / F7 executes one instruction and then transfers control back to the debugger. Take care when stepping into API calls, some Windows versions (95 family) won't like or even allow it. When stepping through external module code you will see the module name in the debug dialog caption.
If the next instruction is a 'CALL' you have the possibility to Step Over the call, which allows the debuggee to run until the call has returned. This is also possible for looped instructions 'REPxx …'. F8 is even effective if the menu item is not available, having the same effect as Step Into, sometimes referred as 'auto-step-over'.
With Return to Caller / Ctrl+F7 you can step out of the current code and return to the caller in the process' code. This is useful if you're lost in deeply nested calls, or outside of the process' code. It won't work if the current code was called by the OS, like 'Main' or any 'WndProc'.
Terminate Debuggee / Ctrl+F6 lets you kill the debuggee at any time. First it kindly asks the debuggee to exit, if this does not happen within a few seconds the debuggee is terminated the hard way. It's also used to close the debuggee after an exception has occurred.
Source editor integration
RosAsm debugger operates on the source level. What does that mean in the context of assembly language? It means that you have full access to all symbols (code & data labels, equates) and the tracing takes place in the source editor. When stepping, the instruction which is executed next is highlighted. In case of instructions which have been generated by macros or pre-parsers the statement is highlighted from which the instruction was generated. To keep track of the progress inside the statement, the disassembled instruction is shown in the caption of the debug dialog.
If single-stepping multiple instruction statements is not wished, you can switch to 'Source level stepping' in the debug dialog menu settings. In this mode the debuggee is continued until the next source statement is reached.
One of the benefits of operating on the source level, is the possibility of mouse sensitive data observation. When you move the mouse over an addressing expression (e.g. D$eax+8) in the source editor while debugging, you'll see the resolved virtual address (e.g. 010008 if eax=010000) and (if it is a valid address), the 32-bit value at this address in various data representations (hex, unsigned & signed decimal). Observable expressions start with D$, W$, B$, F$, R$, T$ and may contain registers, numbers, plain data labels & equates, segment selectors and '+', '-', '*' as operators. Q$, X$ and U$ are not yet supported. The size specifier determines the quantity and quality of the displayed memory contents. For example, D$ and F$ both reference 32-bit values but the latter is represented as floating point. Expressions which contain local labels or equates can only be observed if those belong to the procedure currently being executed. In other words, when you step through 'Foobar' you can observe the labels and equates local to Foobar in the form 'D$@Local' or the more common 'D@Local'. Examples for legal observable statements:
B$eax
F$DataLabel+ecx*4+EQUATE
W@Local+2 ; only if CurrentLabel@Local is defined
D$fs:8
To view the locals of the caller function(s) you can use the call stack described later in this document.
Data viewer
The data viewer shows all data symbols and their virtual addresses declared in your source. When selecting a symbol you can see the content with different representations in the window below the label list. The representations comprise Dword, Word, Byte sizes in Hexadecimal and Decimal (signed and unsigned) notation, floating point in single and double precision, and, if the data stream consists only of printable chars, the ASCII representation.
When right-clicking on a data symbol you can choose to view the content in the memory inspector, or, if the Dword content of the data is a valid address in the process' address space you can view the referenced memory. You can also search the declaration in the source, change the sort order of the symbols (by name, by address) or set watchpoints.
Watchpoints
Watchpoints can be assigned to data symbols. They are useful to observe write and/or read accesses to data, therefore they are sometimes referred as data breakpoints. To set a watchpoint, right click the symbol you want to observe in the data viewer and select 'Break On Write Access' or 'Break On Read/Write Access'.
Watched data symbols are highlighted red (write) or orange (read/write). In the current implementation you cannot set multiple Watchpoints at the same time. Therefore, if you assign another watchpoint to a different symbol you will lose the old watchpoint.
When a watched access is observed the debuggee is halted, the title shows 'WP ...', the data viewer is activated and the watched data symbol is selected. Some implementation specific details:
Watchpoints utilize hardware debug register that are not handled correctly under old Windows version. Do not use watchpoints on Windows 95/98!
Access means, that at least one of the first 4 bytes (starting at the data address) is written to or read from.
Watchpoints only work on Dword aligned data.
Memory inspector
With the memory inspector you can view the memory contents of the allocated memory of your process. The memory is displayed in 4kB chunks which corresponds to the typical page size on x86 systems. The edit box shows the virtual address of the page in hex notation. Each list item contains an offset (e.g. +3F8) and the memory contents at this address.
To view content at a specific address just enter it in the edit box and press return or use the virtual page table to select another region. You can also use segment overrides: e.g. FS:8 displays the TEB and goes to offset 8.
The memory is shown aligned on 8 byte boundaries.
Call stack
The call stack shows the called procedures (labels) along with their parameters and local data. As the name implies it is derived from the stack content. When right-clicking on a function name you can show the invocation or declaration. The call stack is built using advanced interpretation mechanisms and should also show function calls inside modules, functions which don't setup stack-frames, ... However, it is only an interpretation. If your code or the modules you use make dynamic stack allocations (sub esp eax) or use jump tables the success rate will drop significantly. (It also does not handle spaghetti code very well) .
Best results are achieved if you follow these rules:
Always use ret to return to the caller and to remove parameters from the stack
Only use jumps to navigate inside your functions and not across your whole source
Adhere to the standard code sequence to enter functions with local data (push ebp | mov ebp esp | sub esp x)
Function calls which belong to different modules (referenced code is outside your source) are grayed for clarity. If the information given is yet too detailed you can filter the output by right-clicking on any function name and selecting 'Hide module calls' or 'Hide intra-module calls'.
Debug log
The traditional way to debug code when no debugger is available is to log information to the console or a file. This might even make sense if using a debugger: For example, when the applications working is time-dependent and halting the program for inspection is not feasible because it would tamper with the output. Win32 offers a function for applications to pass strings to a possibly attached debugger: 'OutputDebugString'. When the debuggee calls 'OutputDebugString' the debugger is invoked and adds the string to the log tab and a log file is created aside the application with the name '[AppName]_dbg.log'.
call 'Kernel32.OutputDebugString' {'Hello big brother' 0}
For convenience the log tab also lists mapped & unmapped modules and the creation and destruction of threads. Note that 'OutputDebugString' causes a context-switch to the debugger and thus is an expensive operation.
Address space
The address space tree shows all user accessible virtual memory pages of the debuggee. These are comprised of the mapped PE, the process environment block (PEB), the thread environment blocks (TEB), the stacks, the imported modules, the modules loaded by LoadLibrary, memory allocated by VirtualAlloc and the environment.
The root nodes can be regarded as the 'logical groups' in which the memory was reserved while the leaf nodes represent the actual 4kB pages represented through the virtual start address and page properties (eXecute, Read, Write, Copy on write, Guard, No cache). When you double-click on a leaf node the page is loaded in the memory inspector.
~~~~~~~