mdb [-fkmuwyAFKMSUW] [±o option] [-b VM] [-p pid] [-s distance] [-I path] [-L path] [-P prompt] [-R root] [-V dis-version] [-e expr] [object [core] | core | suffix]
Debugging is the process of analyzing the execution and state of a software program in order to remove defects. Traditional debugging tools provide facilities for execution control so that programmers can re-execute programs in a controlled environment and display the current state of program data or evaluate expressions in the source language used to develop the program.
Unfortunately, these techniques are often inappropriate for debugging complex software systems such as an operating system, where bugs might not be reproducible and program state is massive and distributed, for programs that are highly optimized, have had their debug information removed, or are themselves low-level debugging tools, or for customer situations where the developer can only access post-mortem information.
mdb provides a completely customizable environment for debugging these programs and scenarios, including a dynamic module facility that programmers can use to implement their own debugging commands to perform program-specific analysis. Each mdb module can be used to examine the program in several different contexts, including live and post-mortem.
A debugger command, or dcmd (pronounced dee-command) in mdb terminology, is a routine in the debugger that can access any of the properties of the current target. mdb parses commands from standard input, and then executes the corresponding dcmds. Each dcmd can also accept a list of string or numerical arguments, as shown in the syntax description below. mdb contains a set of built-in dcmds, described below, that are always available. You can also extend the capabilities of mdb itself by writing your own dcmds, as described in the Modular Debugger Guide.
A walker is a set of routines that describe how to walk, or iterate, through the elements of a particular program data structure. A walker encapsulates the data structure's implementation from dcmds and from mdb itself. You can use walkers interactively, or use them as a primitive to build other dcmds or walkers. As with dcmds, you can extend mdb by implementing your own walkers as part of a debugger module.
A debugger module, or dmod (pronounced dee-mod), is a dynamically loaded library containing a set of dcmds and walkers. During initialization, mdb attempts to load dmods corresponding to the load objects present in the target. You can subsequently load or unload dmods at any time while running mdb. mdb ships with a set of standard dmods for debugging the kernel. The Modular Debugger Guide contains more information on developing your own debugger modules.
A macro file is a text file containing a set of commands to execute. Macro files are typically used to automate the process of displaying a simple data structure. mdb provides complete backward compatibility for the execution of macro files written for adb(1), and illumos includes a set of macro files for debugging the kernel that can be used with either tool.
A metacharacter is one of the following characters:
[ ] | ! / \ ? = > $ : ; NEWLINE SPACE TAB
A blank is a TAB or a SPACE. A word is a sequence of characters separated by one or more non-quoted metacharacters. Some of the metacharacters only function as delimiters in certain contexts, as described below. An identifier is a sequence of letters, digits, underscores, periods, or backquotes beginning with a letter, underscore, or period. Identifiers are used as the names of symbols, variables, dcmds, and walkers. Commands are delimited by a NEWLINE or semicolon ( ; ).
A dcmd is denoted by one of the following words or metacharacters:
/ \ ? = > $character :character ::identifier
dcmds named by metacharacters or prefixed by a single $ or : are provided as built-in operators, and implement complete compatibility with the command set of the legacy adb(1) utility. Once a dcmd has been parsed, the /, \, ?, =, >, $, and : characters are no longer recognized as metacharacters until the termination of the argument list.
A simple-command is a dcmd followed by a sequence of zero or more blank-separated words. The words are passed as arguments to the invoked dcmd, except as specified under Quoting and Arithmetic Expansion below. Each dcmd returns an exit status that indicates it was either successful, failed, or was invoked with invalid arguments.
A pipeline is a sequence of one or more simple commands separated by |. Unlike the shell, dcmds in mdb pipelines are not executed as separate processes. After the pipeline has been parsed, each dcmd is invoked in order from left to right. Each dcmd's output is processed and stored as described under dcmd Pipelines below. Once the left-hand dcmd is complete, its processed output is used as input for the next dcmd in the pipeline. If any dcmd does not return a successful exit status, the pipeline is aborted.
An expression is a sequence of words that is evaluated to compute a 64-bit unsigned integer value. The words are evaluated using the rules described under Arithmetic Expansion below.
pipeline [! word ...] [ ; ]
expression pipeline [! word ...] [ ; ]
expression , expression pipeline [! word ...] [ ; ]
, expression pipeline [! word ...] [ ; ]
expression [! word ...] [ ; ]
expression, expression [! word ...] [ ; ]
, expression [! word ...] [ ; ]
! word ... [ ; ]
Expressions can contain any of the following special words:
The increment is a global variable that stores the total bytes read by the last formatting dcmd. For more information on the increment, refer to the discussion of Formatting dcmds below.
Unary operators are right associative and have higher precedence than binary operators. The unary operators are:
Binary operators are left associative and have lower precedence than unary operators. The binary operators, in order of precedence from highest to lowest, are:
The following variables are defined as persistent:
In addition, the mdb kernel and process targets export the current values of the representative thread's register set as named variables. The names of these variables depend on the target's platform and instruction set architecture.
Additionally, mdb provides a private user-defined symbol table that is searched prior to any of the target symbol tables. The private symbol table is initially empty, and can be manipulated using the ::nmadd and ::nmdel dcmds. The ::nm -P option can be used to display the contents of the private symbol table. The private symbol table allows the user to create symbol definitions for program functions or data that were either missing from the original program or stripped out. These definitions are then used whenever mdb converts a symbolic name to an address, or an address to the nearest symbol.
As targets contain multiple symbol tables, and each symbol table can include symbols from multiple object files, different symbols with the same name can exist. mdb uses the backquote (`) character as a symbol name scoping operator to allow the programmer to obtain the value of the desired symbol in this situation. The programmer can specify the scope used to resolve a symbol name as either: object`name, or file`name, or object`file`name. The object identifier refers to the name of a load object. The file identifier refers to the basename of a source file that has a symbol of type STT_FILE in the specified object's symbol table. The object identifier's interpretation depends on the target type.
The mdb kernel target expects object to specify the basename of a loaded kernel module. For example, the symbol name
evaluates to the value of the _init symbol in the specfs kernel module.
The mdb process target expects object to specify the name of the executable or of a loaded shared library. It can take any of the following forms:
The process target also accepts any of the four forms described above preceded by an optional link-map id (lmid). The lmid prefix is specified by an initial "LM" followed by the link-map id in hexadecimal followed by an additional backquote. For example, the symbol name
evaluates to the value of the _init symbol in the libc.so.1 library that is loaded on link-map 0 (LM_ID_BASE). The link-map specifier can be necessary to resolve symbol naming conflicts in the event that the same library is loaded on more than one link map. For more information on link maps, refer to the Linker and Libraries Guide and dlopen(3C). Link-map identifiers are displayed when symbols are printed according to the setting of the showlmid option, as described under OPTIONS.
In the case of a naming conflict between symbols and hexadecimal integer values, mdb attempts to evaluate an ambiguous token as a symbol first, before evaluating it as an integer value. For example, the token f can either refer to the decimal integer value 15 specified in hexadecimal (the default base), or to a global variable named f in the target's symbol table. If a symbol with an ambiguous name is present, the integer value can be specified by using an explicit 0x or 0X prefix.
If module m1 were now unloaded, the next dcmd on the global definition list (m2`d) would be promoted to global visibility. The current definition of a dcmd or walker can be determined using the ::which dcmd, described below. The global definition list can be displayed using the ::which -v option.
mdb executes each dcmd in the pipeline in order from left to right. The leftmost dcmd is executed using the current value of dot, or using the value specified by an explicit expression at the start of the command. When a | operator is encountered, mdb creates a pipe (a shared buffer) between the output of the dcmd to its left and the mdb parser, and an empty list of values. As the dcmd executes, its standard output is placed in the pipe and then consumed and evaluated by the parser, as if mdb were reading this data from standard input. Each line must consist of an arithmetic expression terminated by a NEWLINE or semicolon (;). The value of the expression is appended to the list of values associated with the pipe. If a syntax error is detected, the pipeline is aborted.
When the dcmd to the left of a | operator completes, the list of values associated with the pipe is then used to invoke the dcmd to the right of the | operator. For each value in the list, dot is set to this value and the right-hand dcmd is executed. Only the rightmost dcmd in the pipeline has its output printed to standard output. If any dcmd in the pipeline produces output to standard error, these messages are printed directly to standard error and are not processed as part of the pipeline.
The editing mode also interprets the following user-defined sequences as editing commands. User defined sequences can be read or modified using the stty(1) command.
On keyboards that support an extended keypad with arrow keys, mdb interprets these keystrokes as editing commands:
>> More [<space>, <cr>, q, n, c, a] ?
The following key sequences are recognized by the pager:
n, N, NEWLINE, RETURN
q, Q, ^C, ^\
The formatting dcmds are:
In addition to dot, mdb keeps track of another global value called the increment. The increment represents the distance between dot and the address following all the data read by the last formatting dcmd. For example, if a formatting dcmd is executed with dot equal to address A, and displays a 4-byte integer, then after this dcmd completes, dot is still A, but the increment is set to 4. The + character (described under Arithmetic Expansion above) would now evaluate to the value A + 4, and could be used to reset dot to the address of the next data object for a subsequent dcmd.
Most format characters increase the value of the increment by the number of bytes corresponding to the size of the data format, shown in the table. The table of format characters can be displayed from within mdb using the ::formats dcmd. The format characters are:
|+||increment dot by the count (variable size)|
|-||decrement dot by the count (variable size)|
|B||hexadecimal int (1 byte)|
|C||character using C character notation (1 byte)|
|D||decimal signed int (4 bytes)|
|E||decimal unsigned long long (8 bytes)|
|F||double (8 bytes)|
|G||octal unsigned long long (8 bytes)|
|H||swap bytes and shorts (4 bytes)|
|I||address and disassembled instruction (variable size)|
|J||hexadecimal long long (8 bytes)|
|K||hexadecimal uintptr_t (4 or 8 bytes)|
|O||octal unsigned int (4 bytes)|
|P||symbol (4 or 8 bytes)|
|Q||octal signed int (4 bytes)|
|R||binary unsigned long long (8 bytes)|
|S||string using C string notation (variable size)|
|U||decimal unsigned int (4 bytes)|
|V||decimal unsigned int (1 byte)|
|W||default radix unsigned int (4 bytes)|
|X||hexadecimal int (4 bytes)|
|Y||decoded time32_t (4 bytes)|
|Z||hexadecimal long long (8 bytes)|
|^||decrement dot by increment * count (variable size)|
|a||dot as symbol+offset|
|b||octal unsigned int (1 byte)|
|c||character (1 byte)|
|d||decimal signed short (2 bytes)|
|e||decimal signed long long (8 bytes)|
|f||float (4 bytes)|
|g||octal signed long long (8 bytes)|
|h||swap bytes (2 bytes)|
|i||disassembled instruction (variable size)|
|j||jazzed-up binary unsigned long long (8 bytes)|
|o||octal unsigned short (2 bytes)|
|p||symbol (4 or 8 bytes)|
|q||octal signed short (2 bytes)|
|s||raw string (variable size)|
|u||decimal unsigned short (2 bytes)|
|v||decimal signed int (1 byte)|
|w||default radix unsigned short (2 bytes)|
|x||hexadecimal short (2 bytes)|
|y||decoded time64_t (8 bytes)|
|z||write whose size is inferred by CTF info (variable size)|
The /, \, and ? formatting dcmds can also be used to write to the target's virtual address space, physical address space, or object file by specifying one of the following modifiers as the first format character, and then specifying a list of words that are either immediate values or expressions enclosed in square brackets preceded by a dollar sign ($[ ]).
The write modifiers are:
The /, \, and ? formatting dcmds can also be used to search for a particular integer value in the target's virtual address space, physical address space, and object file, respectively, by specifying one of the following modifiers as the first format character, and then specifying a value and optional mask. The value and mask are each specified as either immediate values or expressions enclosed in square brackets preceded by a dollar sign. If only a value is specified, mdb reads integers of the appropriate size and stops at the address containing the matching value. If a value V and mask M are specified, mdb reads integers of the appropriate size and stops at the address containing a value X where (X & M) == V. At the completion of the dcmd, dot is updated to the address containing the match. If no match is found, dot is left at the last address that was read.
The search modifiers are:
|l||Search for the specified 2-byte value.|
|L||Search for the specified 4-byte value.|
|M||Search for the specified 8-byte value.|
Notice that for both user and kernel targets, an address space is typically composed of a set of discontiguous segments. It is not legal to read from an address that does not have a corresponding segment. If a search reaches a segment boundary without finding a match, it aborts when the read past the end of the segment boundary fails.
A software event is a state transition in the target program that is observed by the debugger. For example, the debugger can observe the transition of a program counter register to a value of interest (a breakpoint) or the delivery of a particular signal.
A software event specifier is a description of a class of software events that is used by the debugger to instrument the target program in order to observe these events. The ::events dcmd is used to list the software event specifiers. A set of standard properties is associated with each event specifier, as described under ::events, below.
The debugger can observe a variety of different software events, including breakpoints, watchpoints, signals, machine faults, and system calls. New specifiers can be created using ::bp, ::fltbp, ::sigbp, ::sysbp, or ::wp. Each specifier has an associated callback (an mdb command string to execute as if it had been typed at the command prompt) and a set of properties, as described below. Any number of specifiers for the same event can be created, each with different callbacks and properties. The current list of traced events and the properties of the corresponding event specifiers can be displayed using the ::events dcmd. The event specifier properties are defined as part of the description of the ::events and ::evset dcmds, below.
The execution control built-in dcmds, described below, are always available, but issues an error message indicating they are not supported if applied to a target that does not support execution control. For more information about the interaction of exec, attach, release, and job control with debugger execution control, refer to NOTES, below.
If the event callbacks themselves contain one or more commands to continue the target (for example, ::cont or ::step), these commands do not immediately continue the target and wait for it to stop again. Instead, inside of an event callback, the continue dcmds note that a continue operation is now pending, and then return immediately. Therefore, if multiple dcmds are included in an event callback, the step or continue dcmd should be the last command specified. Following the execution of all event callbacks, the target immediately resumes execution if all matching event callbacks requested a continue. If conflicting continue operations are requested, the operation with the highest precedence determines what type of continue occurs. The order of precedence from highest to lowest is: step, step-over (next), step-out, continue.
The mdb kernel target exports the virtual address of the corresponding internal thread structure as the identifier for a given thread. The Modular Debugger Guide provides more information on debugging support for threads in the kernel. The mdb process target provides proper support for examination of multi-threaded user processes that use the native lwp_* interfaces, /usr/lib/libthread.so or /usr/lib/lwp/libthread.so. When debugging a live user process, mdb detects if a single threaded process dlopens or closes libthread and automatically adjusts its view of the threading model on-the-fly. The process target thread identifiers corresponds to either the lwpid_t, thread_t, or pthread_t of the representative, depending on the threading model used by the application.
If mdb is debugging a user process target and the target makes use of compiler-supported thread-local storage, mdb automatically evaluates symbol names referring to thread-local storage to the address of the storage corresponding to the current representative thread. The ::tls built-in dcmd can be used to display the value of the symbol for threads other than the representative thread.
Notice that these operators do not perform a cast. Instead, they fetch the specified number of low-order bytes (on little-endian architectures) or high-order bytes (big-endian architectures). Modifiers are provided for backwards compatibility; the mdb */modifier/ and %/modifier/ syntax should be used instead.
[ address ] $C [ count ]
[ base ] $d
[ pid ] ::attach [ core | pid
[ pid ] :A [ core | pid ]
[address] ::bp [-/-dDesT]
[-c cmd] [-n count] sym ...
address :b [cmd ...]
::cat filename ...
::cont [ SIG ]
:c [ SIG ]
When the user requests a context switch from the kernel target, mdb constructs a new target representing the specified user process. Once the switch occurs, the new target interposes its dcmds at the global level: thus the / dcmd now formats and displays data from the virtual address space of the user process, the ::mappings dcmd displays the mappings in the address space of the user process, and so on. The kernel target can be restored by executing 0::context.
[ address ] ::delete [ id | all
[ address ] :d [ id | all ]
[ address ] ::dis [ -fw ] [ -n count ] [ address ]
::dismode [ mode ]
$V [ mode ]
::dmods [ -l ] [ module-name ]
[ address ] ::dump [ -eqrstu ] [
#sp;#sp;[ -g bytes ] [ -w paragraphs ]
::echo [ string | value ...]
::events [ -av ]
$b [ -av ]
> ::events ID S TA HT LM Description Action ----- - -- -- -- -------------------------------- ------ [ 1 ] - T 1 0 stop on SIGINT - [ 2 ] - T 0 0 stop on SIGQUIT - [ 3 ] - T 0 0 stop on SIGILL - ... [ 11] - T 0 0 stop on SIGXCPU - [ 12] - T 0 0 stop on SIGXFSZ - [ 13] - 2 0 stop at libc`printf ::echo printf >
The following table explains the meaning of each column. A summary of this information is available using ::help events.
[id] ::evset [-/-dDestT] [-c cmd] [-n count] id ...
A summary of this information is available using ::help evset.
[flt] ::fltbp [-/-dDestT] [-c cmd] [-n count] flt ...
[ thread ] ::fpregs
[ thread ] $x, $X, $y, $Y
::help [ dcmd-name ]
[ address ] ::list type member [ variable-name ]
::load [ -s ] module-name
::log [ -d | [ -e ] filename ]
$> [ filename ]
[ address ] ::mappings [ name ]
[ address ] $m [ name ]
::next [ SIG ]
:e [ SIG ]
[ address ] ::nm [ -DPdghnopuvx ] [
-t types ]
#sp;#sp;[ -f format ] [ object ]
-t type[,type ... ]
-f format[,format ... ]
value ::nmadd [ -fo ] [ -e end ] [ -s size ] name
::objects [ -v ]
::offsetof type member
address ::print [ -aCdiLptx ] [
-c lim ]
#sp;#sp;[ -l lim ] [ type [ member ... ] ]
If the -a option is present, the address of each member is displayed. If the -p option is present, ::print interprets address as a physical memory address instead of a virtual memory address. If the -t option is present, the type of each member is displayed. If the -d or -x options are present, all integers are displayed in decimal (-d) or hexadecimal (-x). By default, a heuristic is used to determine if the value should be displayed in decimal or hexadecimal. The number of characters in a character array that is read and displayed as a string can be limited with the -c option. If the -C option is present, no limit is enforced. The number of elements in a standard array that is read and displayed can be limited with the -l option. If the -L option is present, no limit is enforced and all array elements are shown. The default values for -c and -l can be modified using ::set or the -o command-line option as described under OPTIONS.
If the -i option is specified, the address value is interpreted as an immediate value to be printed. You must give a type with which to interpret the value. If the type is smaller than 64 bits, the immediate value is interpreted as if it were the size of the type. The -i option cannot be used in conjunction with the -p option. If the -a option is given, the addresses shown are byte offsets starting at zero.
[ thread ] ::regs
[ thread ] $r
::release [ -a ]
:R [ -a ]
::run [ args ... ]
:r [ args ... ]
::set [ -wF ] [ -/-o
option ] [ -s distance ] [ -I path ]
#sp;#sp;[ -L path ] [ -P prompt ]
::showrev [ -pv ]
[signal] ::sigbp [-/-dDestT]
[-c cmd] [-n count] SIG ...
[signal] :t [-/-dDestT] [-c cmd] [-n count] SIG ...
[ address ] ::stack [ count ]
[ address ] $c [ count ]
::step [ over | out ] [ SIG ]
:s [ SIG ]
:u [ SIG ]
[ syscall ] ::sysbp [ -/-dDestT
] [ -io ] [ -c cmd ]
#sp;#sp;[ -n count ] syscall...
thread ::tls symbol
::typeset [ -/-t] variable-name ...
::unset variable-name ...
::vars [ -npt]
address ::vtop [-a as]
When examining a kernel target from the kernel context, the -a option can be used to specify the address (as) of an alternate address space structure that should be used for the virtual to physical translation. By default, the kernel's address space is used for translation. This option is available for active address spaces even when the dump content only contains kernel pages.
[ address ] ::walk walker-name [ variable-name ]
::whence [ -v ] name ...
::which [ -v ] name ...
addr [ ,len ]::wp [
-/-dDestT ] [ -rwx ] [ -c cmd ]
#sp;#sp; [ -n count ]
addr [ ,len ] :a [ cmd ... ]
addr [ ,len ] :p [ cmd ... ]
addr [ ,len ] :w [ cmd ... ]
The default include path for 32-bit mdb is:
The default include path for 64-bit mdb is:
|ATTRIBUTE TYPE||ATTRIBUTE VALUE|
Linker and Libraries Guide
Modular Debugger Guide:
The debugger can also need to re-execute itself when debugging a 32-bit process that execs a 64-bit process, or vice-versa. The handling of this situation is discussed in more detail under Interaction with Exec, below.
If a 32-bit victim process execs a 64-bit program, then "stop" returns to the command prompt, but the debugger is no longer able to examine the process because it is now using the 64-bit data model. To resume debugging, execute the ::release -a dcmd, quit mdb, and then execute mdb -p pid to re-attach the 64-bit debugger to the process.
If a 64-bit victim process execs a 32-bit program, then "stop" returns to the command prompt, but the debugger only provides limited capabilities for examining the new process. All built-in dcmds work as advertised, but loadable dcmds do not since they do not perform data model conversion of structures. The user should release and re-attach the debugger to the process as described above in order to restore full debugging capabilities.
mdb releases the current running process (if any) when the :R, ::release, :r, ::run, $q, or ::quit dcmds are executed, or when the debugger terminates as the result of an EOF or signal. If the process was originally created by the debugger using :r or ::run, it is forcibly terminated as if by SIGKILL when it is released. If the process was already running prior to attaching mdb to it, it is set running again when it is released. A process can be released and left stopped and abandoned using the ::release -a option.
The header file <sys/mdb_modapi.h> contains prototypes for the functions in the MDB Module API, and distributions may provide source code for an example module in the directory /usr/demo/mdb.
|May 20, 2020||OmniOS|