| MDB(1) | User Commands | MDB(1) |
mdb - modular debugger
mdb [-fkmuwyAFKMSUW] [±o option] [-b VM] [-p pid] [-s distance]
[-I path] [-L path] [-P prompt] [-R root]
[-V dis-version] [-e expr] [object [core] | core | suffix]
The mdb utility is an extensible utility for low-level debugging and editing of the live operating system, operating system crash dumps, user processes, user process core dumps, and object files. For a more detailed description of mdb features, refer to the Modular Debugger Guide.
Debugging is the process of analyzing the execution and state of a software program in order to remove defects. Traditional debugging tools provide facilities for execution control so that programmers can re-execute programs in a controlled environment and display the current state of program data or evaluate expressions in the source language used to develop the program.
Unfortunately, these techniques are often inappropriate for debugging complex software systems such as an operating system, where bugs might not be reproducible and program state is massive and distributed, for programs that are highly optimized, have had their debug information removed, or are themselves low-level debugging tools, or for customer situations where the developer can only access post-mortem information.
mdb provides a completely customizable environment for debugging these programs and scenarios, including a dynamic module facility that programmers can use to implement their own debugging commands to perform program-specific analysis. Each mdb module can be used to examine the program in several different contexts, including live and post-mortem.
The target is the program being inspected by the debugger. mdb currently provides support for the following types of targets: user processes, user process core files, live bhyve VMs, the live operating system (via /dev/kmem and /dev/ksyms), operating system crash dumps, user process images recorded inside an operating system crash dump, ELF object files, and raw binary files. Each target exports a standard set of properties, including one or more address spaces, one or more symbol tables, a set of load objects, and a set of threads that can be examined using the debugger commands described below.
A debugger command, or dcmd (pronounced dee-command) in mdb terminology, is a routine in the debugger that can access any of the properties of the current target. mdb parses commands from standard input, and then executes the corresponding dcmds. Each dcmd can also accept a list of string or numerical arguments, as shown in the syntax description below. mdb contains a set of built-in dcmds, described below, that are always available. You can also extend the capabilities of mdb itself by writing your own dcmds, as described in the Modular Debugger Guide.
A walker is a set of routines that describe how to walk, or iterate, through the elements of a particular program data structure. A walker encapsulates the data structure's implementation from dcmds and from mdb itself. You can use walkers interactively, or use them as a primitive to build other dcmds or walkers. As with dcmds, you can extend mdb by implementing your own walkers as part of a debugger module.
A debugger module, or dmod (pronounced dee-mod), is a dynamically loaded library containing a set of dcmds and walkers. During initialization, mdb attempts to load dmods corresponding to the load objects present in the target. You can subsequently load or unload dmods at any time while running mdb. mdb ships with a set of standard dmods for debugging the kernel. The Modular Debugger Guide contains more information on developing your own debugger modules.
A macro file is a text file containing a set of commands to execute. Macro files are typically used to automate the process of displaying a simple data structure. mdb provides complete backward compatibility for the execution of macro files written for adb(1), and illumos includes a set of macro files for debugging the kernel that can be used with either tool.
The debugger processes commands from standard input. If standard input is a terminal, mdb provides terminal editing capabilities. mdb can also process commands from macro files and from dcmd pipelines, described below. The language syntax is designed around the concept of computing the value of an expression (typically a memory address in the target), and then applying a dcmd to that address. The current address location is referred to as dot, and its value is referenced using ``.''.
A metacharacter is one of the following characters:
[ ] | ! / \ ? = > $ : ;
NEWLINE SPACE TAB
A blank is a TAB or a SPACE. A word is a sequence of characters separated by one or more non-quoted metacharacters. Some of the metacharacters only function as delimiters in certain contexts, as described below. An identifier is a sequence of letters, digits, underscores, periods, or backquotes beginning with a letter, underscore, or period. Identifiers are used as the names of symbols, variables, dcmds, and walkers. Commands are delimited by a NEWLINE or semicolon ( ; ).
A dcmd is denoted by one of the following words or metacharacters:
/ \ ? = > $character :character ::identifier
dcmds named by metacharacters or prefixed by a single $ or : are provided as built-in operators, and implement complete compatibility with the command set of the legacy adb(1) utility. Once a dcmd has been parsed, the /, \, ?, =, >, $, and : characters are no longer recognized as metacharacters until the termination of the argument list.
A simple-command is a dcmd followed by a sequence of zero or more blank-separated words. The words are passed as arguments to the invoked dcmd, except as specified under Quoting and Arithmetic Expansion below. Each dcmd returns an exit status that indicates it was either successful, failed, or was invoked with invalid arguments.
A pipeline is a sequence of one or more simple commands separated by |. Unlike the shell, dcmds in mdb pipelines are not executed as separate processes. After the pipeline has been parsed, each dcmd is invoked in order from left to right. Each dcmd's output is processed and stored as described under dcmd Pipelines below. Once the left-hand dcmd is complete, its processed output is used as input for the next dcmd in the pipeline. If any dcmd does not return a successful exit status, the pipeline is aborted.
An expression is a sequence of words that is evaluated to compute a 64-bit unsigned integer value. The words are evaluated using the rules described under Arithmetic Expansion below.
A command is one of the following:
pipeline [! word ...] [ ; ]
expression pipeline [! word ...] [ ; ]
expression , expression pipeline [! word ...] [ ; ]
, expression pipeline [! word ...] [ ; ]
expression [! word ...] [ ; ]
expression, expression [! word ...] [ ; ]
, expression [! word ...] [ ; ]
! word ... [ ; ]
A word beginning with // causes that word and all the subsequent characters up to a NEWLINE to be ignored.
Arithmetic expansion is performed when an mdb command is preceded by an optional expression representing a start address, or a start address and a repeat count. Arithmetic expansion can also be performed to compute a numerical argument for a dcmd. An arithmetic expression can appear in an argument list enclosed in square brackets preceded by a dollar sign ($[ expression ]), and is replaced by the value of the expression.
Expressions can contain any of the following special words:
integer
0[tT][0-9]+.[0-9]+
'cccccccc'
<identifier
identifier
(expression)
.
&
+
^
The increment is a global variable that stores the total bytes read by the last formatting dcmd. For more information on the increment, refer to the discussion of Formatting dcmds below.
Unary operators are right associative and have higher precedence than binary operators. The unary operators are:
#expression
~expression
-expression
%expression
%/[csil]/expression
%/[1248]/expression
*expression
*/[csil]/expression
*/[1248]/expression
Binary operators are left associative and have lower precedence than unary operators. The binary operators, in order of precedence from highest to lowest, are:
*
%
#
+
-
<<
>>
==
!=
&
^
|
Each metacharacter described above (see Syntax) terminates a word unless quoted. Characters can be quoted (forcing mdb to interpret each character as itself without any special significance) by enclosing them in a pair of single (' ') or double (" ") quote marks. A single quote cannot appear within single quotes. Inside double quotes, mdb recognizes the C programming language character escape sequences.
The ! character can be used to create a pipeline between an mdb command and the user's shell. If the $SHELL environment variable is set, mdb forks and execs this program for shell escapes; otherwise /bin/sh is used. The shell is invoked with the -c option followed by a string formed by concatenating the words after the ! character. The ! character takes precedence over all other metacharacters, except semicolon (;) and NEWLINE. Once a shell escape is detected, the remaining characters up to the next semicolon or NEWLINE are passed as is to the shell. The output of shell commands can not be piped to mdb dcmds. Commands executed by a shell escape have their output sent directly to the terminal, not to mdb.
A variable is a variable name, a corresponding integer value, and a set of attributes. A variable name is a sequence of letters, digits, underscores, or periods. A variable can be assigned a value using the > dcmd or ::typeset dcmd, and its attributes can be manipulated using the ::typeset dcmd. Each variable's value is represented as a 64-bit unsigned integer. A variable can have one or more of the following attributes: read-only (cannot be modified by the user), persistent (cannot be unset by the user), and tagged (user-defined indicator).
The following variables are defined as persistent:
0
9
b
d
e
m
t
hits
thread
In addition, the mdb kernel and process targets export the current values of the representative thread's register set as named variables. The names of these variables depend on the target's platform and instruction set architecture.
As explained in the Syntax description above, a symbol identifier present in an expression context evaluates to the value of this symbol. The value typically denotes the virtual address of the storage associated with the symbol in the target's virtual address space. A target can support multiple symbol tables including, but not limited to, a primary executable symbol table, a primary dynamic symbol table, a run-time link-editor symbol table, and standard and dynamic symbol tables for each of a number of load objects (such as shared libraries in a user process, or kernel modules in the kernel). The target typically searches the primary executable's symbol tables first, and then one or more of the other symbol tables. Notice that ELF symbol tables only contain entries for external, global, and static symbols; automatic symbols do not appear in the symbol tables processed by mdb.
Additionally, mdb provides a private user-defined symbol table that is searched prior to any of the target symbol tables. The private symbol table is initially empty, and can be manipulated using the ::nmadd and ::nmdel dcmds. The ::nm -P option can be used to display the contents of the private symbol table. The private symbol table allows the user to create symbol definitions for program functions or data that were either missing from the original program or stripped out. These definitions are then used whenever mdb converts a symbolic name to an address, or an address to the nearest symbol.
As targets contain multiple symbol tables, and each symbol table can include symbols from multiple object files, different symbols with the same name can exist. mdb uses the backquote (`) character as a symbol name scoping operator to allow the programmer to obtain the value of the desired symbol in this situation. The programmer can specify the scope used to resolve a symbol name as either: object`name, or file`name, or object`file`name. The object identifier refers to the name of a load object. The file identifier refers to the basename of a source file that has a symbol of type STT_FILE in the specified object's symbol table. The object identifier's interpretation depends on the target type.
The mdb kernel target expects object to specify the basename of a loaded kernel module. For example, the symbol name
specfs`_init
evaluates to the value of the _init symbol in the specfs kernel module.
The mdb process target expects object to specify the name of the executable or of a loaded shared library. It can take any of the following forms:
The process target also accepts any of the four forms described above preceded by an optional link-map id (lmid). The lmid prefix is specified by an initial "LM" followed by the link-map id in hexadecimal followed by an additional backquote. For example, the symbol name
LM0`libc.so.1`_init
evaluates to the value of the _init symbol in the libc.so.1 library that is loaded on link-map 0 (LM_ID_BASE). The link-map specifier can be necessary to resolve symbol naming conflicts in the event that the same library is loaded on more than one link map. For more information on link maps, refer to the Linker and Libraries Guide and dlopen(3C). Link-map identifiers are displayed when symbols are printed according to the setting of the showlmid option, as described under OPTIONS.
In the case of a naming conflict between symbols and hexadecimal integer values, mdb attempts to evaluate an ambiguous token as a symbol first, before evaluating it as an integer value. For example, the token f can either refer to the decimal integer value 15 specified in hexadecimal (the default base), or to a global variable named f in the target's symbol table. If a symbol with an ambiguous name is present, the integer value can be specified by using an explicit 0x or 0X prefix.
As described earlier, each mdb dmod provides a set of dcmds and walkers. dcmds and walkers are tracked in two distinct, global namespaces. mdb also keeps track of a dcmd and walker namespace associated with each dmod. Identically named dcmds or walkers within a given dmod are not allowed: a dmod with this type of naming conflict fails to load. Name conflicts between dcmds or walkers from different dmods are allowed in the global namespace. In the case of a conflict, the first dcmd or walker with that particular name to be loaded is given precedence in the global namespace. Alternate definitions are kept in a list in load order. The backquote character (`) can be used in a dcmd or walker name as a scoping operator to select an alternate definition. For example, if dmods m1 and m2 each provide a dcmd d, and m1 is loaded prior to m2, then:
::d
::m1`d
::m2`d
If module m1 were now unloaded, the next dcmd on the global definition list (m2`d) would be promoted to global visibility. The current definition of a dcmd or walker can be determined using the ::which dcmd, described below. The global definition list can be displayed using the ::which -v option.
dcmds can be composed into a pipeline using the | operator. The purpose of a pipeline is to pass a list of values, typically virtual addresses, from one dcmd or walker to another. Pipeline stages might be used to map a pointer from one type of data structure to a pointer to a corresponding data structure, to sort a list of addresses, or to select the addresses of structures with certain properties.
mdb executes each dcmd in the pipeline in order from left to right. The leftmost dcmd is executed using the current value of dot, or using the value specified by an explicit expression at the start of the command. When a | operator is encountered, mdb creates a pipe (a shared buffer) between the output of the dcmd to its left and the mdb parser, and an empty list of values. As the dcmd executes, its standard output is placed in the pipe and then consumed and evaluated by the parser, as if mdb were reading this data from standard input. Each line must consist of an arithmetic expression terminated by a NEWLINE or semicolon (;). The value of the expression is appended to the list of values associated with the pipe. If a syntax error is detected, the pipeline is aborted.
When the dcmd to the left of a | operator completes, the list of values associated with the pipe is then used to invoke the dcmd to the right of the | operator. For each value in the list, dot is set to this value and the right-hand dcmd is executed. Only the rightmost dcmd in the pipeline has its output printed to standard output. If any dcmd in the pipeline produces output to standard error, these messages are printed directly to standard error and are not processed as part of the pipeline.
The debugger ignores the PIPE and QUIT signals. The INT signal aborts the command that is currently executing. The debugger intercepts and provides special handling for the ILL, TRAP, EMT, FPE, BUS, and SEGV signals. If any of these signals are generated asynchronously (that is, delivered from another process using kill(2)), mdb restores the signal to its default disposition and dump core. However, if any of these signals are generated synchronously by the debugger process itself and a dcmd from an externally loaded dmod is currently executing, and standard input is a terminal, mdb provides a menu of choices allowing the user to force a core dump, quit without producing a core dump, stop for attach by a debugger, or attempt to resume. The resume option aborts all active commands and unload the dmod whose dcmd was active at the time the fault occurred. It can then be subsequently re-loaded by the user. The resume option provides limited protection against buggy dcmds. Refer to WARNINGS, Use of the Error Recovery Mechanism, below for information about the risks associated with the resume option.
The text of the last HISTSIZE (default 128) commands entered from a terminal device are saved in memory. The in-line editing facility, described next, provides key mappings for searching and fetching elements from the history list.
If standard input is a terminal device, mdb provides some simple emacs-style facilities for editing the command line. The search, previous, and next commands in edit mode provide access to the history list. Only strings, not patterns, are matched when searching. In the table below, the notation for control characters is caret (^) followed by a character shown in upper case. The notation for escape sequences is M- followed by a character. For example, M-f (pronounced meta-eff) is entered by depressing ESC followed by 'f', or by depressing Meta followed by 'f' on keyboards that support a Meta key. A command line is committed and executed using RETURN or NEWLINE. The edit commands are:
^F
M-f
^B
M-b
^A
^E
^D
M-^H
^K
^L
^T
^N
^P
^R[string]
The editing mode also interprets the following user-defined sequences as editing commands. User defined sequences can be read or modified using the stty(1) command.
erase
intr
kill
quit
suspend
werase
On keyboards that support an extended keypad with arrow keys, mdb interprets these keystrokes as editing commands:
up-arrow
down-arrow
left-arrow
right-arrow
mdb provides a built-in output pager. The output pager is enabled if the debugger's standard output is a terminal device. Each time a command is executed, mdb pauses after one screenful of output is produced and displays a pager prompt:
>> More [<space>, <cr>, q, n, c, a] ?
The following key sequences are recognized by the pager:
SPACE
a, A
c, C
n, N, NEWLINE, RETURN
q, Q, ^C, ^\
The /, \, ?, and = metacharacters are used to denote the special output formatting dcmds. Each of these dcmds accepts an argument list consisting of one or more format characters, repeat counts, or quoted strings. A format character is one of the ASCII characters shown in the table below. Format characters are used to read and format data from the target. A repeat count is a positive integer preceding the format character that is always interpreted in base 10 (decimal). A repeat count can also be specified as an expression enclosed in square brackets preceded by a dollar sign ($[ ]). A string argument must be enclosed in double-quotes (" "). No blanks are necessary between format arguments.
The formatting dcmds are:
/
\
?
=
In addition to dot, mdb keeps track of another global value called the increment. The increment represents the distance between dot and the address following all the data read by the last formatting dcmd. For example, if a formatting dcmd is executed with dot equal to address A, and displays a 4-byte integer, then after this dcmd completes, dot is still A, but the increment is set to 4. The + character (described under Arithmetic Expansion above) would now evaluate to the value A + 4, and could be used to reset dot to the address of the next data object for a subsequent dcmd.
Most format characters increase the value of the increment by the number of bytes corresponding to the size of the data format, shown in the table. The table of format characters can be displayed from within mdb using the ::formats dcmd. The format characters are:
| + | increment dot by the count (variable size) |
| - | decrement dot by the count (variable size) |
| B | hexadecimal int (1 byte) |
| C | character using C character notation (1 byte) |
| D | decimal signed int (4 bytes) |
| E | decimal unsigned long long (8 bytes) |
| F | double (8 bytes) |
| G | octal unsigned long long (8 bytes) |
| H | swap bytes and shorts (4 bytes) |
| I | address and disassembled instruction (variable size) |
| J | hexadecimal long long (8 bytes) |
| K | hexadecimal uintptr_t (4 or 8 bytes) |
| N | newline |
| O | octal unsigned int (4 bytes) |
| P | symbol (4 or 8 bytes) |
| Q | octal signed int (4 bytes) |
| R | binary unsigned long long (8 bytes) |
| S | string using C string notation (variable size) |
| T | horizontal tab |
| U | decimal unsigned int (4 bytes) |
| V | decimal unsigned int (1 byte) |
| W | default radix unsigned int (4 bytes) |
| X | hexadecimal int (4 bytes) |
| Y | decoded time32_t (4 bytes) |
| Z | hexadecimal long long (8 bytes) |
| ^ | decrement dot by increment * count (variable size) |
| a | dot as symbol+offset |
| b | octal unsigned int (1 byte) |
| c | character (1 byte) |
| d | decimal signed short (2 bytes) |
| e | decimal signed long long (8 bytes) |
| f | float (4 bytes) |
| g | octal signed long long (8 bytes) |
| h | swap bytes (2 bytes) |
| i | disassembled instruction (variable size) |
| j | jazzed-up binary unsigned long long (8 bytes) |
| n | newline |
| o | octal unsigned short (2 bytes) |
| p | symbol (4 or 8 bytes) |
| q | octal signed short (2 bytes) |
| r | whitespace |
| s | raw string (variable size) |
| t | horizontal tab |
| u | decimal unsigned short (2 bytes) |
| v | decimal signed int (1 byte) |
| w | default radix unsigned short (2 bytes) |
| x | hexadecimal short (2 bytes) |
| y | decoded time64_t (8 bytes) |
| z | write whose size is inferred by CTF info (variable size) |
The /, \, and ? formatting dcmds can also be used to write to the target's virtual address space, physical address space, or object file by specifying one of the following modifiers as the first format character, and then specifying a list of words that are either immediate values or expressions enclosed in square brackets preceded by a dollar sign ($[ ]).
The write modifiers are:
v
w
W
Z
The /, \, and ? formatting dcmds can also be used to search for a particular integer value in the target's virtual address space, physical address space, and object file, respectively, by specifying one of the following modifiers as the first format character, and then specifying a value and optional mask. The value and mask are each specified as either immediate values or expressions enclosed in square brackets preceded by a dollar sign. If only a value is specified, mdb reads integers of the appropriate size and stops at the address containing the matching value. If a value V and mask M are specified, mdb reads integers of the appropriate size and stops at the address containing a value X where (X & M) == V. At the completion of the dcmd, dot is updated to the address containing the match. If no match is found, dot is left at the last address that was read.
The search modifiers are:
| l | Search for the specified 2-byte value. |
| L | Search for the specified 4-byte value. |
| M | Search for the specified 8-byte value. |
Notice that for both user and kernel targets, an address space is typically composed of a set of discontiguous segments. It is not legal to read from an address that does not have a corresponding segment. If a search reaches a segment boundary without finding a match, it aborts when the read past the end of the segment boundary fails.
mdb provides facilities for controlling and tracing the execution of a live running program. Currently, only the user process target provides support for execution control. mdb provides a simple model of execution control: a target process can be started from within the debugger using ::run, or mdb can attach to an existing process using :A, ::attach, or the -p command-line option, as described below. A list of traced software events can be specified by the user. Each time a traced event occurs in the target process, all threads in the target stop, the thread that triggered the event is chosen as the representative thread, and control returns to the debugger. Once the target program is set running, control can be asynchronously returned to the debugger by typing the user-defined interrupt character (typically ^C).
A software event is a state transition in the target program that is observed by the debugger. For example, the debugger can observe the transition of a program counter register to a value of interest (a breakpoint) or the delivery of a particular signal.
A software event specifier is a description of a class of software events that is used by the debugger to instrument the target program in order to observe these events. The ::events dcmd is used to list the software event specifiers. A set of standard properties is associated with each event specifier, as described under ::events, below.
The debugger can observe a variety of different software events, including breakpoints, watchpoints, signals, machine faults, and system calls. New specifiers can be created using ::bp, ::fltbp, ::sigbp, ::sysbp, or ::wp. Each specifier has an associated callback (an mdb command string to execute as if it had been typed at the command prompt) and a set of properties, as described below. Any number of specifiers for the same event can be created, each with different callbacks and properties. The current list of traced events and the properties of the corresponding event specifiers can be displayed using the ::events dcmd. The event specifier properties are defined as part of the description of the ::events and ::evset dcmds, below.
The execution control built-in dcmds, described below, are always available, but issues an error message indicating they are not supported if applied to a target that does not support execution control. For more information about the interaction of exec, attach, release, and job control with debugger execution control, refer to NOTES, below.
The ::evset dcmd and event tracing dcmds allow you to associate an event callback (using the -c option) with each event specifier. The event callbacks are strings that represent mdb commands to execute when the corresponding event occurs in the target. These commands are executed as if they had been typed at the command prompt. Before executing each callback, the dot variable is set to the value of the representative thread's program counter and the "hits" variable is set to the number of times this specifier has been matched, including the current match.
If the event callbacks themselves contain one or more commands to continue the target (for example, ::cont or ::step), these commands do not immediately continue the target and wait for it to stop again. Instead, inside of an event callback, the continue dcmds note that a continue operation is now pending, and then return immediately. Therefore, if multiple dcmds are included in an event callback, the step or continue dcmd should be the last command specified. Following the execution of all event callbacks, the target immediately resumes execution if all matching event callbacks requested a continue. If conflicting continue operations are requested, the operation with the highest precedence determines what type of continue occurs. The order of precedence from highest to lowest is: step, step-over (next), step-out, continue.
mdb provides facilities to examine the stacks and registers of each thread associated with the target. The persistent "thread" variable contains the current representative thread identifier. The format of the thread identifier depends on the target. The ::regs and ::fpregs dcmds can be used to examine the register set of the representative thread, or of another thread if its register set is currently available. In addition, the register set of the representative thread is exported as a set of named variables. The user can modify the value of one or more registers by applying the > dcmd to the corresponding named variable.
The mdb kernel target exports the virtual address of the corresponding internal thread structure as the identifier for a given thread. The Modular Debugger Guide provides more information on debugging support for threads in the kernel. The mdb process target provides proper support for examination of multi-threaded user processes that use the native lwp_* interfaces, /usr/lib/libthread.so or /usr/lib/lwp/libthread.so. When debugging a live user process, mdb detects if a single threaded process dlopens or closes libthread and automatically adjusts its view of the threading model on-the-fly. The process target thread identifiers corresponds to either the lwpid_t, thread_t, or pthread_t of the representative, depending on the threading model used by the application.
If mdb is debugging a user process target and the target makes use of compiler-supported thread-local storage, mdb automatically evaluates symbol names referring to thread-local storage to the address of the storage corresponding to the current representative thread. The ::tls built-in dcmd can be used to display the value of the symbol for threads other than the representative thread.
mdb provides a set of built-in dcmds that are always defined. Some of these dcmds are only applicable to certain targets: if a dcmd is not applicable to the current target, it fails and prints a message indicating "command is not supported by current target". In many cases, mdb provides a mnemonic equivalent (::identifier) for the legacy adb(1) dcmd names. For example, ::quit is provided as the equivalent of $q. Programmers who are experienced with adb(1) or who appreciate brevity or arcana can prefer the $ or : forms of the built-ins. Programmers who are new to mdb might prefer the more verbose :: form. The built-ins are shown in alphabetical order. If a $ or : form has a ::identifier equivalent, it is shown underneath the ::identifier form. The built-in dcmds are:
> variable-name
>/modifier/variable-name
c
s
i
l
Notice that these operators do not perform a cast. Instead, they fetch the specified number of low-order bytes (on little-endian architectures) or high-order bytes (big-endian architectures). Modifiers are provided for backwards compatibility; the mdb */modifier/ and %/modifier/ syntax should be used instead.
$< macro-name
$<< macro-name
$?
[ address ] $C [ count ]
[ base ] $d
$e
$P prompt-string
distance $s
$v
width $w
$W
[ pid ] ::attach [ core | pid
]
[ pid ] :A [ core | pid ]
[address] ::bp [-/-dDesT]
[-c cmd] [-n count] sym ...
address :b [cmd ...]
::cat filename ...
::cont [ SIG ]
:c [ SIG ]
address ::context
address $p
When the user requests a context switch from the kernel target, mdb constructs a new target representing the specified user process. Once the switch occurs, the new target interposes its dcmds at the global level: thus the / dcmd now formats and displays data from the virtual address space of the user process, the ::mappings dcmd displays the mappings in the address space of the user process, and so on. The kernel target can be restored by executing 0::context.
::dcmds
[ address ] ::delete [ id | all
]
[ address ] :d [ id | all ]
[ address ] ::dis [ -fw ] [ -n count ] [ address ]
::disasms
::dismode [ mode ]
$V [ mode ]
::dmods [ -l ] [ module-name ]
[ address ] ::dump [ -eqrstu ] [
-f|-p ]
#sp;#sp;[ -g bytes ] [ -w paragraphs ]
-e
-f
-g bytes
-p
-q
-r
-s
-t
-u
-w paragraphs
::echo [ string | value ...]
::eval command
::events [ -av ]
$b [ -av ]
> ::events
ID S TA HT LM Description Action ----- - -- -- -- -------------------------------- ------ [ 1 ] - T 1 0 stop on SIGINT - [ 2 ] - T 0 0 stop on SIGQUIT - [ 3 ] - T 0 0 stop on SIGILL -
... [ 11] - T 0 0 stop on SIGXCPU - [ 12] - T 0 0 stop on SIGXFSZ - [ 13] - 2 0 stop at libc`printf ::echo printf >
The following table explains the meaning of each column. A summary of this information is available using ::help events.
ID
S
-
+
*
!
TA
t
T
d
D
s
HT
LM
Description
Action
[id] ::evset [-/-dDestT] [-c cmd] [-n count] id ...
-d
-D
-e
-s
-t
-T
-c
-n
A summary of this information is available using ::help evset.
::files
$f
[flt] ::fltbp [-/-dDestT] [-c cmd] [-n count] flt ...
[ thread ] ::fpregs
[ thread ] $x, $X, $y, $Y
::formats
::grep command
::help [ dcmd-name ]
signal :i
$i
::kill
:k
$l
$L
[ address ] ::list type member [ variable-name ]
::load [ -s ] module-name
::log [ -d | [ -e ] filename ]
$> [ filename ]
::map command
[ address ] ::mappings [ name ]
[ address ] $m [ name ]
::next [ SIG ]
:e [ SIG ]
[ address ] ::nm [ -DPdghnopuvx ] [
-t types ]
#sp;#sp;[ -f format ] [ object ]
-D
-P
-d
-g
-h
-n
-o
-p
-u
-v
-x
-t type[,type ... ]
noty
objt
func
sect
file
comm
tls
regi
-f format[,format ... ]
ndx
val
size
type
bind
oth
shndx
name
ctype
obj
value ::nmadd [ -fo ] [ -e end ] [ -s size ] name
-e
-f
-o
-s
::nmdel name
::objects [ -v ]
::offsetof type member
address ::print [ -aCdiLptx ] [
-c lim ]
#sp;#sp;[ -l lim ] [ type [ member ... ]
]
If the -a option is present, the address of each member is displayed. If the -p option is present, ::print interprets address as a physical memory address instead of a virtual memory address. If the -t option is present, the type of each member is displayed. If the -d or -x options are present, all integers are displayed in decimal (-d) or hexadecimal (-x). By default, a heuristic is used to determine if the value should be displayed in decimal or hexadecimal. The number of characters in a character array that is read and displayed as a string can be limited with the -c option. If the -C option is present, no limit is enforced. The number of elements in a standard array that is read and displayed can be limited with the -l option. If the -L option is present, no limit is enforced and all array elements are shown. The default values for -c and -l can be modified using ::set or the -o command-line option as described under OPTIONS.
If the -i option is specified, the address value is interpreted as an immediate value to be printed. You must give a type with which to interpret the value. If the type is smaller than 64 bits, the immediate value is interpreted as if it were the size of the type. The -i option cannot be used in conjunction with the -p option. If the -a option is given, the addresses shown are byte offsets starting at zero.
::quit
$q
[ thread ] ::regs
[ thread ] $r
::release [ -a ]
:R [ -a ]
::run [ args ... ]
:r [ args ... ]
::set [ -wF ] [ -/-o
option ] [ -s distance ] [ -I path ]
#sp;#sp;[ -L path ] [ -P prompt ]
-F
-I
-L
-o
-P
-s
-w
::showrev [ -pv ]
[signal] ::sigbp [-/-dDestT]
[-c cmd] [-n count] SIG ...
[signal] :t [-/-dDestT] [-c
cmd] [-n count] SIG ...
::sizeof type
[ address ] ::stack [ count ]
[ address ] $c [ count ]
::status
::step [ over | out ] [ SIG ]
:s [ SIG ]
:u [ SIG ]
[ syscall ] ::sysbp [ -/-dDestT
] [ -io ] [ -c cmd ]
#sp;#sp;[ -n count ] syscall...
thread ::tls symbol
::typeset [ -/-t] variable-name ...
::unload module-name
::unset variable-name ...
::vars [ -npt]
::version
address ::vtop [-a as]
When examining a kernel target from the kernel context, the -a option can be used to specify the address (as) of an alternate address space structure that should be used for the virtual to physical translation. By default, the kernel's address space is used for translation. This option is available for active address spaces even when the dump content only contains kernel pages.
[ address ] ::walk walker-name [ variable-name ]
::walkers
::whence [ -v ] name ...
::which [ -v ] name ...
addr [ ,len ]::wp [
-/-dDestT ] [ -rwx ] [ -c cmd ]
#sp;#sp; [ -n count ]
addr [ ,len ] :a [ cmd ... ]
addr [ ,len ] :p [ cmd ... ]
addr [ ,len ] :w [ cmd ... ]
::xdata
:z
The following options are supported:
-A
-b VM
-e expr
-f
-F
-I path
%i
%o
%p
%r
%t
The default include path for 32-bit mdb is:
%r/usr/platform/%p/lib/adb:%r/usr/lib/adb
The default include path for 64-bit mdb is:
%r/usr/platform/%p/lib/adb/%i:%r/usr/lib/adb/%i
-k
-K
-L path
-m
-M
-o option
adb
array_mem_limit=limit
array_str_limit=limit
autowrap
follow_exec_mode=mode
ask
follow
stop
follow_fork_mode=mode
ask
parent
child
ignoreeof
nostop
pager
repeatlast
showlmid
-p pid
-P prompt
-R root
-s distance
-S
-u
-U
-V version
-w
-W
-y
The following operands are supported:
object
core
suffix
mdb processes all input files (including scripts, object files, core files, and raw data files) in a large file aware fashion. See largefile(7) for more information about the processing of large files, which are files greater than or equal to 2 Gbytes (2^31 bytes).
The following exit values are returned:
0
1
2
HISTSIZE
HOME
SHELL
$HOME/.mdbrc
/dev/kmem
/dev/ksyms
/proc/pid/*
/usr/lib/adb
/usr/platform/platform-name/lib/adb
/usr/lib/mdb
/usr/platform/platform-name/lib/mdb
See attributes(7) for descriptions of the following attributes:
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
| Interface Stability | Evolving |
adb(1), gcore(1), pgrep(1), proc(1), ps(1), stty(1), truss(1), uname(1), _lwp_self(2), exec(2), fork(2), pipe(2), vfork(2), dlopen(3C), signal(3C), thr_self(3C), elf(3ELF), signal.h(3HEAD), libc_db(3LIB), libkvm(3LIB), libthread(3LIB), ksyms(4D), mem(4D), core(5), proc(5), attributes(7), largefile(7), threads(7), coreadm(8), dumpadm(8), savecore(8)
Linker and Libraries Guide
Modular Debugger Guide:
https://illumos.org/books/mdb/
The debugger and its dmods execute in the same address space, and thus it is quite possible that a buggy dmod can cause mdb to dump core or otherwise misbehave. The mdb resume capability, described above under Signal Handling, provides a limited recovery mechanism for these situations. However, it is not possible for mdb to know definitively whether the dmod in question has corrupted only its own state, or the debugger's global state. Therefore a resume operation cannot be guaranteed to be safe, or to prevent a subsequent crash of the debugger. The safest course of action following a resume is to save any important debug information, and then quit and restart the debugger.
The use of the debugger to modify (that is, write to) the address space of live running operating system is extremely dangerous, and can result in a system panic in the event the user damages a kernel data structure.
By default, text sections and read-only data are included in core files. While ill-advised, users can configure their processes to exclude that information from core files using coreadm(8). Such core files will be missing the information necessary to interpret them; to debug them using mdb, the object file shold be specified in addition to the core file. The instruction set architecture of the machine running mdb must match the instruction set architecture of the machine that generated the core file.
If a crash dump from one operating system release is examined using the dmods from a different operating system release, changes in the kernel implementation can prevent some dcmds or walkers from working properly. mdb issues a warning message if it detects this condition. The instruction set architecture of the machine running mdb must match the instruction set architecture of the machine that generated the crash dump.
mdb provides support for debugging both 32-bit and 64-bit programs. Once it has examined the target and determined its data model, mdb automatically re-executes the mdb binary that has the same data model as the target, if necessary. This approach simplifies the task of writing debugger modules, because the modules that are loaded use the same data model as the primary target. Only the 64-bit debugger can be used to debug 64-bit target programs. The 64-bit debugger can only be used on a system that is running the 64-bit operating environment.
The debugger can also need to re-execute itself when debugging a 32-bit process that execs a 64-bit process, or vice-versa. The handling of this situation is discussed in more detail under Interaction with Exec, below.
When a controlled process performs a successful exec(2), the behavior of the debugger is controlled by the ::set -o follow_exec_mode option, as described above. If the debugger and victim process have the same data model, then the "stop" and "follow" modes determine whether mdb automatically continues the target or returns to the debugger prompt following the exec. If the debugger and victim process have a different data model, then the "follow" behavior causes mdb to automatically re-exec the mdb binary with the appropriate data model and to re-attach to the process, still stopped on return from the exec. Not all debugger state is preserved across this re-exec.
If a 32-bit victim process execs a 64-bit program, then "stop" returns to the command prompt, but the debugger is no longer able to examine the process because it is now using the 64-bit data model. To resume debugging, execute the ::release -a dcmd, quit mdb, and then execute mdb -p pid to re-attach the 64-bit debugger to the process.
If a 64-bit victim process execs a 32-bit program, then "stop" returns to the command prompt, but the debugger only provides limited capabilities for examining the new process. All built-in dcmds work as advertised, but loadable dcmds do not since they do not perform data model conversion of structures. The user should release and re-attach the debugger to the process as described above in order to restore full debugging capabilities.
If the debugger is attached to a process that is stopped by job control (that is, it stopped in response to SIGTSTP, SIGTTIN, or SIGTTOU), the process can not be able to be set running again when it is continued by a continue dcmd. If the victim process is a member of the same session (that is, it shares the same controlling terminal as mdb), mdb attempts to bring the associated process group to the foreground and to continue the process with SIGCONT to resume it from job control stop. When mdb is detached from such a process, it restores the process group to the background before exiting. If the victim process is not a member of the same session, mdb cannot safely bring the process group to the foreground, so it continues the process with respect to the debugger, but the process remains stopped by job control. mdb prints a warning in this case, and the user must issue an "fg" command from the appropriate shell in order to resume the process.
When mdb attaches to a running process, the process is stopped and remains stopped until one of the continue dcmds is applied, or the debugger quits. If the -o nostop option is enabled prior to attaching the debugger to a process with -p, or prior to issuing an ::attach or :A command, mdb attaches to the process but does not stop it. While the process is still running, it can be inspected as usual (albeit with inconsistent results) and breakpoints or other tracing flags might be enabled. If the :c or ::cont dcmds are executed while the process is running, the debugger waits for the process to stop. If no traced software events occur, the user can send an interrupt (^C) after :c or ::cont to force the process to stop and return control to the debugger.
mdb releases the current running process (if any) when the :R, ::release, :r, ::run, $q, or ::quit dcmds are executed, or when the debugger terminates as the result of an EOF or signal. If the process was originally created by the debugger using :r or ::run, it is forcibly terminated as if by SIGKILL when it is released. If the process was already running prior to attaching mdb to it, it is set running again when it is released. A process can be released and left stopped and abandoned using the ::release -a option.
The ::list, ::offsetof, ::print, and ::sizeof dcmds require that one or more load objects contain compressed symbolic debugging information suitable for use with mdb. This information is currently only available for certain kernel modules.
The Modular Debugger Guide provides a more detailed description of mdb features, as well as information for debugger module developers.
The header file <sys/mdb_modapi.h> contains prototypes for the functions in the MDB Module API, and distributions may provide source code for an example module in the directory /usr/demo/mdb.
| February 21, 2023 | OmniOS |