OVERLAY(7) Standards, Environments, and Macros OVERLAY(7)

overlay — Overlay Devices

Overlay devices are a GLDv3 device that allows users to create overlay networks that can be used to form the basis of network virtualization and software defined networking. Overlay networks allow a single physical network, often called an network, to provide the means for creating multiple logical, isolated, and discrete layer two and layer three networks on top of it.

Overlay devices are administered through dladm(8). Overlay devices themselves cannot be plumbed up with IP, , or any other protocol. Instead, like an etherstub, they allow for VNICs to be created on top of them. Like an etherstub, an overlay device acts as a local switch; however, when it encounters a non-local destination address, it instead looks up where it should send the packet, encapsulates it, and sends it out another interface in the system.

A single overlay device encapsulates the logic to answer two different, but related, questions:

  1. How should a packet be transformed and put on the wire?
  2. Where should a transformed packet be sent?

Each of these questions is answered by a plugin. The first question is answered by what's called an . The second question is answered by what's called a search plugin. Packets are encapsulated and decapsulated using the encapsulation plugin by the kernel. The search plugins are all user land plugins that are consumed by the varpd service whose FMRI is . This separation allows for the kernel to be responsible for the data path, while having the search plugins in userland allows the system to provide a much more expressive interface.

Overlay devices come in two different flavors, one where all packets are always sent to a single address, the other, where the destination of a packet varies based on the target MAC address of the packet. This information is maintained in a , which is independent and unique to each overlay device. We call the plugins that send traffic to a single location, for example a single unicast or multicast IP address, a overlay and the overlay devices that can send traffic to different locations based on the MAC address of that packet a dynamic overlay. The plugin type is determined based on the type of the search plugin. These are all fully listed in the section Plugins and their Properties.

Both encapsulation and search plugins define the kinds of destinations that they know how to support. An encapsulation plugin always has a single destination type that's determined based on how the encapsulation is defined. A search plugin, on the other hand, can support multiple combinations of destinations. A search plugin must support the destination type of the encapsulation device. The destination may require any of the following three pieces of information, depending on the encapsulation plugin:

An Ethernet MAC address is required to determine the destination.
IP Address
An IP address is required. Both IPv4 and IPv6 addresses are supported.
An IP protocol level (TCP, UDP, SCTP, etc.) port is required.

The list of destination types that are supported by both the search and encapsulation plugins is listed in the section Plugins and their Properties.

The varpd service, mentioned above, is responsible for providing the virtual ARP daemon. Its responsibility is conceptually similar to ARP. It runs all instances of search plugins in the system and is responsible for answering the kernel's ARP-like questions for where packets should be sent.

The varpd service, svc:/network/varpd:default, must be enabled for overlay devices to function. If it is disabled while there are active devices, then most overlay devices will not function correctly and likely will end up dropping traffic.

Properties fall into three categories in the system:

  1. Generic properties all overlay devices have
  2. Properties specific to the encapsulation plugin
  3. Properties specific to the search plugin

Each property in the system has the following attributes, which mirror the traditional dladm(8) link properties:

The name of a property is namespaced by its module and always structured and referred to as as module/property. This allows for both an encapsulation and search plugin to have a property with the same name. Properties that are valid for all overlay devices and not specific to a module do not generally use a module prefix.

For example, the property vxlan/listen_ip is associated with the vxlan encapsulation module.

Each property in the system has a type. dladm(8) takes care of converting between the internal representation and a value, but the type influences the acceptable input range. The types are:
A signed integer that is up to eight bytes long ().
An unsigned integer that is up to eight bytes long ().
Either an IPv4 or IPv6 address in traditional string form. For example, 192.168.128.23 or 2001:470:8af4::1:1. IPv4 addresses may also be encoded as IPv4-mapped IPv6 addresses.
A string of ASCII or UTF-8 encoded characters terminated with a byte. The maximum string length, including the terminator, is currently 256 bytes.
Each property has permissions associated with it, which indicate whether the system considers them read-only properties or read-write properties. A read-only property can never be updated once the device is created. This generally includes things like the overlay's encapsulation module.
This property indicates whether the property is required for the given plugin. If it is not specified during a call to dladm create-overlay, then the overlay cannot be successfully created. Properties which have a will use that value if one is not specified rather than cause the overlay creation to fail.
The current value of a property, if the property has a value set. Required properties always have a value set.
The default value is an optional part of a given property. If a property does define a default value, then it will be used when an overlay is created and no other value is given.
Value ranges are an optional part of a given property. They indicate a range or set of values that are valid and may be set for a property. A property may not declare such a range as it may be impractical or unknown. For example, most properties based on IP addresses will not declare a range.

The following sections describe both the modules and the properties that exist for each module, noting their name, type, permissions, whether or not they are required, and if there is a default value. In addition, the effects of each property will be described.

The vxlan module is a UDP based encapsulation method. It takes a frame that would be put on the wire, wraps it up in a VXLAN header and places it in a UDP packet that gets sent out on the underlying network. For more details about the specific format of the VXLAN header, see vxlan(4P).

The vxlan module requires both an IP address and port to address it. It has a 24-bit virtual network ID space, allowing for virtual network identifiers that range from 0 - .

The vxlan module has the following properties:

Type: IP | Permissions: Read/Write | Required
The vxlan/listen_ip property determines the IP address that the system will accept VXLAN encapsulated packets on for this overlay.
Type: UINT | Permissions: Read/Write | Required
Default Value: 4789 | Range: 0 - 65535
The vxlan/listen_port property determines the UDP port that the system will listen on for VXLAN traffic for this overlay. The default value is 4789, the IANA assigned port for VXLAN.

The vxlan/listen_ip and vxlan/listen_port properties determine how the system will accept VXLAN encapsulated packets for this interface. It does not determine the interface that packets will be sent out over. Multiple overlays that all use VXLAN can share the same IP and port combination, as the virtual network identifier can be used to tell the different overlays apart.

Because search plugins may support multiple destinations, they may have more properties listed than necessarily show up for a given overlay. For example, the direct plugin supports destinations that are identified by both an IP address and a port, or just an IP address. In cases where the device is created over an overlay that only uses an IP address for its destination, then it will not have the direct/dest_port property.

The direct plugin is a point to point module that can be used to create an overlay that forwards all non-local traffic to a single destination. It supports destinations that are a combination of an IP Address and a port.

The direct plugin has the following properties:

Type: IP | Permissions: Read/Write | Required
The direct/dest_ip property indicates the IP address that all traffic will be sent out. Traffic will be sent out the corresponding interface based on traditional IP routing rules and the configuration of the networking stack of the global zone.
Type: UINT | Permissions: Read/Write | Required
Default Value: - | Range: 0 - 65535
The direct/dest_port property indicates the TCP or UDP port that all traffic will be directed to.
The files plugin implements a dynamic plugin that specifies where traffic should be sent based on a file. It is a glorified version of /etc/ethers. The dynamic plugin does not support broadcast or multicast traffic, but it has support for proxy ARP, NDP, and DHCPv4. For the full details of the file format, see overlay_files(5).

The files plugin has the following property:

Type: String | Permissions: Read/Write | Required
The files/config property specifies an absolute path to a file to read. The file is a JSON file that is formatted according to overlay_files(5).

Each overlay has the following properties which are used to give additional information about the system. None of these properties may be specified as part of a dladm create-overlay, instead they come from other arguments or from internal parts of the system.

String | Permissions: Read Only
The encap property contains the name of the encapsulation module that's in use.
UINT | Permissions: Read/Write
Default Value: 1400 | Range:
The mtu property describes the maximum transmission unit of the overlay. The default value is 1400 bytes, which ensures that in a traditional deployment with an MTU of 1500 bytes, the overhead that is added from encapsulation is all accounted for. It is the administrator's responsibility to ensure that the device's MTU and the encapsulation overhead does not exceed that of the interfaces that the encapsulated traffic will be sent out of.

To modify the mtu property, use dladm set-linkprop.

String | Permissions: Read Only
The search property contains the name of the search plugin that's in use.
String | Permissions: Read Only
The varpd/id property indicates the identifier which the varpd service uses for this overlay.
UINT | Permissions: Read/Write
The vnetid property has the virtual network identifier that belongs to this overlay. The valid range for the virtual network identifier depends on the encapsulation engine.

Overlay devices are wired into FMA, the illumos fault management architecture, and generates error reports depending on the search plugin in use. Due to limitations in FMA today, when a single overlay enters a degraded state, meaning that it cannot properly perform look ups or another error occurred, then it degrades the overall pseudo-device driver.

For more fine-grained information about which overlay is actually in a state, one should run dladm show-overlay -f. In addition, for each overlay in a degraded state a more useful diagnostic message is provided which describes the reason that caused this overlay to enter into a degraded state.

The overlay driver is self-healing. If the problem corrects itself on its own, it will clear the fault on the corresponding device.

vxlan(4P), overlay_files(5), dladm(8)

March 30, 2022 OmniOS