overlay —
Overlay Devices
Overlay devices are a GLDv3 device that allows users to create overlay networks
that can be used to form the basis of network virtualization and software
defined networking. Overlay networks allow a single physical network, often
called an underlay network, to provide the means for
creating multiple logical, isolated, and discrete layer two and layer three
networks on top of it.
Overlay devices are administered through
dladm(8). Overlay devices themselves
cannot be plumbed up with IP, vnd, or
any other protocol. Instead, like an etherstub, they allow
for VNICs to be created on top of them. Like an etherstub,
an overlay device acts as a local switch; however, when it encounters a
non-local destination address, it instead looks up where it should send the
packet, encapsulates it, and sends it out another interface in the
system.
A single overlay device encapsulates the logic to answer two
different, but related, questions:
- How should a packet be transformed and put on the wire?
- Where should a transformed packet be sent?
Each of these questions is answered by a plugin. The first
question is answered by what's called an encapsulation
plugin. The second question is answered by what's called a
search plugin. Packets are encapsulated and decapsulated
using the encapsulation plugin by the kernel. The search plugins are all
user land plugins that are consumed by the varpd service whose FMRI is
svc:/network/varpd:default. This separation allows for the
kernel to be responsible for the data path, while having the search plugins
in userland allows the system to provide a much more expressive
interface.
Overlay devices come in two different flavors, one where all packets are always
sent to a single address, the other, where the destination of a packet varies
based on the target MAC address of the packet. This information is maintained
in a target table, which is independent and unique to each
overlay device. We call the plugins that send traffic to a single location,
for example a single unicast or multicast IP address, a point to
point overlay and the overlay devices that can send traffic to different
locations based on the MAC address of that packet a dynamic
overlay. The plugin type is determined based on the type of the
search plugin. These are all fully listed in the section
Plugins and their
Properties.
Both encapsulation and search plugins define the kinds of destinations that they
know how to support. An encapsulation plugin always has a single destination
type that's determined based on how the encapsulation is defined. A search
plugin, on the other hand, can support multiple combinations of destinations.
A search plugin must support the destination type of the encapsulation device.
The destination may require any of the following three pieces of information,
depending on the encapsulation plugin:
- MAC Address
-
An Ethernet MAC address is required to determine the
destination.
- IP Address
-
An IP address is required. Both IPv4 and IPv6 addresses are
supported.
- Port
-
An IP protocol level (TCP, UDP, SCTP, etc.) port is
required.
The list of destination types that are supported by both the
search and encapsulation plugins is listed in the section
Plugins and their
Properties.
The varpd service, mentioned above, is responsible for providing the virtual ARP
daemon. Its responsibility is conceptually similar to ARP. It runs all
instances of search plugins in the system and is responsible for answering the
kernel's ARP-like questions for where packets should be sent.
The varpd service, svc:/network/varpd:default, must be enabled for
overlay devices to function. If it is disabled while there are active
devices, then most overlay devices will not function correctly and likely
will end up dropping traffic.
Properties fall into three categories in the system:
- Generic properties all overlay devices have
- Properties specific to the encapsulation plugin
- Properties specific to the search plugin
Each property in the system has the following attributes, which
mirror the traditional dladm(8) link
properties:
- Name
-
The name of a property is namespaced by its module and
always structured and referred to as as module/property. This allows for
both an encapsulation and search plugin to have a property with the same
name. Properties that are valid for all overlay devices and not specific
to a module do not generally use a module prefix.
For example, the property vxlan/listen_ip is
associated with the vxlan encapsulation module.
- Type
-
Each property in the system has a type.
dladm(8) takes care of converting
between the internal representation and a value, but the type influences
the acceptable input range. The types are:
- INT
- A signed integer that is up to eight bytes long
(int64_t).
- UINT
- An unsigned integer that is up to eight bytes long
(uint64_t).
- IP
- Either an IPv4 or IPv6 address in traditional string form. For
example, 192.168.128.23 or 2001:470:8af4::1:1. IPv4 addresses may also
be encoded as IPv4-mapped IPv6 addresses.
- STRING
- A string of ASCII or UTF-8 encoded characters terminated with a
NUL byte. The maximum string length, including the
terminator, is currently 256 bytes.
- Permissions
-
Each property has permissions associated with it, which
indicate whether the system considers them read-only properties or
read-write properties. A read-only property can never be updated once the
device is created. This generally includes things like the overlay's
encapsulation module.
- Required
-
This property indicates whether the property is required for
the given plugin. If it is not specified during a call to
dladm create-overlay, then the overlay cannot be
successfully created. Properties which have a default
will use that value if one is not specified rather than cause the overlay
creation to fail.
- Current Value
-
The current value of a property, if the property has a value
set. Required properties always have a value set.
- Default Value
-
The default value is an optional part of a given property.
If a property does define a default value, then it will be used when an
overlay is created and no other value is given.
- Value ranges
-
Value ranges are an optional part of a given property. They
indicate a range or set of values that are valid and may be set for a
property. A property may not declare such a range as it may be impractical
or unknown. For example, most properties based on IP addresses will not
declare a range.
The following sections describe both the modules and the
properties that exist for each module, noting their name, type, permissions,
whether or not they are required, and if there is a default value. In
addition, the effects of each property will be described.
- vxlan
- The vxlan module is a UDP based encapsulation method. It
takes a frame that would be put on the wire, wraps it up in a VXLAN header
and places it in a UDP packet that gets sent out on the underlying
network. For more details about the specific format of the VXLAN header,
see vxlan(4P).
The vxlan module requires both an
IP address and port to address it.
It has a 24-bit virtual network ID space, allowing for virtual network
identifiers that range from 0 -
16777215.
The vxlan module has the following
properties:
- vxlan/listen_ip
-
Type: IP | Permissions:
Read/Write | Required
The vxlan/listen_ip property
determines the IP address that the system will accept VXLAN
encapsulated packets on for this overlay.
- vxlan/listen_port
-
Type: UINT | Permissions:
Read/Write | Required
Default Value: 4789 | Range:
0 - 65535
The vxlan/listen_port property
determines the UDP port that the system will listen on for VXLAN
traffic for this overlay. The default value is 4789,
the IANA assigned port for VXLAN.
The vxlan/listen_ip and
vxlan/listen_port properties determine how the system
will accept VXLAN encapsulated packets for this interface. It does not
determine the interface that packets will be sent out over. Multiple
overlays that all use VXLAN can share the same IP and port combination,
as the virtual network identifier can be used to tell the different
overlays apart.
Because search plugins may support multiple destinations, they may have more
properties listed than necessarily show up for a given overlay. For example,
the direct plugin supports destinations that are identified
by both an IP address and a port, or just an IP address. In cases where the
device is created over an overlay that only uses an IP address for its
destination, then it will not have the direct/dest_port
property.
- direct
- The direct plugin is a point to point module that can be
used to create an overlay that forwards all non-local traffic to a single
destination. It supports destinations that are a combination of an
IP Address and a port.
The direct plugin has the following
properties:
- direct/dest_ip
-
Type: IP | Permissions:
Read/Write | Required
The direct/dest_ip property
indicates the IP address that all traffic will be sent out. Traffic
will be sent out the corresponding interface based on traditional IP
routing rules and the configuration of the networking stack of the
global zone.
- direct/dest_port
-
Type: UINT | Permissions:
Read/Write | Required
Default Value: - | Range:
0 - 65535
The direct/dest_port property
indicates the TCP or UDP port that all traffic will be directed
to.
- files
- The files plugin implements a dynamic
plugin that specifies where traffic should be sent based on a file. It is
a glorified version of /etc/ethers. The dynamic plugin
does not support broadcast or multicast traffic, but it has support for
proxy ARP, NDP, and DHCPv4. For the full details of the file format, see
overlay_files(5).
The files plugin has the following
property:
- files/config
-
Type: String | Permissions:
Read/Write | Required
The
files/config property specifies
an absolute path to a file to read. The file is a JSON file that is
formatted according to
overlay_files(5).
Each overlay has the following properties which are used to give additional
information about the system. None of these properties may be specified as
part of a dladm create-overlay, instead they come from other
arguments or from internal parts of the system.
- encap
-
String | Permissions: Read
Only
The encap property contains the name of
the encapsulation module that's in use.
- mtu
-
UINT | Permissions:
Read/Write
Default Value: 1400 | Range:
576 - 9000
The
mtu property describes the maximum
transmission unit of the overlay. The default value is
1400 bytes, which ensures that in a traditional
deployment with an MTU of 1500 bytes, the overhead that is added from
encapsulation is all accounted for. It is the administrator's
responsibility to ensure that the device's MTU and the encapsulation
overhead does not exceed that of the interfaces that the encapsulated
traffic will be sent out of.
To modify the mtu property, use
dladm set-linkprop.
- search
-
String | Permissions: Read
Only
The search property contains the name
of the search plugin that's in use.
- varpd/id
-
String | Permissions: Read
Only
The varpd/id property indicates the
identifier which the varpd service uses for this
overlay.
- vnetid
-
UINT | Permissions:
Read/Write
The vnetid property has the virtual
network identifier that belongs to this overlay. The valid range for the
virtual network identifier depends on the encapsulation engine.
Overlay devices are wired into FMA, the illumos fault management architecture,
and generates error reports depending on the search plugin
in use. Due to limitations in FMA today, when a single overlay enters a
degraded state, meaning that it cannot properly perform look ups or another
error occurred, then it degrades the overall overlay
pseudo-device driver.
For more fine-grained information about which overlay is actually
in a degraded state, one should run dladm
show-overlay -f. In addition, for each overlay in a degraded state a
more useful diagnostic message is provided which describes the reason that
caused this overlay to enter into a degraded state.
The overlay driver is self-healing. If the problem corrects itself
on its own, it will clear the fault on the corresponding device.