Component Metadata Protocol (WIP)

WARNING

This service is still marked as "work in progress", and should not be relied upon in production. It has also evolved since first created as the "Component Information Protocol".

The Component Metadata Protocol is a MAVLink service for requesting metadata from (and about) MAVLink components. It is intended to provide autopilot- and version- independent feature discovery and configuration, allowing a GCS to configure its UI and/or a device without knowing anything about the connected system.

Information shared using this service may include:

• What types of component information are supported (by this component).
• What MAVLink commands are supported (both in missions and in other modes).
• Parameter metadata for parameters supported by the vehicle.
• Metadata about events emitted by the system
• Self-describing configuration UIs (i.e. similar to MAVLink camera configuration files).
• Translations of other metadata.

Component metadata is specified in appropriately formatted JSON files (which may be .xz compressed). The component metadata protocol is used to request the location of the general metadata file file, which is then parsed to get the location of other metadata files supported by the component.

Information supplied by the service is assumed to be invariant after boot. There is no mechanism, for example, to provide an update if the set of supported parameters was to change after boot.

Message/Enum Summary

Message	Description
COMPONENT_METADATA	Message providing a download url and CRC for the general metadata component information file. The message is requested using MAV_CMD_REQUEST_MESSAGE.
MAV_CMD_REQUEST_MESSAGE	Use this command to request that a component emit COMPONENT_METADATA. Use param1=397 (the message id of COMPONENT_METADATA).

Enum	Description
COMP_METADATA_TYPE	Types of component metadata supported by the protocol - e.g. general information, parameter metadata, supported commands, events, peripherals, etc. The type identifiers are used in the "general" metadata file to identify the sections that provide information about each supported type of metadata.

INFO

COMPONENT_INFORMATION is not used by thie service (it is a deprecated legacy version of COMPONENT_METADATA).

Component Information File Format (Schema)

Component information files are written in JSON and must conform to the schema definitions found in the folder /component_metadata. The component information file types and schema are (at time of writing):

Type	Type Id	Schema	Description
General metadata	COMP_METADATA_TYPE_GENERAL	general.schema.json	General information about the component including hardware/firmware vendor version. This metadata includes information about all the other metadata types are supported by the component and where their metadata files are located. This metadata type must be supported if this protocol is supported and the file must be present on vehicle and delivered by MAVLink FTP. Note however that you never actually need to specify this type!
Parameter metadata	COMP_METADATA_TYPE_PARAMETER	parameter.schema.json	Information about parameters supported by the vehicle (on boot).
Command protocol metadata	COMP_METADATA_TYPE_COMMANDS	TBD	Information about which commands and command paramters are supported in via the command protocol.
Peripheral metadata	COMP_METADATA_TYPE_PERIPHERALS	peripherals.schema.json	Information about non-MAVLink peripherals connected to vehicle (on boot).
Event metadata	COMP_METADATA_TYPE_EVENTS	TBD	Information about events interface support by the vehicle.
Event metadata	COMP_METADATA_TYPE_ACTUATORS	actuators.schema.json	Metadata for actuator configuration (motors, servos and vehicle geometry) and testing.

All schema files are versioned using a version integer.

Schema versions are backwards compatible - i.e. a ground station that was able to populate its UI from a file based on an older schema should be able to do so from a newer version (albeit losing information provided by the newer format).

Generally this means that new versions may add fields but should not delete them, and also that a recipient can ignore fields that it does not understand.

The schema are currently a work in progress and can be modified as needed. Once accepted, they will be under change control (managed in a similar way to standard MAVLink messages).

File Locations/URLs

General metadata files must be stored on the device, and will usually be .xz compressed. The location of these files is returned in the COMPONENT_METADATA uri field.

Other component information files may be hosted on either the device or on the internet.

INFO

Where permitted by memory constraints you should host component information on the device (so that it is always available and cannot get out of sync).

Files on the device are downloaded using MAVLink FTP. The URI format is defined in MAVLink FTP URL Scheme. A typical parameter metadata URI might look like this: mftp:///component_metadata/parameters.json.xz.

Files on the Internet are downloaded using HTTPS or HTTP via a normal web URL (e.g. https://some_domain/component_metadata/parameters.json).

Metadata Caching (CRC)

The COMPONENT_METADATA message includes the file_crc field, which contain CRC32 values calculated for the file referenced in the uri field. A ground station should cache downloaded component metadata and only update it if the CRC value changes.

The general metadata file similarly provides file locations for other metadata supported by a component. It will also include CRC32 values any files that contain only static data (no CRC32 should be supplied for metadata files that might be updated dynamically).

File Compression

Component information files may be .xz compressed (this is recommended for files that are hosted on the device).

INFO

The prototype implementation generates and compresses component information files at build time. No compression library is required within the flight stack itself.

WARNING

Systems that request component information must support extraction of .xz-compressed JSON files.

TIP

The Tukaani Project XZ Embedded library is an easy-to-use XZ compression library for embedded systems and cross-platform C/C++ projects.

Sequences

Component Discovery

A GCS can broadcast the MAV_CMD_REQUEST_MESSAGE specifying param1=397; all components that support the protocol should respond with COMPONENT_METADATA.

A GCS can further discover all components in the system by monitoring the channel for HEARTBEAT ids, and then send the request to each of them to verify whether the protocol is supported. The broadcast approach is recommended for GCSes that don't track all components on the link.

Check if Protocol is Supported

A system can query whether another component supports the protocol by sending the command MAV_CMD_REQUEST_MESSAGE (specifying the COMPONENT_METADATA message).

The component will respond with COMPONENT_METADATA.uri containing a valid URI if the protocol is supported. If the protocol is not supported the component will ACK that the message with MAV_RESULT_UNSUPPORTED, MAV_RESULT_DENIED or MAV_RESULT_FAILED, or return a null value in uri.

INFO

A component that supports this service must return a general metadata file URI that is hosted on the vehicle (accessed using MAVLink FTP).

Get MetaData

The basic sequence is shown below.

In summary:

1. GCS (client) sends MAV_CMD_REQUEST_MESSAGE to component (server) specifying param1=397.
   • This is a normal command protocol request with timeouts and resends based on the ACK.
2. The component will ACK the command and immediately send the requested COMPONENT_METADATA message (populated with URI and CRC for the general metadata file).
   • A CMD_ACK of anything other than MAV_RESULT_ACCEPTED indicates the protocol is not supported (sequence completes).
3. GCS waits for the COMPONENT_METADATA message
   • If not recieved the GCS should resend the request (typically in the application level).
   • Once information is received:
     • the GCS checks if COMPONENT_METADATA.file_crc matches its cached CRC value. If so, there is no need to download the general metadata file (or other files it references) as it has not changed since the last download. If the cached values do not match the associated files should be downloaded and parsed (vehicle firmware has updated).
4. GCS downloads the file specified in the uri using MAVLink FTP.
5. GCS parses the general metadata for other supported metadata locations, and then downloads the files via MAVFTP or HTTP(s). This may be done immediately, or as needed.

Metadata Types

Actuators (COMP_METADATA_TYPE_ACTUATORS)

The actuators metadata allows a GCS to create a UI to configure and test actuators, and configure vehicle geometries, without having to understand anything about the underlying flight stack.

INFO

The mechanism works similarly to camera definition files. It can be used for flight stacks that allow outputs and geometry definition to be configured using parameters. If flight stack outputs or geometries cannot be configured using parameters, the mechanism can still be used to display the current geometry and output mappings, and to test outputs.

The definition contains information about actuator output drivers (e.g. PWM MAIN or UAVCAN), the functions that can be assigned to each output channel, supported geometries, and their configuration parameters. Detailed information can be found in the schema file, and there's also an example.

Specifically, the following information is provided:

• A list of actuator output drivers (e.g. PWM MAIN or UAVCAN): outputs_v1. This can be hardware-specific. Each output driver contains one or more groups of output channels. A group contains a common set of configuration parameters, indexed for each channel. A parameter may be assigned a specific meaning, e.g. disarmed. A GCS can use this information to provide specific actions for these (without having to know all disarmed parameters a priori).

• Actuator output functions: functions_v1. A list of the output functions (hardware) that can be connected to a particular output channel, for example: Motor 1, Landing Gear, Camera capture. Each actuator output channel is expected to provide a parameter that can be used to configure its output function.

• A geometry/mixer section: mixer_v1. A list of frame geometries, where at most one geometry is selected (via parameter), and a list of actuator types. An actuator type (e.g. servo or motor) contains limits and defaults for actuator testing, and a set of output function values that can be assigned to this type. A GCS may use the type to render the geometry, so it can display different images depending on the type.

  Each mixer contains one or more groups of actuators, where each group belongs to an actuator type. The group can contain a fixed or configurable number of actuators and a set of indexed configuration parameters. If the size is fixed, the actuator group can contain lists of fixed values, e.g. to provide position information for non-configurable actuators. As with the actuator outputs, parameters may be assigned a specific meaning, e.g. posx, which hints to the GCS that this parameter/value defines the x position of the actuator. This can be used to dynamically render a vehicle's geometry.

  Additionally there is an optional list of rules. Rules are used to constrain or hide/disable geometry parameters depending on the value of a selection parameter. For example there could be a parameter to select the control surface type, and three parameters to configure the roll, pitch and yaw torque. When the user sets the type to 'Left Aileron', certain restrictions to roll and pitch torque are applied, and the yaw torque is hidden.

A GCS can provide a UI for testing outputs based on the configured output functions, by iterating over all output channels and collecting the configured actuator output functions, and then utilizing the MAV_CMD_ACTUATOR_TEST command.

Translation

At high-level, metadata translation works as follows:

• The metadata provider sets the translationUri in general metadata file for each metadata type that supports translation. Note that the URL has no associated CRC as the translation data can change independently of metadata (for example, adding or changing translations).
• The translationUri URL points to a summary JSON file that contains links to the separate files that contain each translation of the particular metadata type. The translation summary JSON file also contains modification timestamps for each linked translation file so that a GCS can determine whether a particular file has been updated. The translation files are in TS file format, and may optionally be compressed in .xz format.
• A client (GCS) downloads the summary file then uses it to locate and download the translation file(s) it is interested in.
• The client can then apply the downloaded translations to the metadata json file(s) (which contains annotations for which tags to translate).

Caching

The following caching strategy is recommended for clients:

• Locally cache the downloaded translation files. These should be used until successfully replaced with a newer version.
• After 3 days attempt to download the summary JSON file again.
• Translation files can either be downloaded whenever the summary is downloaded or only when needed (because a modification timestamp has changed in the summary).

File Formats

The metadata json contains a translation section, such as this one. The translation section follows this schema, which is used to extract the translation strings into a TS file (see below for a script), and by the client to know which strings to translate. The TS file may be xz compressed.

This allows to add new metadata without having to change the translation implementation in the client.

The summary json has the following form:

{
  "<locale>": {
    "url": "<file url>.ts.xz",
    "last-modified": "<ISO 8601 timestamp>"
  }
  // ...
}

For example:

{
  "fr_FR": {
    "url": "https://raw.githubusercontent.com/PX4/PX4-Metadata-Translations/main/translated/parameters_fr_FR.ts.xz",
    "last-modified": "2023-03-22T06:15:59.203476+00:00"
  },
  "de_DE": {
    "url": "https://raw.githubusercontent.com/PX4/PX4-Metadata-Translations/main/translated/parameters_de_DE.ts.xz",
    "last-modified": "2023-03-22T06:15:59.199476+00:00"
  }
}

Hosting Translations

Any server can be used to host translations. The following example uses github.com, as it is easy to set up, automate, and download files.

The example repository is https://github.com/PX4/PX4-Metadata-Translations:

• metadata/ contains the untranslated metadata JSON files.
• to_translate contains the TS files to translate. This is generated from the files in metadata/ using scripts/prepare_ts.py.
• A translation service, such as crowdin can be used to translate the files
• translated/ contains translated metadata TS files (in this case synced back from Crowdin)
• scripts/update_summary.py is executed to update the summary JSON file with translation file locations and modification dates.

Open Issues

Schema Management

Schema management has not yet been signed off.

Guaranteed Availablility of Component Information

There is a concern that vehicles reliant on internet-hosted component information files may stop working if the hosting ceases.

This can generally be avoided by hosting the files compressed on-vehicle.

We propose that manufacturers that use autopilots with limited flash (1MB or below) and do custom firmware development should host the files in github.