Custom Mavlink Messages
JoshuaHintze opened this issue · comments
Hello, I'm looking to convert our python source code over to typescript. Currently we use pymavlink to load in our XML definitions of our custom message types. All standard messages from common/arduino parse properly such as Ping, Heartbeat, SysStatus but uncommon ones fail here in this section of code:
private validatePacket(buffer: Buffer, Protocol: MavLinkProtocolConstructor) {
const protocol = new Protocol()
const header = protocol.header(buffer)
// @ts-ignore
const magic = MSG_ID_MAGIC_NUMBER[header.msgid]
if (magic) {
....
else {
...
// unknown message (as in not generated from the XML sources)
this.log.debug(`Unknown message with id ${header.msgid} (magic number not found) - skipping`)
return PacketValidationResult.UNKNOWN <---
}
I noticed a closed issue here that was asking some of the same: #16
If I return back VALID (ignoring a crc check) for the result it continues onto the PacketParser stream and finally to my on('data') method and things just work fine.
So my question is this....is there a way to add to the MSG_ID_MAG_NUMBER array with these custom messages? Or what would you suggest? Thanks
Ha! I didn't foresee custom messages being added to the mix! But it is a great idea. Let me think about it. I can't promise it'll be available in a day or two, but I like the idea, it's a completely valid case and I think, in the end, it needs to work.
Thank you @padcom . For now I just forked the repo and did the following:
switch (this.validatePacket(buffer, Protocol)) {
case PacketValidationResult.UNKNOWN:
case PacketValidationResult.VALID:
this.log.debug('Found a valid packet');
this._validPackagesCount++;
this.push({ buffer, timestamp: this.timestamp });
// truncate the buffer to remove the current message
this.buffer = this.buffer.slice(expectedBufferLength);
break;
case PacketValidationResult.INVALID:
this.log.debug('Found an invalid packet - skipping');
this._invalidPackagesCount++;
// truncate the buffer to remove the wrongly identified STX
this.buffer = this.buffer.slice(1);
break;
// case PacketValidationResult.UNKNOWN:
// this.log.debug('Found an unknown packet - skipping');
// this._unknownPackagesCount++;
// // truncate the buffer to remove the current message
// this.buffer = this.buffer.slice(expectedBufferLength);
// break;
}
Basically making UNKNOWN and VALID the same for now. However a suggestion would be that you expose a method or function in the mavlink library that we could push() onto the array of MSG_ID_MAGIC_NUMBER. Even better would be if that method took a export const messageRegistry = new Map<number, new (system_id: number, component_id: number) => MavLinkData>([ Map that already gets generated from the pymavlink code generator.
FYI we found a number of issues with the typescript generation and submitted a PR here: ArduPilot/pymavlink#778
Hello, this feauture would be great. I've tried using pymavlink typescript/javascript generators but they both have flaws, aside from the fact that I was not successful in getting the "official" node-mavlink library to work at all.
Also, @GimpMaster could you share how you managed to generate proper message mappings from the xml? How did you provide the correct CRCs for the MessageParser? At the moment all the my custom messages that have an id shared with the default ones are marked as invalid.
Thanks
I created a PR where, with some simple non-breaking modification, we should be able to use custom message registers: #22
Please, @padcom let me now what you think. If the pymavlink PR from @GimpMaster will be accepted as well, it would be awesome. However, it seems like the pymavlink community is not very active regarding pull requests...
@BearToCode Hi! I've looked over your proposed PR. I think it needs careful consideration - that's quite a big architectural change and I need to think about it if it is a step in the right direction. I'll keep you posted, but don't hold your breath.
Out of curiosity @GimpMaster @BearToCode, have you tried importing the MSG_ID_MAGIC_NUMBER constant and registering your command in there? Of course you need to know both the MSG_ID of your new command as well as the magic value, but I believe you'll be able to figure that out.
registering your command in there
What I mean by that is:
import { MSG_ID_MAGIC_NUMBER } from 'mavlink-mappings'
MSG_ID_MAGIC_NUMBER['99999'] = 42
That way, your new message will have a valid definition of the magic number and when deserializing the line
const magic = MSG_ID_MAGIC_NUMBER[header.msgid]
if (magic) {
...
will have a valid value for magic and everything else should automatically fall in place. I very much prefer that solution to the introduction of a customized registry and flipping everything upside down.
Hi, I still can't test this as the PR from @GimpMaster still has not been accepted (I might contact a contributor privately if they do not respond in the following days).
Anyway, this feels more like an hack than an intended feature. I think it would be better to have a more solid implementation, which could also not be the custom registries I implemented if you don't like the library to work that way
For example, if you like this approach we could add something like this:
export function setCustomRegistry(registry: MavlinkPacketRegistry) {
// clear MSG_ID_MAGIC_NUMBER
// for packet in registry
// MSG_ID_MAGIC_NUMBER[msgid] = magic
}This way the user does not need to know of the existance of MSG_ID_MAGIC_NUMBER at all
Hey guys, I know I have been somewhat silent. My PR works great for me. I would love to see the PR merged just so typescript works out of the box.
The only modification I made for magic numbers for custom messages was what I put in the first post, that is let unrecognized messages pass to the handlers and let them figure out if they can use that message or not.
For @padcom suggestion I'm not sure if MSG_ID_MAGIC_NUMBER adjustment would work as I think its a constant const but I haven't tried it to verify.
For example, if you like this approach we could add something like this:
export function setCustomRegistry(registry: MavlinkPacketRegistry) { // clear MSG_ID_MAGIC_NUMBER // for packet in registry // MSG_ID_MAGIC_NUMBER[msgid] = magic }This way the user does not need to know of the existance of
MSG_ID_MAGIC_NUMBERat all
@BearToCode I like the idea. There are just a couple of gotchas, though. Currently, the table of magic values for all known (thus precalculated) message types. This means that you may be reading packages that you didn't subscribe to which leads to you as the user being asked to tell which message types should be recognized at any given time...
I'm just thinking out loud here... bear with me..
I didn't really understand what you meant here:
For example, if you like this approach we could add something like this:
export function setCustomRegistry(registry: MavlinkPacketRegistry) { // clear MSG_ID_MAGIC_NUMBER // for packet in registry // MSG_ID_MAGIC_NUMBER[msgid] = magic }This way the user does not need to know of the existance of
MSG_ID_MAGIC_NUMBERat all@BearToCode I like the idea. There are just a couple of gotchas, though. Currently, the table of magic values for all known (thus precalculated) message types. This means that you may be reading packages that you didn't subscribe to which leads to you as the user being asked to tell which message types should be recognized at any given time...
I'm just thinking out loud here... bear with me..
Anyway, I updated my PR, tell me what you think now.
I tested the GimpMaster fork with the generated definitions and it works fine, so I adjusted the PR to use the generated registry from that. I also contacted a pymavlink mantainer, @khancyr, hoping he will take a look at the request.
That was an intense day!
A newer version of the mappings has been published! 1.0.15-20230328-0
This is a major refactoring of the generator, something that was long overdue. Now cool things will be possible (at least more possible than they were before).
Now I can properly start working on the problem of reading custom messages. @GimpMaster I'll revisit your code once again - since it's working let's give it a shot. Could you be so kind as to create a minimalistic example that would show reading a custom message? I want to be sure I am working with something you guys see too
@GimpMaster I think I know what's up now. Let me explain:
In your PR #22 your idea to allow for custom messages is to make another dependency than the MSG_ID_MAGIC_NUMBER one. The use of MSG_ID_MAGIC_NUMBER as one single place to understand message magic numbers without having a dependency on the actual message classes. That's, however, what your PR introduces and that's why I am not going to merge it in.
The official way of adding custom messages will come in the form of registration custom message magic number (in progress)
The API for it will be more/less as follows:
function registerCustomMessageMagicNumber(msgid: string, magic: number) { ...}
The registerCustomMessageMagicNumber will also check if the magic number has not been already registered and if that is the case, it will throw an error.
There are cases I can think of, where replacement of a predefined message (or a fix for one, who knows?!). For that the following API will be available:
function overrideMessageMagicNumber(msgid: string, magic: number) { ...}
Similarly to the previous function, it will check if the message is registered with magic numbers and if not it will throw an error.
Version 1.6.0 released
- updated mavlink mappings to the latest stand
- added API for managing magic values for custom messages
Closing - if that API won't work for you please reopen this issue.
@GimpMaster I think I know what's up now. Let me explain:
In your PR #22 your idea to allow for custom messages is to make another dependency than the
MSG_ID_MAGIC_NUMBERone. The use ofMSG_ID_MAGIC_NUMBERas one single place to understand message magic numbers without having a dependency on the actual message classes. That's, however, what your PR introduces and that's why I am not going to merge it in.The official way of adding custom messages will come in the form of registration custom message magic number (in progress)
The API for it will be more/less as follows:
function registerCustomMessageMagicNumber(msgid: string, magic: number) { ...}The
registerCustomMessageMagicNumberwill also check if the magic number has not been already registered and if that is the case, it will throw an error.There are cases I can think of, where replacement of a predefined message (or a fix for one, who knows?!). For that the following API will be available:
function overrideMessageMagicNumber(msgid: string, magic: number) { ...}Similarly to the previous function, it will check if the message is registered with magic numbers and if not it will throw an error.
Ok, but in my case I only have custom messages, of which I have many. The user(me) should not check that the message id is already used in a default registry :/
Magic numbers are used in the packet splitter and that's why it needs to be available everywhere. Otherwise you'll get massive memory usage while the buffer tries to find the next message tripping over the missing magic numbers all the time.
That why, even though it is technically feasible, you shouldn't do it.
Ok, but in my case I only have custom messages, of which I have many. The user(me) should not check that the message id is already used in a default registry :/
So if that is the case you're much better off using the generator! It's really easy!
- clone the mavlink-mappings repository and install dependencies
- edit it's
package.jsonand add your mapping there (or even remove all other mappings if you want!) - call
npm run regenerate
That way you'll get a set of mavlink mappings you can use directly in your code, including properly calculated magic numbers!
Once you have generated your mappings you can publish them in a git repository and reference those mappings instead of the ones published to NPM. This means that you'll be able to properly version your mappings which is quite important if more than one project uses a lib.
Ok, I understand, but having this design with a registerCustomMessageMagicNumber and a overrideMessageMagicNumber is kinda bad. I do not want to wrap my everything with a try/catch. Maybe just make a single function that accepts a list of options, that include the one to overwrite already present definitions. If a message is found and the options is not set, an error is thrown
Once you have generated your mappings you can publish them in a git repository and reference those mappings instead of the ones published to NPM. This means that you'll be able to properly version your mappings which is quite important if more than one project uses a lib.
Thanks, but I was hoping to use the generated definitions from @GimpMaster 's PR, if it gets accepted
Like I said, IMHO your best option, because you only have custom messages and you have a lot of them, is to use the generator and forget about all the magic madness. That PR you mentioned has been rejected.
I was talking about this one: ArduPilot/pymavlink#778 It's on pymavlink official repo, not this one
So I see you'll be generating things eitherway, right?
Yeah, but my team insists on using the "official" definitions... They will be used for a rocket, we want some extra safety
Cool! Good luck with your project!
Thanks, but could please consider introducing the pattern I talked before? It will work exactly as you want to, it's just a better implementation.
This one:
Ok, I understand, but having this design with a
registerCustomMessageMagicNumberand aoverrideMessageMagicNumberis kinda bad. I do not want to wrap my everything with a try/catch. Maybe just make a single function that accepts a list of options, that include the one to overwrite already present definitions. If a message is found and the options is not set, an error is thrown
I was just taking a peek at the pymavgen PR you mentioned and I am not sure I understand what's going on there... Is this generating classes that are based on the node-mavlink package? Am I seeing that right? Is this the "official" backend for generated mavlink thingies now?
I was just taking a peek at the pymavgen PR you mentioned and I am not sure I understand what's going on there... Is this generating classes that are based on the node-mavlink package? Am I seeing that right? Is this the "official" backend for generated mavlink thingies now?
Well, If it gets accepted it will be included in the official mavlink generation tool. @GimpMaster made it work with your library.
PS: please read the message above...
I don't know, guys... I mean, I'm flattered and everything, but I think it's not time yet to be making those kinds of decisions. I mean... let some pet projects cut their teeth first, maybe destroy a plane or two, before we put it in 🚀.
Nevertheless, AFAICR the deal was that node-mavlink and mavlink-mappings land under the Ardupilot org, but they won't replace the default generator (which is ugly as f.. very ugly - everywhere you look). So if you feel a bit adventurous and don't mind, no 100% feature completeness and no 100% feature parity (let's be honest, there's just the link exchange implementation and some definition classes) then you might want to look favorably on node-mavlink. Maybe it'll suit you, perhaps not.
It is very opinionated, and it will stay this way, even if it means a few broken dishes :)
Ok, I respect your decision. Anyway, the official "node-mavlink" (https://github.com/ifrunistuttgart/node-mavlink) does not work AT ALL, so I'll stick to your library. Also, the good thing of this library is that it does a great job at validating packets and does not need all the messages, but only the magic numbers. So, if the PR to the pymavlink gets accepted, we could use the officially generated messages and still stick to this library, as I already tested locally.
The little thing I was suggesting was just a minor design change, as I personally do not like having to deal with exceptions. Only that :)
Still, this is the only working NodeJS library that I was able to find.
I think we need to give a bit of respect to the butt-ugly pymavgen. AFAIK the guy writing all that code was not really a typescript developer, rather a Python dev that pulled together something that kinda worked. And since the system isn't used by JS devs then that rusty bit kinda stuck there.
I remember some time ago I was faced with the same problem as you are. I stopped at there's no npm package named node-mavlink and started hacking. My goal was initially to just send a couple of messages for the drone to perform some maneuvers like a mission, but calculated based on what happened. I learned that the Mavlink protocol is actually pretty nicely described and found the butt-ugly pymavgen (the king is dead - long live the king). I was literally SHOCKED I was allowed to release a package of that kind of name. And that's why you're using it - because the official code generator is broken and... I am not a python developer nor am I planning on becoming one so.. That's how you arrived at node-mavlink :)
I think I know how I can solve your problem in a way that won't hurt everyone. It will require just a tiny more work on your end but you will be able to specify a record of string and number to define msgid to magic resolution. Gimme a second.
Hey guys sorry I haven't been able to respond more timely. I'm seeing all the conversation which is great.
Just for my quick summary. Let me know if I'm wrong....
@padcom just released 1.6.0 which adds a couple new methods for adjusting the registry of messages. Awesome. I can try it out in a couple weeks when I get back to this project....
@BearToCode - As far as knowing what my magic numbers are.... I just ran my pymavlink fork and looked at the generated folder. Inside there are class for each of my custom messages. For example here is one...
import {MavLinkData, MavLinkPacketField} from 'node-mavlink';
/*
Sent at a set periodic interval by a component to report its status to other components.
**BROADCAST MESSAGE**
*/
/* uptime Uptime of the source component in ms uint32_t */
export class R1ComponentStatus extends MavLinkData {
public uptime!: number;
static MSG_ID: number = 42900;
static MSG_NAME: string = 'R1_COMPONENT_STATUS';
static MAGIC_NUMBER: number = 212;
static FIELDS: MavLinkPacketField[] = [
/* Uptime of the source component in ms */
new MavLinkPacketField('uptime', 'uptime', 0, false, 4, 'uint32_t', '[ms]'),
];
static PAYLOAD_LENGTH: number = 4;
}
In there I have both the MSG_ID and the MAGIC_NUMBER. Sadly I don't see a file that gives me a straight dump of <MSG_ID, MAGIC_NUMBER> in an array format.
There is a message-registry.ts file that looks like this...
export const messageRegistry = new Map<number, new (system_id: number, component_id: number) => MavLinkData>([
[42900, R1ComponentStatus],
....
]);
So maybe I'll just loop over all this and generate my own array of those values for passing into @padcom new functions.
Again looking forward to trying in next version.
It would be cool to get my fork merged into pymavlink so we can have stable base for typescript generation.
OK, so keeping things separated yet override-able here's how you would work with the 2.0 API:
const source = ... // some kind of stream
const reader = source
.pipe(new MavLinkPacketSplitter({}, { magicNumbers: { '0': 50 } }))
.pipe(new MavLinkPacketParser())I think it is kind of similar to what you wanted to introduce, just with a bigger change in the API where the second parameter for the splitter is not a callback but a configuration object.
Check out 2.0.0-alpha.0 - it's got the change in the splitter constructor parameters. It may be what you need!
You can try the 2.0.0-alpha.1 - it has added a bit of documentation to it about customizing things
https://www.npmjs.com/package/node-mavlink/v/2.0.0-alpha.1#registering-custom-messages
In there I have both the MSG_ID and the MAGIC_NUMBER. Sadly I don't see a file that gives me a straight dump of <MSG_ID, MAGIC_NUMBER> in an array format.
@GimpMaster That's how you get at the default generated list of magic numbers:
import { MSG_ID_MAGIC_NUMBER } from 'mavlink-mappings'
// or
import { MSG_ID_MAGIC_NUMBER } from 'node-mavlink'Both point to the same registry (key: string, value: number).
I just ran my pymavlink fork and looked at the generated folder. Inside there are class for each of my custom messages
Well, the generation of magic numbers is described here: https://mavlink.io/en/guide/serialization.html#crc_extra
@GimpMaster For your convenience, I've exposed the function that calculates the correct magic value:
https://github.com/ArduPilot/node-mavlink-mappings/blob/master/lib/utils.ts#L30
So, all you need to do is basically install the mavlink-mappings@latest package (version >= 1.0.16-20230329-0) and import that function. It's fully typed (I hope!) so you'll know what goes where. And magically, it spits outs the correct magic value (aka crc_extra, but both are magic words, so...)
console.log(calculateCrcExtra('MAV_TMP12', [
{ extension: false, fieldSize: 4, itemType: 'uint32_t', source: { name: 'a_b_c', type: 'uint32_t' } },
]));)
I need to put some documentation around it, especially describing the expected data types and how arrays are handled... coming soon.
Thank you @padcom.
Here is how I managed to piece this all together (@GimpMaster PR and the magicNumbers packet splitter option from the library).
import { messageRegistry } from 'path/to/generated/message-registry';
import {
MavLinkPacketParser,
MavLinkPacketSplitter,
MavLinkDataConstructor,
} from 'node-mavlink';
function getCustomMagicNumbers() {
const record: Record<string, number> = {};
Array.from(messageRegistry.entries())
.forEach(([msgId, constructor]) => {
record[msgId.toString()] = (constructor as MavLinkDataConstructor<any>).MAGIC_NUMBER;
});
return record;
}
const MAGIC_NUMBERS = getCustomMagicNumbers();
// ...
const mavlink = port
.pipe(new MavLinkPacketSplitter({}, { magicNumbers: MAGIC_NUMBERS }))
.pipe(new MavLinkPacketParser());I think this issue can be finally considered solved :) . @GimpMaster , it would be great from you to add the getCustomMagicNumbers() function directly in the generated definitions, so users will find everything packed and ready! You may also mark this answer as the "correct answer", so people looking for this will immediately find how to get this working.
And that looks awesome! It finally puts it in the hands of the user to decide what the system should and shouldn't do with the option to redefine each and every packet if one needs to
@GimpMaster @BearToCode I'd appreciate if you could confirm for me that the version alpha I published yesterday (or was it already Today?) works and we can go out of alpha into beta, rc and final (or I'll just skip the red tape and go straight to 2.0 GA)
it would be great from you to add the getCustomMagicNumbers() function directly in the generated definitions, so users will find everything packed and ready
That'll work for you, but not for everyone. The core concept is that I don't care where the magic numbers come from and it needs to stay this way. But adding it to the docs - no problem. Maybe you want to make a PR for the README.md? There's already a section that touches on the topic of custom messages, so maybe you can add it there... Or even create an example with a custom-made message?
@GimpMaster @BearToCode I'd appreciate if you could confirm for me that the version alpha I published yesterday (or was it already Today?) works and we can go out of alpha into beta, rc and final (or I'll just skip the red tape and go straight to 2.0 GA)
Can confirm 2.0.0-alpha.1 to be working for me!
it would be great from you to add the getCustomMagicNumbers() function directly in the generated definitions, so users will find everything packed and ready
That'll work for you, but not for everyone. The core concept is that I don't care where the magic numbers come from and it needs to stay this way. But adding it to the docs - no problem. Maybe you want to make a PR for the README.md? There's already a section that touches on the topic of custom messages, so maybe you can add it there... Or even create an example with a custom-made message?
The suggestion was for @GimpMaster , the library is perfectly fine as of now. I'd wait to see what happens to the pymavlink PR before updating any docs
Guys, since you're the only ones that brought up the issue of code generation, I think this is the right forum to let you know that there are changes coming, cool new things for the very few that would benefit from it.
https://github.com/ArduPilot/node-mavlink-mappings/#generating-custom-definitions
So, if you upgrade mavlink-mappings to 1.0.17-20230329-0 it's all yours.
Awesome. Thank you for your work!
@GimpMaster @BearToCode I'd appreciate if you could confirm for me that the version alpha I published yesterday (or was it already Today?) works and we can go out of alpha into beta, rc and final (or I'll just skip the red tape and go straight to 2.0 GA)
I will give it a try today and report back shortly.
Released with node-mavlink 2.0.1
I'm getting this error
Error: Cannot find module 'xml2js'
It's listed as a dev-dependency but not a full dependency. Is that correct?
I can confirm that after manually adding xml2js to my project the custom messages DO WORK! Nice job. Also thank you @BearToCode . I think I might add in the getCustomMagicNumbers() method to PR I have going with pymavlink making it extremely easy for anyone to get it.
I need to add that information about the xml2js to the docs. Good catch!
Done. mavlink-mappings 1.0.17-20230329-1
It's listed as a dev-dependency but not a full dependency. Is that correct?
Do you think it'd be better for the parser to be an optional peer dependency?
I couldn't even run without loading it. I would probably just add it as a regular dependency.
@BearToCode - PR updated with your generateCustomMessages function and comments.
I finished cleaning up the code and its dependencies. Currently we have not 2 but 3 packages:
- node-mavlink
- mavlink-mappings
- mavlink-mappings-gen
Everything related to actual code generation has been moved to mavlink-mappings-gen but at the same time everything is re-exported by mavlink-mappings and node-mavlink, so whichever package you're going to get the generator is always there for you.
That concludes work on extending the mavlink library with implementation of custom messages.