Plans for v3.0 - discussion

Question

Plans for v3.0 - discussion

Jacajack opened this issue 4 years ago · comments

Hi!

It's been over 2 years since the v2.0 release, the last major update to liblightmodbus. I think that many things can be still improved and cleaned up. Since there are apparently quite a few people using this library, I feel that I owe you all a next release. But before I break anything, I'd like to hear your suggestions and ideas.

An overview of currently planned changes can be found here. If you have anything to suggest, please write a new comment below or open a new issue.

Please let me know what you think.

ehughes · Answer 1 · Tue Nov 17 2020 22:30:57 GMT+0800 (China Standard Time)

Hello Jack. I am using the library in some embedded targets.

https://resources.altium.com/p/demand-digital-manufacturing-and-32-bit-microcontroller-everyone

In my use case, I am using the experimental callback mechanism as registers in use are no contiguous. In one case, I have a register that is writable (a special parameter) only if another register is set to a specific value. What I would really like is some more control in the callback to return an error to flag a return message of "Illegal Function" or "Illegal Data".

When the callback is called with "MODBUS_REGQ_W_CHECK", it would be nice of the "value" field was set to what was going to be written. That way I could reject based upon specific values of the register, not just the address.

It is very possible that this code exists now and I am not properly using the calllback.

Either way, thanks for the library. I really like how the physical layer is separated. It made integration much easier as compared to some of the other libraries. We did some extensive testing/fuzzing on it and it works well.

Jacek Wieczorek · Answer 2 · Wed Nov 18 2020 06:10:17 GMT+0800 (China Standard Time)

@ehughes Hello and thank you for your kind words. I'm very happy to hear that the library serves you well.

I really like the idea of register value being passed to callback function during write protection check - I've never thought of that. Sadly, in the current version, the value passed to the callback during access check is always set to 0.

I suppose it's worth mentioning that all kinds of interdependent registers introduce some problems when it comes to writing multiple values at once. Perhaps the safest approach is to disable functions 16 and 15 on such registers. That means that the callback function should also accept Modbus function ID as a parameter.

Giving the user more control over what kind of error frame is generated is a good idea too. I suppose we could have the callback return ModbusError though an additional parameter. The only requirement would be that the callback function returns the error during the access check phase, because once writing registers starts, there's no turning back.

Thank you for your insight. I've updated the TODO list.

saeidou · Answer 3 · Thu Jan 14 2021 23:02:57 GMT+0800 (China Standard Time)

Hi Jack. I am using the library, too.
May I know why you merged the functions 01-02 and 03-04?

Initially I tried to use dynamic memory but for unknown reasons on atxmega32a4 I got some mysterious behaviors. However, by changing that to static memory it became fine.

Jacek Wieczorek · Answer 4 · Fri Jan 15 2021 00:18:50 GMT+0800 (China Standard Time)

Hi,

It's because I wanted to avoid code repetition. The logic behind reading discrete inputs and coils is exactly the same, only the source array is different.

Initially I tried to use dynamic memory but for unknown reasons on atxmega32a4 I got some mysterious behaviors. However, by changing that to static memory it became fine.

I've never worked with ATXmegas before, but it's good to hear that you managed to solve that issue. This seems to prove the point that memory management should be left to the user in v3.0.

Matteo Franceschini · Answer 5 · Thu Feb 18 2021 02:06:35 GMT+0800 (China Standard Time)

Hi Jack! Thanks for the library, I really like the optimizations and how it's written.

I use your library on a dsPIC33CK and, after some time, I was able to make it work.

Here my two cents:

I actually don't use cmake for configuration (I have your library imported in my MPLABX project), I'd rather prefer to have a "libconf.h" written by hand that has everything I need. Took me a while to understand which exact defines are needed to make everything work. I may help you write down some documentation, if needed, for those like me.
not every compiler has "inttypes.h", XC16 does not have it: I ended adding a fake "inttypes.h" that has just a line "#include <stdint.h>" and from what I've seen, it works well. I'm not using dynamic allocation, though. If the whole "inttypes.h" is not needed, probably stdint is enough and AFAIK is more common to have it already there.
dsPICs have 16-bit memory alignment: it took me a loong time to understand why the lib was not working with registers. Sadly, it was due to the "response0304" struct that, when packed, breaks the two bytes alignment with the values field, I don't think that memory management should be left to the user, but I guess that writing by default one byte at the time could be the best option for many.
I don't know if you're aware of its existance, but the CException library could be a wonderful resource when dealing with error handling. You might take a look, if interested.

With respect to issue number 3, I'd say that it probably took me more time than needed, because you already gave me the solution, but I was too blind to see it: you had already patched a misalignment issue on the coil library, I just needed to copy it on the registers. I'll may send a PR with this fix, but I haven't looked at the parts I don't use (dynamic allocation and modbus master) so, for now, I just post here the solution for those that, like me, have memory issues with static allocation: just copy these lines in the sregs.c file, from line 120 to 126.

THIS

#else
		for ( i = 0; i < count; i++ )
			builder->response0304.values[i] = modbusMatchEndian( ( parser->base.function == 3 ? status->registers : status->inputRegisters )[index + i] );
	#endif

	//Calculate crc
	builder->response0304.values[count] = modbusCRC( builder->frame, frameLength - 2 );

BECOMES THIS

 #else
    for (i = 0; i < builder->response0304.length; i++)
    {
        uint16_t temp_value = modbusMatchEndian((parser->base_bytes.function == 3 ? status->registers : status->inputRegisters)[index + i]);
        memcpy(builder->response0304.values + i, &temp_value, 2);
    }
#endif

    //Calculate crc
    uint16_t crc = modbusCRC(builder->frame, frameLength - 2);
    memcpy(builder->frame + frameLength - 2, &crc, 2);

Cheers!

Jacek Wieczorek · Answer 6 · Thu Feb 18 2021 06:53:25 GMT+0800 (China Standard Time)

Hi! Thanks for your very detailed insight - I appreciate it very much. 👍

I actually don't use cmake for configuration (I have your library imported in my MPLABX project), I'd rather prefer to have a "libconf.h" written by hand that has everything I need. Took me a while to understand which exact defines are needed to make everything work. I may help you write down some documentation, if needed, for those like me.

It's seems that perhaps CMake wasn't the right choice for this project. A lot of people seem to be having trouble with integrating the library into their projects because of it.

I've never given it much thought, but now it struck me - maybe we should go header only? This seems to be the best choice for a platform and environment agnostic library. This may also give the compiler a better chance to do some optimizations. Configuration would be a bit simpler too - the config file could be replaced with just a bunch of macros defined before including the library:

#define LIGHTMODBUS_F01S
#define LIGHTMODBUS_BIG_ENDIAN
...
#define LIGHTMODBUS_IMPL
#include "lightmodbus.h"

That obviously doesn't stop us from having a CMake config allowing anyone who uses it to simply write find_package(lightmodbus REQUIRED) and easily integrate with the library.

not every compiler has "inttypes.h", XC16 does not have it...

Very good point. stdint.h should be sufficient across the library code. Frankly, I didn't realize that not all compilers had inttypes.h, but if that's really the case, this needs to be fixed.

dsPICs have 16-bit memory alignment: it took me a loong time to understand why the lib was not working with registers...

I'm sorry to hear that. The fault lies definitely on my side here. I'll fix that right away.

What I meant by "leaving memory allocation to the user" was that instead of having the library allocate memory or define static buffers for you, you just need to provide it a pointer to some buffer you allocated. I think that would significantly simplify the design and allow me to clean up the code from all the complicated preprocessor stuff. The library code will still have to be able to safely read words from the provided memory region, though.

EDIT: Perhaps it's a good idea to stop relying on __attribute__((__packed__)) at all in future versions? I suppose that's not a very portable solution anyway.

I don't know if you're aware of its existence, but the CException library could be a wonderful resource when dealing with error handling. You might take a look, if interested.

I've never used CException, but, funnily enough, I once wrote similar library just for fun - cex.

Having 'real' exceptions would be really handy for dealing with errors. I'm quite worried about mixing longjmp() and interrupt code, though. I might give it a go on some testing branch sometime, but I have a feeling that this could cause some trouble down the road.

Thanks and please let me know you think of all that.

ehughes · Answer 7 · Fri Feb 19 2021 22:16:19 GMT+0800 (China Standard Time)

Hello Jack:

One thing I am looking at is Modbus TCP. I am doing some work on a cellular application (NRF9160) and will be doing both ends. The device will be a Modbus-RTU device and be a Modbus TCP master. For me, simply having helper functions to put on the TCP header would be enough. This functionality might exist but I thought I would bring it up.

ehughes · Answer 8 · Fri Feb 19 2021 22:18:42 GMT+0800 (China Standard Time)

Also, I am using cmake (but not very skilled). Some sort of integration work to make including the cmake code from your library would be helpful. Right now I copy/paste sections into my own cmakelists.txt

Jacek Wieczorek · Answer 9 · Sat Feb 20 2021 03:08:01 GMT+0800 (China Standard Time)

Hello,

I've never worked with Modbus TCP so liblightmodbus doesn't currently support it. However, if converting frames from RTU to TCP is just a matter of prepending a proper header I think it could be implemented as an extension in future versions. I'd consider that something of a secondary priority, though.

I agree, for people already using CMake a config for "including" the library would be really helpful. It could be as simple as defining a single interface target with proper include directories if I transform the library to header-only.

ehughes · Answer 10 · Sat Feb 20 2021 03:48:24 GMT+0800 (China Standard Time)

Yes, Modbus TCP is simply a different header (and no CRC at the end).

Jacek Wieczorek · Answer 11 · Sun Feb 21 2021 06:43:24 GMT+0800 (China Standard Time)

The Modbus TCP header seems to much simpler than I anticipated. I don't see any reasons why not to implement it in the core library functionality :)

Matteo Franceschini · Answer 12 · Fri Feb 26 2021 18:12:08 GMT+0800 (China Standard Time)

It's seems that perhaps CMake wasn't the right choice for this project. A lot of people seem to be having trouble with integrating the library into their projects because of it.

I mostly write for embedded, in C, and I've never used CMake for it. I would personally never use it, but maybe someone out there is finding it useful... I find myself closer to the "header" idea, because this would be the easiest way for everyone to integrate it. In my everyday life, I also define a good part of the "configuration constants" while calling the compiler: this is because I use the same code for different chips, but MplabX allows me to set different configurations with different #define and this is useful to choose (for example) how large are the buffers, which functions to include and which not, etc.

What I meant by "leaving memory allocation to the user" was that instead of having the library allocate memory or define static buffers for you, you just need to provide it a pointer to some buffer you allocated. I think that would significantly simplify the design and allow me to clean up the code from all the complicated preprocessor stuff. The library code will still have to be able to safely read words from the provided memory region, though.

If you think that would ease up the job of the library.. yeah, why not? On the other hand, I really liked the idea of having it like it is right now because once it works for a chip, it works for every different program on it. Let's say: I'm not adding new bugs every time I start a new project, because I just need to say "hey I want the buffer this long" and the tests are already done. But it's just my two cents

EDIT: Perhaps it's a good idea to stop relying on __attribute__((__packed__)) at all in future versions? I suppose that's not a very portable solution anyway.

I guess that we could add a test that checks whether or not the memory alignement causes issues and warn the user about it, with possible different solutions. On the other hand, in my experience, if we write things one byte at the time we shouldn't have issues.

I don't know if you're aware of its existence, but the CException library could be a wonderful resource when dealing with error handling. You might take a look, if interested.

I've never used CException, but, funnily enough, I once wrote similar library just for fun - cex.

Having 'real' exceptions would be really handy for dealing with errors. I'm quite worried about mixing longjmp() and interrupt code, though. I might give it a go on some testing branch sometime, but I have a feeling that this could cause some trouble down the road.

haha cool! You might be right, I have not tested enough to feel safe about that.

Jacek Wieczorek · Answer 13 · Tue Mar 02 2021 03:26:54 GMT+0800 (China Standard Time)

I mostly write for embedded, in C, and I've never used CMake for it. I would personally never use it, but maybe someone out there is finding it useful... I find myself closer to the "header" idea, because this would be the easiest way for everyone to integrate it. In my everyday life, I also define a good part of the "configuration constants" while calling the compiler: this is because I use the same code for different chips, but MplabX allows me to set different configurations with different #define and this is useful to choose (for example) how large are the buffers, which functions to include and which not, etc.

The only reason I decided to use CMake was to replace an awful "custom build system" written in bash. While it's really a fantastic tool for building bigger projects that depend on other CMake libraries, I agree that for embedded systems it's an overkill. Changing liblightmodbus to header-only seems to be settled then :)

If you think that would ease up the job of the library.. yeah, why not? On the other hand, I really liked the idea of having it like it is right now because once it works for a chip, it works for every different program on it. Let's say: I'm not adding new bugs every time I start a new project, because I just need to say "hey I want the buffer this long" and the tests are already done. But it's just my two cents

You're right. What about a custom allocator callback like this void *modbusAlloc(uint16_t size, modbusBufferType purpose)? The library could then guarantee that no two buffers with the same purpose will be requested at the same time, so the function can simply return a pointer to a static array. The default callback would obviously just use malloc() or calloc(). This setup still allows for both static and dynamic allocation, but simplifies the code a lot.

It's only a rough idea, though. I literally just came up with this when replying.

I guess that we could add a test that checks whether or not the memory alignement causes issues and warn the user about it, with possible different solutions. On the other hand, in my experience, if we write things one byte at the time we shouldn't have issues.

What kind of checks do you mean? I think that reading one byte at a time is the best possible solution, because of it's simplicity. Maybe add just a couple of helper functions for reading words and dealing with endianness, and that would be it.

Jacek Wieczorek · Answer 14 · Mon Apr 26 2021 16:09:53 GMT+0800 (China Standard Time)

Hi,

I just wanted to let you know that I pushed some changes to the dev-v3.0 branch. It's still a bit messy and I haven't implemented ~~majority~~ some of the functions yet, but I think it gives the rough idea of how things are going to look (see demo.c).

The main changes are:

Library is header-only now
Introduced allocator callbacks allowing for both dynamic and static memory allocation
All register operations are performed through the register callback
Parsing functions now only operate on PDU allowing for unified function implementation for both Modbus RTU and TCP
All 16-bit memory operations are performed byte by byte (to avoid misaligned access)
By default, functions returning ModbusError are marked with __attribute__((warn_unused_result)) to encourage error checking. This can be disabled by defining empty LIGHTMODBUS_WARN_UNUSED.
I think I managed to improve the overall structure of the code and simplify a lot of things. Not having preprocessor macros interleaved with the code everywhere is certainly very nice.
EDIT: I just ran avr-size on the output and as it turns out, we have a massive improvement in the binary size. Full slave code takes up around 7400 flash bytes with v2.0. Now it's only 3144 bytes! 🎉

Also, I wanted to ask you whether you've actually ever found ModbusSlave.parseError useful. I'm wondering if it's something worth keeping in this release or if it's just a permanently inbuilt debugging feature.

Please let me know what you think :)

Jacek Wieczorek · Answer 15 · Wed Jul 28 2021 06:35:40 GMT+0800 (China Standard Time)

I have a little update, just so you know that something is still happening.

I think that at this point most of the hard work and thinking is done. After a bit of struggling to find the best possible solutions and a couple of dilemmas, I successfully managed to implement all of the biggest and most important changes to the library.

New changes:

ModbusErrorInfo type providing richer error information. Errors caused by frame contents can now be more easily distinguished from actually serious errors.
Callback arguments are now passed as structs providing more flexibility.
Consistent type naming (PascalCase)
I implemented some debug utilities (enabled with LIGHTMODBUS_DEBUG)
modbusBuildRequestXX() now are now available in PDU, RTU and TCP variants (generated with some preprocessor magic)

There are still a couple things to do, though:

Finishing touches to the code
New extensive testing suite, fix coverage testing
Fuzzing
Update examples
Finalize function/struct documentation, write a couple of user manual pages and a "Porting existing code to v3.0" guide
Update ESP-IDF config (#18)
Perhaps put up an announcement about the update before merging into master?

EDIT: You can preview docs for the new version here: https://jacajack.github.io/liblightmodbus/v3.0/

Jacek Wieczorek · Answer 16 · Fri Aug 13 2021 06:47:53 GMT+0800 (China Standard Time)

Hi there, I have another update.

I think v3.0 is pretty much ready. As you may have noticed, I already published some release candidates here and put up an announcement about the new version in the readme. The new release should be published at the end of this month. 🚀

In the meantime, I'm planning to focus mainly on more extensive fuzzing. I don't think I will be making any significant changes to the code.

Please feel free to try out the new version and let me know if anything bothers you. You can also take a look at the new examples and the documentation. Thank you for all the feedback and ideas I got so far.