Not everything starts 00 01 00 02 it seems...
nbertram opened this issue · comments
Hi,
I have a newer Growatt Wifi module (PN MR00,0004601, Shine Wifi) which doesn't seem to expose a web interface (or listen on any ports). I've managed to intercept its comms by MITMing DNS for server.growatt.com, but then found I couldn't make it communicate with your server...
Here's an excerpt of the comms if I intercept the real client and server:
[client].07446-[proxy].05279:
0000: 0001 0005 000c 0116 0e27 2d44 5945 4c77 4056 c38c 0001 0005 000c 0116 0e27 2d44 .........'-DYELw@V...........'-D
0020: 5945 4c77 4056 c38c YELw@V..
[proxy].05279-[client].07446:
0000: 0001 0005 0010 0119 0e27 2d44 5945 4c77 4056 7765 7461 2db0 .........'-DYELw@Vweta-.
[proxy].05279-[client].07446:
0000: 0001 0005 000c 0116 0e27 2d44 5945 4c77 4056 c38c 0001 0005 000c 0116 0e27 2d44 .........'-DYELw@V...........'-D
0020: 5945 4c77 4056 c38c YELw@V..
[client].07446-[proxy].05279:
0000: 0001 0005 0023 0119 0e27 2d44 5945 4c77 4056 777e 7467 7542 5f47 4c44 456a 425e .....#...'-DYELw@Vw~tguB_GLDEjB^
0020: 5751 454e 7743 5547 50b5 1b WQENwCUGP..
[proxy].05279-[client].07446:
0000: 0001 0005 0023 0118 0e27 2d44 5945 4c77 4056 777e 7467 7542 5e4f 4c45 446a 405b .....#...'-DYELw@Vw~tguB^OLEDj@[
0020: 5750 4c4e 7646 5546 53b7 71 WPLNvFUFS.q
[client].07446-[proxy].05279:
0000: 0001 0005 000f 0118 0e27 2d44 5945 4c77 4056 777e 74b9 4f00 0100 0500 1101 190e .........'-DYELw@Vw~t.O.........
0020: 272d 4459 454c 7740 5677 6574 7572 6df2 '-DYELw@Vweturm.
[client].07446-[proxy].05279:
0000: 0001 0005 0011 0119 0e27 2d44 5945 4c77 4056 7764 7475 7652 f2 .........'-DYELw@VwdtuvR.
[client].07446-[proxy].05279:
0000: 0001 0005 0012 0119 0e27 2d44 5945 4c77 4056 7767 7476 7440 5123 .........'-DYELw@Vwgtvt@Q#
[client].07446-[proxy].05279:
0000: 0001 0005 0016 0119 0e27 2d44 5945 4c77 4056 7766 7472 7640 5c43 5442 617c .........'-DYELw@Vwftrv@\CTBa|
[client].07446-[proxy].05279:
0000: 0001 0005 001a 0119 0e27 2d44 5945 4c77 4056 7769 747e 0e27 2d44 5945 4c77 4056 .........'-DYELw@Vwit~.'-DYELw@V
0020: bf7c .| .|
[client].07446-[proxy].05279:
0000: 0001 0005 0013 0119 0e27 2d44 5945 4c77 4056 7768 7477 755c 5f33 35 .........'-DYELw@Vwhtwu\_35
[client].07446-[proxy].05279:
0000: 0001 0005 0011 0119 0e27 2d44 5945 4c77 4056 776b 7475 7786 30 .........'-DYELw@Vwktuw.0
[client].07446-[proxy].05279:
0000: 0001 0005 0014 0119 0e27 2d44 5945 4c77 4056 776a 7470 0927 233b 3a1b .........'-DYELw@Vwjtp.'#;:.
[client].07446-[proxy].05279:
0000: 0001 0005 001e 0119 0e27 2d44 5945 4c77 4056 776d 747a 7542 5d59 5842 5a76 415b .........'-DYELw@VwmtzuB]YXBZvA[
0020: 5950 4747 f55c YPGG.\
[client].07446-[proxy].05279:
0000: 0001 0005 0011 0119 0e27 2d44 5945 4c77 4056 776c 7475 71f0 b1 .........'-DYELw@Vwltuq..
[proxy].05279-[client].07446:
0000: 0001 0005 0010 0119 0e27 2d44 5945 4c77 4056 777e 746b 2d40 .........'-DYELw@Vw~tk-@
[client].07446-[proxy].05279:
0000: 0001 0005 0023 0119 0e27 2d44 5945 4c77 4056 777e 7467 7542 5e4f 4c45 446a 405b .....#...'-DYELw@Vw~tguB^OLEDj@[
0020: 5750 4c4e 7646 5545 53aa 7000 0100 0500 1001 190e 272d 4459 454c 7740 5677 6f74 WPLNvFUES.p.........'-DYELw@Vwot
0040: 74e0 51 t.Q
[client].07446-[proxy].05279:
0000: 0001 0005 0010 0119 0e27 2d44 5945 4c77 4056 776e 7474 2000 .........'-DYELw@Vwntt .
[client].07446-[proxy].05279:
0000: 0001 0005 00d9 0103 0e27 2d44 5945 4c77 4056 3427 3537 7f40 5c47 514d 7447 726f .........'-DYELw@V4'57.@\GQMtGro
0020: 7761 7674 4772 4376 6074 7547 726f 1361 1053 5772 6fd3 717a 6406 3a5e 5951 7474 wavtGrCv`tuGro.a.SWro.qzd.:^YQtt
0040: 4772 6f77 6174 7547 766c 9f61 3573 4f78 e366 f560 2404 342e 3459 4647 7742 5667 GrowatuGvl.a5sOx.f.`$.4.4YFGwBVg
0060: 7185 3c47 736f 7761 7474 4772 6f70 697e ca56 e67b 2769 627e 0163 9762 7974 a645 q.<GsowattGropi~.V.{'ib~.c.byt.E
0080: 736f 5a61 2d73 a572 6577 7974 6647 7c6f 6262 9c77 af71 8777 0577 9c47 166c 9f61 soZa-s.rewytfG|obb.w.q.w.w.G.l.a
00a0: 1033 351d 1816 1500 540e 1c19 1213 0011 3513 0715 0045 4076 406f 7761 7775 7672 .35.....T.......5....E@v@owawuvr
00c0: 6f77 6074 7447 726f 7761 7474 4661 ce77 a974 7447 726f 7761 7474 4772 6f77 61e2 ow`ttGrowattFa.w.ttGrowattGrowa.
00e0: 00 .
[proxy].05279-[client].07446:
0000: 0001 0005 0003 0103 472f da ........G/.
[client].07446-[proxy].05279:
0000: 0001 0005 0137 0150 0e27 2d44 5945 4c77 4056 3427 3537 7f40 5c47 514d 664d 6a7f .....7.P.'-DYELw@V4'57.@\GQMfMj.
0020: 7360 7774 4772 4377 6174 7447 7263 3f61 7474 4772 6f7f 2474 7447 726f 7761 7474 s`wtGrCwattGrc?attGro.$ttGrowatt
0040: 4761 ed7e 0574 7447 726f 7761 7474 4772 6f77 6174 7447 726f 7761 7474 4772 0277 Ga.~.ttGrowattGrowattGrowattGr.w
0060: 6170 ed47 7805 f060 2674 4772 6f77 6174 7447 726f 7761 7474 4773 177b 1574 7447 ap.Gx..`&tGrowattGrowattGs.{.ttG
0080: 726f 5a61 2d3a 6772 6f77 6174 7447 4d6f 7763 b274 4772 4177 6175 a947 726b d461 roZa-:growattGMowc.tGrAwau.Grk.a
00a0: 7474 4772 6f77 6174 7447 726f 7761 7474 4772 6f77 6174 7447 726f 7761 7474 4772 ttGrowattGrowattGrowattGrowattGr
00c0: 6f77 6174 7447 726f 7761 7474 4772 6f77 6174 7447 726f 7761 7474 4772 6f77 6174 owattGrowattGrowattGrowattGrowat
00e0: 2e47 f46f 6e73 7e6c 5770 4b77 6174 6d56 736e 7764 4f74 4772 6f66 6075 7447 726f .G.ons~lWpKwatmVsnwdOtGrof`utGro
[client].07446-[proxy].05279:
0000: 7761 7465 4673 6f77 6174 7447 727e 7660 7474 4772 6f77 6174 7447 726f 7761 7474 wateFsowattGr~v`ttGrowattGrowatt
0020: 4772 6f77 6174 7447 726f 7761 7474 4772 6f77 6174 7447 726f 7761 7474 47da 2b GrowattGrowattGrowattGrowattG.+
[proxy].05279-[client].07446:
0000: 0001 0005 0003 0150 47df e6 .......PG..
etc etc etc
I haven't delved in to try to see if this protocol is similar yet, but the first stumbling block is the header seems to always be 0001 0005 in both directions. If you haven't seen this before, I guess I'll be trying to decipher it tomorrow. Just thought I'd ask first!
I haven't seen this before, but actually since my server is running happily I haven't paid much attention to newer developments.
From your dumps certain parts are recognizable, e.g. 0001 0005 000c 0116 XXXXX is clearly a 'PING'. It seems that 0001 0005 imply the protocol version (1.5) as opposed to the traditional 0001 0002 (1.2) version.
It may take some time and patience to reverse engineer this...
Yeah I thought I'd try this as an interim measure before I build something to do modbus over RS232 instead (which is apparently an option). The inverter is a 4200MTL.
If you think there's value in reverse engineering this new protocol, I'm happy to go through it and see how similar it is to what you've implemented. I did see the PING in there, but saw the response from the server was IDENTIFY without an ACK?
I guess the other option is to simply identify the DATA packet format and just continue to proxy all unknown traffic and just extract the bit I actually need.
Anyway, thanks very much for posting your work, it's very refreshing to see a protocol reverse engineering implementation taken far enough to publish!
I stumbled upon this by chance trying to get some useful info out of the newer module. My tcpdump output looked similarly puzzling. I'll post on this issue again if I manage to extract some useful info from it.
I'd like to point out that you CAN get the web interface still on this revision of the module, I don't fully recall how I accesed it but you can access it in an unconfigured state. There you can set the target IP etc.
EDIT: at least some part of the messages are xor'd with the dongle key. When I tried this on one of the smaller messages I got GrowattGro. This was pretty obvious from some of the larger messages which read just GrowattGrowattGrowatt when there are large areas with zeroed out fields. Headers of mesages seem to not be xor'd as far as I can tell. I am not sure why they made this change but it shouldn't dissuade anyone attempting to circumvent it.
Possibly a typically IoT approach to encryption... or obfuscation?
I gave up on using the Shine adapter, removed its guts and put in an ESP8266 acting as a modbus TCP bridge instead. Works fine, though this inverter seems unable to have things like a date/time register set, which means a day starts when it sees light after a long pause and ends when the inverter shuts down. 99% of the time it's right...
I think if I buy another one of these inverters I'll skip buying the wifi adapter for it, especially after the Shine app crashing constantly while trying to configure its connection.
Nice to hear you found the web interface. I was completely unable to after a large amount of prodding/resetting it, and had to persevere with the unstable app.
Yea I don't fully trust the adapter itself to be reliable either. For me the main reason I started looking in to it was it seemingly losing connection with the server but not the wifi connection.
I currently have a pretty clear idea of how this protocol works.
The first 2 bytes are a counter, This gets incremented for every time a data update is sent to the server. The next two bytes are the version ID, this seems to be 5 for my dongle. After that we get the type and message ID with 2 bytes each as in the original protocol spec done by Sciurius. The rest of the data is XOR'd with the dongle ID and the inverter ID. Messages seem to have a "random" two bytes trailing data.
I have not yet made sure if this bit of data will be enough to decode the large main data packet but the logic seems to hold for the smaller periodic packets.
message payload is xored with GrowattGrowatt... last 2 bytes seems to be a crc, just no clue how to calculate and over what
Thanks for reminding me about this thread, I wrote an implementation of the protocol which I have been using for the past two weeks or so. The protocol is basically ModbusTCP, growatt uses a different version ID. After the modbus header we have a packet identifier, followed by the packet data. I too could not find the logic for generating the CRC. I wrote the server by intercepting traffic so I don't need to be able to generate good replies for now. I'll probably upload my implementation to Github after some more cleanup.
it's CRC-16/MODBUS over the complete original data (ex crc)
found this site for many crc results https://crccalc.com/
I have received my Growatt Inverter today, with a "Shine LAN Box", which seems to be using the same "1.5" protocol. I've written a system in PHP to monitor my "SmartMeter" and I've made an extension to it to monitor the Growatt. I am not intercepting, but have fully taken over the role of the server.
I've come so far as requesting the config information (Query) and I'm getting some Query Replies, but apparently my Acks to those are not good enough, because I'm not getting past that point. Can you, @Soreil share your implementation and possibly a few wireshark dumps?
Sure thing Jeroen, I'll take a look at it over the weekend. I have some dumps but they did not record the system configuration since I wasn't monitoring for those steps, it's all the regular data communcation between dongle and server. I'll also upload those PCAP for you to play with.
@Soreil thanks, but it's no longer needed. I decided to let my inverter talk to "China" for a few minutes and PCAP'ed that, with a little bit of puzzling I managed to get it fully working. It's now (well, not now, sun's set) happily sending data to the database. Once I've removed all the debugging code and cleaned it up a little I will share my code on Gitlab.
@jeroenrnl Could you share your code for protocol v1.5?
@Tristan79 yes. It has been working without any problems for the past month or so. I will see if I can upload it to my Gitlab repo this evening, but be aware it is somewhat "alpha" state. I'll be happy to help to get it working though.
Tof! alpha state is ok.
I received my shinelan today (paid 3 bucks + 11 euro shipping for it ;-), stupid question but what is the login/password for the web interface, I am googling far too long with no answer. I also have the ShineWifi version. For some reason I got 2 times in the web interface after that, whatever I do (reset, cut power, etc) I can't access it... and why it has 5 different mac addresses? (I can not change the dns on the device itself, luckily I have a dns server :-) So anyway, the lan version must have the same options, but what is the password, login?
Will take me a couple of days to test your software, looking forward to it, thanks!
Try
admin
admin
or just "ok", without filling anything in
For wifi is is there an ssid with the serial of the wifi dongle?
If Yes, connect to that password 12345678
After that the gui should be admin:admin as well
I just was going to try that but now the ip/mac is gone..., probably have to push the button like on the wifi to get it into that mode, I notice something different...
even thou 1.5 is not supported this github app still gives raw data,,, and the following i notice
for the wifi I got 5 networks...
84:0d:8e:94:01:1e
84:0d:8e:88:8d:a8
84:0d:8e:88:bd:88
cc:50:e3:27:31:6a
84:0d:8e:88:c9:a5
and this github app for the lan, I got the some mac addresses...
For some strange reason it uses normal mode 5 mac addresses (listed above) to communicate. Why? We will never know :-)
Correction:
For some strange reason it uses in normal mode, 2 of those 5 mac addresses (3th and 5th listed above) to communicate.
98:XX:63:0e:47:03 of the lan is also used
is probably equivalent to wifi cc:50:e3:27:31:6a (guessing)
strange to get the same mac addresses
also strange to get so much mac addresses with either device, you would expect just one :-)
anyway keep you updated on progress!
each (virtual)network interface card must have it's own mac address, for switches & routers to work properly
Lan and wifi would each have their own MAC, virtual wifi accesspoints would also likely have their own mac.
If 2 devices on the same network would have the same mac routers/switches would send any data for that mac to both devices...
Luckily I do not have 2 growatt devices (the "what it is called?" where you stick in the lan/wifi extension which I got both)... cuz swithcing from wifi to lan giving the same mac addresses, would it clash 2 "the what it is called?" devices sending from the same mac their data..???. that is a problem. And unless someone test 2 growatt "the what it is called?" devices, with either lan or wifi, this is probably a bug.
Luckily most of us got only one "the what it is called?" device :-) So don't worry!
the what it is called? -> inverters i think they are called
deep trailer voice:
this summer...
...two inverters...
...one shared (virtual) mac address...
...confusing around...
sub starring
john cleese
:-)
serious, 2 random bought growatt lan and wifi devices sharing the same mac addresss?? Hopefully the firmware of either compensate for that, or there will be clashes! You can not have two inverters with wifilan on the same network as a result of this behaviour.
Unless the firmware compensate for it, which I highly doubt it. It's probably far less then <1% of the use cases... someone with more then one inverter on the same network :-)
If the Mac address stays the same when plugging in the other interface i would expect the interface to use the serial of the inverter to generate a Mac.
As for the networking layers May I suggest you read up on the OSI model
Seems my lan version is v2 not v5 as my wifi is.
So...
I combined the data & server script and added/hacked in some http to communicate with domoticz, since it is http url to update domoticz you probably could use that with other domotica controllers software like openhab by changing the urls.
https://github.com/Tristan79/Growatt-WiFi-Tools
I have a ShineWifi-X on a 5000TL-X inverter and it's using this newer protocol so the perl scripts don't work at all.
@jeroenrnl your code works (mostly) and I'll pop some notes in your gitlab.
XOR with "Growatt" is a funny touch - I was wondering why it was spewing Growatt over and over again in to the tcpdump for no apparent reason. :D
So a question... I have the ShineLanBox and of course get the same growattgrowattgrowatt data. I saw thaty they used to have an Open Api http://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-Open-API-protocol-standards.pdf but it seem closed now.
I have a few questions:
- Can you give step by step to implement whatever code you have put here? So to saw newbie but I know how to capture things with Wireshark and I have 30 years of IT experience, but not programming.
- Can you only read or would you be able to change Priority through the Modbus or via the ShineLanBox? I wanna be able to tell it to not discharge from battery when charging my EVs. Or tell to charge the batteries from the grid.
- jeroenrnl: I guess you but another listener on the bus from the SmartMeter, but what do you use to view that data? Can you send it to an MQTT to be used in Home Assistant? Can you use NodeRed to do this?
As far as I know, the API is to talk to Growatt's servers. So, you let your Growatt box send the data to their service and then pull some data from there.
-
I think the documentation at https://github.com/sciurius/Growatt-WiFi-Tools/blob/master/README is pretty clear
-
I don't have a battery in my system and my inverter doesn't support connecting to one, so I don't know.
-
No, I am not connecting Growatt to my smart meter, I just added Growatt support to a tool that I previously built to read out the smart meter. However, my program is a totally different program than the one by @sciurius - I only built on his research to decode the data, but it is a complete separate implementation, that doesn't share any code with his. If you want support for nrg, please head over to nrg's site.