Broken training file(s)

Question

Broken training file(s)

ababacz opened this issue 9 years ago · comments

Hello there!

my V800 crashed during the training. I got FATAL ERROR on it.
Currently, I'm trying to recover GPS record from that one at least.

I have managed to download number of files from that particular training but v800_downloader gives me some error during conversion to TCX.

Question, which of the files contains GPS records?
Is it simply ROUTE.GZB? Is there any way to convert it to any popular GPS file format?

Thanks

Paul Colby · Answer 1 · Sun Aug 02 2015 13:13:41 GMT+0800 (China Standard Time)

my V800 crashed during the training. I got FATAL ERROR on it.

That sucks! I know how important that data can be!

but v800_downloader gives me some error during conversion to TCX.

What's the actual error message?

ababacz · Answer 2 · Sun Aug 02 2015 15:52:11 GMT+0800 (China Standard Time)

Hello,

The error is just:
20150730155424: Error making Bipolar names.

In the result program manages to download attached files (see attachment.
Directory is packed with zip and extension is changed to r__ to pass email
filters)

Inside the directory, i see key Route file which i'm even able to un-gzip.
Again, not all the decompressors are able to do it... Maybe file itself is
corrupted.

Finaly i have ROUTE unpacked. How may i concert it to any standard gps file
known for endo or strava? :)

Thank you in advance for help! :)

Pozdrawiam
Aleksander Babacz

my V800 crashed during the training. I got FATAL ERROR on it.

That sucks! I know how important that data can be!

but v800_downloader gives me some error during conversion to TCX.

What's the actual error message?

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

Paul Colby · Answer 3 · Mon Aug 03 2015 05:35:05 GMT+0800 (China Standard Time)

Disclaimer: I'm not all that familiar with the V800_download code... (other than the code shared with Bipolar, of course).

20150730155424: Error making Bipolar names.

It appears that this occurs when the V800export::make_bipolar_names function fails to rename files to match those expect by the Bipolar code.

Specifically, it appears to require all of the following files:

TSESS.BPB
PHYSDATA.BPB
BASE.BPB
ROUTE.GZB

Presumably one (or more) of these are missing due to the "FATAL ERROR" that occurred on the V800.

It will take an awful lot of work to convert the ROUTE.GZB to GPX by hand, so the best option is going to be use either V800_downloader or Bipolar, if possible, both probably requiring some modification to deal with the missing data. But it really does depend on which data is missing. For example, if either TSESS.BPB or BASE.BPB is missing, then its going to me much harder. Whereas, if PHYSDATA.PBP is missing, then it'll probably just mean some info would be missing from TCX output (not normally included in GPX anyway).

see attachment. Directory is packed with zip and extension is changed to r__ to pass email
filters

Unfortunately the attachment didn't come through (I suspect github doesn't allow email attachments, and if it did, I'm pretty sure they only allow images to be attached to issues anyway).

If you email the attachment directly to me, then I'd be happy to take a look sometime: git at colby dot id dot au

Cheers.

ababacz · Answer 4 · Mon Aug 03 2015 16:07:50 GMT+0800 (China Standard Time)

Hello Paul,

here is mentioned archive (your personal emmail address id TO’ed)

Thank you for your help.

Pozdrawiam,
Aleksander Babacz
aleksander.babacz@gmail.com
mob 603134040

From: Paul Colby
Sent: Sunday, August 02, 2015 11:35 PM
To: profanum429/v800_downloader
Cc: ababacz
Subject: Re: [v800_downloader] Broken training file(s) (#11)

Disclaimer: I'm not all that familiar with the V800_download code... (other than the code shared with Bipolar, of course).

20150730155424: Error making Bipolar names.

It appears that this occurs when the V800export::make_bipolar_names function fails to rename files to match those expect by the Bipolar code.

Specifically, it appears to require all of the following files:

a.. TSESS.BPB
b.. PHYSDATA.BPB
c.. BASE.BPB
d.. ROUTE.GZB
Presumably one (or more) of these are missing due to the "FATAL ERROR" that occurred on the V800.

It will take an awful lot of work to convert the ROUTE.GZB to GPX by hand, so the best option is going to be use either V800_downloader or Bipolar, if possible, both probably requiring some modification to deal with the missing data. But it really does depend on which data is missing. For example, if either TSESS.BPB or BASE.BPB is missing, then its going to me much harder. Whereas, if PHYSDATA.PBP is missing, then it'll probably just mean some info would be missing from TCX output (not normally included in GPX anyway).

see attachment. Directory is packed with zip and extension is changed to r__ to pass email
filters

Unfortunately the attachment didn't come through (I suspect github doesn't allow email attachments, and if it did, I'm pretty sure they only allow images to be attached to issues anyway).

If you email the attachment directly to me, then I'd be happy to take a look sometime: git at colby dot id dot au

Cheers.

—
Reply to this email directly or view it on GitHub.

Paul Colby · Answer 5 · Mon Aug 03 2015 16:29:07 GMT+0800 (China Standard Time)

Got it. Thanks.

So it looks like the TSESS.BPB and PHYSDATA.BPB files were both present, and successfully renamed (to *-create and *-physical-information respectively). And ROUTE.GZB is obviously present, as you said.

The problem is that BASE.BPB is missing... the file provides all of the non-sample data for the actual exercise (the watch / flow.polar.com protocol regards a session as consisting of one or more exercises).

I'll have a look at what (if anything) can be done without the base / exercise data being present...

Paul Colby · Answer 6 · Mon Aug 03 2015 17:44:26 GMT+0800 (China Standard Time)

Finaly i have ROUTE unpacked.

How did you manage to unpack it? Normally, the following should work:

cat ROUTE.GZB | gunzip | protoc --decode_raw 

gzip: stdin: unexpected end of file
Failed to parse input.

Or if you've managed to unpack it first:

cat ROUTE | protoc --decode_raw 
// lots of protocol buffer data here...

(note, the above examples are using the protoc compiler from the Google Protocol Buffers project).

At this stage, I can't decompress the data (since gunzip won't recognise it). But if you can (as you suggested above), then let me know how you did so. Then we can see if the uncompressed data is valid protobuf data or not.

Cheers.

ababacz · Answer 7 · Mon Aug 03 2015 18:29:07 GMT+0800 (China Standard Time)

Hello Paul,

I used both 7-zip and one more program (I cannot recall the name of it – I have it on my second PC) however both of them gave me different results as decompressed files have 2 different file sizes.
Attached ROUTE and SAMPLES are taken from 7-zip

Later one I will send you files from that second decompressor.

Thank you.

Pozdrawiam,
Aleksander Babacz
aleksander.babacz@gmail.com
mob 603134040

From: Paul Colby
Sent: Monday, August 03, 2015 11:44 AM
To: profanum429/v800_downloader
Cc: ababacz
Subject: Re: [v800_downloader] Broken training file(s) (#11)

Finaly i have ROUTE unpacked.

How did you manage to unpack it? Normally, the following should work:

cat ROUTE.GZB | gunzip | protoc --decode_raw

gzip: stdin: unexpected end of file
Failed to parse input.
Or if you've managed to unpack it first:

cat ROUTE | protoc --decode_raw
// lots of protocol buffer data here...
(note, the above examples are using the protoc compiler from the Google Protocol Buffers project).

At this stage, I can't decompress the data (since gunzip won't recognise it). But if you can (as you suggested above), then let me know how you did so. Then we can see if the uncompressed data is valid protobuf data or not.

Cheers.

—
Reply to this email directly or view it on GitHub.

Paul Colby · Answer 8 · Tue Aug 04 2015 06:25:04 GMT+0800 (China Standard Time)

The bad news is that both the ROUTE and SAMPLE files are corrupt.

for f in ROUTE SAMPLES; do echo -n "$f: "; cat $f | protoc --decode_raw; done
ROUTE: Failed to parse input.
SAMPLES: Failed to parse input.

The good news, is that the corruption appears to be a truncation, so we should be able to get most of the samples...

for i in `seq 131902 -1 1`; do echo -n "$i: "; cat ROUTE | dd bs=1 count=$i 2>/dev/null | protoc --decode_raw >/dev/null && echo good && break; done
131902: Failed to parse input.
131901: Failed to parse input.
131900: Failed to parse input.
131899: Failed to parse input.
131898: Failed to parse input.
131897: Failed to parse input.
131896: Failed to parse input.
131895: good

for i in `seq 200635 -1 1`; do echo -n "$i: "; cat SAMPLES | dd bs=1 count=$i 2>/dev/null | protoc --decode_raw >/dev/null && echo good && break; done
200635: Failed to parse input.
200634: Failed to parse input.
200633: Failed to parse input.
200632: Failed to parse input.
200631: Failed to parse input.
200630: good

Now we'll either need to manually truncate the files to valid protobuf boundaries, or make the shared Bipolar code ignore trailing broken protobuf entries (if it doesn't already).

ababacz · Answer 9 · Tue Aug 04 2015 07:49:15 GMT+0800 (China Standard Time)

Hi @ababacz,

I have:

manually renamed the files that v800_downloader didn't get to (due to
the parse error).
added a fake zero-byte -exercise--create file (ie would have come from
the missing BASE.BPB file).
uncompressed the ROUTE and SAMPLE files (the other files were already
raw protobuf, without compression)
truncated ROUTE and SAMPLE at 131895 and 200630 bytes respectively.
processed the session with Bipolar.

The resulting GPX file looks good. As do both HRM files. The TCX file is
essentially empty - unfortunately, it will be very hard to get the TCX
output without the proper exercise-create (ie BASE.BPB) file.

I've uploaded the GPX file to Strava - looks good. It works out to be
25.7km... so hopefully the watch didn't lose too much of your activity.

The resulting GPX, HRM and TCX files are attached in 7zip, zip, and
TAR/bzip2 formats. Hopefully one of those will work for you.

Cheers.

pc.

On Mon, Aug 3, 2015 at 8:29 PM Aleksander Babacz <
aleksander.babacz@gmail.com> wrote:

Hello Paul,

I used both 7-zip and one more program (I cannot recall the name of it – I
have it on my second PC) however both of them gave me different results as
decompressed files have 2 different file sizes.
Attached ROUTE and SAMPLES are taken from 7-zip

Later one I will send you files from that second decompressor.

Thank you.

Pozdrawiam,
Aleksander Babacz
aleksander.babacz@gmail.com
mob 603134040

From: Paul Colby notifications@github.com
Sent: Monday, August 03, 2015 11:44 AM
To: profanum429/v800_downloader v800_downloader@noreply.github.com
Cc: ababacz aleksander.babacz@gmail.com
Subject: Re: [v800_downloader] Broken training file(s) (#11)

Finaly i have ROUTE unpacked.

How did you manage to unpack it? Normally, the following should work:

cat ROUTE.GZB | gunzip | protoc --decode_raw

gzip: stdin: unexpected end of file
Failed to parse input.

Or if you've managed to unpack it first:

cat ROUTE | protoc --decode_raw
// lots of protocol buffer data here...

(note, the above examples are using the protoc compiler from the Google
Protocol Buffers project).

At this stage, I can't decompress the data (since gunzip won't recognise
it). But if you can (as you suggested above), then let me know how you did
so. Then we can see if the uncompressed data is valid protobuf data or not.

Cheers.

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

Paul Colby · Answer 10 · Tue Aug 04 2015 07:50:32 GMT+0800 (China Standard Time)

Hmm.. not sure why github recorded the above comment as being from @ababacz ... it was from me ;)

Cheers.

ababacz · Answer 11 · Tue Aug 04 2015 13:39:49 GMT+0800 (China Standard Time)

Hello Paul,

you are amazing! It works! Strava has successfully imported GPX file so
something from me left on few very nice segments ;)
Thank you for your time and efforts.
Do guys from Polar know about you and your skills? :)

Thank you and regards from Poland! :)

2015-08-04 1:50 GMT+02:00 Paul Colby notifications@github.com:

Hmm.. not sure why github recorded the above comment as being from
@ababacz https://github.com/ababacz ... it was from me ;)

Cheers.

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

Pozdrawiam,
Aleksander Babacz
aleksander.babacz@gmail.com
Mob. 603134040

Paul Colby · Answer 12 · Tue Aug 04 2015 14:30:35 GMT+0800 (China Standard Time)

Hi @ababacz,

Strava has successfully imported GPX file

I'm glad we could get some data for you 😄 I know how incredibly frustrating it is to even come close to losing this sort of data!! I've come close a couple of times 😦

Thank you for your time and efforts.

You're very welcome. It was actually fun to get back into the protobuf level for a bit (I've been stuck at much higher / more abstract levels of software engineering lately).

Do guys from Polar know about you and your skills? :)

No idea. They are aware of the v800_downloader and Bipolar projects based on chatter on their forums 😉

Cheers.

profanum429 · Answer 13 · Fri Aug 07 2015 08:52:11 GMT+0800 (China Standard Time)

I've been away for the past few weeks, sorry I couldn't reply but it looks like Paul got things sorted out for you!

ababacz · Answer 14 · Fri Aug 07 2015 17:06:36 GMT+0800 (China Standard Time)

Hello,

that is correct. Thank you. I close this issue.