Error while processing multi-segment file - Following segment redefines not found, Please check fields exist

Question

Error while processing multi-segment file - Following segment redefines not found, Please check fields exist

eapframework opened this issue 4 years ago · comments

I am trying to process a multi-segment file. Please find the copybook attached.

Code below:

val df = spark.read.format("cobol").option("copybook", "copybook-flap.txt").option("pedantic", "false").option("segment_field", "FLAP-MTHD-OVER-RIDE-NR")
.option("redefine_segment_id_map:0", "FLAP_RECORD.FLAP-ITEM.FLAP-MTHD-OVER-RIDE.FLAP-MTHDS.REDEFINE-STR1 => 1")
.option("redefine-segment-id-map:1", "FLAP_RECORD.FLAP-ITEM.FLAP-MTHD-OVER-RIDE.FLAP-MTHDS.REDEFINE-STR2 => 2")
.option("redefine-segment-id-map:2","FLAP_RECORD.FLAP-ITEM.FLAP-MTHD-OVER-RIDE.FLAP-MTHDS.REDEFINE-STR3 => 3")
.option("redefine-segment-id-map:3","FLAP_RECORD.FLAP-ITEM.FLAP-MTHD-OVER-RIDE.FLAP-MTHDS.REDEFINE-STR4 => 4")
.load("mcy_flap1_Dec19.dat")

I gave the full path for Redefines fields - FLAP_RECORD.FLAP-ITEM.FLAP-MTHD-OVER-RIDE.FLAP-MTHDS.REDEFINE-STR1

But still getting error -

Following segment redefines not found, Please check fields exist and are redefines/redefined by.

Please help!

Ruslan Yushchenko · Answer 1 · Fri Mar 27 2020 14:40:41 GMT+0800 (China Standard Time)

Thanks for your request. Will take a look.

Ruslan Yushchenko · Answer 2 · Mon Mar 30 2020 20:12:44 GMT+0800 (China Standard Time)

Sorry for the delay. Will get to this soon. Fro the first glance it looks like it is related to the depth of nesting of segment redefines. If that is the case, it is a bug and we will fix it. Will let you know more soon.

eapframework · Answer 3 · Mon Mar 30 2020 23:02:05 GMT+0800 (China Standard Time)

Thanks for the update. Even I am working on resolving the issue. So far no luck. Will update if I am able to resolve the issue.

eapframework · Answer 4 · Wed Apr 01 2020 15:59:01 GMT+0800 (China Standard Time)

Hi yruslan, I am able to resolve the issue by changing the depth of nesting of segment redefines and clearing the cached files in my spark cluster. I have a question - what happens if pass segment_field(FLAP-MTHD-OVER-RIDE-NR) as 0. Can we skip without allocating bytes to any segment?
Thanks!

Ruslan Yushchenko · Answer 5 · Wed Apr 01 2020 16:28:03 GMT+0800 (China Standard Time)

I'm glad you've found a workaround. But I'm going to reopen the issue so that we understand why there is a limitation of the depth of segment redefines and if we can remove it.

If the value of a segment id is not in the list of segment redefine mappings all segment-specific fields should be empty in the dataset. But the record itself won't be skipped.