Trajectory scan problems
georgekontogiorgos opened this issue · comments
Hi,
We are getting a segmentation fault error when running the IOC with the pmacControllerTrajectory.template. Using gdb debugger we realized that profileUser_ and profileVelMode_ pointer were not initialized and the free() functions at lines 2416 and 2421 of pmacController.cpp, respectively, probably deallocates forbidden memory region causing the problem. We could not find errors on our setup so we initialized those variables as NULL at constructor pmacController::pmacController at lines 223, 224 and 225:
// Initialize pointer values as NULL
profileUser_=NULL;
profileVelMode=NULL;Have we forget some initial configuration or is this a bug?
Another problem we are facing is that most of the times pointer can not return to buffer A after finishing buffer B at first cycle:
Buffer A -> Buffer B -> Stop here.
The GUI shows this failure immediately after I press Execute Profile. We generated 6000 points but we are using just 2500 to debug this buffer change problem.
I had to change manually the Version inside PowerPMAC to 2.0 in order to fix the incompatibility. I could not find an updated version of motion program (compatible with driver 2.0) here: trajectory_scan_code_ppmac.pmc and here trajectory_scan_definitions_ppmac.pmh.
Thanks
George
Hi George, I'm going to post the IOC (dls specific I'm afraid) that I use for testing trajectory scan on ppmac.
I'm afraid that we are still only just beginning to adopt ppmac at DLS and I'm aware that we have not kept the ppmac trajectory scan pmcs up to date.
Before I help you debug this I'd like to get the very latest code merged upstream and do some tests of trajectory scan on ppmac here. I will have some time late next week to do so hopefully.
In the meantime perhaps you can take a look at my IOC and see if you have neglected to initialise something.
Hi Giles, I'm going to adapt your IOC to our paths and test it still today.
Thank you.
Hi @gilesknap, we just found out that the trajectory scan (tag 2-4-10) works fine for PowerPC,APM86xxx CPU and does not work for arm,LS1021A CPU (we have both architectures on LNLS).
I suspect this is caused by some race condition due to better performance of ARM processor, since the debug output shows that scan have not started yet but the motors move and suddenly stop when buffer B must be changed to buffer A again (next cycle).
The pmacDebug output from ARM processor (unsuccessful):
2021/05/18 15:31:25.756 pmacController::writeFloat64Array Initialising CS trajectory scan interface
epics> 2021/05/18 15:32:54.288 pmacController::tScanBuildProfileArray Called for axis => 0
2021/05/18 15:32:54.609 pmacController::sendTrajectoryDemands Command => M4044=1000
2021/05/18 15:32:59.337 pmacController::trajectoryTask Trajectory scan started
2021/05/18 15:32:59.343 pmacController::trajectoryTask Reading from buffer => 0
2021/05/18 15:32:59.343 pmacController::trajectoryTask Send next demand set to PMAC
2021/05/18 15:32:59.679 pmacController::sendTrajectoryDemands Command => M4045=1000
2021/05/18 15:32:59.680 pmacController::trajectoryTask Waiting for scan to start
The pmacDebug output from PowerPC processor (successful):
2021/05/18 14:54:04.793 pmacController::writeFloat64Array Initialising CS trajectory scan interface
epics> 2021/05/18 14:54:53.869 pmacController::sendTrajectoryDemands Command => M4044=1000
2021/05/18 14:55:04.068 pmacController::trajectoryTask Trajectory scan started
2021/05/18 14:55:04.086 pmacController::trajectoryTask Reading from buffer => 0
2021/05/18 14:55:04.086 pmacController::trajectoryTask Send next demand set to PMAC
2021/05/18 14:55:04.824 pmacController::sendTrajectoryDemands Command => M4045=1000
2021/05/18 14:55:07.233 pmacController::trajectoryTask Reading from buffer => 1
2021/05/18 14:55:07.233 pmacController::trajectoryTask Send next demand set to PMAC
2021/05/18 14:55:08.000 pmacController::sendTrajectoryDemands Command => M4044=1000
2021/05/18 14:55:10.216 pmacController::trajectoryTask Reading from buffer => 0
2021/05/18 14:55:10.216 pmacController::trajectoryTask Send next demand set to PMAC
2021/05/18 14:55:10.914 pmacController::sendTrajectoryDemands Command => M4045=1000
2021/05/18 14:55:13.320 pmacController::trajectoryTask Reading from buffer => 1
2021/05/18 14:55:13.320 pmacController::trajectoryTask Send next demand set to PMAC
2021/05/18 14:55:13.645 pmacController::sendTrajectoryDemands Command => M4044=575
2021/05/18 14:55:16.354 pmacController::trajectoryTask Reading from buffer => 0
2021/05/18 14:55:16.354 pmacController::trajectoryTask Send next demand set to PMAC
2021/05/18 14:55:16.354 pmacController::sendTrajectoryDemands Command => M4045=0
2021/05/18 14:55:18.363 pmacController::trajectoryTask Waiting for scan to start
Furthermore, I have done another test with the ARM PowerPMAC: I used one global variable DEBUG and set a number for it on each state of trajectory_scan_code_ppmac.pmc. The variable DEBUG did not appear at every code state on plot at phase rate, which strongly indicates that ARM processor works fast. I am going to do the same test with PowerPC to compare the results and share with you.
I have been trying, unsuccessfully yet, to compile version 2-4-11 with our paths. I would like to check if the fixes of this latest version could solve the problem for ARM processor. I will continue to work on fixing those paths.
Hi George,
I'm afraid you are breaking new ground here. I have as yet not had an ARM to test. However I'm sure we do have some incoming at DLS.
@hir12111 , @LeeHudsonDLS Are the devices you are looking at ARM based? Thanks.
@georgekontogiorgos I have released version 2-5-2 of the pmac driver and this included lots of changes to the ppmac trajectory scan code to bring it up to date with the driver. I was able to run my trajectory scan tests against a power clipper (power pc architecture).
Please can you try this version?
Thanks,
giles