Kalman Filter prediction rate appears lower than expected
rbarcham opened this issue · comments
For a TTC deployment, our metrics suggests that the Kalman Filter is generating a limited proportion of predictions - hovering around 50% or total, with TTC falling back to the traditional historical algorithm in other cases. Investigation is needed to dig into whether this is caused by data, configuration, or code issues.
We can supply a data snapshot as needed for investigation.
Thank you for raising this issue,
The following items are needed to better understand the issue and identify potential solutions:
- Config files for TheTransitClock and caching
- Latest configParams.log file
- Pull-request for instrumentation of Kalman Filter to tc_issue_212 branch of TheTransitClock. Please ensure that only the code specific to the instrumentation feature is included in the pull request.
We'll share 1) and 2) and add 3) to our backlog, thanks!
The consistent low PredictionKalmanHeadwayHitRate that is seen in the graph above is not repeating on our test setup which is based on the config provided and the code from the tc_issue_212 branch of TheTransitClock. See attached file.
@rbarcham Would it be possible to provide a screenshot (or the raw values) of the graph for this value from your system on Cloudwatch for the same period yesterday, say from 4pm to 9pm?
In the config files provided, the transitclock.core.terminatePredictionsAtTripEnd parameter is set to true. This parameter is not implemented in the develop version of TheTransitClock. Could you send a pull-request with the code implementing this parameter so we can do further analysis.
In addition, we suggest testing the system with this parameter set to false and provide the associated instrumentation including the PredictionKalmanHeadwayHitRate.
transitclock.core.terminatePredictionsAtTripEnd=false
@simonberrebi we believe terminatePredictionsAtTripEnd should only impact our GTFS-RT caching, and was set as a requirement from our partners. Are there differences in the latest prediction generation code? We may have to chat further offline.
There is a check to see if the direction of the last vehicle when it traversed the stop-path matches the direction of the vehicle we are currently generating the prediction for. The issue is that once we try to predict past the end of a trip (Trip 1) within a block, the direction no longer matches. The first, second, and subsequent stops of the next trip (Trip 2) will be going in the opposite direction. This is causing the KalmanHeadwayHit to be false for all stop-paths on Trip 2.
It seems that the parameter we asked for in the issue (transitclock.core.terminatePredictionsAtTripEnd) causes predictions to be generated for all stop-paths in a trip that is predicted to start within a configurable time limit. I.e. we expect the CS fork of TTC to be generating predictions for all stop-paths in Trip 2 whenever a bus is reasonably close to the end of Trip 1. Therefore, we believe that the low hit-rate is due to the direction on Trip 2 causing PredictionKalmanHeadwayHit to be set to false.
The reason we requested a pull request (see above) is to ensure we understood the intention of this parameter and to ensure it was indeed causing the difference between the two forks.
If it is the reason for this difference, then it would raise the question of whether the Kalman Filter is suitable to make predictions past the end of a trip. This should have an impact on prediction accuracy and on system performance, which should be considered.
Thank you so much for bringing this to my attention @simonberrebi . You misunderstand the parameter and its intentions -- the parameter merely allows predictions to be generated beyond the configurable time window until the end of the trip is reached. By extension then, if the issue indeed is as you say, it exists whenever the prediction window exceeds the (remaining) trip length and therefore will occur independent of the terminatePredictionsAtTripEnd parameter. Delivering predictions beyond the current trip is native to TTC, its not something I added. Or do I misunderstand?
If I'm understanding your text above correctly, the direction comparison is in error and should be relative to the current index and not the vehicle state? Would you agree? If so, would a fix such as this:
possibly resolve the issue? I'm eager to hear your thoughts...
Sheldon
Thank you for clarifying @sheldon,
The develop branch of TTC generates predictions beyond the current trip when it is expected to end within the configurable time window. In its current form, the direction comparison ensures that predictions made on subsequent trips are based on the Historical Average method instead of the Kalman Filter. We fully expect that modifying the direction comparison will increase the Kalman hit-rate. However, it could cause downstream issues in prediction accuracy and performance. The effect on both should be assessed.
Alternatively, as mentioned above, setting the terminatePredictionsAtTripEnd parameter to false would also increase the hit rate. However, it would not produce predictions until the end of the next trip.
Based on the information you provided, @sheldonabrown was able to resolve the issue. Thanks!