Calculation froze during transition probabilities (37%)
amirforouhar opened this issue · comments
Hello,
test2.zip
I am currently working on a Toronto city and I need to do map matching for GPS points. I am really interested in using your valuable plugin, but I encountered a problem in running the plugin.
Since I am working on a very big network, first I tried to run the plugin for a small part of my network. But the calculation froze during transition probabilities (37%). Since I tried it with a very small part of my network and limited GPS points and the problem remained, I guess there is something wrong with my data. Also, I tested your plugin with another network and GPS point (in the Netherlands) with the same network size and it worked correctly.
I would be really grateful if you could look at my attached data (a very small part of my network and limited GPS points) and help me regarding my data problem. Thanks in advance.
Export_Output_3.shp: GPS point
Export_Output_4.shp: network shapefile
Hi Amir,
I know that my plugin has a problem with the computation time (it has a more or less exponential increase, Python is a very slow computation language). Now and during Christmas holidays I have no access to my PC, but after Christmas I will have a look on your data.
Sincerely (and happy Christmas 🎄🎅🏻😄),
Christoph
I found the problem 😀
Your data has the CRS EPSG:4326, i.e. geographical coordinates. Using such coordinates can produce a lot of candidate points and transitions between them. My plugin calculates distances for all transitions and the computation time will increase exponentially with the number of transitions and candidates. Because of this a metric CRS is recommended. To provide a metric search distance is easier and the results will be better, because the other parameters are also metric values. I reprojected your data to EPSG:3978 and it just needs 20 seconds for the map matching.
Please tell me if this solved your problem.
Sincerely,
Christoph
P.S. a Screenshot of the map matching with your reprojected data:
Hi Christoph,
Thank you so much, indeed. I will run it again and let you know soon.
Hi Christoph,
Thank you for your help, indeed. I reprojected the CRS to EPSG:3978 and I put the value of 300 for maximum distance search and it worked well and got the same results as you have already attached. But when I am gonna do it for the whole Toronto network (the network includes 61786 segment and 1475839 GPS points), I put the values of 50, 100 and 200 for maximum distance search and the result says that "the maximum search distance is too low for one trajectory point to find at least one candidate". I put the value of 300 for maximum distance search and it again froze in 12% for 5 hours.
I have two questions: can the map matching ever be done with this plugin for such a heavy network?? and how can I realize that what value is suitable for the maximum distance search?
Great to hear that the first problem is solved :)
1.5 million GPS points are an enormous trajectory oO. Is it really only one trajectory? I also worked with long trajectories, but they were located on highways, i.e. on a network with less junctions, crossings and segments. Such trajectories do not need a high density. Also standing times are a big problem. For the map matching process standing times have no further information (more or less), so GPS points with the same coordinates should be removed before starting the plugin. The plugin has a function to reduce the density of the trajectory (vector menu - Offline-MapMatching - Preprocessing).
Another possibility should be to split your trajectory in multiple parts (e.g. using a timestamp). than you could split the network for each sub-trajectory to reduce the count of network segments. this will increase the calculations a lot. A function to clip the network is available in the preprocessing submenu too. All functions of the plugin are accessible via the toolbox, so you can automate your processings with the model builder or python.
According your question to the maximum search distance:
This value will be used for finding candidate points on the network. For each gps point the closest point on each segment will be calculated. If the distance between the gps and the calculated point is less or equal the maximum search distance, the point will be a candidate. It must be found at least one candidate per gps point, otherwise the plugin will quit. The plugin writes the index of the gps point into the QGIS log, for which no candidate can be found. You have tge following options: increase the maximum search distance or remove this point or correct its position (if it is an outlier).
If something is not understandable, please let me know ^^
Dear Christoph,
Thank you so much for your comprehensive explanation. These GPS points are not related to one trajectory, they obtained from different bicycle trips and the GPS points have a trip id for each trip. shall I do map-matching for each trip separately? I would be really grateful if you have any suggestions on how can I do map-matching for these enormous trips.
Thank you for your supports
in fact, the dataset includes 67000 trips. Since it is obvious I can not do map-matching separately, is this plugin able to calculate the trajectories based on the trip id field?
No, sorry, the plugin has not the ability to split your trips into multiple trajectories using a trip id or something else. At the moment it has to be done by the user before running the plugin. You could use the function „Split Vector Layer“ from the toolbox or from Vector-Data Management Tools-menu for this task. Then you could start a map matching for each trajectory using the preprocessing function „Clip Network“ before. This should decrease the computation time a lot. You can automate this task using the graphic modeller or python, i.e. the plugin is also available via the processing framework (to avoid running the plugin manually for several times ^^).
I am facing a similar issue, my transition probability crashes at 2%. Could you tell me how to re-project the data into a metric CRS ?
P.S: I am new to QGIS!
@rampage-rider QGIS has a function called "Reproject layer". You can find it by searching it in the processing toolbox or via the vector menu (see the screenshots below):
closed because of inactivity, please reopen this issue if necessary