Create connectors for the OSM network.
zhwlxl opened this issue · comments
Hi Pedro,
Thanks for your great work on Aequilibrae.
I am trying to make a highway transport network from the openstreet map, but I am stuck at the step of creating the connectors. It did create the connectors but also a new B_node which is overlaping the existing node in the network. Please see the screenshot below:
Therefore, when I was trying to do skims, it always show some huge numbers, which is no connections between zones. Is that possible you can have a look on this issue, I am not sure if I did something wrong or not. Also, is there a way to speed up the creating connectors process, for about 360 zones, it takes few hours to create the connectors.
Thanks a lot.
Regards,
Hongwei
Hi @zhwlxl , Cna you share the dataset you are using, as well as the operating system and AequilibraE version you are using?
Hi Pedro, thanks for your reply. Please find the link of the project file https://1drv.ms/u/s!ApCAzDcnT4qagfgXaRD9rO2_RdfrTQ?e=lghssj. It's created from QGIS 3.10.5 on windows 10. I also tried on my Mac, however the QGIS is crached when I load any project file from AE GUI on MacOS, therefore, I copied out the script to run in python which gives the same results as the connector B_node is created overlapping the existing node. here is the script in jupyter format https://1drv.ms/u/s!ApCAzDcnT4qagfgYk_L-PCiiGeSXyQ?e=jq3NEw
Many thanks
@zhwlxl , running it in Python will probably not work due to the lack of the Rtree Sqlite extension on most Python distributions (this is not the Rtree module), so I will investigate the issue with the plugin first and get back to you
Investigating a little more...
The procedure is NOT creating overlapping nodes, but rather inserting the wrong coordinates for the link extremity that corresponds to the existing network link
The code seems pretty air-tight, as all the operations are done by Spatialite. HOWEVER, there is clearly a problem with collecting the nodes with AsText(nodes.geometry) and then inserting the connector using GeomFromText ("LINESTRING…
The solution is probably not to transform the data in any way, but rather retrieve binary from Spatialite and then re-insert as binary. Not sure if it is possible.
The issue was that AsText(nodes.geometry) does not yield good precision. The solution was to replace it with AsWKT(nodes.geometry)
The solution will be shipped with version 0.6.3
@zhwlxl
Version 0.6.3 (experimental) includes the fix for this bug. It has already been uploaded to the QGIS repository and is waiting for approval.
Hi @pedrocamargo,
Thanks for the quick fix. I have done a run test using the python script by replace to AsWKT(nodes.geometry). It worked on the most of connectors, however, for some of them, it provide the different y values. see below, the script read the node 32609 y value with lots of 99999s at the end.
but actually the y value is 52.6002868 from the network. The screenshot below shows the existing node (32609) y value and the added new node (232330) y value.
Here attached the project file (with samples of centroids) of this network.
https://1drv.ms/u/s!ApCAzDcnT4qagfh2ccM29Zc9J_sStA?e=ajZ3bt
Then, I find the OSM node x,y are always 7digits, therefore I change the script to AsWKT(nodes.geometry,7), and it works. I attached here the link for script I used https://1drv.ms/u/s!ApCAzDcnT4qagfh01Fk7PtLEL8MUrw?e=54GCXN
By the way, I used the python script in my Mac, it seems works fine. I think I install libspatialite in my Mac. is that what you mean for the Rtree Sqlite extension?
Many thanks,
Thanks for investigating further and so soon, @zhwlxl This looks like a floating point issue, so I will have to investigate it a bit further. The Rtree extension to sqlite is different than libspatialite (https://sqlite.org/rtree.html), and that is not shipped with Python for Windows AFAIK.
This is a little trickier to solve, so I will need some help here from @arobrien .
@arobrien , My first reaction is to apply ST_SnapToGrid to all geometries (basically having that inside the query that checks geometries for new nodes) and using some ridiculous precision from a real world perspective but that would still avoid this type of numerical issue (e.g. 1e-9 degrees).
The other possibility is to expand the node overlap trigger to include its surroundings (to similar precision as above).
What do you think?
Hey @zhwlxl , can you do a test with this network? https://1drv.ms/u/s!Atqjh7mmOwHFkBd9jrSLhP6Zw79f?e=PElgo2
In this one I just changed the trigger that operates on the nodes' geometries, which would be a simple solution for the problem
@arobrien , I attempted the second alternative I suggested
What do you think?
CREATE TRIGGER cannibalise_node BEFORE UPDATE OF geometry ON nodes WHEN (SELECT count(*) FROM nodes WHERE node_id != new.node_id AND geometry = new.geometry AND ROWID IN ( SELECT ROWID FROM SpatialIndex WHERE f_table_name = 'nodes' AND search_frame = ST_Buffer(new.geometry,0.000000001))) > 0 BEGIN UPDATE links SET a_node = new.node_id WHERE a_node = (SELECT node_id FROM nodes WHERE node_id != new.node_id AND geometry = new.geometry AND ROWID IN ( SELECT ROWID FROM SpatialIndex WHERE f_table_name = 'nodes' AND search_frame = ST_Buffer(new.geometry,0.000000001))); UPDATE links SET b_node = new.node_id WHERE b_node = (SELECT node_id FROM nodes WHERE node_id != new.node_id AND geometry = new.geometry AND ROWID IN ( SELECT ROWID FROM SpatialIndex WHERE f_table_name = 'nodes' AND search_frame = ST_Buffer(new.geometry,0.000000001))); DELETE FROM nodes WHERE node_id != new.node_id AND geometry = new.geometry AND ROWID IN ( SELECT ROWID FROM SpatialIndex WHERE f_table_name = 'nodes' AND search_frame = ST_Buffer(new.geometry,0.000000001)); END;
@pedrocamargo my first inclination would be to keep the data as binary, rather than do any sort of snapping. I'll have a look.
Hey @zhwlxl , can you do a test with this network? https://1drv.ms/u/s!Atqjh7mmOwHFkBd9jrSLhP6Zw79f?e=PElgo2
In this one I just changed the trigger that operates on the nodes' geometries, which would be a simple solution for the problem
Hi @pedrocamargo, I tried the modified network you provided, it still provided the same results with creating a new B_node of the connector.
Thanks,
@pedrocamargo I've prototyped a little code in QGIS python console working with purely binary data.
Is there any problem with relying on Shapely, which exists in at least the Windows QGIS install? This would allow elimination of the binary->text conversion completely:
Note: you should also switch to using placeholder ? instead of so much string formatting, as per https://docs.python.org/3/library/sqlite3.html
from shapely import wkb
from shapely.geometry import LineString
sql = "Select node_id, asbinary(geometry) from nodes"
cursor.excute(sql)
nodes = cursor.fetchall()
p1 = wkb.loads(nodes[0][1]) # returns a shapely Point
# ...
geometry = LineString([p1, p2])
cursor.execute('insert into links (link_id, modes, geometry) values (?, ?, SetSRID(GeomFromWKB(?), ?))', (link_id, modes, wkb.dumps(geometry), srid))
conn.commit()Actually, it looks like we could do this with qgis geometries instead of shapely, for the same benefits:
https://qgis.org/pyqgis/3.10/core/QgsGeometry.html#qgis.core.QgsGeometry.fromWkb
I used @arobrien 's idea and dropped the need for Shapely or even QGIS's geometry handling, and used only WKB throughout, letting Spatialite itself handle the creation of the connector with MakeLine.
In my machine, however, QGIS is closing down after successfully adding all connectors, and I have absolutely no idea why, Can you test it, @zhwlxl ?
There is a pull request for the fix #168
Hi, I did the test on the the python script I used before with the WKB changes, and it worked prefectly. I also test the GUI in QGIS both Mac and Win. In Mac, I cann't open project file as it always crash out. In Win10, it woked, but QGIS is closing down after adding all connectors. Thanks.
The issue has been solved in the Python code and the capability has been moved out of QGIS