max_threads - TypeError: replace() argument 2 must be str, not int
PedroVenancio opened this issue · comments
Hi @jblindsay
I'm trying WbW for QGIS but I'm getting this error in several tools:
QGIS version: 3.34.3-Prizren
QGIS code revision: 47373234ac
Qt version: 5.15.3
Python version: 3.9.18
GDAL version: 3.8.3
GEOS version: 3.12.1-CAPI-1.18.1
PROJ version: Rel. 9.3.1, December 1st, 2023
PDAL version: 2.6.0 (git-version: 3fced5)
Algorithm started at: 2024-02-21T10:51:36
Algorithm 'Slope' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'TEMPORARY_OUTPUT', 'units2' : 0, 'z_factor3' : 1 }
Traceback (most recent call last):
File "C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis\whitebox_workflows_for_qgis_algorithm.py", line 260, in processAlgorithm
scriptString = scriptString.replace("max_threads", max_threads)
TypeError: replace() argument 2 must be str, not int
Execution failed after 0.20 seconds
Loading resulting layers
Algorithm 'Slope' finished
The problem seems to be in
max_threads = ProcessingConfig.getSetting('WBW_MAX_THREADS')
if max_threads == None or int(max_threads) < 1:
max_threads = '-1' # All available threads
scriptString = scriptString.replace("max_threads", max_threads)
By default, it puts all the threads available in my machine:
This is an odd error indeed, in that I cannot seem to replicate it on my testing systems. Can I ask what version of the frontend you are using? Also, what happens if you enter an output file name rather than simply using the temporary file option?
Hi @jblindsay
It's the same with the path to the output:
QGIS version: 3.34.3-Prizren
QGIS code revision: 47373234ac
Qt version: 5.15.3
Python version: 3.9.18
GDAL version: 3.8.3
GEOS version: 3.12.1-CAPI-1.18.1
PROJ version: Rel. 9.3.1, December 1st, 2023
PDAL version: 2.6.0 (git-version: 3fced5)
Algorithm started at: 2024-02-21T13:00:01
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'C:/Users/Agif/AGIF/Testes/Aspect.tif', 'z_factor2' : 1 }
Traceback (most recent call last):
File "C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis\whitebox_workflows_for_qgis_algorithm.py", line 260, in processAlgorithm
scriptString = scriptString.replace("max_threads", max_threads)
TypeError: replace() argument 2 must be str, not int
Execution failed after 0.25 seconds
Loading resulting layers
Algorithm 'Aspect' finished
I also tried to install whitebox-workflows
C:\Program Files\QGIS 3.34.3>pip show whitebox-workflows
Name: whitebox_workflows
Version: 1.2.0
Summary: whitebox_workflows is a Python library for advanced spatial analysis.
Home-page:
Author: Whitebox Geospatial Inc.
Author-email: support@whiteboxgeo.com
License:
Location: c:\users\agif\appdata\roaming\python\python39\site-packages
but I keep with the same error.
I'm using
| QGIS version | 3.34.3-Prizren | QGIS code revision | 47373234ac |
|---|---|---|---|
| Qt version | 5.15.3 | ||
| Python version | 3.9.18 | ||
| GDAL/OGR version | 3.8.3 | ||
| PROJ version | 9.3.1 | ||
| EPSG Registry database version | v10.098 (2023-11-24) | ||
| GEOS version | 3.12.1-CAPI-1.18.1 | ||
| SQLite version | 3.41.1 | ||
| PDAL version | 2.6.0 | ||
| PostgreSQL client version | 15.2 | ||
| SpatiaLite version | 5.1.0 | ||
| QWT version | 6.1.6 | ||
| QScintilla2 version | 2.13.4 | ||
| OS version | Windows 10 Version 2009 | ||
| Active Python plugins | |||
| active_fire | 0.3 | ||
| AemetOpenDataDownloader | 0.1 | ||
| ArrNorm | 23.4b | ||
| attributeBasedClustering | 2.2.1 | ||
| BackupLayer | 0.3 | ||
| buffer_without_overlaps | 0.2 | ||
| cartography_tools | 1.2.1 | ||
| ContrastHomogenizer | 3.2 | ||
| Coregistration | 23.11 | ||
| crayfish | 3.6.0 | ||
| d3datavis | 3.0.6 | ||
| DataPlotly | 4.1.0 | ||
| densityanalysis | 2024.1.23 | ||
| downloadgeostatportugal | 0.2 | ||
| DSGTools_Op | 2.2.0 | ||
| dzetsaka | 3.70 | ||
| edr_plugin | 0.9.0 | ||
| ee_plugin | 0.0.6 | ||
| export_to_sql_server | 0.4 | ||
| FreehandRasterGeoreferencer | 0.8.3 | ||
| geetimeseriesexplorer | 2.0 | ||
| GeoCoding | 2.19 | ||
| getools | 1.0.0 | ||
| go2streetview | 8.7 | ||
| gribdownloader | 0.5 | ||
| GroupStats | 2.2.7 | ||
| HCMGIS | 24.1.12 | ||
| Hqgis | 1.2.3 | ||
| inspireatomclient | 0.8.1 | ||
| kmltools | 3.1.32 | ||
| layer2kmz-processing | 2.0.0 | ||
| LecoS | 3.0.1 | ||
| loadthemall | 3.4 | ||
| localmaxfilter | 1.5.7 | ||
| los_tools | 1.0 | ||
| lrs | 1.2.3 | ||
| mapswipetool_plugin | 1.2 | ||
| mmqgis | 2021.9.10 | ||
| MultiDistanceBuffer | 3.2.4 | ||
| nominatim_locator_filter | 0.3.2 | ||
| ntv2_transformations | 0.20 | ||
| OSMDownloader | 1.0.3 | ||
| photo2shape | 3.0 | ||
| pointsamplingtool | 0.5.4 | ||
| postgis_geoprocessing | 0.9 | ||
| PotentialSlopeFailure | 0.7 | ||
| processing_gnm | 0.2.0 | ||
| ProcessX | 1.7 | ||
| profiletool | 4.2.6 | ||
| qchainage | 3.0.1 | ||
| qconsolidate | 1.1.0 | ||
| qfieldsync | v4.7.1 | ||
| Qgis2threejs | 2.7.3 | ||
| qgis2web | 3.18.2 | ||
| qgis_gee_data_catalog | 0.4.3 | ||
| qgis_stac | 1.1.1 | ||
| QNEAT3 | 1.0.5 | ||
| QuickWKT | 3.1 | ||
| quick_map_services | 0.19.34 | ||
| raster_cutter | 0.6 | ||
| raster_tracer | 0.3.3 | ||
| refFunctions | 1.6 | ||
| ReverseFlow | 0.2 | ||
| road_slope_calculator | 0.5 | ||
| SemiAutomaticClassificationPlugin | 8.2.2 | ||
| sentinel2_removed_cloud_rec | 0.4 | ||
| SentinelHub | 2.0.2 | ||
| simplesvg | 3.0.3 | ||
| simplewcs2 | 0.2 | ||
| SpreadsheetLayers | 2.1.1 | ||
| SRTM-Downloader | 3.2.1 | ||
| statist | 3.2 | ||
| SwapVectorDirection | 0.9 | ||
| TerrainShading | 0.9.4 | ||
| timemanager | 3.6 | ||
| timezone_expressions | v0.2 | ||
| valuetool | 3.0.17 | ||
| Video_UAV_Tracker | 2.1 | ||
| ViewshedAnalysis | 1.8 | ||
| VoGisProfilTool | 3.0.2 | ||
| wbt_for_qgis | 1.0.9 | ||
| whitebox_workflows_for_qgis | 1.0.1 | ||
| db_manager | 0.1.20 | ||
| grassprovider | 2.12.99 | ||
| MetaSearch | 0.3.6 | ||
| otbprovider | 2.12.99 | ||
| processing | 2.12.99 |
Again, thank you for reporting this issue and providing so much information. It is puzzling to me how you are receiving this error but I am not. Given the nature of the error that you're reporting, it should affect everyone! Okay, can you perform a bit of an experiment for me please? If you open your profile folder (Settings => User Profile => Open Active Profile Folder), you should find the file whitebox_workflows_for_qgis_algorithm.py. If you could please open that file in a text editor and then look for the line:
scriptString = scriptString.replace("max_threads", max_threads)
which should be line 260 (although if you are just using a normal text editor, you may not have line numbers). Once found, if you could modify this line to read:
scriptString = scriptString.replace("max_threads", str(max_threads))
Then re-start QGIS and try running the tool again. Please let me know if this resolves the issue for you.
Hi @jblindsay
Thank you very much for your attention!
That change indeed worked, and it passes that step, but now I have another error:
Algorithm started at: 2024-02-21T14:12:17
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'C:/Users/Agif/AGIF/Testes/Aspect.tif', 'z_factor2' : 1 }
WbW Script:
import sys
path = 'C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis'
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = 'C:\Users\Agif\AGIF\Testes\'
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.aspect(raster_1, 1.0)
wbe.write_raster(outputRaster, 'Aspect.tif', True)
Process "py" failed to start. Either "py" is missing, or you may have insufficient permissions to run the program.
Execution failed after 0.07 seconds
Loading resulting layers
Algorithm 'Aspect' finished
Any hint?
Another curious one, as I don't use Windows machines very often, but I have tested that this frontend works on Windows. Regardless, try this fix: in the same file that I indicated before (whitebox_workflows_for_qgis_algorithm.py) find the lines:
if platform.system() == 'Windows':
command = "py"and change it to:
if platform.system() == 'Windows':
command = "python"The key line should be around line 304. Please let me know if that works.
Here is the new output:
Algorithm started at: 2024-02-21T14:39:52
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste_Aspect.tif', 'z_factor2' : 1 }
WbW Script:
import sys
path = 'C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis'
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = 'C:\Users\Agif\AGIF\Testes\'
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.aspect(raster_1, 1.0)
wbe.write_raster(outputRaster, 'MDT_Teste_Aspect.tif', True)
File "<string>", line 2
path = 'C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis'
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
Process returned error code 1
Execution completed in 1.10 seconds
Results:
{'fnOutput': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/Agif/AGIF/Testes/MDT_Teste_Aspect.tif, 'createOptions': {'fileEncoding': 'windows-1252'}}>}
Loading resulting layers
The following layers were not correctly generated.
• C:/Users/Agif/AGIF/Testes/MDT_Teste_Aspect.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
In the whitebox_workflows_for_qgis/function_code_files there should be a file called aspect.py. If you open that file up in a text/code editor and modify the second line to read:
path = r'plugin_path'That is, place an 'r' in front of the quotation. Please try this and then run the Aspect tool once more and let me know if that works. If so, then I can make all of these changes and (all of the function files will need to be modified) and update the frontend.
Hi @jblindsay
I think we are close to the fix! Now the problem is with the working_directory:
Algorithm started at: 2024-02-21T16:05:43
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'C:/Users/Agif/AGIF/Testes/Aspect.tif', 'z_factor2' : 1 }
WbW Script:
import sys
path = r'C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis'
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = 'C:\Users\Agif\AGIF\Testes\'
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.aspect(raster_1, 1.0)
wbe.write_raster(outputRaster, 'Aspect.tif', True)
File "<string>", line 10
wbe.working_directory = 'C:\Users\Agif\AGIF\Testes\'
^
SyntaxError: EOL while scanning string literal
Process returned error code 1
Execution completed in 0.38 seconds
Results:
{'fnOutput': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/Agif/AGIF/Testes/Aspect.tif, 'createOptions': {'fileEncoding': 'windows-1252'}}>}
Loading resulting layers
The following layers were not correctly generated.
• C:/Users/Agif/AGIF/Testes/Aspect.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
'C:\Users\Agif\AGIF\Testes\' -> the last slash \ seems to be escaping the '
Yes indeed we are! Okay, instead of placing the 'r' in front, let's do a proper job of handling the Windows file separator character. I have updated all of the function files here:
https://github.com/jblindsay/whitebox_workflows_for_qgis/tree/main/function_code_files
Can you replace your files in the plugin folder with these files in the repository and try again? You'll notice that I used os.path.normpath for both the plugin path and the working directory in each of these new files. This should, fingers crossed, fix the issue for you.
Hi @jblindsay
With the new code I get (I changed all folder with the code from the last commit):
Algorithm started at: 2024-02-21T16:54:29
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'TEMPORARY_OUTPUT', 'z_factor2' : 1 }
WbW Script:
import os, sys
path = os.path.normpath("C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis")
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = os.path.normpath("C:\Users\Agif\AGIF\Testes\")
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.aspect(raster_1, 1.0)
wbe.write_raster(outputRaster, 'C:\Users\Agif\AppData\Local\Temp\processing_bGvrnu\4d4bce8ef98d437bba7b4e05414894bd\fnOutput.tif', True)
File "<string>", line 2
path = os.path.normpath("C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis")
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
Process returned error code 1
Execution completed in 0.31 seconds
Results:
{'fnOutput': <QgsProcessingOutputLayerDefinition {'sink':TEMPORARY_OUTPUT, 'createOptions': {'fileEncoding': 'windows-1252'}}>}
Loading resulting layers
The following layers were not correctly generated.
• C:/Users/Agif/AppData/Local/Temp/processing_bGvrnu/4d4bce8ef98d437bba7b4e05414894bd/fnOutput.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
Then I changed the line 2 to path = os.path.normpath(r"plugin_path")
and...
Algorithm started at: 2024-02-21T16:59:07
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'C:/Users/Agif/AGIF/Testes/Aspect.tif', 'z_factor2' : 1 }
WbW Script:
import os, sys
path = os.path.normpath(r"C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis")
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = os.path.normpath("C:\Users\Agif\AGIF\Testes\")
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.aspect(raster_1, 1.0)
wbe.write_raster(outputRaster, 'Aspect.tif', True)
File "<string>", line 10
wbe.working_directory = os.path.normpath("C:\Users\Agif\AGIF\Testes\")
^
SyntaxError: EOL while scanning string literal
Process returned error code 1
Execution completed in 0.42 seconds
Results:
{'fnOutput': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/Agif/AGIF/Testes/Aspect.tif, 'createOptions': {'fileEncoding': 'windows-1252'}}>}
Loading resulting layers
The following layers were not correctly generated.
• C:/Users/Agif/AGIF/Testes/Aspect.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
It seems to have the same behaviour than before.
Goodness, I really dislike Windows and an OS. Okay, I would have thought that os.path.normpath would have been a better approach to resolving this than using r-strings, but alas...Windows! I'm so sorry that you've had to do all this testing here, but I am very grateful for your patience. Alright, can you also add an 'r' in front of the working directory line string well, as in wbe.working_directory = os.path.normpath(r"wk_dir").
Oh I'm glad if I can help to tackle this issue!
Thanks for your effort trying to fix it!
Changing to wbe.working_directory = os.path.normpath(r"wk_dir") I still get
Algorithm started at: 2024-02-21T17:15:15
Algorithm 'Aspect' starting…
Input parameters:
{ 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'C:/Users/Agif/AGIF/Testes/Aspect.tif', 'z_factor2' : 1 }
WbW Script:
import os, sys
path = os.path.normpath(r"C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis")
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = os.path.normpath(r"C:\Users\Agif\AGIF\Testes\")
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.aspect(raster_1, 1.0)
wbe.write_raster(outputRaster, 'Aspect.tif', True)
File "<string>", line 10
wbe.working_directory = os.path.normpath(r"C:\Users\Agif\AGIF\Testes\")
^
SyntaxError: EOL while scanning string literal
Process returned error code 1
Execution completed in 0.33 seconds
Results:
{'fnOutput': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/Agif/AGIF/Testes/Aspect.tif, 'createOptions': {'fileEncoding': 'windows-1252'}}>}
Loading resulting layers
The following layers were not correctly generated.
• C:/Users/Agif/AGIF/Testes/Aspect.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
I've been able to replicate the issue now and I'll continue to work on a solution and will report back once I have found one. Thanks again for all of your help with this and for taking the time to report the issue.
Okay, I think I have a fix in place. At least it seems to be working on my old Windows machine. Do you want to try to download all of the files from the Github repo once again and replace the files in your QGIS plugin folder and see if this works?
@jblindsay Perfect!! It works now!!
Okay, excellent! I've just uploaded the new version (v1.2.0) of the frontend to the QGIS plugin repo and it awaits approval. I'm marking this issue as closed now, but if you encounter any further related problems, please feel free to re-open. I am very grateful for everything that you did today to help with this.
By the way, that's one beautiful hillshade raster! You might want to give the multidtreciton hillshade tool (particularly in 360-degree mode) a try as well.
Wow really nice effect of multidireciton hillshade!!!
It seems to be working great now, the only thing I see is when using a TEMPORARY_OUTPUT:
Algorithm 'Hillshade' starting…
Input parameters:
{ 'altitude3' : 30, 'azimuth2' : 315, 'dem1' : 'C:/Users/Agif/AGIF/Testes/MDT_Teste.tif', 'fnOutput' : 'TEMPORARY_OUTPUT', 'z_factor4' : 1 }
WbW Script:
import os, sys
path = os.path.normpath(r"C:\Users/Agif/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\whitebox_workflows_for_qgis")
if path not in sys.path:
sys.path.append(path)
from whitebox_workflows import WbEnvironment
wbe = WbEnvironment()
wbe.verbose = True
wbe.max_procs = 8
wbe.working_directory = os.path.normpath(r"C:\Users\Agif\AGIF\Testes")
raster_1 = wbe.read_raster('MDT_Teste.tif')
outputRaster = wbe.hillshade(raster_1, 315.0, 30.0, 1.0)
wbe.write_raster(outputRaster, 'fnOutput.tif', True)
********************************************
* Welcome to hillshade *
* Powered by Whitebox Workflows for Python *
* www.whiteboxgeo.com *
********************************************
Process completed successfully.
Execution completed in 0.79 seconds
Results:
{'fnOutput': <QgsProcessingOutputLayerDefinition {'sink':TEMPORARY_OUTPUT, 'createOptions': {'fileEncoding': 'windows-1252'}}>}
Loading resulting layers
The following layers were not correctly generated.
• C:/Users/Agif/AppData/Local/Temp/processing_NCqQqw/d6e129d8fc7e468cb1b57971159cd34d/fnOutput.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
Thank you very much @jblindsay for this amazing collection of tools and for your effort fixing this issue!!!
That is an even more beautiful hillshade! As for temporary files, I've never really used that option in QGIS before today's discussion. What I've come to realize is that, at least on my computer, the default is to output a '*.nc' file, which I think is a NetCDF file type. Unfortunately, this file type is unsupported by Whitebox and so it throws an error. I would probably just avoid using the QGIS temporary raster file outputs with Whitebox and specify a proper output file name instead. That's probably good practice anyhow.
the default is to output a '*.nc' file, which I think is a NetCDF file type.
The defaut output raster and vector file types are defined on Processing General Settings:
In my case the default raster output is defined to tif. You should have nc (NetCDF).
But even with tif format, the folders are created on TEMP/TMP folder when I run the algorithm with TEMPORARY_OUTPUT, but they are empty, the fnOutput.tif is not created.
Hi @jblindsay
I see now that when I use the TEMPORARY_OUTPUT, the output file is in fact created
but inside the working_directory, and not in the TEMP folder. QGIS tries to open it from the TEMP, and fails as it is not there.
Anyway, the issue of this ticket is definitively fixed! Thank you very much!!
Now that is an excellent tip! Thanks for this. I figured it must be configurable, but I couldn't see where.