Database error: BLAST and Diamond

Question

Database error: BLAST and Diamond

s4251484 opened this issue 5 years ago · comments

Hi

I installed SUPER-FOCUS using miniconda today. The database_PKs.txt and organisms2subsystem.txt were not in the db folder, which I manually downloaded and put into the db folder.

ls /Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db
100_clusters 98_clusters
90_clusters clusters organisms2subsystem.txt
95_clusters database_PKs.txt

Here are the two database errors I encountered:

1. BLAST
BLAST Database error: No alias or index file found for protein database [/Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db/static/blast/90.db] in search path

[2019-03-12 15:57:29,724 - INFO] Parsing Alignments
Traceback (most recent call last):
File "/Users/ME/miniconda2/envs/super-focus/bin/superfocus", line 11, in
load_entry_point('superfocus==0.31', 'console_scripts', 'superfocus')()
File "/Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/superfocus.py", line 322, in main
minimum_identity, minimum_alignment, subsystems_translation, aligner)
File "/Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/do_alignment.py", line 171, in parse_alignments
update_results(results, sample_index, temp_results, normalise, number_samples)
UnboundLocalError: local variable 'temp_results' referenced before assignment

2. DIAMOND
#CPU threads: 4
Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1)
Temporary directory: /Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db/tmp
Opening the database... No such file or directory
[0.000117s]
Error: Error opening file /Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db/static/diamond/90.db

I reckon there is something not right with my db folder, but I don't know how to make it right. Please help. Looking forward for your reply. Thanks.

Geni Silva · Answer 1 · Tue Mar 12 2019 22:23:33 GMT+0800 (China Standard Time)

Hi @s4251484 , Any reason to be using SUPER-FOCUS 0.31 rather than the latest?

Also, did you can the program to download the database (https://github.com/metageni/SUPER-FOCUS#database)?

Best

s4251484 · Answer 2 · Wed Mar 13 2019 05:19:48 GMT+0800 (China Standard Time)

Hi @s4251484 , Any reason to be using SUPER-FOCUS 0.31 rather than the latest?

Also, did you can the program to download the database (https://github.com/metageni/SUPER-FOCUS#database)?

Best

hi @metageni, 0.31 was the version I got when I downloaded SUPER-FOCUS via miniconda. and it was not updated even after I "conda update super-focus"
and yes I downloaded both blast and diamond databases.

UPDATE I: I install SUPER-FOCUS using pip3 intall instead and got the latest version 0.33. however it didn't manage to format the diamond database successfully, my guess is there is problem with the makedb. it doesn't seem to be able to read and iterate the individual faa files and make a db out of the faa files. Hope my experience helps you.

Other the other hand, downloading and formatting blast database (with super-focus intalled via pip3) was successful, I'm now running the superfocus with blast database, hopefully comes back with good news.

UPDATE II: both (pip3) and (miniconda) installed super-focus are not able to build diamond db for me. errors came out after unzipping the db.zip. hence, I went on to make the diamond db manually (by referring to diamond manual) instead of using the superfocus_downloadDB.py. Briefly, first use cat to join the multiple faa files into one fasta file, then use diamond makedb to format fasta into diamond db. Move the diamond db files to a folder "db/static/diamond". then the diamond *.db files will be usable with superfocus.py.

PS: dont forget to download the database_PKs.txt and organisms2subsystems.txt from github and transfer them to the db folder if yours are not auto-generated. this happened to me.

Cheers!

Geni Silva · Answer 3 · Thu Mar 14 2019 01:44:32 GMT+0800 (China Standard Time)

@s4251484

I just took a look into the problem, and it seems that version 0.33 would fix it. For some reason, bioconda is installing v0.31.

I just downloaded version 0.33 on git and installed it. I downloaded the database and everything worked fine.

For now, I would recommend you do the same.

I will follow up with the bioconda team on the issue.

Sorry about it

s4251484 · Answer 4 · Sat Mar 16 2019 21:14:12 GMT+0800 (China Standard Time)

@metageni that will be great! thank you~ thank you for developing useful tools like this. They are great for bio students who dont have a programming background, eg ME!

Florentin Constancias · Answer 5 · Sat Mar 30 2019 15:28:02 GMT+0800 (China Standard Time)

I am experiencing the same issue running superfocus installed using conda.

Geni Silva · Answer 6 · Sun Mar 31 2019 01:22:13 GMT+0800 (China Standard Time)

Hi @fconstancias, I will see if this weekend I can create a new version that deals with the database installation in a different way. Stay tuned!

Geni Silva · Answer 7 · Tue Apr 02 2019 14:53:20 GMT+0800 (China Standard Time)

@fconstancias @s4251484 I have released a new version (v0.34) of superfocus (https://github.com/metageni/SUPER-FOCUS/releases) where users can now manually download the database and format it using a script I wrote.

It should be live on bioconda in 1-2 days. I will see if I can push it tomorrow into pip.

For now, you can pip it using (https://github.com/metageni/SUPER-FOCUS/releases).

Please read (https://github.com/metageni/SUPER-FOCUS#database) on how to download and format the database.

Any feedback is welcome.

Deleted user · Answer 8 · Fri Apr 05 2019 01:46:16 GMT+0800 (China Standard Time)

I have tried installing v0.34 but at the end, it is installing 0.0.0 version.

[root@rameshgoel ~]# cd /srv/data/SUPER-FOCUS-0.34/
[root@rameshgoel SUPER-FOCUS-0.34]# python setup.py install
/usr/bin/miniconda3/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'use_scm_version'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
creating superfocus.egg-info
writing superfocus.egg-info/PKG-INFO
writing dependency_links to superfocus.egg-info/dependency_links.txt
writing entry points to superfocus.egg-info/entry_points.txt
writing requirements to superfocus.egg-info/requires.txt
writing top-level names to superfocus.egg-info/top_level.txt
writing manifest file 'superfocus.egg-info/SOURCES.txt'
reading manifest file 'superfocus.egg-info/SOURCES.txt'
writing manifest file 'superfocus.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/superfocus_app
copying superfocus_app/superfocus_downloadDB.py -> build/lib/superfocus_app
copying superfocus_app/init.py -> build/lib/superfocus_app
copying superfocus_app/do_alignment.py -> build/lib/superfocus_app
copying superfocus_app/superfocus.py -> build/lib/superfocus_app
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/superfocus_app
copying build/lib/superfocus_app/superfocus_downloadDB.py -> build/bdist.linux-x86_64/egg/superfocus_app
copying build/lib/superfocus_app/init.py -> build/bdist.linux-x86_64/egg/superfocus_app
copying build/lib/superfocus_app/do_alignment.py -> build/bdist.linux-x86_64/egg/superfocus_app
copying build/lib/superfocus_app/superfocus.py -> build/bdist.linux-x86_64/egg/superfocus_app
byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/superfocus_downloadDB.py to superfocus_downloadDB.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/init.py to init.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/do_alignment.py to do_alignment.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/superfocus.py to superfocus.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying superfocus.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating dist
creating 'dist/superfocus-0.0.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing superfocus-0.0.0-py3.6.egg
removing '/usr/bin/miniconda3/lib/python3.6/site-packages/superfocus-0.0.0-py3.6.egg' (and everything under it)
creating /usr/bin/miniconda3/lib/python3.6/site-packages/superfocus-0.0.0-py3.6.egg
Extracting superfocus-0.0.0-py3.6.egg to /usr/bin/miniconda3/lib/python3.6/site-packages
superfocus 0.0.0 is already the active version in easy-install.pth
Installing superfocus script to /usr/bin/miniconda3/bin
Installing superfocus_downloadDB script to /usr/bin/miniconda3/bin

Installed /usr/bin/miniconda3/lib/python3.6/site-packages/superfocus-0.0.0-py3.6.egg
Processing dependencies for superfocus==0.0.0
Searching for scipy==1.2.1
Best match: scipy 1.2.1
Processing scipy-1.2.1-py3.6-linux-x86_64.egg
scipy 1.2.1 is already the active version in easy-install.pth

Using /usr/bin/miniconda3/lib/python3.6/site-packages/scipy-1.2.1-py3.6-linux-x86_64.egg
Searching for numpy==1.16.2
Best match: numpy 1.16.2
Processing numpy-1.16.2-py3.6-linux-x86_64.egg
numpy 1.16.2 is already the active version in easy-install.pth
Installing f2py script to /usr/bin/miniconda3/bin
Installing f2py3 script to /usr/bin/miniconda3/bin
Installing f2py3.6 script to /usr/bin/miniconda3/bin

Using /usr/bin/miniconda3/lib/python3.6/site-packages/numpy-1.16.2-py3.6-linux-x86_64.egg
Finished processing dependencies for superfocus==0.0.0

[root@rameshgoel SUPER-FOCUS-0.34]# superfocus -v
SUPER-FOCUS 0.0.0

Why it's not installing v0.34?

Geni Silva · Answer 9 · Fri Apr 05 2019 01:54:49 GMT+0800 (China Standard Time)

Genis-MacBook-Pro-2:geni.silva$ superfocus -v
SUPER-FOCUS 0.34

Bizzare! I wonder if it is something to do with bioconda.

I will need to investigate it

Deleted user · Answer 10 · Fri Apr 05 2019 02:05:28 GMT+0800 (China Standard Time)

Hi @metageni : I was able to install v0.34 using super user by logging as root.
But I can't access the superfocus when log in as user. How to fix this?

Geni Silva · Answer 11 · Fri Apr 05 2019 18:27:29 GMT+0800 (China Standard Time)

Problem solved here #49

Geni Silva · Answer 12 · Fri Apr 05 2019 18:32:29 GMT+0800 (China Standard Time)

Closing this issue, and leaving (#24) so people can ask questions there.

Florentin Constancias · Answer 13 · Wed Apr 10 2019 16:33:43 GMT+0800 (China Standard Time)

I have just installed super focus using conda conda create -n super-focus -y -c bioconda super-focus and I also got

(super-focus)$ superfocus --version
SUPER-FOCUS 0.0.0

Geni Silva · Answer 14 · Thu Apr 11 2019 02:44:23 GMT+0800 (China Standard Time)

@fconstancias For some reason, superfocus is giving this back when using -v. Were you able to run the program? I will add a fix to it on next release.

You can also install the too with pip.