OperationalError: database is locked

Question

OperationalError: database is locked

AHCChan opened this issue 9 years ago · comments

Hi,

I'm using a pipeline implemented by Ruffus that calls several other programs. I'm getting an error when Ruffus calls "bamM make".

This error only occurs when the data is stored on some parts of the server but not others. Any troubleshooting help would be greatly appreciated.

The error message is as follows:

Original exception:

Exception #1
  '<class 'ruffus.ruffus_exceptions.RethrownJobError'>

    Exceptions generating parameters for

    task = 'transcriptm.pipeline.QC_output'

Original exception:

    Exception #1
      'sqlite3.OperationalError(database is locked)' raised in ...
       Task = def transcriptm.pipeline.QC_output(...):


    Traceback (most recent call last):
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5118, in parameter_generator
        job_history, verbose_abbreviated_path):
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 4923, in job_needs_to_run
        verbose_abbreviated_path=verbose_abbreviated_path)
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/file_name_parameters.py", line 548, in needs_update_check_modify_time
        if os.path.relpath(p) not in job_history and p not in set_incomplete_files:
      File "/srv/sw/python/2.7.4/lib/python2.7/_abcoll.py", line 348, in __contains__
        self[key]
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/dbdict.py", line 174, in __getitem__
        (key, )).fetchone()
    OperationalError: database is locked

    ' raised in ...
Traceback (most recent call last):
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5887, in pipeline_run
    verbose)
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5283, in fill_queue_with_job_parameters
    for params in job_parameters:
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5215, in parameter_generator
    raise errt
RethrownJobError: 

    Exceptions generating parameters for

    task = 'transcriptm.pipeline.QC_output'

Original exception:

    Exception #1
      'sqlite3.OperationalError(database is locked)' raised in ...
       Task = def transcriptm.pipeline.QC_output(...):


    Traceback (most recent call last):
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5118, in parameter_generator
        job_history, verbose_abbreviated_path):
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 4923, in job_needs_to_run
        verbose_abbreviated_path=verbose_abbreviated_path)
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/file_name_parameters.py", line 548, in needs_update_check_modify_time
        if os.path.relpath(p) not in job_history and p not in set_incomplete_files:
      File "/srv/sw/python/2.7.4/lib/python2.7/_abcoll.py", line 348, in __contains__
        self[key]
      File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/dbdict.py", line 174, in __getitem__
        (key, )).fetchone()
    OperationalError: database is locked

Leo Goodstadt · Answer 1 · Thu Feb 18 2016 22:18:00 GMT+0800 (China Standard Time)

I did a quick google and found this:
http://stackoverflow.com/questions/2740806/python-sqlite-database-is-locked

The checkpointing uses an SQLite database file (defaults to
.ruffus_history.sqlite) that requires file locking semantics to ensure
database integrity. The error message is telling you this has gone wrong.
Here are some suggestions:

Your underlying file system does not provide file locking semantics.
The stack overflow article says this can fail if your directory is samba
enabled....?
Run sqlite manually on the checkpoint database
$ sqlite3 .ruffus_history.sqlite
Check you don't have a hung pipeline process sitting on the file.
$ fuser .ruffus_history.sqlite
Check you don't have multiple pipeline running instances using the
same database file. (This is not sensible on so many levels...)
Check there isn't a crashed session that has not been cleaned up
properly
$ ls .ruffus_history.sqlite-journal
Check that the database is not corrupt
$ sqlite3 .ruffus_history.sqlite "pragma integrity_check;"
Try renaming the database
$ mv .ruffus_history.sqlite .ruffus_history.sqlite.backup

Please tell us if any of this works.
If your file system does not support placing the checkpoint database in
your working directory, you can always put it elsewhere
Leo

On 17 February 2016 at 05:43, AHCChan notifications@github.com wrote:

Hi,

I'm using a pipeline implemented by Ruffus that calls several other
programs. I'm getting an error when Ruffus calls "bamM make".

This error only occurs when the data is stored on some parts of the server
but not others. Any troubleshooting help would be greatly appreciated.

The error message is as follows:

Original exception:

Exception #1
'<class 'ruffus.ruffus_exceptions.RethrownJobError'>
Exceptions generating parameters for

task = 'transcriptm.pipeline.QC_output'
Original exception:
Exception #1
  'sqlite3.OperationalError(database is locked)' raised in ...
   Task = def transcriptm.pipeline.QC_output(...):


Traceback (most recent call last):
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5118, in parameter_generator
    job_history, verbose_abbreviated_path):
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 4923, in job_needs_to_run
    verbose_abbreviated_path=verbose_abbreviated_path)
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/file_name_parameters.py", line 548, in needs_update_check_modify_time
    if os.path.relpath(p) not in job_history and p not in set_incomplete_files:
  File "/srv/sw/python/2.7.4/lib/python2.7/_abcoll.py", line 348, in __contains__
    self[key]
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/dbdict.py", line 174, in __getitem__
    (key, )).fetchone()
OperationalError: database is locked

' raised in ...
Traceback (most recent call last):
File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5887, in pipeline_run
verbose)
File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5283, in fill_queue_with_job_parameters
for params in job_parameters:
File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5215, in parameter_generator
raise errt
RethrownJobError:
Exceptions generating parameters for

task = 'transcriptm.pipeline.QC_output'
Original exception:
Exception #1
  'sqlite3.OperationalError(database is locked)' raised in ...
   Task = def transcriptm.pipeline.QC_output(...):


Traceback (most recent call last):
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 5118, in parameter_generator
    job_history, verbose_abbreviated_path):
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/task.py", line 4923, in job_needs_to_run
    verbose_abbreviated_path=verbose_abbreviated_path)
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/file_name_parameters.py", line 548, in needs_update_check_modify_time
    if os.path.relpath(p) not in job_history and p not in set_incomplete_files:
  File "/srv/sw/python/2.7.4/lib/python2.7/_abcoll.py", line 348, in __contains__
    self[key]
  File "/srv/sw/ruffus/2.6.3/lib/python2.7/site-packages/ruffus/dbdict.py", line 174, in __getitem__
    (key, )).fetchone()
OperationalError: database is locked
—
Reply to this email directly or view it on GitHub
#63.

Andreas Heger · Answer 2 · Thu May 24 2018 03:57:30 GMT+0800 (China Standard Time)

Thanks , closing for now.