Unable to restore backup
Gert-P opened this issue · comments
Hi
I urgently need to restore a backup of a ceph rdb image on proxmox. I have lost a pool and had to recreate it and I am now restoring all the images that was on that pool from a backup, however one image does not want to restore all the way, it goes fine till around 30% and then it just stops and displays "Killed"
I tried an earlier version of the same image with the same results. This is very urgent that I restore that backup. Please help
.
I have enabled DEBUG mode and now I get this:
I'm very sorry that my response took to long and I'd assume that you already found a solution or gave up on this. The logging by Benji doesn't reveal anything that might help to narrow this down. I'd suspect that the process might be killed by the kernel because of on out-of-memory condition. I'd be interested in the outcome of this so if you have the time please let me know. Thanks.
I just tried to restore the same VM from a new backup and ran into the same problem. I checked dmesg and as you suspected it was killed due to "Out of memory", not sure why cause I have shut down all the VMs on that node and as mentioned previously the server have 128GB of ram.
I was monitoring the RAM usage while the restore was running and it got higher and higher and reached 100% when the restore reached 9%, then it was killed and immediately after the ram usage dropped again back to normal.
I'm very sorry for the big inconveniences this has caused you and appreciate that you're still willing to work on a solution with me.
I've just done a test restore of a 60GB volume which contains a production database and memory usage of Benji stayed stable over the whole restore run at about 500MB:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
518 root 20 0 2473924 480432 30156 R 142.0 0.2 2:51.91 benji [Restoring version <redacted> to rbd:rbd/csi-vol-7ef85515-949f-11ec-b812-de350e70c1d2 (98.7%)]
I tried running Benji with a memory profiler (https://github.com/pythonspeed/filprofiler) but after some initial problems with the easy install script wrapper that Benji uses all I got from the tool was a segmentation fault. https://pypi.org/project/memory-profiler/ at least provided some output. Maybe using memory-profiler would be a good starting point for further investigation but there are other memory profilers out there. I suspect that the multi-threadedness of Benji might make this process more difficult. It might also be difficult to get the collected output if the process is killed by the kernel OOM killer, so some code changes would be needed to generate the report before that happens. Maybe it would suffice to get some statistics of the objects on the heap to point us into the right direction (see https://github.com/zhuyifei1999/guppy3/ for example).
Unfortunately I don't have the time to work on this at the moment in my spare time and as shown I can't reproduce it for my main use case and customer. I will advise my boss to contact you. Maybe we can work something out, so that I can provide paid support to you.
Thank you for the reply, The image I am trying to restore now is around 150GB and I am restoring to RBD. I understand that you don't have time to work on it now. I am looking forward to your boss contacting me and I will be very happy if you can then provide me with paid support.
Thank you for your understanding. I'm feeling bad because I actually do want to help but I can't do this in my spare time at the moment. In the mean time can you provide the information from the bug template (https://github.com/elemental-lf/benji/blob/master/.github/ISSUE_TEMPLATE/bug_report.md). I'm mainly interested in your Benji version and configuration as well as your Ceph version and the kind of storage you're using for the backups.
I have an external device formatted ext4 I use for the backup, the backup is an additional mount point of a LXC container.
Benji Version:
INFO: Benji version: 0.15.0.
INFO: Configuration version: 1.0.0, supported >=1,<2.
INFO: Metadata version: 3.0.0, supported >=1,<4.
INFO: Object metadata version: 2.0.0, supported >=1,<3.
Proxmox Version 7.1.7
ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable)
Python 3.9.2
pip list
Package Version
---------------- --------
alembic 1.6.5
argcomplete 1.12.3
benji 0.15.0
Cerberus 1.3.4
colorama 0.4.4
dateparser 0.7.6
diskcache 5.2.1
greenlet 1.1.1
Mako 1.1.4
MarkupSafe 2.0.1
pip 20.3.4
pkg-resources 0.0.0
prettytable 0.7.2
psycopg2-binary 2.9.1
pycryptodome 3.10.1
pyparsing 2.4.7
python-dateutil 2.8.2
python-editor 1.0.4
pytz 2021.1
rados 2.0.0
rbd 2.0.0
regex 2021.8.3
ruamel.yaml 0.15.100
semantic-version 2.8.5
setproctitle 1.2.2
setuptools 44.1.1
six 1.16.0
sortedcontainers 2.4.0
sparsebitfield 0.2.3
SQLAlchemy 1.4.22
structlog 21.1.0
tzlocal 2.1
zstandard 0.15.2
benji.yaml
configurationVersion: '1'
logFile: /var/log/benji.log
databaseEngine: sqlite:////mnt/backup/benji/benji.sqlite
blockSize: 4194304
processName: benji
disallowRemoveWhenYounger: 0
defaultStorage: usb
transforms:
- name: compression
module: zstd
configuration:
level: 3
# dictDataFile:
storages:
- name: usb
storageId: 1
module: file
configuration:
path: /mnt/backup/benji-data/
activeTransforms:
- compression
# - name: b2
# storageId: 2
# module: b2
# configuration:
# accountId: f164bb96cda5
# applicationKey: 0029fcd28d1b84d2702985e43e4fcdb945a88cc0e2
# bucketName: hugetest
# writeObjectAttempts: 2
# uploadAttempts: 5
# bandwidthRead: 0
# bandwidthWrite: 0
# simultaneousWrites: 20
# simultaneousReads: 20
ios:
- name: rbd
module: rbdaio
configuration:
cephConfigFile: /etc/ceph/ceph.conf
clientIdentifier: admin
newImageFeatures:
- RBD_FEATURE_LAYERING
- RBD_FEATURE_EXCLUSIVE_LOCK
- RBD_FEATURE_STRIPINGV2
simultaneousReads: 6
- name: file
module: file
Command: benji restore -f --sparse vm-104-disk-3.h-jtlbgd rbd:ceph-hdd/vm-105-disk-0
In the mean time I am also trying to mount the image using RBD with no success:
(benji) root@node4:~# benji nbd
INFO: Starting to serve NBD on 127.0.0.1:10809
INFO: Incoming connection from 127.0.0.1:52466.
INFO: [127.0.0.1:52466] Negotiated export: vm-104-disk-3.h-jtlbgd.
INFO: [127.0.0.1:52466] Version vm-104-disk-3.h-jtlbgd has been opened.
INFO: [127.0.0.1:52466] Export is read/write.
INFO: Active transforms for storage usb: compression (zstd).
ERROR: [127.0.0.1:52466] 0 bytes read on a total of 28 expected bytes
INFO: Fixating version nbd-cow-vm-104-disk-3.h-jtlbgd-ct94y1 with 1 blocks, please wait.
INFO: Set status of version nbd-cow-vm-104-disk-3.h-jtlbgd-ct94y1 to valid.
INFO: Marked version nbd-cow-vm-104-disk-3.h-jtlbgd-ct94y1 as protected.
INFO: Backed up metadata of version nbd-cow-vm-104-disk-3.h-jtlbgd-ct94y1.
INFO: Finished.
root@node4:~# modprobe nbd
root@node4:~# nbd-client 127.0.0.1 10809 /dev/nbd0 -b 512 -t 10 -N vm-104-disk-3.h-jtlbgd
Negotiation: ..size = 153600MB
Connected /dev/nbd0
root@node4:~# fdisk -l /dev/nbd0
Disk /dev/nbd0: 150 GiB, 161061273600 bytes, 314572800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@node4:~# mount /dev/nbd0 /mnt/temp
mount: /mnt/temp: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.
root@node4:~# fdisk -l /dev/nbd0
fdisk: cannot open /dev/nbd0: Input/output error
I've got good news, Gert. I found the problem thanks to the debugging output you provided. You can either switch from the rbdaio module to the older but better tested rbd module or you can try current master which includes 8b836a6.
The problem was that the io module was accepting write requests without limit and so was eating up all memory. This didn't show up in my tests because my storage's read speed is the limiting factor and in other contexts I only tested with smaller images. In your setup the storage seems to be faster than writing to an RBD image.
Please report back if you can restore your backups now. Thanks.
Lars, that is fantastic news. I have tested it using the updated rbdaio.py, however it took quite al long time before showing:
INFO: Active transforms for storage usb: compression (zstd)
Then it appeared to just hang there and not actually starting the restore. I then did as you suggested and switched from rbdaio to rbd and that work as the restoration started promptly and completed without any problems. It did however took some hours for the 150GB image to restore so it seems to be slow, but at least it worked perfectly otherwise.
Thank you SO SO much for looking into this and helping me to resolve this problem as it was quite a big one. Regarding the module, what is the difference between rbd and rbdaio? Do you suggest just sticking to rbdaio
Thank you for reporting back your findings. It's a bummer that rbdaio is still flawed, guess I'll need to do some more testing. rbd is the old multi-threaded implementation that I ported over from Backy2, rbdaio is a reimplementation that uses asynchronous io. In theory the new implementation should have less overhead and consume less CPU resources. I've seen some speed minor speed improvements, too. rbd is the safe choice.
As you seem to be limited by your Ceph's writing speed you could try to increase simultaneousWrites and see if there is any improvement (the default is 3). The network connection of the host doing the restore might also be a limiting factor.
Hi Gert,
I think I found the bug that broke the restore with rbdaio. The fix is in commit b79b0a7 and has been merged into the master branch. I'm going to close this issue now as everything has been resolved I hope.