Add a sync client daemon
DrHyde opened this issue · comments
I have several headless client machines on which I can't run the normal client, because not only do they require X11, they also require a "tray", whatever that might be (is it a Gnome thing?). Having a client that I can just run in the background like I can with the Dropbox client would be great.
You know about owncloudcmd?
Yeah. It just does a single sync. The GUI client (and the Dropbox CLI client) sit in the background waiting for changes to happen and then syncing them straight away in the background.
@DrHyde Is there any reason that owncloudcmd could not be set up as a cron job?
Understandably, this would not work on Windows, but because you mention X11/Gnome I assume you're on Linux.
I presume that I'd have to write Stuff to make sure only one instance was running at a time?
But in any case, that just seems so ... inelegant.
[adds "write owncloudcmddaemon'" to his to-do list]
@DrHyde You could strictly decouple the sync algorithm from the GUI and make them into two processes, but then you'd need to invent IPC between the two. That's feasible, although not trivial.
That said, your usecase is not what owncloudcmd was designed for. The primary use case is testing, benchmarking and "special operations" that would be more cumbersome with the normal client.
@DrHyde for the cron script method, it's a 3 step job. I don't forsee any need to make a daemon or anything.
- you'd simply need to create a one-line bash script that calls the owncloudcmd with the appropriate flags, just as you do now for the single sync.
- Once the script is made, you simply tell cron to execute the script however often you please. (every minute, every day?)
- Boom, done. 🍺
This cron stuff is a trick, not a proper sycn daemon.
There's a difference from "update periodically" to "synchronize".
Owncloud should have a proper daemon. OMG, even Dropbox has it!
This cron stuff is a trick, not a proper sycn daemon.
There's a difference from "update periodically" to "synchronize".
Yes, and I acknowledged that in my post above
Owncloud should have a proper daemon. OMG, even Dropbox has it!
And I agree. I am happy to assist you with a patch that 100% separates the sync logic from the GUI and forms a daemon on its own. I'll also review it.
It sounds great! Many thanks.
@brihuega This was to say "if you write the patch..."
Related #1581
I came here looking for a headless sync daemon as well.
Just one comment: I agree with @danimo that the cleanest solution would be to separate the GUI from the daemon and implement IPC to make them work together. That way the daemon can run in the background, but still be controlled by the user whenever they log in.
But it seems most people are looking for a plain daemon process because their machine doesn't have X installed in the first place. So they would never actually require the IPC to work.
Why not just make the GUI part optional (as per good old C-style preprocessor macros, for example), and then compile and distribute two versions? This should be fairly easy to do, and not require major changes in the code.
By the way, I'm actually on OS X, not Linux. But same idea there.
@guruz: Yes, I know about the commandline client. It does a single sync run, and then exits. This is not the same as a daemon, which is what I and other people were looking for. Or are you trying to tell me something else that I'm missing?
why not make the sync daemon accessible via an API, so someone can make a webclient for it (would be nice for a headless machine) and someone can make a real gui for desktops?
@idef1x There is the SocketAPI already, feel free to extend.. http://code.woboq.org/owncloud/client/src/gui/socketapi.h.html#_ZN3OCC9SocketApi30command_RETRIEVE_FOLDER_STATUSERK7QStringP12QLocalSocket
Although I would argue that the webclient is the normal ownCloud server?
@guruz We're talking about the client here, not the server. The "normal" ownCloud is something else. People here are talking about syncing files from a (nother) server, maybe for backup for exemple, toward the main owncloud server.
any progress here?
@danimo could this feature be somehow prioritised a bit more? It would make ownCloud more useful for a lot of folks working with servers, virtual machines and docker containers.
Nope, sorry. Stuff like multi-account is significantly more important. If you really need it, either help us in development or subscribe with owncloud.com.
FWIW, I needed a solution quicker than I could write it, so I paid Dropbox.
I'd like to pay Owncloud.com instead, but it's not clear that doing so would actually provide what I want, and they don't publish prices, which usually means that it's arse-rapingly expensive, and it would come with a significant time cost of re-syncing a humungous amount of data across many devices.
Would it be possible to have a way to install the command line client without the GUI client? I'm not particularly sanguine about installing a bunch of X requirements on a headless server.
I thought the repo on opensuse was the official one - that one doesn't seem to have separate packages either.
@Targren We have the client installed on a headless CentOS 6.6 server VM. No X. Credentials are in a .netrc file in the ~ dir of the appropriate local user account on the server. We have a simple bash script to run the client with a pid to make sure that the cron job that runs every minute will only be running one instance of the client at a time. Pretty simple setup.
We are syncing every 60 seconds vs every 30. That's the only practical difference I can find between what we have and what is being discussed here with a demonized process.
@scolebrook I don't have any problem with running it via cron job rather than its own daemon, I was just hoping for a cleaner packaging job. I don't know who's in charge of the OpenSUSE isv:ownCloud:desktop/owncloud-client packaging though. I was under the impression that the it was a project-run page.
@scolebrook mind to share your bash script? That sounds like a nice additions which we could put in a contrib repo if you don't mind.
@Targren you're right, the OBS repos are driven by the project. About the packaging: Yes, we could break that up, however, my priorities are elsewhere. If you can motivate yourself or somebody else to send a submit request for that change in OBS, I am fine to take it. Thanks!
Guys, I think we're not talking about the same here:
- Installing owncloudcmd without X dependencies: Currently impossible, since libowncloudsync currently has dependencies on X. As @dragotin stated, this is not a priority for us currently, but if someone feels like investing some time on the issue, we'd be happy to accept patches.
- Packaging owncloud and owncloudcmd separately: This only really makes sense until 1 is fixed, i.e. libowncloudsync no longer directly depends on X.
I don't understand why it's closed. The ticket is not about packaging owncloudcmd separately (i agree this is a different issue)
It is about having a sync daemon for synchronizing (headless) servers as we do now with desktops. This is still not available, is it ? Maybe the scrip from scolebrook is that, but it's not merged/integrated, and is kinda hacky, isn't it ? (but still useful, no offense intended @scolebrook )
Yes, this should probably not be closed.
@orzel None taken. You're right, it is kind of hacky. But it gets the job done reliably until something better comes along.
@dragotin Here's the bash script I use. I take no credit for it. It's a mash up of a couple of helpful forum posts (thank you google). As it was intended for our internal use I don't have a list of the smart people who posted the examples I plagarized for attribution. Sorry to the smart people. 😳
#!/bin/bash
SCRIPTNAME=`basename $0`
PIDFILE=/tmp/${SCRIPTNAME}.pid
if [ -f ${PIDFILE} ]; then
#verify if the process is actually still running under this pid
OLDPID=`cat ${PIDFILE}`
RESULT=`ps -ef | grep ${OLDPID} | grep ${SCRIPTNAME}`
if [ -n "${RESULT}" ]; then
echo "Script already running! Exiting"
exit 255
fi
fi
#grab pid of this process and update the pid file with it
PID=`ps -ef | grep ${SCRIPTNAME} | head -n1 | awk ' {print $2;} '`
echo ${PID} > ${PIDFILE}
/opt/qt-4.8/bin/owncloudcmd -n --trust --silent --non-interactive <path to dir to sync> https://<your server>/remote.php/webdav/<path to server side dir>
if [ -f ${PIDFILE} ]; then
rm -f ${PIDFILE}
fiWe use this to sync just a sub directory in an account. We have a local user in a CentOS VM with the credentials for that ownCloud account in a .netrc file. This script lives in that users home. Cron runs the script every minute as that user. owncloudcmd only gets run if there's no PID file so you don't have cron triggering multiple sync jobs that will conflict with each other if there's a large file to upload that takes more than a minute.
You could add multiple owncloudcmd lines to sync multiple directories, put credentials in the script instead of .netrc, whatever you like. Just remember that if you use .netrc you need to run your cron job as that user otherwise it will look for /root/.netrc. Also make sure you consistently run the script as the same user to avoid ownership issues with the synced files or with the ownCloud journal db and log files.
Hello,
We have loads of requests for this feature from our users. I know a workaround is to run owncloudcmd but this requires some clarification:
- this is so far an internal tool: if we expose this to the users we should make sure the command line options and the CLI is stable and supported long term
- the unnecessary dependencies to X libs should be removed
However, I understand that the 1.8 client is getting smarter in using filesystem notifications and it does not need to scan the whole local tree all the time. This means that the real --daemon mode would be helpful -- because running owncloudcmd in a loop periodically would not be as efficient. Is it feasible to have --daemon mode some time in the future?
kuba
I came here looking for a sync daemon too. Just to raise awarenes that this is a needed thing.
I too came here through Google while looking for a non GUI sync daemon that can run on a headless Ubuntu server and allow me to sync 1 (or a selection of) folder(s) from this Ubuntu server to my cloud hosted owncloud server.
Unfortunately I don't know enough to be able to contribute code, so I too would just like to raise awareness about the need for such feature.
+1 for a sync client without gui (as a windows service). To be used on a server where there is no user session.
+1
I'm surprised this hasn't actually been one of the first things to be implemented.
Instead of +1, you could post a bounty: https://www.bountysource.com/issues/3316729-add-a-sync-client-daemon
For reference: #3089 could be solved more easily by a backend-daemon serparation as well.
indeed, fixed
+1
As above, came looking for a sync daemon.
+1 to headless daemon as well, found this through a google search for the subject.
I see a highly practical application for owncloud in an industrial LAN environment to get programs in and out of CNC controllers without having to depend on proprietary solutions or specialized hardware.
Many of the machines are Windows based PC-CNC type solutions, but are designed to store their part files on their own hard drives. It is possible to run them over a network share, but I've had issues with network latency causing the machine to hang in the middle of a run.
Using owncloud to sync from our main fileserver to the machine hard drives would be an ideal solution, because it would give the convenience of the network share for the engineers writing the part programs while eliminating the latency issues by executing the program from the in-machine storage device.
Ideally though this solution needs a headless command line client configured to monitor the network shares that the engineers use and sync files on them with the machine controllers.
For the time being I can work around the issue using the cronjob trick, or alternately use a Windows VM on the server to just run the Windows client for this automatic syncing functionality. Doing so is less efficient than a stand alone headless daemon would be.
Dropbox has one. Crashplan has one. Bittorrent Sync has one. Syncthing has one.
It's the only reason why I do not use owncloud.
According to this you can use Xvfb to fake having a Display and such.
Install:
sudo apt-get install xvfbStart: (you need to demonize this somehow, thus the suffix &)
Xvfb :1 -screen 0 800x600x16&Start owncloud (configured graphically before via ssh -X)
export DISPLAY=":1.0"
owncloud
I yet have to find out how to configure a headless password wallet.
Notes:
maybe this or this might help here further.
I found a "dirty" solution for sync:
Install Xvfb, X11vnc and fluxbox (small and fast window manager):
sudo aptitude install xvfb x11vnc fluxboxCreate a script to launch all. I put in scripts folder into my home. The contents is:
#!/bin/bash
#Create virtual buffer at display 5
Xvfb :5 -screen 0 1024x768x16 &
#Export DISPLAY var
export DISPLAY=:5
#OPTIONAL for external control and checking
#sleep 20 seconds to start all ok
sleep 20
#launch vnc server into loopback interface
x11vnc -display :5 -bg -nopw -forever -listen 127.0.0.1 -xkb &
#launch fluxbox
startfluxbox &For a good start, its necessary than configure fluxbox to start owncloud sync client and other daemons, for this, launch fluxbox
edit startup file, I put all my startup file and comments:
#!/bin/sh
#
# fluxbox startup-script:
#
# Lines starting with a '#' are ignored.
# Change your keymap:
xmodmap "/home/pepramon/.Xmodmap"
# Applications you want to run with fluxbox.
# MAKE SURE THAT APPS THAT KEEP RUNNING HAVE AN ''&'' AT THE END.
#
# unclutter -idle 2 &
# wmnd &
# wmsmixer -w &
# idesk &
#PLACE TO PUT STARTS!!!!!!
#First start kwalletd, this promotes than owncloud client don't ask password. If you no have, install and conf without key
/usr/bin/kwalletd &
#Launch owncloud, the & it's important
/usr/bin/owncloud &
#
# Debian-local change:
# - fbautostart has been added with a quick hack to check to see if it
# exists. If it does, we'll start it up by default.
which fbautostart > /dev/null
if [ $? -eq 0 ]; then
fbautostart
fi
# And last but not least we start fluxbox.
# Because it is the last app you have to run it with ''exec'' before it.
exec fluxbox
# or if you want to keep a log:
# exec fluxbox -log "/home/pepramon/.fluxbox/log"
one time to create folder conf in you user. Folder name is .fluxbox.finally, it's necesary to start all when the server starts, I solve with crontab, just add:
@reboot /home/pepramon/scripts/fluxbox.sh
this execute the first script and launch all.
BONUS
I use vcn to configure all, you can use vinagre rdc to conect via ssh tunneling :)
An example running in other computer :-)
Great, that seems like a possible workaround.
When all we needed would only be a config file which declares the server folder <-> harddrive folder relationship and an stores the password securely.
kwalletd stores passwords secure, encrypts the password and conf file is only readable by owner.
You can use without fluxbox, just call kwalletd, wait 5 or 10 seconds and starts owncloud. In my opinion, have a minimalistic gui can help to start the service and solve problems. Other option is launch other gui services inside fluxbox (for example, I have dropbox too).
The x11vnc is optional, you can comment the line of the script and don't start and launch manually later. If you prefer, you can run x11vnc -storepasswd in console and set a password for vnc session, later just change the launch command to:
x11vnc -display :5 -bg -rfbauth ~/.vnc/passwd -forever -listen 127.0.0.1 -xkb &
password is stored encrypted to.
remember conect via ssh redirect!!!!!!!
For me, this solution works fine, but I prefer a native daemon :-(
hi @uncovery :) how do i do it with syncthing ?
@yavama try this here as a guideline: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-syncthing-to-synchronize-directories-on-ubuntu-14-04
Dropbox has one. Crashplan has one. Bittorrent Sync has one. Syncthing has one.
+1 for a headless sync daemon!
Came here from google. Now that Copy.com is shutting down, which had a complete headless sync client, I was kinda hoping I could recreate the same environment with owncloud. To my surprise, I can not.
I used to sync flat directories with renders from different software to a single Dropbox directory like this:
#!/usr/bin/env bash
echo "Inotify is watching " `pwd` " for changes..."
DIR=.
while EVENT=`inotifywait -re modify $DIR`; do
FILE=${EVENT#?*MODIFY }
echo $FILE changed.
pv "$FILE" > "/home/redsandro/Dropbox/backup/gfx/$FILE"
done
Now, this only watches for changes. It doesn't really care about new files and created directories, which would make this more complex since you cannot do one simple event regex anymore.
So I'm just leaving this here, in case someone clever wants to wrap it to the script by @scolebrook.
Ideally you will pass the subdirectory in which there were changes only, so owncloudcmd won't sync 10.000 files, but only the relevant subselection. 👼
#!/usr/bin/env bash
echo "Inotify is watching " `pwd` " for changes..."
DIR=.
while EVENT=`inotifywait -r -e modify -e create -e delete $DIR`; do
FILE=${EVENT#?*MODIFY }
echo "EVENT:" $EVENT
echo $FILE changed. # Broken if not for MODIFY event
echo ""
done
Have fun.
I was playing around with python and webdav and probably watchdog.
Will tell if I got something in a working state.
(I know it is off topic, but does anyone know a good cross platform python package for detecting directory changes? I only found watchdog)
Hi,
I've wrote a simple but effective way of "creating" a sync deamon. I've combined the owncloudcmd and supervisor to keep syncing my owncloud directory. You can read about it at my blog. Small sidenote: the instructions writing there are specifically with the use of Transip.com's Owncloud service, but can also be applied to your own environment / provider.
http://caro.ga/owncloudcustomsyncsetup
@Caroga But in your blog post, we see that owncloudcmd still requires owncloud-client, and owncloud-client requires GUI stuff. So, it's useless and is off topic as said in the first post. ownCloud is often used in a server environment without X11 & Co.
@HLFH Thanks for calling it useless 👍 !
Owncloud-client is a dependency yes, but you do not need to use it as said yourself: its an environment without X11 & Co. As I wrote, the only thing you will use is the owncloudcmd lib, with supervisor. Without any graphical interface whatsoever.
But thank you for your really positive feedback.
@Caroga You're blog won't let me comment (I got told: "You\re spam! Isn't you?") but that doesn't seem like a solution to the problem of this issue—at least not one that is practical for most installations. It's going to sit there spamming the network and bombard the ownCloud server with sync status requests as fast as it can respawn the process. That is basically a while true; do... loop and not a responsible substitute for a daemon that gets notified of changes (local & remote) and only takes action on such events.
It doesn't really matter that you're technically correct about owncloudcmd not being dependent on running X (even if the packaging is crazy, that's not uncommon); the point is we can't run it as a daemon. What we really need out of this issue is a version of the owncloud-client with the gui stuff ripped out that actually can run as a background daemon.
@alerque Now that's some constructive feedback, @HLFH could learn from this.
I understand what you are saying and am agreeing with you. This is not a replacement for the functionality as requested in the most pure form, but you could run it every 5 minutes.
For me this was a acceptable solution as my ownCloud provider wouldnt let me mount it using webdav so I needed a automated way of transferring around 500gb of data...
Side: "You\re spam! Isn't you?" will look into this, thank you for telling me.