Make localScript.sh that is called by cron

Question

Make localScript.sh that is called by cron

msyriac opened this issue 8 years ago · comments

Mathew S. Madhavacheril commented 8 years ago

Mathew S. Madhavacheril · Answer 1 · Thu Jan 19 2017 08:05:07 GMT+0800 (China Standard Time)

Something weird is happening. syncScript.sh works correctly when run from Cori, but when run from my local computer like in localScript.sh , the ssh authentication doesn't go through, presumably because the key forwarding isn't working correctly. I've tried to abstract out what's happening here:
http://unix.stackexchange.com/questions/338491/rsync-ssh-agent-forwarding-through-3-remote-systems-doesnt-work-with-single-com

Any idea what's going on @mhasself @amaurea ?

In that example, local is my machine, remote1 is Cori, remote2 is the Scinet login node, and remote3 is datamover1.

amaurea · Answer 2 · Thu Jan 19 2017 10:22:15 GMT+0800 (China Standard Time)

Are nerscUserName and nerscPath files on the local computer? What is the result if you do put an echo in front of the whole ssh command? Are you running localcScript.sh directly or through cron when it doesn't work?

Mathew S. Madhavacheril · Answer 3 · Thu Jan 19 2017 10:39:59 GMT+0800 (China Standard Time)

Both nerscUserName and nerscPath are on the local computer. I've tried typing out the ssh command explicitly on the command line instead of using localScript.sh and the same error happens. I am running it locally, not yet using cron.

You can reproduce this exactly by running the example in the stackexchange question I posted, but replacing remote1 with cori.nersc.gov, remote2 with login.scinet.utoronto.ca and remote3 with datamover1.

amaurea · Answer 4 · Thu Jan 19 2017 11:16:41 GMT+0800 (China Standard Time)

I can't reproduce the problem, but I'm not using ssh agent forwarding to forward access to my key, and I think that's where your problem lies, based on your error message. Without ssh agent I can successfully run a script on cori that will rsync a file to scinet.

Mathew S. Madhavacheril · Answer 5 · Sat Jan 21 2017 00:53:23 GMT+0800 (China Standard Time)

@amaurea This has now been fixed but I don't understand how.
db37a58

I noticed that not sending the rsync process to the background fixed it. So I sent it to the background and added a wait command. This fixed it even though only one process is currently submitted (syncList.txt has only one item), so I am puzzled.

Matthew Hasselfield · Answer 6 · Sat Jan 21 2017 01:00:48 GMT+0800 (China Standard Time)

Oh cool... that must be because rsync actually needs to talk to your local computer's ssh client session, to perform authentication with your key. When you & and exit (without waiting) there's no pipe to your local computer for that authentication to take place on.

…

On 20 January 2017 at 11:53, Mathew S. Madhavacheril < ***@***.***> wrote: @amaurea <https://github.com/amaurea> This has now been fixed but I don't understand how. db37a58 <db37a58> I noticed that not sending the rsync process to the background fixed it. So I sent it to the background and added a wait command. This fixed it even though only one process is currently submitted (syncList.txt has only one item), so I am puzzled. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABAZDGOKrOgT53t5TpFrsU-L99lZ9L-cks5rUOaEgaJpZM4Lna56> .

Mathew S. Madhavacheril · Answer 7 · Sat Jan 21 2017 07:19:25 GMT+0800 (China Standard Time)

Ahh that's it. I know you're right because when I run localscript.sh, the ssh connection stays open.

What I have learnt from this should have been obvious from the start, the SSH agent doesn't move anything to remote connections, but needs to be running on the local computer and communicating with remote.

This means that the local computer has to maintain a connection for the duration of the rsync. So it's probably not a good idea to set this up on my laptop. And setting it up on Feynman would mean moving my private keys there, which I don't want to do.

Luckily, I have a home server set up and I'm sure it will be happy to knock on Cori to do these syncs every night or week or so. If I stop running it, anyone else who has an always-up system that they administer (and so would be happy to put their keys on) can set this repo up.

Mathew S. Madhavacheril · Answer 8 · Fri Apr 28 2017 06:35:35 GMT+0800 (China Standard Time)

I wrote a Python daemon which is now running on a Raspberry Pi at home. I've set it to do nightly syncs at 3am EST. Closing.