Giters

mitro-co / mitro

Repository for all Mitro client & server code

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Building/running a server on Linux

wilsonk opened this issue · comments

commented

@evanj @vijayp

Hello everyone. I have managed to build and run mitro-core on Scientific Linux 6.2 (yes that is old...but any newer release should work also), and access it from a separate machine on my internal network. This has some overlap with #53 but I thought I should file it separately. Here are my notes on the process:

For Scientific Linux (or CentOs or any Redhat Enterprise Linux derivative)

git clone https://github.com/mitro-co/mitro

sudo yum install nodejs.x86_64; yum install postgresql.x86_64; yum install postgresql-server; yum install postgresql-contrib

Install java sdk from website as described in mitro-core/README.md
use the jdk-7u67-linux-x64.rpm

sudo rpm -Uvh jdk-7u67-linux-x64.rpm
sudo yum install npm.noarch

Remember to
export NODE_PATH=/<path-to-mitro>/mitro/browser-ext/api/build/node/lib/node_modules
in your .bashrc or .profile

Run sysctl commands as per mitro-core/README, if you want to run multiple postgres instances
Run build.sh as per mitro-core/README and then the other commands to set up a postgresql db.

If you get an error with:
psql -c 'create database mitro;'
like >>> FATAL: database "<user_name> does not exist'
then run
createdb <user_name> -U <user_name> followed by psql -c 'create database mitro;' again
Once that is finished, then build the db:
postgres -d build/postgres
(you may want an & after that to make it run in the background as this process won't return)

And the run the server:
ant server
in the mitro-core directory. Things should be ready to go (you might also want an & here to run in the background)

If you have a problem with 'ant server' indicating 'invalid source release' then you need to install java-1.7.0-openjdk.x86_64 and java-1.7.0-openjdk-devel.x86_64 via yum because the rpm didn't install properly earlier

Test
Connect to: https://localhost:8443/mitro-core/api/BuildMetadata with a browser and you should see

"commit: ... some commit hash
describe: fatal: No names found, cannot describe anything.
build time: ... some date"

Now build the extension and run tests:
cd browser-ext/api
./build.sh

cd js/cli
./runtests.sh FAST && echo "SUCCESS"

To test out the extension in a browser:

Build the 'safari/firefox/chrome'-debug extension in browser-ext/login with 'make safari-debug' (or firefox/chrome)

If you run into problems with 'nopt' not being found then you need to
npm install -g nopt

If that still doesn't work then you can run
npm install nopt
in the browser-ext/third_party/hogan.js/bin directory

Install the extension into the browser (I chose safari...for a chrome install, see the browser-ext/README)

If you don't have a safari extension developer certificate then you will need to sign up for one

Turn on Safari Developer Tools under 'Safari->Preferences'
Under 'Develop' in the tool bar you choose 'Show Extension Builder'
Click on the little '+' to add the safari-debug extension
Navigate to browser-ext/login/build/safari/ and choose the debug.safariextension directory
Click 'install' and the extension should be ready to use
Follow the regular "Sign Up" workflow to add a new user to the postgres db

NOTE: You won't get an email sent to your login email account because emailer2.py is still missing from ansible
NOTE: if you want to use a different machine on your network to test the extension, rather than the mitro server machine, then edit the browser-ext/login/common/config/cofig.debug.js file and change the two 'localhost' entries to an ip address like '192.168.1.66', or wherever the server is running
commented

This is great. Can you modify the readme with these details? Thanks!
On 1 Sep 2014 12:19, "wilsonk" notifications@github.com wrote:

@evanj https://github.com/evanj @vijayp https://github.com/vijayp
Hello everyone. I have managed to build and run mitro-core on Scientific
Linux 6.2 (yes that is old...but any newer release should work also), and
access it from a separate machine on my internal network. This has some
overlap with #53 #53 but I
thought I should file it separately. Here are my notes on the process: For
Scientific Linux (or CentOs or any Redhat Enterprise Linux derivative)

git clone https://github.com/mitro-co/mitro

sudo yum install nodejs.x86_64;
yum install postgresql.x86_64;
yum install postgresql-server;
yum install postgresql-contrib

Install java sdk from website as described in mitro-core/README.md
use the jdk-7u67-linux-x64.rpm

sudo rpm -Uvh jdk-7u67-linux-x64.rpm
sudo yum install npm.noarch

Remember to
export
NODE_PATH=//mitro/browser-ext/api/build/node/lib/node_modules
in your .bashrc or .profile

Run sysctl commands as per mitro-core/README, if you want to run multiple
postgres instances
Run build.sh as per mitro-core/README and then the other commands to set
up a postgresql db.

If you get an error with:
psql -c 'create database mitro;'
like >>> FATAL: database "<user_name> does not exist'
then run
createdb <user_name> -U <user_name> followed by psql -c 'create database
mitro;' again
Once that is finished, then build the db:
postgres -d build/postgres

(you may want an & after that to make it run in the background as this
process won't return)

And the run the server:
ant server
in the mitro-core directory. Things should be ready to go (you might
also want an & here to run in the background)

If you have a problem with 'ant server' indicating 'invalid source
release' then you need to install java-1.7.0-openjdk.x86_64 and
java-1.7.0-openjdk-devel.x86_64 via yum because the rpm didn't install
properly earlier

Test
Connect to: https://localhost:8443/mitro-core/api/BuildMetadata with a
browser and you should see

"commit: ... some commit hash
describe: fatal: No names found, cannot describe anything.
build time: ... some date"

Now build the extension and run tests:
cd browser-ext/api
./build.sh

cd js/cli
./runtests.sh FAST && echo "SUCCESS"
To test out the extension in a browser:

Build the 'safari/firefox/chrome'-debug extension in browser-ext/login
with 'make safari-debug' (or firefox/chrome)

If you run into problems with 'nopt' not being found then you need to
npm install -g nopt

If that still doesn't work then you can run
npm install nopt
in the browser-ext/third_party/hogan.js/bin directory
Install the extension into the browser (I chose safari...for a chrome
install, see the browser-ext/README)

If you don't have a safari extension developer certificate then you will
need to sign up for one

Turn on Safari Developer Tools under 'Safari->Preferences'
Under 'Develop' in the tool bar you choose 'Show Extension Builder'
Click on the little '+' to add the safari-debug extension
Navigate to browser-ext/login/build/safari/ and choose the
debug.safariextension directory
Click 'install' and the extension should be ready to use
Follow the regular "Sign Up" workflow to add a new user to the postgres db

NOTE: You won't get an email sent to your login email account because
emailer2.py is still missing from ansible NOTE: if you want to use a
different machine on your network to test the extension, rather than the
mitro server machine, then edit the
browser-ext/login/common/config/cofig.debug.js file and change the two
'localhost' entries to an ip address like '192.168.1.66', or wherever the
server is running

—
Reply to this email directly or view it on GitHub
#56.

commented

I'm trying to get it to run on Debian 7 atm. Waiting for #35 to get the mail issue done. Once everything is running I'll post a howto.

commented

okay on EC2/Amazon Linux AMI/ 2014.03.2...

I had to update the Python from 2.6.something to 2.7.5 to get the browser-exts to work - that was a bit tricky. BUT then I had to install the pip and virtualenv for 2.7.5 and then the "mitro/emailer" ./build.sh script worked just now.

Of course, glancing at the sources... it looks like there is a "localhost" and a port 8125 that needs to be open. I wonder if there are MANDRILL environment variables someplace or settings required to make it work.

commented

So we kind of got the mail stuff working now. I'm going to write everything down and add the changes to my fork so you can take a look at it.

@vijayp I found traces of mandril, local smtp and aws ses for sending mail. Can you tell me which one you actually used?

commented

And once more - documentation is up at https://www.hashtagsecurity.com/mitro-login-manager-on-premise-2/

I'll try to publish our findings on the mailer as soon as I can get to it.

Btw, should we keep this as an issue or document our findings in the wiki?

commented

So I am at this point at well. I get 404 from jetty when I try to hit the root url, and I get the

commit: b3b9895b4d73ae36f14e7227eb0a3de6e33909de
describe: fatal: No names found, cannot describe anything.
build time: 2014-09-22T18:31:20Z

when I touch the BuildMetadata page

commented

also, when I try to run the emailer2 I get

$ build/venv/bin/python emailer2.py --enable_email --mandrill_api_key=S0methingRealHere123
Traceback (most recent call last):
  File "emailer2.py", line 33, in <module>
    from auth import statsd
ImportError: cannot import name statsd

hmmm.

commented

@kristofer The emailer appears to be terribly broken. I've gotten it up and running, but there are emails being placed in the email table it doesn't know how to handle. It's as if they didn't release the correct version of the emailer.

commented

I'm currently working on that - try this for now https://github.com/fredericmohr/mitro/tree/master/mitro-mail

Once I'm happy with everything, I'll make a pull request to mitro.

Edit: Mandrill and AWS SES not yet supported, but it should run with SMTP (at least it did on my debian box) - you can set your SMTP server settings in mitro/mitro.cfg :)

commented

I decided to just rewrite the emailer from (mostly) scratch. https://github.com/croemmich/mitro-emailer
I'm still missing a number of templates, but I'm not actually sure what they are supposed to look like, as they were missing from the official repo.

commented

@croemmich Nice, I should have waited a few more hours with mine... yours is much better :)

Can you make a pull request to mitro, so we can continue to use the official repository?

BTW, you can get the mails by creating a mitro.co account and trigger the actions that send the mails. Then you can recreate the templates based on the mails.

commented

@fredericmohr Since this a rewrite, it would make more sense to keep it out of the mitro-co/mitro repo. Not to mention, storing everything in one repo is just annoying. It'd be nice to get it in to something like mitro-co/mitro-emailer however.

commented

Ha, I actually like having this in one repository. It means you only ever need to grab one chunk of code, instead of "please check out these six repositories".

That said: If you would like, I'm totally happy to create a new repo, and add you as a collaborator?

commented

@evanj: I think I'd prefer a separate repo. I'll also help to separate issues.

commented

I would prefer having different repositories for server side and client, as I think it's unnecessary to checkout all the code if you only need the server stuff on your server and the client stuff on your dev or build machine. I'm not sure about splitting the mailer from the mitro-core part... as those are both needed to run mitro server.

commented

Typically if you are working on the extension, you want a local dev server to test your changes, but I don't want to have a religious debate about it.

I just created https://github.com/mitro/emailer with croemmich as a collaborator. Should I add some other people to this?

This reminds me: we should move our last private repo out of this account so I can convert it to an open source organization ...

commented

For future reference, the new mailer can be found here: https://github.com/mitro-co/emailer

Great work!

commented

Hi Wilson, I've followed the above as best I can on a clean RedHat 6.5 server but the "ant test" never completes. It runs fine until "[junit] Testsuite: co.mitro.core.servlets.MitroServletTest" but that never finishes.

I'm using nodesource.com as the repo for nodejs and I'm using ant 1.9.6 downloaded directly from Apache. I tried using "ant" from the RedHat repo but that pulls in older versions of java packages which then override the jdk rpm installed from Oracle.

Do you have any idea what might be wrong or is there more information that you need which might help diagnose the problem?

Thanks in advance...

commented

Hello rogerwlucas,

I just ran 'ant test' here again (I seem to remember that it didn't complete when I wrote this up last year, but I am not totally sure). It has been running for 10 minutes without finishing (still a little activity with registering with 'top', but not much, so I just terminated it), so I will say that it doesn't appear to work for me either.

That seems to be the only test that doesn't complete, because I get "SUCCESS" if I just remove that one test for the mitro-core and mitro-core-android tests.

It looks like this might be a database problem? Rights issue when inserting? Not sure. Maybe someone else can chime in, because I am not really sure what the problem is.

If it is critical that this test passes for you, then please reply back and I may be able to look into it some more.

Thanks,
Kelly

P.S. Things seem to hang here (just so there is a more complete error message to search on for others):

[junit] Testsuite: co.mitro.core.servlets.MitroServletTest
[junit] Expected: a string containing "Please retry"
[junit]      but: was null)
[junit] Expected: a string containing "read-only transaction"
[junit]      but: was "Unable to run insert stmt on object userid:10; name:readonly@example.com: INSERT INTO "identity" ("id" ,"verification_uid" ,"analytics_id" ,"verified" ,"name" ,"publicKeyString" ,"encryptedPrivateKeyString" ,"keyserverSignatureOfIdAndKey" ,"force_password_change" ,"twoFactorSecret" ,"backup1" ,"backup2" ,"backup3" ,"backup4" ,"backup5" ,"backup6" ,"backup7" ,"backup8" ,"backup9" ,"backup10" ,"lastAuthMs" ,"numAvailableBackups" ,"enabledTFAMs" ,"referrer" ,"guid_cookie" ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)")
commented

Hi Wilsonk,
Thanks for the quick reply. I've tried setting Mitro up on Ubuntu 14.04.2 LTS as well as on RedHat 6.5. I had more success on Ubuntu as the "ant test" in the mitro-core ran without error but I still got errors when running the tests in the browser-ext/api/js/cli. On Ubuntu, I can build and run Chrome extension but still cannot sign in to the Mitro server - I see an initial connection and then nothing more.

On RedHat, I don't even get that far because the Chrome Extension crashes with an error on startup.

I suspect that I've got something fundamental wrong with my config on both systems but either the build+test isn't generating a warning/error or I'm missing it.

This was why I was questioning whether the tests ran on your install... if they did then that would give me a clue. If they didn't, but you could still connect the Chrome Extension to the server, sign in, manage passwords, etc then it's even more weird.

I'll keep digging as I would really like to get Mitro working...

Thanks again,
Roger

commented

Hi Wilsonk,
We've done more digging here. The problem with the "ant test" not running in mitro-core was a PostgreSQL version issue. RedHat 6 comes with PG 8.x but Java was using drivers for PG 9.2. We upgraded PG to 9.2 using the PostgreSQL official RPMs and the tests now pass reliably.

After more playing with Chrome, "npm install ...", etc we also have the browser-ext cli tests passing too.

The final problem is that we cannot sign up new users. We have installed the debug version of Mitro into Chrome and have seen a connection to the local ant server "GET /mitro-core/ServerRejects?deviceId=/BYMlIJkvuUwGhgN4vESf29B/To=" so it's obviously doing something as soon as the Mitro extension starts.

But, when we go through the "Sign Up" process in the Mitro Chrome extension, we enter username + password then click "Sign Up". We get the spinner but it never completes.

We don't see any more messages on the "ant server" console either so I suspect that the signup request is not happening for some reason.

Any ideas?

Thanks,

Roger

commented

Hello Roger,

Sorry but I haven't really worked on mitro in a while (and don't have a currently running system), so I can't remember if I ran into this problem before. I just remember that the emailer wasn't available when I set up mitro the first time, so I had massage things around that when signing up.

I might look at setting up a new install here, at some point soon, so if you are still having problems then let me know and I might be able to look into this.

Thanks,
Kelly

commented

In case these are of use to anyone reading this thread, below are the commands that I used to install Mitro on a RedHat 6.6 server. They assume that the user is not root and that it can "sudo".

Install Java 7 from Oracle

Download JDK from Oracle - version 7u67 has been tested and works. The latest version 8u51 does not work so you will need to register to get access to the earlier v7 releases. You need the appropriate 32- or 64-bit RPM for your system...

Once downloaded, install it...

Log in as "root"

rpm -i /home/labuser/jdk-7u67-linux-x64.rpm

Install PosgreSQL v9.2 (not 8.x or any other version)

wget http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/pgdg-redhat92-9.2-7.noarch.rpm
rpm -i pgdg-redhat92-9.2-7.noarch.rpm
yum install -y postgresql92 postgresql92-contrib postgresql92-odbc postgresql92-server postgresql92-devel

ln -s /usr/pgsql-9.2/bin/initdb /usr/bin/
ln -s /usr/pgsql-9.2/bin/postgres /usr/bin/
ln -s /usr/pgsql-9.2/bin/pg_ctl /usr/bin/
ln -s /usr/pgsql-9.2/bin/pg_config /usr/bin/

Install the latest version of Java Ant

wget https://www.apache.org/dist/ant/binaries/apache-ant-1.9.6-bin.tar.gz
tar -zxf apache-ant-1.9.6-bin.tar.gz -C /usr/
ln -s /usr/apache-ant-1.9.6/bin/ant /usr/bin/ant

Install node.js and npm

curl --silent --location https://rpm.nodesource.com/setup | bash -
yum install -y nodejs

Install other tools that you will need to run the test suites

yum install -y git mlocate python-setuptools bind-utils

easy_install argparse mandrill sqlalchemy statsd tornado virtualenv

Install development tools used by some regression tests

yum install -y python-devel
yum groupinstall -y "Development Tools"

Update "locate" database

updatedb

Add "mitro" user and switch to this user

useradd mitro
su - mitro

Get the latest Mitro code

git clone https://github.com/mitro-co/mitro
cd mitro/mitro-core/
ant test

This should pass, it takes ~25 seconds on Xeon E5-2643 @ 3.3 GHz

cd mitro/browser-ext/api
./build.sh

echo "export NODE_PATH=/home/mitro/mitro/browser-ext/api/build/node/lib/node_modules" >> ~/.bashrc
export NODE_PATH=/home/mitro/mitro/browser-ext/api/build/node/lib/node_modules

cd js/cli
./runtests.sh

The early tests should pass but then fail to start Chrome because it is an SSH connection (assuming you are connecting over SSH). It may work if you are connected via a X terminal session.

cd ../../../login

Set your server FQDN and port in the client browser configuration file

Edit common/config/config.release.js
Change MITRO_HOST, MITRO_AGENT_HOST and FAILOVER_MITRO_HOST to point to your server

Build release versions for Firefox and Chrome

make firefox release

Start the PostgreSQL server

cd ~/mitro/mitro-core
./build.sh
postgres -D pwd/build/postgres

Complete the PostgreSQL configuration

Open a new session as "mitro" user

cd ~/mitro/mitro-core
psql -c 'create database mitro;' postgres

Generate local static keys for Mitro instance

If you don't do this, you will get new keys generated each time Mitro starts. This will reset all your client accounts.

mkdir -p mitrocore_secrets/sign_keyczar
java -cp build/mitrocore.jar org.keyczar.KeyczarTool create --location=mitrocore_secrets/sign_keyczar --purpose=sign
java -cp build/mitrocore.jar org.keyczar.KeyczarTool addkey --location=mitrocore_secrets/sign_keyczar --status=primary

Create the new server self-signed certificate

You will want to do this so that you don't get connection errors every time your clients connect. You can then load this certificate into your certificate store as as trusted certificate.

openssl genrsa -des3 -out server.key 2048
openssl req -new -sha256 -key server.key -out server.csr
openssl x509 -req -days 3650 -in server.csr -signkey server.key -out server.crt
openssl pkcs12 -export -inkey server.key -in server.crt -name mitro_server -out server.p12
/usr/java/jdk1.7.0_67/bin/keytool -importkeystore -srckeystore server.p12 -srcstoretype pkcs12 -srcalias mitro_server -destkeystore server.jks -deststoretype jks -deststorepass password -destalias jetty

cp server.jks build/java/src/co/mitro/core/server/debug_keystore.jks
cp server.jks java/server/src/co/mitro/core/server/debug_keystore.jks

ant server

Set up the system to auto-run at boot

cat <> /etc/rc.local

######## Start PostgreSQL as mitro
echo -n "Starting PostgreSQL Mitro server..."
su --login mitro -c "postgres -D mitro/mitro-core/build/postgres" &> /var/log/mitro-postgres.log &
echo "OK (logging to /var/log/mitro-postgres.log)"

######## Start Mitro server
echo -n "Starting Mitro core server..."
su --login mitro -c "cd mitro/mitro-core && ant server" &> /var/log/mitro-core.log &
echo "OK (logging to /var/log/mitro-core.log)"

EOF

commented

I have dockerized mitro mainly following this guidelines, so if somebody finds it useful you can found it at http://github.com/servomac/mitro

Patches and issues are welcomed (be carefull about the self-signed certs, i'm waiting a little to add a simple call to a letencrypt client container or something); this is still a work in progress.