This document describes security enhanced Linux installation. ## Security policy ### Mediawiki setup mediawiki, modrewrite, https ### Track setup ### Communication - Use jitsi for secure voip and chat - Use Adium + OTR for chat - Use Retroshare for secure P2P ### RabbitMQ etc ## Cluster simulation in Virtualbox https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1 ## Kickstart ## BIOS Security ### Admin password - Set user password for top secret machines. - Disable usb, and boot device other than disk. ### Hyper-threading - Check hyper-threading and CPUs cpumap - Disable script http://superuser.com/questions/368620/should-i-disable-hyperthreading ## BMC and IPMI - Separated management network on lan. TODO ## Naming conventions https://computing.llnl.gov/linux/genders.html - Based on SGI naming according to physical location. - Label each node. - SELinux policy is enforced/targeted<= Gender role | Internal Name | Security | Purpose login | loginN | high | user login and confinement admin | adminN | high | cluster administration manager | mgmntN | very high | BMC login low-level administration compute | rLbMnN | normal | compute node for HPC or cloud fs | rLbMnN | normal | compute node for fs cluster (ceph node) L: rack, M: blade, N: node log | logN | high | log and audit aggregator noc | nocN | high | ganglia aggregator ### Network zones - Consider security domains for ipsec and cipso. - User login allowed on login nodes. - Admin login allowed on separate mgmnt node eg. shell control box http://www.subnetmask.info/ FW Zone | A Range / 255.255.0.0 | Description ext_n | - | External network to login and manager bmc_n | 10.1.0.0 | Internal network for ipmi, GE overlap w/ sys sys_n | 10.2.0.0 | Internal network for system services, auth, log, ganglia, slurm etc. mpi_n | 10.3.0.0 | Internal network for mpi, IB (but use OFED for production) nfs_n | 10.4.0.0 | Internal network for nfs, IB or GE bootp and provision login: ext_n, bmc_n, sys_n, nfs_n admin: bmc_n, sys_n, nfs_n, mpi_n In a virtual environment eg. VirtualBox ext_n NAT or Bridged bmc_n N/A sys_n Host-only mpi_n Intnet nfs_n Intnet ## Partitions https://www.centos.org/docs/5/html/5.2/Cluster_Logical_Volume_Manager/ https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/index.html - Create partitions with lvm. - Use full disk encryption for workstations - Consider GRUB password. LVM Label | Mount point - | /boot vg_host* lv_swap | - lv_tmp | /tmp lv_log | /var/log lv_audit | /var/log/audit lv_home | /home lv_root | / lv_opt | /opt lv_install | /install * host is the internal name ### Mount Options - nodev,nosuid,noexec,xattr Mount | Options /tmp | nodev,nosuid,noexec /dev/shm | nodev,nosuid,noexec - Bind mount in /etc/fstab /tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0 ## Boot Loader options - Disable kdump - Disable usb: nosub ### Kernel Modules - Edit /etc/modprobe.d/disabled.conf install usb-storage /bin/true install cramfs /bin/true install freevxfs /bin/true install jffs2 /bin/true install hfs /bin/true install hfsplus /bin/true install squashfs /bin/true install udf /bin/true install dccp /bin/true install sctp /bin/true install rds /bin/true install tipc /bin/true (reboot) ## Script bootstrap - Append to $HOME/.bash_profile GITBASH=https://raw.github.com/hornos/config/master/centos function gitbash() { if [ -z "$*" ]; then exit 1; fi; curl -s $GITBASH/$* | bash } source $HOME/.bash_profile ## Packages www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf ### Check Red Hat PGP key rpm -q --queryformat "%{SUMMARY}\n" gpg-pubkey for i in /etc/pki/rpm-gpg/RPM-GPG-KEY-*; do gpg --quiet --with-fingerprint $i;done OR gitbash sbin/check_package_keys - List package content rpm -ql $(rpm -q PACKAGE) ### Add DVD repo http://www.cyberciti.biz/tips/redhat-centos-fedora-linux-setup-repo.html Load the DVD or attach the iso to the virtual drive. mkdir /mnt/cdrom mount /dev/cdrom /mnt/cdrom - Edit /etc/yum.repos.d/dvd.repo [DVD] name=DVD baseurl=file:///mnt/cdrom/Server enabled=1 ### Activate EPEL repo http://www.cyberciti.biz/faq/fedora-sl-centos-redhat6-enable-epel-repo/ http://www.tecmint.com/how-to-enable-epel-repository-for-rhel-centos-6-5/ #### RH 5.X wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm rpm -ivh epel-release-5-4.noarch.rpm yum install git - List package groups yum grouplist #### Centos 6 yum install wget curl -s https://raw.github.com/hornos/config/master/centos/sbin/activate_epel_repo | bash yum update yum install man man-pages mc git sudo yum install selinux-policy-strict selinux-policy-mls policycoreutils-python OR gitbash sbin/initial_update - Check GPG repo keys for i in /etc/yum.conf /etc/yum.repos.d/*; do echo -n "$i ";cat $i|grep gpgcheck|wc -l;done OR gitbash check_repo_keys - RPM diff check rpm -qVa OR gitbash sbin/check_rpm_diff ### Activate Globus repo http://www.globus.org/toolkit/docs/5.2/5.2.3/admin/install/#q-bininst #### RH 5.X wget http://www.globus.org/ftppub/gt5/5.2/5.2.3/installers/repo/Globus-5.2.stable-config.redhat-5Server-1.noarch.rpm rpm -ivh Globus-5.2.stable-config.redhat-5Server-1.noarch.rpm - Install packages yum grouplist yum groupinstall globus-data-management-client globus-data-management-server yum groupinstall globus-gram5 globus-gridftp globus-gsi yum groupinstall globus-resource-management-client globus-resource-management-server ## Initial settings http://www.server-world.info/en/note?os=CentOS_6&p=initial_conf&f=1 http://www.server-world.info/en/note?os=CentOS_6&p=initial_conf&f=8 ### Clone the config repo cd git clone git://github.com/hornos/config.git Original files should be saved with .orig suffix. - Add admin user Admin user is your last resort. Please use strong password. useradd -m -G wheel admin passwd admin semanage login -a -s unconfined_u admin restorecon -r /home/admin OR $HOME/config/centos/sbin/create_admin_user admin - Edit /etc/pam.d/su auth required pam_wheel.so use_uid - Edit /etc/sudoers.d/wheel %wheel ALL=(ALL) ALL chmod 440 /etc/sudoers.d/wheel OR yum uninstall sudo - Filter security incidents tail -f /var/log/secure | grep -e "sudo.*sudoers" - Edit /etc/aliases root: admin - Edit /etc/selinux/config SELINUX=enforcing SELINUXTYPE=targeted semanage login -m -S targeted -s "user_u" -r s0 __default__ - Translation service chkconfig mcstrans on service mcstrans start - Relabel filesystems touch /.autorelabel reboot OR gitbash sbin/reboot_to_selinux ### Initial SSHD Setup http://www.softec.lu/site/DevelopersCorner/HowToRegenerateNewSsh - Generate ssh key for admin on client. cd $HOME/.ssh ssh-keygen -f admin Copy admin.pub to .ssh/authorized_keys on admin. cd chmod -R go-rwx .ssh - Reset ssh server key. cd /etc/ssh sudo ssh-keygen -f ssh_host_rsa_key -b 2048 -N "" -t rsa sudo ssh-keygen -f ssh_host_dsa_key -b 1024 -N "" -t dsa - Distribute fingerprints in a secure channel to users. ssh-keygen -lf ssh_host_rsa_key.pub ssh-keygen -lf ssh_host_dsa_key.pub OR gitbash sbin/create_sshd_keys Normal ssh service should be used only for the wheel group and from authorized networks. User login ssh is controlled by xinetd. See the SSHD Hardening section. ### Install Additional Packages config/centos/sbin/install_packageset admin config/centos/sbin/install_packageset base ## Random number generator and entropy check http://www.howtoforge.com/helping-the-random-number-generator-to-gain-enough-entropy-with-rng-tools-debian-lenny yum install rng-tools - Edit /etc/sysconfig/rngd EXTRAOPTIONS="-r /dev/urandom" chkconfig rngd on service rngd start ### Disable ipv6 - Edit /etc/sysctl.conf net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 ### TCP Wrapper - Edit /etc/hosts.allow sshd: ALL ALL: localhost - Edit /etc/hosts.deny ALL: ALL - Check for application support: ldd /path/to/daemon | grep libwrap.so reboot ## Logging ### NTP http://www.cyberciti.biz/faq/rhel-fedora-centos-configure-ntp-client-server/ The admin nodes should be used as an internal NTP server for the cluster. yum install ntp - On admin edit /etc/ntpd.conf server pool.ntp.org restrict default noquery nomodify notrap restrict 10.2.0.0 mask 255.255.0.0 nomodify notrap - On login edit /etc/ntpd.conf server loghost server pool.ntp.org restrict default noquery nomodify notrap restrict 10.2.0.0 mask 255.255.0.0 nomodify notrap TODO: restrict interface chkconfig ntpd on service ntpd start ### tlsdate https://github.com/ioerror/tlsdate ### rsyslog Custom configuration should go to separate conf files in /etc/rsyslog.d http://www.rsyslog.com/doc/rsyslog_conf_templates.html yum install rsyslog chkconfig syslog off chkconfig rsyslog on OR gitbash sbin/change_to_rsyslog - Edit /etc/rsyslog.conf $ActionFileDefaultTemplate RSYSLOG_FileFormat ### Extra logs http://www.rsyslog.com/doc/rsyslog_conf_filter.html - Edit /etc/rsyslog.d/shorewall.conf :msg, contains, "Shorewall" /var/log/firewall - Edit /etc/rsyslog.d/avc.conf :msg, contains, "avc:" /var/log/avc.log - Edit /etc/logrotate.d/syslog /var/log/firewall /var/log/avc.log service rsyslog restart ### Host-based logging http://wiki.rsyslog.com/index.php/DailyLogRotation http://www.rsyslog.com/storing-messages-from-a-remote-system-into-a-specific-file/ http://wiki.rsyslog.com/index.php/OffPeakHours Admin machines are loghost for the cluster on sys_n network. - On admin edit /etc/rsyslog.d/loghost.conf $ModLoad imtcp.so $InputTCPServerRun 514 $template MonthlyPerHostLogs,"/var/log/loghost/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/messages.log" if $fromhost-ip startswith '10.2.' then -?MonthlyPerHostLogs & ~ *.* -?MonthlyPerHostLogs service rsyslog restart TODO: logrotate, RELP, encryption ### Send logs - On non-admin nodes edit /etc/hosts 10.2.1.1 admin loghost - Edit /etc/rsyslog.d/send.conf ### begin ### $WorkDirectory /var/lib/rsyslog $ActionQueueFileName fwloghost $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionResumeRetryCount -1 *.* @@loghost ### end ### ### logwatch http://www.stellarcore.net/logwatch/tabs/docs/HOWTO-Customize-LogWatch.html yum install logwatch logwatch --print TODO: firewall & cluster logs ### Log.io http://logio.org/ git clone git://github.com/hornos/Log.io.git cd Log.io npm install TODO: Graylog Logstash Scribe ## UPS UPS should be con]nected to admin nodes. ## Cluster management with xcat http://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_iDataPlex_Cluster_Quick_Start cd /etc/yum.repos.d wget http://xcat.sourceforge.net/yum/xcat-core/xCAT-core.repo wget http://xcat.sourceforge.net/yum/xcat-dep/rh6/x86_64/xCAT-dep.repo yum clean metadata yum install xCAT yum install net-snmp net-snmp-utils.x86_64 OR gitbash sbin/install_xcat ### RH 5.X wget http://xcat.sourceforge.net/yum/xcat-dep/rh5/x86_64/xCAT-dep.repo - Restore rsyslog.conf cd /etc mv rsyslog.conf rsyslog.xcatmod cp rsyslog.conf.XCATORG rsyslog.conf - SNMP switch config for auto-discovery - SNMP switch check snmpwalk -v 3 -u xcat -a SHA -A PASS -X cluster -l authnoPriv SWIP .1.3.6.1.2.1.2.2.1.2 - Setup mysql backend yum install mysql-server mysql mysql-bench mysql-devel mysql-connector-odbc chkconfig mysqld on service mysqld start /usr/bin/mysql_secure_installation ## Firewall - Check rules: iptables -vnL --line-numbers ### Shorewall Based on: http://www.cyberciti.biz/faq/centos-rhel-shorewall-firewall-configuration-setup-howto-tutorial/ yum install shorewall - Edit ... shorewall check chkconfig iptables off service shorewall start chkconfig shorewall on shorewall show shorewall show capabilities shorewall show macros tail -f /var/log/messages ### Test the firewall nmap -p 20-22 $FW TODO: ssh syslog ntp nfs ldap ## Cluster Boot order ### Startup 1. Manager nodes 2. Admin nodes 3. Login nodes 4. Compute nodes ### Shutdown 1. Compute nodes 2. Login nodes 3. Admin nodes 4. Manager nodes ## Groups & Users ### LDAP directory entry: DN + attributes: RDN <- objectclass defs in schema LDIF: ascii entry format yum install openldap openldap-clients openldap-servers pam_ldap nss-pam-ldapd LDAP should be used for user authentication on admin nodes. pam ldap timeout ### Central LDAP on noc ## Modules http://pkgs.repoforge.org/environment-modules/ yum install tcl.x86_64 libX11.x86_64 rpm -ivh http://pkgs.repoforge.org/environment-modules/environment-modules-3.2.8a-2.el6.rfx.x86_64.rpm The module system is based on ESZR and UNITE. https://github.com/hornos/eszr http://apps.fz-juelich.de/unite/index.php/UNITE_Introduction ### NUCE - NIIF Unified Computing Environment yum install git genders Nodeattr syntax: nuce node -q compute Comment all entry in /usr/share/Modules/init/.modulespath Edit $HOME/.bash_profile or .profile module purge module use /site/nuce/mod module load nuce/global module load etc/... source $NUCE_ROOT/etc/alias TODO: complete for modules ## SSH ### Ciphers http://blog.famzah.net/2010/06/11/openssh-ciphers-performance-benchmark/ secure: 3des-cbc normal: aes-128-ctr fast: arcfour256 ssh -c Cipher ### PAM http://www.cyberciti.biz/tips/howto-deny-allow-linux-user-group-login.html http://www.cyberciti.biz/tips/linux-unix-bsd-openssh-server-best-practices.html Edit /etc/pam.d/sshd account required pam_listfile.so onerr=succeed item=user sense=deny file=/etc/restrict/sshd.deny.user account required pam_listfile.so onerr=succeed item=group sense=deny file=/etc/restrict/sshd.deny.group account required pam_listfile.so onerr=fail item=group sense=allow file=/etc/restrict/sshd.allow.group ### TCP wrappers ### sftp chroot http://www.debian-administration.org/articles/590 ### Mosh http://mosh.mit.edu/ ## Su & Sudo ### Admin users ## Disable user password change Use unknown random passwords. ### Lock a system user usermod -L ACCT usermod -s /sbin/nologin ACCT ## PAM ### Password quality - Edit /etc/pam.d/system-auth password requisite pam_cracklib.so try_first_pass retry=3 password requisite pam_passwdqc.so min=disabled,disabled,16,12,8 ### Lockout of failed attempts - Edit /etc/pam.d/system-auth auth required pam_tally2.so deny=5 onerr=fail unlock_time=900 account required pam_tally2.so /sbin/pam_tally2 --user username --reset ### Deny a service auth requisite pam_deny.so ### Limits - Edit /etc/security/limits.conf * hard core 0 - Edit /etc/sysctl.conf fs.suid_dumpable = 0 ## Users & Groups ### libuser.conf and login.defs ### skel Unified skel ### LDAP Edit /etc/nsswitch.conf #### SLES - Self-signed ldap problem. Edit /etc/openldap/ldap.conf TLS_REQCERT allow - Edit /etc/ldap.conf tls_checkpeer no ### NFS home - Edit /etc/auto.master /net -hosts -fstype=nfs,tcp,mountproto=tcp,soft,intr,rsize=8192,wsize=8192,nolock chkconfig autofs on service autofs start mount | grep ^admin| awk '{print $1,$2,$3}' ### Umask - Edit /etc/sysconfig/init umask 027 PROMPT=no - Edit /etc/login.defs umask 007 ### Shell timeout and lock - Edit /etc/profile.d/tmout.sh TMOUT=900 readonly TMOUT export TMOUT yum install vlock ### Prelink - Edit /etc/sysconfig/prelink: PRELINKING=no /usr/sbin/prelink -ua ### Enable & Disable (nologin) ## Files - Sticky bits find PART -xdev -type d \( -perm -0002 -a ! -perm -1000 \) -print - World-writable find PART -xdev -type f -perm -0002 -print - SUID find PART -xdev \( -perm -4000 -o -perm -2000 \) -type f -print - Find and Repair Unowned Files find PART -xdev \( -nouser -o -nogroup \) -print - Verify that All World-Writable Directories Have Proper Ownership find PART -xdev -type d -perm -0002 -uid +500 -print ## Services chkconfig --list | grep '3:on' ## Auditd ## IDS: AIDE ## Apparmour ## Quota ## Monitoring ### Ganglia #### SLES - Install packages on clients zypper in ganglia-gmond ganglia-gmond-python chkconfig gmond on service gmond restart ## Slurm - Edit /etc/sysconfig/slurm ulimit -l unlimited ulimit -v unlimited ulimit -n 16384 scontrol update NodeName=... State=DOWN Reason=hung_completing ## Kernel ### SLES - Install kernel source zypper in kernel-source - Make a copy of the source and cd into it cp -vi /boot/config-$(uname -r) .config make menuconfig ## SELinux - Boot options for relabeling enforcing=0 single autorelabel - Uninstall uneccessary packages chkconfig setroubleshoot off yum erase setroubleshoot - Check for context errors in audit logs ausearch -m AVC,USER_AVC -sv no - Restore security context restorecon -v FILE - Generate TE for policy modules ausearch -m AVC -sv no -ts recent | audit2allow [-M localmodule] semodule -i localmodule.pp - Set default mapping semanage login -m -S targeted -s "user_u" -r s0 __default__ semanage fcontext -a -t type "/home(/.*)? restorecon -R -v /home ### Update Repo Setup Integrate update process with AIDE. chkconfig rhnsd off yum check-update yum update chkconfig yum-updatesd off - Run yum update from cron. - Check and uncompress a package rpm --import KEY rpm --checksig RPM rpm2cpio RPM | cpio -id ## SELinux Access Control MLS is a Mandatory Access Control (MAC) security scheme. Processes are Subjects (clearances); files, sockets and other passive operating system entities Objects (classifications). Both are labeled with Security Levels (SLs): - Sensitivity: hierarchical attribute - Categories: set of non hierarchical attribute SELinux uses Bell-LaPadula (BLP), with Type Enforcement (TE) for integrity. SELinux contexts follow the SELinux user:role:type:level syntax. Type Enforcement then separates each domain. ### Security levels https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Security-Enhanced_Linux/chap-Security-Enhanced_Linux-Introduction.html http://www.isode.com/whitepapers/security-labels-clearance.html http://james-morris.livejournal.com/5020.html http://james-morris.livejournal.com/8228.html http://james-morris.livejournal.com/5583.html https://fedoraproject.org/wiki/SELinux/FedoraMLSHowto http://www.gentoo.org/proj/en/hardened/selinux/selinux-handbook.xml DAC 1st SELinux 2nd yum install selinux-policy-mls - Edit /etc/selinux/config SELINUX=enforcing SELINUXTYPE=mls - Edit /etc/selinux/mls/setrans.conf s0 - Public s1 - Unclassified s2 - Restricted s3 - Secret s4 - Top Secret touch /.autorelabel - Check user mappings semanage login -l - Check status sestatus - Switch mode setenforce 0/1 - List booleans semanage boolean -l - Set booleans getsebool boolean-name setsebool boolean-name x Make permanent setsebool -P boolean-name on - Change file context permanently semanage fcontext -a -t type file semanage fcontext -d file - Recursive change semanage fcontext -a -t type "/dir(/.*)? restorecon -R -v /dir - Mount mount -o context="..." -o defcontext="..." - Check context match matchpathcon -V path - Tar yum install star tar --xattrs --selinux star -xattr H=exustar ### Monitoring avcstat seinfo - Number of confined processes seinfo -adomain -x | wc -l - Number of unconfined seinfo -aunconfined_domain_type -x | wc -l - Number of permissive seinfo --permissive -x | wc -l - Search sesearch --role_allow -t httpd_sys_content_t sesearch --allow | wc -l sesearch --dontaudit | wc -l - Logging grep "SELinux is preventing" /var/log/messages grep "denied" /var/log/audit/audit.log ### MLS useradd -Z user_u john semanage login --modify --seuser user_u --range s2:c100 john chcon -R -l s2:c100 /home/john ### Polyinstantiated Directories /tmp /var/tmp - Edit /etc/security/namespace.conf /tmp /tmp-inst/ level root,adm /var/tmp /var/tmp/tmp-inst/ level root,adm $HOME $HOME/$USER.inst/ level ### Network allow semanage port -a -t http_port_t -p tcp 9876 ### Aliases alias chtype="chcon -t" alias sebools="semanage boolean -l" alias selogins="semanage login -l" alias seusers="semanage user -l" alias seports="semanage port -l"