UESPWiki:Server Setup/New Servers Feb2011

Files1

• Ran setup script uesp-setup-common files1 10.2.212.14 (no issues)
• Ran setup script uesp-install-memcached 10.2.212.10 (no issues)
• Ran setup script uesp-install-lighttpd (no issues)
• Ran setup script uesp-setup-shares (no issues)
• Copy files from current files server: rsync -avz dave@files1.uesp.net:/shared/* /shared/
• Changed password for user dave
• Ran setup script uesp-update-sshd (no issues)
• Note: Make sure that /etc/init.d/nfslock is running.
• Note: Add 127.0.0.1 to NFS entries in /etc/host.allow.
• Installed Apache (for access to ab).
• Edit /etc/init.d/memcached to start service using the private IP and restart.

Content3

• Ran setup script uesp-setup-content content3 10.2.212.14 (failed to mount file shares)
• Manually mounted shares from the new files1, for example:

• mount newfiles1.uesp.net:/shared/uesp/wikiimages /mnt/test/wikiimages
• Setup script failed to mount due to using files1.uesp.net which points to the current file server.
• Added the new content3 to the entries in /etc/exports and /etc/hosts.allow on old files1.
• Re-ran setup script uesp-setup-mounts (successfully mounted the uesp shares)
• Tested shares as user apache to ensure read-write access (succeeded).
• Added entry files1 10.2.212.14 to the /etc/hosts file on the new files1 (this can be removed after the files1 sub domain points to the new server).
• Re-ran uesp-setup-mounts (no issues)

• Changed password for user dave
• Apache failed to start. Added entry for content3 in /etc/hosts and restarted.
• Ran setup script uesp-setup-zabbixserver 174.142.196.117
• Copied /etc/my.cnf from the old content3.
• Copied /etc/httpd/conf/httpd.conf from the old content3.
• Copied files from /etc/httpd/conf.d/ from the old content3.
• Copied /home/dave from the old content3.
• Copied /home/uesp from the old content3.
• Created the directory /var/log/mysql/
• Ran mysql_install_db
• Created symbolic link: ln -s /tmp/mysql.sock /var/lib/mysql/mysql.sock
• Disabled MySQL slow query log (was preventing the server from starting for some reason).
• Changed ownership in /var/lib/mysql/ to mysql:mysql.
• Started mysql: mysqld_safe --user=mysql
• Ran mysql_secure_installation
• Copied the latest full database dump from db1.
• Ran setup script uesp-update-sshd (no issues)
• Note: Make sure that /etc/init.d/nfslock is running.
• Ran command: chkconfig --level 345 netfs on
• Copied mysql tables from old content3.
• Started zabbix_server.
• Added zabbix_server to /etc/rc.local
• Created link in /home/www/ to relevant web paths.
• Changed Apache ServerName to *:80 and restarted.
• Added date.timezone to php.ini and restarted Apache.
• Changed host IPs in Zabbix

Files1 Migration

• Turned off Wiki uploads.
• Checked that all shared NFS paths on newfiles1 could be mounted by content1/2 and files1.
• Confirmed that files could be locked on newfiles1, for example:

   cd /mnt/phpsessions
   su apache
   touch test
   flock test -c ls

If NFS locking is working on the mount then the above script should immediately display the directory listing. Otherwise it will do nothing and eventually time out.

• Mounted /mnt/phpsessions on content1/2 to newfiles1:/shared/phpsessions
• Synced the phpsessions on newfiles1 from files1.
• Changed the PHP session path on content1/2 to /mnt/phpsessions and restarted Apache.
• Checked content1/2 to ensure pages were still loading correctly.
• Moved /shared/uesp to /shared/uesp-old on files1.
• Mounted UESP shares on files1 from newfiles1.
• Unmounted files1 shares on content1/2.
• Mounted newfiles1 shares on content1/2.
• Changed DNS entries of maps/skins/images/files to the newfiles1.
• Edited local hosts file to check that pages still load fine from the new server.
• Enabled Wiki uploads.
• Tested upload and file deletion.
• Restarted lighttpd on files1.

Content2

• Ran setup script uesp-setup-content content2 10.2.212.14 (didn't run uesp-setup-mounts automatically)
• Ran setup script uesp-setup-mounts (no issues)
• Manually started nfslock: /etc/init.d/nfslock start
• Tested read/write/locking of file shares (no issues)
• Changed password for user dave.
• Copy UESP content files from old content2 into /home/uesp/www.
• Create link from /home/www/uesp pointing to /home/uesp/www.
• Create link from /home/www/uesp.net pointing to /home/uesp/www.
• Install httpd: yum install httpd
• Setup chkconfig: /sbin/chkconfig --level 345 httpd on
• Re-ran setup script: uesp-install-contentapps (no issues)
• Change permissions in UESP content path: chown -R apache:uespadmin /home/uesp/www/*
• Copy/create/modify Apache configuration in /etc/httpd/ (old configuration broken apart into multiple files for easier editing).
• Check Apache config: /etc/init.d/httpd configtest (re-enable several modules until no errors listed)
• Add entry for server in /etc/hosts to ensure Apache binds to public IP address.
• Reboot server to ensure everything comes back up.
• Edit /etc/fstab and change nfs4 entries to nfs and reboot server (files1 shares still not mounted).
• Run the command chkconfig --level 345 netfs on and reboot server (files1 shares now mounted).

Content2 Migration

• Test the new content2 UESP wiki pages for read/write access.
• Changed content2 entries in Squid to point to new IP.
• Restarted Squid.
• Changed DNS entries to point to new server.

Content1

• Ran setup script uesp-setup-content content1 10.2.212.14 (no issues)
• Set passwd for dave
• Rebooted server (success)
• Test read/write/lock of mounts (no issues)
• Changed permissions for backup directory: chmod og+rx /home/backup
• Changed permissions for uesp directory: chmod og+rx /home/uesp
• Copied files from old content1 server
• Copied UESP content into /home/uesp
• Created links to /home/uesp/www/ in /home/www/
• Copied Apache configuration from the new content2
• Added entry for content1 in /etc/hosts
• Restarted httpd (no issues)
• Test web pages

• /server-status OK
• /wiki/Main_Page blank page
• Changed ownership of UESP content: chown -R apache:uespadmin /home/uesp/www/*
• /wiki/Main_Page OK
• /wiki/Special:RecentChanges OK
• Edit page OK
• Benchmark pages OK

• Copy other site configurations (eqwiki, ocp, forum, blog, dhack)
• Copy other site contents
• Update file/directory ownerships for other sites
• Restart Apache
• Test hosting of other sites on new content1

Content1 Migration

• Test the new content1 UESP wiki pages for read/write access.
• Changed content1 entries in Squid to point to new IP.
• Restarted Squid.
• Changed DNS entries to point to new server.
• Changed DNS entries for other sites to point to new server (ocp, forum, blog, dhack, eqwiki).
• Check Apache error log (no errors relevant to migration found).

Squid1

• Changed main file system from ext3 to ext2.
• Added noatime option to main file system.
• Ran setup script uesp-setup-common squid1 10.2.212.14 (no issues)
• Set password for dave
• Installed Squid: yum install squid
• Changed Squid initialization levels: /sbin/chkconfig --level 345 squid on
• Copied Squid configuration from oldsquid1
• Added local host name to /etc/hosts
• Modified configuration for new Squid server

• Changed content1/2 host IPs
• Changed squid1 IP
• Added visible_hostname squid1.uesp.net
• Added cache_swap_log /var/log/squid/cache_swap_log
• Modified cache_dir coss /var/spool/squid 50000 max-size=1000000 maxfullbufs=4 membufs=20 block-size=4096
• Created swap directories: squid -z

• Reboot system

• Squid not started. Starting manually...failed.
• System log message: Cannot open HTTP port
• Modified http_port line to remove specific IP address: http_port 80 vhost defaultsite=www.uesp.net
• Squid started successfully
• Disabled Squid logging
• Restarted Squid

• Installed Apache: yum install httpd (for access to ab)
• Test Squid cache with ab (no issues)
• Installed PHP (for Zabbix custom scripts)

Squid1 Migration

• Change DNS entries for uesp.net and www.uesp.net to new squid1.
• Added new squid1 IP to content1/2/3 wiki configuration.

Db1

• Ran setup script uesp-setup-common db1 10.2.212.14 (no issues)
• Added entry for host in /etc/hosts
• Ran setup script uesp-setup-db (no issues)
• Reverted PHP session path to default (set to mounted share by previous step)
• Ran MySQL install script mysql_install_db
• Started MySQL /etc/init.d/mysqld start (failed)
• Changed ownership of MySQL directory: chown -R mysql:mysql /var/lib/mysql
• Started MySQL /etc/init.d/mysqld start (no issues)
• Ran script mysql_secure_installation (no issues)
• Set password for dave
• Rebooted server (no issues)
• Copied and restored database dumps from old db1.
• Changed server-id to 142 and enabled bin-log. Restarted mysqld.
• Setup and started slave replication on new db1.

Db1 Migration

• Checked new db1 replication to ensure no errors.
• Locked Wiki on content3.
• Changed Wiki database host on content3 and tested page loads.
• Locked Wiki on content1/2 and forums.
• Stopped slave on new db1.
• Tested page loads on content3.
• Made test edit on content3.
• Changed Wiki database host on content1/2 and tested page loads.
• Change forum database host, re-enabled forum, and made test post.
• Unlocked content1/2 and made test edits.
• Changed database host on all secondary services: blog, DaveWiki, EQWiki, simap, obmap, mwmap
• Shutdown MySQL on old db1.
• Tested all services.
• Changed db1 DNS entry.