UESPWiki:Upgrade History/2009

This is an archive of past UESPWiki:Upgrade History discussions. Do not edit the contents of this page, except for maintenance such as updating links.

13 December 2009

• Set Apache on content1 to restart automatically once per day.

11 December 2009

• Changed $wgAutoConfirmCount from 0 to 10 and $wgAutoConfirmAge from 3 to 4 days on content1 and 2.
• Set $wgRateLimits to (20 edits, 360 sec) on content 1 and 2.

14 October 2009

• Forced a restart of content1 by toggling the remote power control off/on. Content1 was unreachable and the webserver on content2 was stalled (probably due to inaccessible memcache server on content1). Content1 successfully restarted. No obvious cause and nothing indicating failure in the logs. Total downtime of about 3 hours.
• Remounted /tmp/phpsessions and /home2/wikiimages on content2.

6 October 2009

• Created a symbolic link /home2/wikiimages/Somerights.png to /home2/wikiimages/images.new/4/4d/Somerights.png so the image displays properly at the bottom left. Should probably update LocalSettings.php at some point as well and delete the symbolic link.

4 October 2009

• Restarted Apache on content1/content2 due to stalled web server (downtime of about 3 hours). Logs show multiple Maximum execution time of 30 seconds exceeded in /home/www/uesp.net/w/includes/diff/DifferenceEngine.php on line 1973 messages probably from this 220kb edit.
• Restarted Apache again at 11:00...same issue.
• Installed Wikidiff2 on content1/2 which should solve this sort of issue.

30 September 2009

• Increased number of open files allowed on db1 from 1024 to 8192 (ulimit -n 8192). Also set the soft/hard limits for mysql in /etc/security/limits.conf to 4096.
• Changed table_cache in my.cnf from 4096 to 3000 (not restarted).

29 September 2009

• Canceled the old content1 server. Effective 17 Oct 2009.

21 September 2009

• Changed the column types of all the "obloc_" columns in the map tables from float to int. Original tables copied to mapdata_old. There is no need to have the accuracy of float coordinates in the maps and it potentially interferes with caching of results in the query and squid caches (although likely a minor performance increase).
• Edited the common maps.js file to convert coordinates being passed to getmaplocs to integers.

16 September 2009

• Fixed updating of the backup1 IP on the new content1 server. File /home/uespkey/backup1-ip is updated from backup1 hourly as well as the /etc/hosts file on content1.
• Setup updating of backup1 on content2/3 as well.
• Can ssh into backup1 but not mysql (can't connect error message). Cannot telnet backup1 3306 either (no route to host). Since backup1 is on a home DSL line they may be blocking port 3306.
• Installed a subversion server on content3. Loaded initial version of UespCustomCode to test.

15 September 2009

• Moved file cache to /home2/wikiimages/cache so file cache is now being shared by content1 and content2 (to hopefully fix problem with out-of-date page versions being shown because file cache is out-of-date but server doesn't know it)
• Refreshed file cache for all pages added to cache since last refresh (Aug 13), to clear out any possible out-of-date files in cache

13 September 2009

• Setup the DungeonHack website on content1 for forwarding purposes.
• Added nephele to 'uesp' group on content1
• Changed group ownership for uespwiki files and directories to 'uesp' on content1 (instead of 'uespadmin')
• Fixed missing quote in SiteSpecialPreferences.php on content1 (now fixed on all servers)
• Fixed database replication errors on content3/backup1 (reloaded last full db backup).

12 September 2009

• Added content1.uesp.net as a ServerAlias to the UESP wiki on content1. This stops the EQWiki from being the default site.
• db1 MySQL thread stats after the thread cache change:

• Uptime = 4 days
• Threads_cache_hitrate_% = 99.61% (was ~98%)
• Threads_created = 15k (was ~100k)
• Threads_cached = 59 (was 16)
• Number of processes (ps ax) = 144
• Number of MySQL processes = 78

• Minor changes to Special:UespSiteStats

• Added ifconfig stats
• Changed to use uespkey user
• Setup proper keys on content1/content2/content3

• Updated Rpeh's wiki user rights and cleared his passwords on all servers.
• Fixed error in log rotation scripts on backup1
• Fixed a missing quote in SiteSpecialPreferences.php on content2
• Made the Apache config directories on content1/2/3 writable by the uespadmin group.
• Added the ocp.uesp.net subdomain.
• Added iptables firewall for memcached on content1. Currently only allows content1/2/3. Rules restored on startup from /etc/uesp-memcached-iptables.

9 September 2009

• Changed MySQL thread_cache_size from 16 to 64.

8 September 2009

• Restarted db1 because mysql server had run out of threads. Problem appears to be caused by forum tables getting locked and the lock is never released, but diagnosis is difficult given that root can't even get into the server to see status details when there are no threads available.
• Added ALTER, CREATE, and DROP privileges for forum user 'uesp'@72.55.146.206 (privileges had previously been added for wrong user; phpbb3 requires these privileges for a few obscure operations). Could this be related to above problem with table locks? (phpbb3 locks table before doing an ALTER, then gets stuck when ALTER command fails?)
• Restarted the process of adding all old posts to the forum's search index (originally started immediately after upgrade, but I didn't realize process had died before completing).
• content1 switched to new server (see UESPWiki:Administrator Noticeboard#Switching to New Content1 Server)

• Recursively changed group permission for phpbb3 directory to apache so forums can function again
• tidy was not installed
• Wrong version of php was being used. Has to be the /virtual/php5s directory in order for eAccelerator and /tmp/phpsessions to be used.
• Fixed issue on content2 with LocalSettings.php (missing 11000 for the memcached server).
• Fixed file/directory permissions in the wiki upload directory.

• Notes about memcached on the old content1:

• Uptime = 32 days
• Bytes = 238MB
• Cmd_Get = 68 million
• Bytes_Read = 76GB
• Bytes_Written = 1650GB
• Hit Rate = 94%
• Total_Malloced = 288MB

5 September 2009

• Upgrade of Special:UespSiteStats extension:

• Uses AJAX calls to load faster and prevent a timeout from preventing the page from displaying.
• Moved statistic configuration into UespSiteStats_conf.php.

2 September 2009

• Fixed a few overly permissive database user permissions (i.e., allowing write access to all dbs).

1 September 2009

• Ordered content1 replacement server.

31 August 2009

• Uploaded new version of MetaTemplate extension to fix issue with #include function
• Uploaded new version of ProtectSection to match Mediawiki version
• Uploaded new versions of many extension files (UespCustomCode, ActiveUsers, UespMap, UespPaypal, ReCaptcha, Icon) with minor changes (in particular, remove blank lines that are problematic in XML)
• Disabled display of all Evony ads.

30 August 2009

• Changed backup1 database backups from individual databases to a single snapshot.
• Checked and setup database backups according to the updated Backup Plan.
• Setup copying of master database backups to content3.
• Database backup on db1 has changed from a daily full snapshot to:

• Daily backup of all databases minus the uesp_net_wiki5.text table.
• Daily backup of the uesp_net_wiki5.text table starting at old_id 443300.
• Weekly full snapshots.
• This improves the backup time from +5 minutes each day where the entire database is locked to <1 minute with each individual database locked as its being backed up.

• Changed log rotation of backups on db1/backup1 from specific files to *.sql.gz.
• Offset crontab schedule slightly on content3/backup1 so as to run after db1 cron scripts.
• Databases set to be backed up individually on backup1.

29 August 2009

• Fixed replication slave error on content3/backup1.

26 August 2009

• Set wgCookieDomain to .uesp.net on content1/content2.

24 August 2009

• Modified MimeMagic.php on content1/2: added a line of code to reduce the chance that binary files are falsely detected as PHP.
• Added 500G hard drive to backup1.
• Changed how db replication is happening on db1, content3, and backup1:

• Removed all binlog-do-db options from db1 (all databases logged).
• Removed all replicate-do-db options from content3/backup1.
• Added replicate-wild-ignore-table=mysql.% to content3/backup1 (all databases except mysql will be mirrored).
• Basic tests seem to indicate this is working as desired.

23 August 2009

• More work on backups:

• The db1 mysql database is backed up seperately on db1, content3 and backup1. Note that content3 and backup1 actually backup the db1:mysql database and not their own local copy.
• UESP content directory on content1 (/home/www/uesp.net/) is backed up weekly onto content3 and backup1.
• Fixed weekly database backup rotations on db1 (maybe).
• Setup backups of database mirror on content3 with standard daily/weekly rotation. Complete backup takes 16m (including database dump and compression).

21 August 2009

• Enabled mod_gzip on content1/content2.
• Added expires headers for favicon.ico and /w/extensions/UespCustomCode/files/<images>.

18 August 2009

• Manually removed offensive entries from the wiki's recentchanges and logging tables; manually modified offensive account names in user table. Deleted information saved in wiki.del_logs.sql.
• Set $wgAccountCreationThrottle to 1 on content1/2

17 August 2009

• Added Google analytics code to forums (reset cache) and OB/MW/SI map pages on content1/content2.
• Fixed permission problem with the uespcounter database user.

14 August 2009

• Re-rebuilding file cache on content1/content2 (using my own clear_cache code, that calls purge/view via API) to:
  1. Remove entries for redirects from file cache (interestingly, only redirects seem to be root-created when using rebuildFileCache?)
  2. Rebuild non-content pages

13 August 2009

• Purged the memcache cache on content1 in an attempt to propagate Google Analytics code update to all pages. Stats before the purge were:

• 131MB in use
• 94% hit rate
• Uptime 504700s

• Rebuilding the file cache (maintenance/rebuildFileCache.php 0 overwrite) on content1/content2.

12 August 2009

• After 1 day the new squid1 server is averaging a 76.5% cache hit rate with a memory usage of 1300MB.
• Fixed issues with Special:UespSiteStats extension on content2.

11 August 2009

• Modified /includes/OutputPage.php (lines 754 and 763) on content1/content2, changing mSquidMaxage to wgSquidMaxage, to fix issue with Squid caching (maybe).
• Setup and tested new squid1 server.
• Switched www.uesp.net and .uesp.net to point to the new squid1 server. Old squid1 averaged a 66% cache hit rate.
• Installed the special page UespSiteStats extension on content1/content2. Currently restricted to sysops and not working correctly on content2.

10 August 2009

• Fixed Google Analytics code on content1/content2.

9 August 2009

• Memcache stats on content1 after 2 days of uptime:

• Memory Used = 78MB
• Hit Rate = 94%

• MySQL stats:

• Uptime = 1 day, 22 hours
• Total Queries = 20,699,227
• free reported 754Mb RAM used
• Handler_read_rnd = 10M
• Handler_read_rnd_next = 2644M
• Qcache_lowmem_prunes = 322k
• Qcache_hits = 144k
• Qcache_not_cached = 5469k
• Qcache_queries_in_cache = 36k
• Qcache_free_memory = 222M
• Slow_launch_threads = 0
• Threads_cached = 15
• Threads_cache_hitrate_% = 99.58%
• Created_tmp_disk_tables = 26k
• Key_reads = 275k
• Key_writes = 582k
• Select_full_join = 6
• Select_range_check = 0
• Opened_tables = 1231
• Table_locks_waited = 163k

• Re-enabled Google Analytics on content1/content2 at least temporarily. Currently the code exists in /includes/Skin.php but should be moved to the UespCustomCode extension at some point.
• Fixed a permission issue with the uespcounter MySQL user.
• Started slave mirroring of db1 on content3.

8 August 2009

• Blocked www.uesp.net and forums.uesp.net from being added to bugmenot.

7 August 2009

• Tweaked MySQL performance settings on db1. MySQL status shortly before the restart was as follows:

• Uptime = 1 day, 3 hours
• Total Queries = 12,395,431
• free reported 140Mb RAM used
• Handler_read_rnd = 2840k
• Handler_read_rnd_next = 4110M
• Qcache_lowmem_prunes = 507k
• Qcache_hits = 70k
• Qcache_not_cached = 3349k
• Qcache_queries_in_cache = 4812
• Slow_launch_threads = 0
• Threads_cached = 7
• Threads_cache_hitrate_% = 99.71%
• Created_tmp_disk_tables = 13k
• Key_reads = 165k
• Key_writes = 328k
• Select_full_join = 2
• Select_range_check = 0
• Opened_tables = 692
• Table_locks_waited = 82k

• Combined memcached servers on content1/content2. Now only running on content1. Changed to run as used nobody and increased memory from 256M to 512M. No immediate errors or issues noted. Memcache stats before restart were:

• Memory Used = 256MB/256MB
• Hit Cache = 65%/55%

• Setup iptables rules on content1 limiting access to the memcached port.
• Started ntpd on db1/content3 and setup to start on server restart.

5 August 2009

• Added the webmail.uesp.net sub-domain (currently points to content3.uesp.net).
• Changed mail.uesp.net to point to content3.uesp.net.
• Setup and tested the e-mail (dave@uesp.net). Both webmail and POP interface works. Cannot send mail via SMTP through mail.uesp.net (probably a good thing).

4 August 2009

• Restarted db slave mirroring on backup1.
• Confirmed that basic onsite and offsite db backups are working.
• MySQL binlogs are restarted on a daily basis when the backup occurs (i.e., a new binlog is started after the backup is finished).
• Installed phpMyAdmin on db1

3 August 2009

• Restored db1 after a new hard drive replacement. Old hard drive was unrecoverable so backup had to be uploaded from the August 2nd backup copy on backup1. This means we lost more than a day of data (August 2nd at 4am to August 3rd at 9am). There is little chance of recovering this data unfortunately.
• Put in some basic daily database backups and offsite copies for db1. Database is not being mirrored yet.
• Increased wgExpensiveParserFunctionLimit from 500 to 1000 (temporary fix, but necessary for pages such as Oblivion:Normal Books).

2 August 2009

• Added new namespace aliases for all namespaces (see Namespace Links).

1 August 2009

• Restarted and repaired db1 from another database failure. No records lost.
• Returned to backing up the UESP wiki database daily.
• Hard disk replacement for db1 scheduled for next week.
• The new PHPBB3 UESP forums database is now being backed up.
• The slave database needs to be updated (probably needs a full restore after the July 19th db1 failure).

20 July 2009

• Missing rows of 'text' recovered from combination of: June 8th daily backup (more recent backups are corrupt or not online); June 16th pre-upgrade test backup; mysqlbinlog
  • mysql binlog cl-t116-160cl-bin.083 corrupt at byte 996000732; valid data found starting at 996152067. Intermediate data could not be recovered.
  • Only unrecoverable record was text.old_id=433673, corresponding to rev_id=442206 of UESPWiki:Reference Desk; a placeholder was inserted to prevent errors.
• All tables checked (again); manual backups done of wiki5 and phpbb3
• UESPWiki taken out of read-only mode.
• All pages cached on content1 and content2 during the outage/problem period were purged and regenerated, which should also clear any problems from squid1

19 July 2009

• Checked all UESP tables in database
  • Cleared multiple instances of "warning  : 2 clients are using or haven't closed the table properly" (all left over from last week's server shutdown)
  • Repaired table "text". Following repair, no errors reported (status OK using both medium and extended checks). Original error message (from command-line mysqlcheck command, rather than a check query) and messages reported by repair:

uesp_net_wiki5.text
warning  : 2 clients are using or haven't closed the table properly
error    : got error: 5 when reading datafile at record: 322209
error    : Corrupt
+---------------------+--------+----------+---------------------------------------------------------+
| Table               | Op     | Msg_type | Msg_text                                                |
+---------------------+--------+----------+---------------------------------------------------------+
| uesp_net_wiki5.text | repair | info     | Read error for block at: 3245841564 (error: 5); Skipped | 
| uesp_net_wiki5.text | repair | warning  | Number of rows changed from 443348 to 322561            | 
| uesp_net_wiki5.text | repair | status   | OK                                                      | 
+---------------------+--------+----------+---------------------------------------------------------+

• Note: Loss of 25% of the table rows, and the missing rows are all the most recent data

• UESPWiki put into read-only mode to work on recovering missing data
• db1 crashed; rebooted by iWeb

8 July 2009

• Checked all UESP tables in database
  • Cleared multiple instances of "warning  : 2 clients are using or haven't closed the table properly" (all left over from recent server shutdown); no other warnings or errors reported

2 July 2009

• Refreshed full text of searchindex for all wiki articles

30 June 2009

• Patched mediawiki code to incorporate all code changes that have been backported to release 1.14 (but never incorporated into the downloadable packages of MW1.14); brought code up to r50661 instead of r47653. Includes a fix to the empty links appearing on Special:Wantedpages. The vast majority of the changes are endless tweaks to dozens of obscure language files. Changes that were important enough to be mentioned in the release notes:
  • (bug 17737) Fixed russian URLs for Special:BookSources
  • (bug 17713) Using links with only an anchor no longer add an dummy entry in the pagelinks table
  • (bug 17897) Fixed string offset error in <pre> tags
  • (bug 17703) Error code and error info were switched for some errors
• Minor change to UespCustomCode (recognize ' ' as empty string, not just '')
• Forums upgraded to phpBB3 (including creation of new forums database which has not yet been added to backup scripts)

28 June 2009

• Installed TitleBlacklist extension
• Set $wgThumbnailEpoch = '20090624000000', to ensure that all thumbnails are regenerated using ImageMagick.

26 June 2009

• Deleted old backups and logs on db1 to free up disk space. Now at 70% used (40GB free).

25 June 2009

• Purged all pages that were processed during the MW1.14 upgrade, or that were processed in the previous days (get rid of site notices and error-filled pages for anonymous visitors)
• Refreshed all titles in the searchindex (still need to refresh full text)
• Fixed some issues with the search function in UespCustomCode
• Revamped the ProtectSection extension (again) to recognize nowiki tags and provide more accurate processing of [edit] tags on protected (and non-protected) pages
• Added protectsection rights for Patrollers

24 June 2009

• Upgraded to Mediawiki 1.14. See UESPWiki:MediaWiki 1.14 Upgrade for what was upgraded. See refreshing caches for notes on complications encountered during upgrade.
• Continuing to work on refreshing the caches and other post-upgrade followup
• Started to set up test of forums phpBB3 upgrade (including duplicates of the forum database that are being used for testing)

23 June 2009

• Checked all UESP tables in database
• Repaired table "text". Following repair, no errors reported (status OK using both medium and extended checks). Original error messages and messages reported by repair:

+---------------------+--------+----------+-----------------------------------------------+
| Table               | Op     | Msg_type | Msg_text                                      |
+---------------------+--------+----------+-----------------------------------------------+
| uesp_net_wiki5.text | check  | warning  | Table is marked as crashed                    |
| uesp_net_wiki5.text | check  | error    | Wrong bytesec: 0-0-0 at linkstart: 4154743104 |
| uesp_net_wiki5.text | check  | error    | Corrupt                                       |
+---------------------+--------+----------+-----------------------------------------------+
+---------------------+--------+----------+---------------------------------------------------+
| uesp_net_wiki5.text | repair | info     | Wrong bytesec:   0-  0-  0 at 4154743104; Skipped |
| uesp_net_wiki5.text | repair | warning  | Number of rows changed from 434847 to 434846      |
| uesp_net_wiki5.text | repair | status   | OK                                                |
+---------------------+--------+----------+---------------------------------------------------+

• Tried to drop obsolete "uesp_net_wiki5_copy" database (left over from last wiki upgrade), but 10 of the tables are "unknown". Nevertheless, some disk space recovered (db1 disk usage now at 89%)

16 June 2009

• Added the Bureaucrat Wiki permission to Nephele.

15 June 2009

• Created "wikicopy" database which is a copy of the actual wiki database (based on a snapshot of the database made this evening) for testing Mediawiki 1.14 setup
• (Note to self: db user rights on localhost are somewhat strange for now, to ensure that I don't accidentally mess up main database while working on the copy. Needs to be reset once MW1.14 testing done.)
• Ran Mediawiki 1.14 upgrade on wikicopy. Problems:
  • Error updating cl_sortkey key for categorylinks, because new key would be more than 500 chars in length. Problem seems to be that the cl_sortkey field is varchar(255), whereas cl_sortkey field should be varchar(86) (MW1.10) or varchar(70) (MW1.14). Checking our database reveals that only 4 records have fields larger than 70; none have records larger than 86 (software has obviously been truncating field to 86 chars). Ran "ALTER TABLE categorylinks MODIFY cl_sortkey varchar(86) binary NOT NULL default ''" on wikicopy (this is the create statement used by MW1.10) -- will need to be done on main db, too.
  • Also need to run install.php for CheckUser
  • Upgrade is taking about 2 hours
• Installed ImageMagick 6.2.8.0 on content2.
• Set up test version of MW1.14 wiki. See MediaWiki 1.14 Upgrade and Testing the Upgrade

14 June 2009

• Updated UespCustomCode to add addImageClear function

12 June 2009

• $wgHashedUploadDirectory set to true on content1 and content2, after reorganizing all images into hashed directories. For now, the new images are located in a subdirectory of the original images directory (images/images.new), but the location is likely to be updated.
  • There were a few transient problems on content1 during the switch (in particular, read permission errors on LocalSettings.php, lasting for a couple minutes). Even after all reported problems have been fixed, performance on content1 continues to be somewhat slow, apparently because all of the new images are being rsync'd to the backup; it's probably going to take a few hours before the 3GB is rsync'd and content1 can recover. On content2, no errors have been reported in the logs and performance appears normal, although there appears to be an increase in image requests (to be expected, given that squid1 also effectively needs to replace all of its cached images).
• Minor changes made to several files on the old UESP site to fix errors reported in the apache error logs (tamrial->tamriel, disabling commented-out SSI commands, etc.)
• Database usernames on db1 updated to allow non-local access to database; programs (in particular, map programs) changed to use program-specific database usernames instead of root

11 June 2009

• Minor update to MetaTemplate on content1 and content2, fixing problems caused by double-parsing function output
• UESP wiki file cache on content1 changed in LocalSettings.php from /tmp1/cache to /tmp1/cache1 so that wiki can use the cache
• Size of table uesp_net_wiki5.text increased before the table ran out of space (just before size increase, table had been at 4120558636/4294967295, or 96% capacity)

9 June 2009

• /tmp on content1 filled up causing problems with PHP sessions. The UESP wiki file cache was moved from /tmp/cache to /tmp1/cache to free up space on the temporary partition. A sym-link /tmp/cache was created to temporarily redirect cache requests (this will break when the server is rebooted, LocalSettings.php should eventually be changed).

7 June 2009

• Fixed some broken links on content2; fixed the original links on content1, so that future syncs won't just recreate the problem.
• Fixed some file permissions on content2. Note that there are some uid and gid mismatches between content1 and content2, notably the nephele user.

6 June 2009

• Ran 'repair table logging' in mysql to fix problems reported by check table. Original error messages:

+------------------------+-------+----------+---------------------------------------+
| Table                  | Op    | Msg_type | Msg_text                              |
+------------------------+-------+----------+---------------------------------------+
| uesp_net_wiki5.logging | check | error    | Can't read key from filepos: 25980928 | 
| uesp_net_wiki5.logging | check | error    | Corrupt                               | 
+------------------------+-------+----------+---------------------------------------+

Repair table was successful, and following repair check table reported status OK (using both medium and extended settings for check).

• Ran 'repair table privmsgs' and 'repair table privmsgs_text' to fix mysql errors. Original error messages:

+-------------------------+--------+----------+--------------------------------------------+
| Table                   | Op     | Msg_type | Msg_text                                   |
+-------------------------+--------+----------+--------------------------------------------+
| uesp_net_phpbb.privmsgs | repair | warning  | Number of rows changed from 21878 to 21877 | 
| uesp_net_phpbb.privmsgs | repair | status   | OK                                         | 
+-------------------------+--------+----------+--------------------------------------------+
+------------------------------+--------+----------+-------------------------------------------------+
| Table                        | Op     | Msg_type | Msg_text                                        |
+------------------------------+--------+----------+-------------------------------------------------+
| uesp_net_phpbb.privmsgs_text | repair | info     | Wrong bytesec:   0-  0-  0 at 11526308; Skipped | 
| uesp_net_phpbb.privmsgs_text | repair | info     | Wrong bytesec:   0-  0-  0 at 11526308; Skipped | 
| uesp_net_phpbb.privmsgs_text | repair | warning  | Number of rows changed from 21878 to 21877      | 
| uesp_net_phpbb.privmsgs_text | repair | status   | OK                                              | 
+------------------------------+--------+----------+-------------------------------------------------+

Repairs of both tables were successful, and subsequent 'check table extended' reported status OK

• Check table was successful on all other databases on db1, except:
  • uesp_net_wiki5_copy which presumably is not actually being used for database queries.
  • warning about uesp_net_wiki5.text, which will be fixed in the near future.
• Changed $wgFileCacheDirectory on content1 to /tmp/cache (LocalSettings.php is again identical on content1 and content2)

2 June 2009

• Updated MetaTemplate to version 0.5 on content1 and content2
• Added tesnexus.com to the new users' valid sites list on the forums
• Changed wgFileCacheDirectory on content2 to /tmp/cache

1 June 2009

• In forum's uesp Board Style, set th_class* parameters equal to "spacer.gif" (no change in forum appearance, but prevents thousands of error messages from flooding logs)
• Removed youtube.com from the new users' valid sites list on the forums (fix reported problems with new users abusing youtube links)
• Created some Elder Scrolls-based avatar galleries on the forums

31 May 2009

• Created accounts for nephele on all servers.
• Created and initalized the blockip/unblockip scripts on all servers.
• Made sure atd was setup to run on all servers.
• Disk Usage Check: content1=60%, content2=10%, content3=2%, squid1=23%, db1=78%, backup1=69%/57%
• Restarted backup1 after a power failure. Corrected a DB slave error. Mirror is now up to date.

• Updated ProtectSection (and EditPage.php) on content1 and content2 (see User talk:Daveh#Updating ProtectSection)
• Updated UespCustomCode to version 0.7 (including updates to Monobook.php, Parser.php, Sanitizer.php, SearchUpdate.php, Skin.php, and Wiki.php) on content1 and content2

26 March 2009

• Started ntp syncs on content2.

25 March 2009

• Installed the modified ActiveUsers extension on content1.
• Installed the first version of UespCustomCode on content1.
• Deleted a bunch of old files and logs on content1. Went from 86% to 58% usage (120GB free).
• Checked free space on squid1: 24% usage (220GB free).
• Checked free space on db1: 72% usage (40GB free).
• Checked free space on content2: 5% usage (130GB free).

2 March 2009

• backup1 replication finally up to date without any errors.
• Reset the Wiki's cache date on content1 (finally).
• Installed tidy on content2 which was the cause of display errors on some pages.

1 March 2009

• Changed the load balancing weight of content2 from 1 to 3 due to a temporary overload in content1. This is a temporary fix until ICP query load balancing can be setup (which chooses the least loaded server).

28 February 2009

• Started round robin mirroring of content1/content2 from squid1. Seems to be working good so far. Will keep an eye for issues this weekend.

27 February 2009

• Basic mirror working on content2.uesp.net (currently set as read-only). Not yet completely setup/tested or set as a load balanced server on squid1.

26 February 2009

• Maxconn settings on squid1 seem to be working (all client connections limited to 12 or fewer per client).
• Received the content2 and util1 servers.
• Reloaded the backup1 database due to replication errors.

25 February 2009

• Maxconn settings on squid1 not working as caught someone with close to 200 concurrent connections. Changed order of ACL settings and restarted squid.
• Corrected a few bugs in backup scripts which were preventing some content from being backed up correctly.

24 February 2009

• Two more servers ordered (cheap 2-for-1 deal from provider). At least one intended for content2.
• Put a maximum connection limit of 12 on squid1. Will try adjusting down to lower limits in the future.

Prev: 2008

Up: Upgrade History

Next: 2010