User:Daveh/Wiki Upgrade 1.22

Notes on the MediaWiki upgrade from 1.19.3 to v1.22.15

Wiki Differences

• extensions/Gadgets/Gadgets_body.php

• Lines 550/551 (end of isValidList()):

          //UESP - Return false on empty array to prevent issue with memcached
     return false; // empty array

• Not propagating...if there's an issue try doing a null edit on [[MediaWiki::Gadgets-definition]].

• includes/logging/LogFormatter.php

• Lines 622 (in PatrolLogFormatter::getMessageParameters()):

     $newParams = array_slice( $params, 0, 3 );

• Lines: 639/640

     $params[3] = Message::rawParam( $revlink );
     return $params;

• Not propagating...code moved to includes/logging/PatrolLogFormatter.php and seems to be due to old wiki code anyways.

• includes/logging/PatrolLog.php

• Line 30 (in PatrolLog::record()):

     global $wgUser;

• Line 35 (in PatrolLog::record()):

     $entry->setPerformer($wgUser, false);

• Not propagating...this appears to have been merged into the mediawiki code.

• skins/MonoBook.php

• Line 63:

    global $wgUser;

• Lines 70-100

     <!-- BEGIN UESP -->
     <?php
             if ( ! $wgUser->isLoggedIn() ) {
     ?>
             <!-- WikiLeaderboardTop -->
     <div class='center' style='width:728px; height:105px; overflow: hidden; margin-left: auto; margin-right: auto;'>
     
     <form method="post" target="_blank" id="form_topad" action="http://www.uesp.net/adreport.php">
     <input name="UespAdContent" type="hidden" value="unknown">
     <input name="UespAdId" type="hidden" value="div-gpt-ad-1344720487368-2" />
     
     <div id='div-gpt-ad-1344720487368-2' style='width:728px; height:90px;'>
     <script type='text/javascript'>
     googletag.cmd.push(function() { googletag.display('div-gpt-ad-1344720487368-2');
      });
     </script>
     </div>
     <div style='float: left;'>
     <small><a href='/wiki/UESPWiki:Site_Support'>What is this Ad?</a></small>
     </div>
     <div style='float: right;'>
     <small><a href="javascript:submitReportAdForm('div-gpt-ad-1344720487368-2', 'form_topad')">Report Ad</a></small>
     </div>
     </form>
     </div>
     <?php
             }
     ?>
     <!-- END UESP -->

• Lines 122-126

     </div>
     <!-- BEGIN UESP -->
     <?php wfRunHooks( 'MonoBookPageBottom', array( &$this ) ); ?>
     <!-- End UESP -->
     </div>

• Lines 150-175

          
     //BEGIN UESP
             if ( ! $wgUser->isLoggedIn() ) {
     ?>
     
           <div id="p-googlead" class="portlet" style="overflow: hidden; width: 162px; height: 625px; ">
                   <form method="post" target="_blank" id="form_sidead" action="http://www.uesp.net/adreport.php">
                   <input name="UespAdContent" type="hidden" value="unknown" />
                   <input name="UespAdId" type="hidden" value="div-gpt-ad-1344720487368-3" />
                   <div style="float: left;">
                           <small> <a href="/wiki/UESPWiki:Site_Support">What is this Ad?</a></small>
                   </div>
                   <div style="float: right;">
                           <small><a href="javascript:submitReportAdForm('div-gpt-ad-1344720487368-3', 'form_sidead')">Report Ad</a></small>
                   </div>
                   <!-- WikiMiddleWideSkyscraper -->
                   <div class='center' id='div-gpt-ad-1344720487368-3' style='width:160px; height:600px; margin-right: auto; margin-left: auto;'>
                           <script type='text/javascript'>googletag.cmd.push(function() { googletag.display('div-gpt-ad-1344720487368-3'); });
                           </script>
                   </div>
                   </form>
             </div>
     <?php
             }
     //END UESP

• All added to the overloaded method UespMonoBookTemplate::execute()'.

• Lines 252-257:

     <!-- Begin UESP -->
           <button id="searchButton" title="Search UESP for this text" name="button" type="submit">
                   <img width="12" height="13" alt="Search" src="/w/extensions/UespCustomCode/files/search-icon.png">
           </button>
     <!-- End UESP -->

• Added to the overloaded UespMonoBookTemplate::searchBox()

• Lines 261-273

     
     <!-- BEGIN UESP -->
             <div id="p-googleplus" class="portlet" style="border: 1px solid #AAAAAA; background:#FBEFD5; margin: 0px; padding: 3px; width: 153px; overflow: hidden;">
             <div class="g-plusone" data-annotation="inline" data-width="139"></div>
     <script type="text/javascript">
       (function() {
         var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true;
         po.src = 'https://apis.google.com/js/plusone.js';
         var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s);
       })();
     </script>
             </div>
     <!-- END UESP -->

• Added to the overloaded UespMonoBookTemplate::searchBox()

• Lines 352-365 (function customBox()):

     <?php                        foreach($cont as $key => $val) {
                                   if( function_exists('efSiteCustomCode') ) {
                                           if (substr($val['text'],0,1)=='*') {
                                                   if (!$isindent)
                                                           echo "\n<ul>\n";
                                                   $isindent = true;
                                                   $val['text'] = trim(substr($val['text'],1));
                                           }
                                           elseif ($isindent) {
                                                   $isindent = false;
                                                   echo "\n</ul>\n";
                                           }
                                   }
     ?>

• Lines 368-370 (function customBox()):

     <?php                 }
                           if ($isindent) echo "\n</ul>\n";
     ?>

• Both added to overloaded method UespMonoBookTemplate::customBox()

Extensions

• Default in MediaWiki 1.19.3

• ConfirmEdit
• Gadgets
• Nuke
• ParserFunctions
• RenameUser
• Vector
• WikiEditor

• Added

• AbuseFilter (u)
• AntiSpoof (u)
• CategoryTree (u)
• CharInsert (u)
• CheckUser (u)
• Cite
• CustomCategory (u)
• DeleteBatch (u)
• DismissableSiteNotice (u)
• DynamicFunctions (u)
• Editcount (u)
• ExpandTemplates (u)
• ImageMap
• InputBox
• LabeledSectionTransclusion (u)
• MediaFunctions (u)
• MobileFrontEnd (u, mobile only)
• MWSearch (u)
• OAI (u)
• Patroller (u)
• ProtectSection (u)
• RegexFunctions (u)
• SimpleAntiSpam
• SpamBlacklist
• StringFunctions (u)
• TitleBlacklist
• TorBlock (u)
• WikiTextLoggedInOut (u)

• UESP Custom

• DailyEdits
• DragonFont
• JobQueue
• MetaTemplate
• qwebirChat
• SearchLog
• UespCustomCode
• UespMap
• UncategorizedUESPPages
• UsersEditCount

• No Longer Needed

• Asirra (built into ConfirmEdit)
• SphinxSearch
• uespdfp
• UespPaypal
• UespSiteStats

• Default in MediaWiki v1.22.2

• Cite
• ConfirmEdit
• Gadgets
• ImageMap
• InputBox
• Interwiki
• LocalisationUpdate
• Nuke
• ParserFunctions
• PdfHandler
• Poem
• Renameuser
• SimpleAntiSpam
• SpamBlacklist
• SyntaxHighlight_GeSHi
• TitleBlackList
• WikiEditor

Database Related

• MediaWiki DB Backup/Restore

• mysqldump --option ...

• Dump Time = 9:09
• Dump Size = 21GB (uncompressed)
• Restore Time = 58:23

• SQL: select/insert into new database (text table only)

• Direct Copy = 32:00
• Copy w/Disabled Indexes = 7:00
• Enable Index = 23:00 (30:00 total for copy and re-index)

• Percona Xtrabackup

• Reference
• Install:

   yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
   yum install percona-xtrabackup-20     //Regular version doesn't support MySQL v5.1?

• Create file cred.cnf

   [client]
   username=...
   password=...
   !include /etc/my.cnf

• Running:

   innobackupex --defaults-file cred.cnf --databases="uesp_net_wiki5"  /backupdir

• Turn on innodb_file_per_table in MySQL

• Backup database with master file/index
• Stop slave
• Stop mysqld
• Edit /etc/my.cnf and add innodb_file_per_table = 1
• Start mysqld
• Restore complete database from backup
• Restart slave

• Full Backup on Content3 = 36:33 (79GB total)
• Wiki Backup = 13:01 (27GB total)
• Wiki Restore = ?
• Notes

• Can't do partial restores unless innodb_file_per_table is on in MySQL (currently off on all UESP MySQL installations)

• Upgrade Method A

• Set wikis to Read-Only (1 min)
• Backup db1 (10 min)
• Run upgrade (? min)
• Test/monitor (5 min)
• Turn writes back on (1 min)
• Total Time = ~20 min?
• In case of problem:

• Restore backup on db1 (1 hour)
• Test/monitor (5 min)
• Turn writes back on (1 min)
• Total Time = ~1 hour

• Comments

• No data lost if upgrade fails
• Potential read-only time of ~15 min on success
• Potential down time of > 1 hour on failure

• Upgrade Method B

• Set wikis to Read-Only (1 min)
• Backup db1 (10 min)
• Turn writes back on (1 min)
• Restore backup to new database on db1 (1 hour)
• Set wikis to Read-Only (1 min)
• Run upgrade (? min)
• Test/monitor (5 min)
• Turn writes back on (1 min)
• Total Time = ~1:20
• In case of problem:

• Switch to new backup database (1 min)
• Test/monitor (5 min)
• Turn writes back on (1 min)
• Total Time = 7 min

• Comments

• Some data lost if upgrade fails (any writes between backup and upgrade)
• Potential read-only time of ~15 min on success
• Potential downtime/read-only time of 1 min on failure (very little perceived downtime)

• Upgrade Method C

• Set wikis to Read-Only (1 min)
• Flush and turn off slave reads on content3 (1 min)
• Run upgrade (? min)
• Test/monitor (5 min)
• Turn writes back on (1 min)
• Restart slave on content3 eventually
• Total Time = 15 min?
• In case of problem:

• Switch to using content3 as primary read database (1 min)
• Total Time = 1 min

• Comments

• No data lost on failure
• Wiki stays in read-only mode on failure until upgrade succeeds (unless you restore a backup on db1)
• Potential downtime/read-only time of 1 min on failure (very little perceived downtime)
• Only 15 min of read-only time on success (all due to upgrade procedure)

Upgrade Procedure

1. Post upgrade notice
2. Set wiki to read-only (edit images/UESP_LOCK_DB file with message)
3. Perform wiki backup on db1
4. Turn read-only mode off (note: edits after this point may be lost if the upgrade fails)
5. Restore backup on db1 to a new database (uesp_net_wikitmp)
6. Set permissions for the new database on db1/db2/content3 (same as existing wiki)
7. Create temporary folder, for the upgraded wiki

• Copy files dev wiki (not including images)
• Create sym-link to shared images
• Copy LocalSettings.php from existing wiki
• Edit the new LocalSettings.php:

• Remove $IP variable (no longer needed since v1.18)
• Remove call to set_include_path() at start
• Remove required include includes/DefaultSettings.php (included by default before LocalSettings)
• Remove check for $wgCommandLineMode
• Set $wgImageMagickConvertCommand = '/usr/bin/convert'; (remove old check)
• Set $wgDefaultSkin = 'uespmonobook';
• Disable extension CatagoryTree
• Disable extension CustomCategory
• Remove extension Asirra (built into ConfirmEdit)
• Remove extension SphinxSearch
• Remove extension uespdfp
• Remove extension UespPaypal
• Remove extension UespSiteStats
• Remove 3 lines related to $wgRC2UDP...
• Update $wgCacheEpochto current date
• Add extension QuestyCaptcha and related configuration lines
• Order extensions alphabetically
• Add APIEditBeforeSave hook code
• Add $wgAPIModules['createaccount'] = 'ApiDisabled';
• Add $wgAccountCreationThrottle = 1;

1. Turn read-only mode back on
2. Copy secrets into a new file secrets/wiki-new.secrets on all content servers
3. Set v1.22.15 of the wiki to use the new secrets file
4. Set live wiki to use the new database (edit secrets/wiki.secrets and change the $uespWikiDB variable)
5. Perform backup of wiki to be upgraded (only needed to save any edits between the previous two read-only periods)
6. Perform upgrade in the v1.22.15 wiki directory
7. Swap v1.22.15 and existing live wiki paths
8. Test wiki to ensure its working (on failure or any issues swap the swaps back so that the live site uses the old, read-only wiki)
9. Turn read-only mode off
10. Post upgrade success message
11. Edit wiki.secrets back to use the main wiki database and revert use of wiki-new.secrets

Notes

• Removed Extensions:

• Asirra (built into ConfirmEdit)

• Include ConfirmEdit.php and Asirra.php from the ConfirmEdit directory

• SphinxSearch
• uespdfp
• UespPaypal
• UespSiteStats

• Renamed Extensions:

• EditCount to Editcount
• Include SpecialEditCount.php to Editcount.php

• Disabled Extensions:

• CustomCategory (has issues with MetaTemplate)
• CategoryTree (not used?)

• New Extensions:
• Issues

• MetaTemplate

• Call to undefined function wfLoadExtensionMessages() in /home/uespdev/www/w/extensions/MetaTemplate/MetaTemplate.php on line 209
• wfLoadExtensionMessages() is no longer used and can be deleted

• SearchLog

• Same issue with wfLoadExtensionMessages

• UespCustomCode

• Failed loading includes/ChangesList.php in SiteChangesList.php line 3
• File moved to includes/changes/

• Double Title on RecentChanges

• There are two system messages, MediaWiki:Recentchangestext and MediaWiki:Recentchanges-summary. According to what is used on Wikipedia the summary message should be the Track the most... and the text message contains more a detailed header text (if required, in our case it would be blank). After upgrade copy the text to the summary message and blank the text message.

• Double check all/none buttons in the advanced search area

• Remove checkboxes from div in extensions/UespCustomCode/SiteSpecialSearch.php (mw code adds buttons via JavaScript when page is loaded)

• Upgrade

• Time required on content3 (v1.22.2) = 2m 20s
• Time required on content3 (v1.22.15) = 5m 30s
• Wiki won't load without running upgrade.php (database error)

• Modified Files

• New directory /skins/UespMonoBook/
• Deleted directory mw-config/
• Modified LocalSettings.php

• As documented at [1]

$wgRC2UDPAddress = '10.2.212.14';
$wgRC2UDPPort = 9390;
$wgRC2UDPPrefix = '';
 
should be moved to
 
$wgRCFeeds

Post Upgrade Notes

• Total Read-Only Time = 17 + 19 = 36 min
• Overall Update Time = 3 hours (2:57)
• MediaWiki Upgrade Time = 200 seconds (0:03:20)
• Wiki/Apache/PHP doesn't like it when you update the secret files from another server. Restarting Apache fixes it.
• With debugging/profiling enabled there are a few "deprecated" messages that may be worth investigating further
• Upgrade Issues/Bugs

• Slow page loads for a few hours (probably due clearing all caches at once and reset the $wgCacheEpoch in combination with the upgrade)
• SearchLog is broken due to database error: mysql_real_escape_string(): Can't connect to local MySQL server through socket...
• Broken gadgets/JS (may just have been a temporary cache/update issue)
• Special:SpecialPages won't load from Squid (unknown reason, added an explicit Squid rule to skip cache for this page)
• Search suggestions only work with main (0) namespace. Caused by a hard coded change to namespace used. Temporarily modified resources/mediawiki/mediawiki.SearchSuggest.js to add UESP content namespaces.
• Database XML dumps broke due to using customized scripts in maintenance/.
• Misc Display Issues

• User CSS doesn't work (have to move User:Name/monobook.css to User:Name/uespmonobook.css)
• Edit Lead Section overlaps protection icons
• Add New Section
• Missing Falmer Font (modify CSS)
• Drop-down menus gadget is broken (due to move to custom skin)
• Page title clipping on some browsers
• Background color in edit, preferences and login screens
• Diff colours changed
• Recent changes duplicate text (as noted above)

• Future Upgrades

• Currently no modified MediaWiki files (other than resources/mediawiki/mediawiki.SearchSuggest.js)
• Copy skins/UespMonoBook/