Monday, November 24, 2003 - Libraries are down
Reported by  Nakor

One of our server hard drives is trying to fail on us. RAID has helped with it so far, but library downloads are suspended while we make some additional snapshots of the current data set. 

Monday, November 24, 2003 - WinCustomize Libraries will remain down Monday night and Tuesday morning
Reported by  Nakor

The server with most of the library downloads is refusing to come back from a reboot. It's possible we're going to have to roll back to an archive, but we're kind of stuck until we hear back from the co-lo. It does not help that this is a holiday week here in the US and lots of people/plans are in flux.

The outage will continue in some fashion into Tuesday (GMT-5)

Sorry, everybody. For those of you in the US, enjoy your holidays. For everyone else, cook a turkey and some cranberries and think about what you're thankful for.

Update: Because of the distance involved (we're in MI, server in CA) restoration is taking awhile. So the libraries will probably be down into tomorrow.

Wednesday, November 26, 2003 - Recovery Status
Reported by Frogboy

Okay, here's the deal...

As some know, last month WinCustomize transitioned from Atlanta to California. The crappy Fujitusi drives are failing. One is already dead and it went out in just a way so that we can't access the RAID easliy making restoring locally easy.

Unfortunately, our next backup version is in Michigan and restoring it remotely is going to take a great deal of time (think how many gigabytes the skin libraries here are and imaging trying to restore over a T1). 

So what we're doing to do is ship a drive there to restore the older skins. But, in the meantime, we will put up the libraries so that new submissions can be made. We will also put in a system so that authors who want their works to be restored faster can restore their work on their own.  This is mainly useful on newer submissions.

The Thanksgiving holiday is what is causing part of our delays as staffing is reduced and one day everything is shut down. So it may be over a week before the full library is restored.

But most people are interested in getting the "Latest" stuff and that will be our focus: Get the libraries back up and restore the files as soon as we can.

Those of you who have recently downloaded files, please keep those handy. It will make the recovery effort smoother for everyone if we can work together and users resubmit recently uploaded files (uploading the .WBA or .ZIP or whatever is only part, the screenshot is necessary too). But if we can work together, we can be up and running at full power again sooner.

Wednesday, November 26, 2003 - Libraries back up, restoration begins
Reported by Frogboy

The libraries are back up. So here's where we're at:

We're going to ship the archive to California to be installed but that will take some time to do. 

In the meantime, everyone who uploaded a skin after October 1 will be receiving an email letting them know how to quick update them. Authors who resubmit their post-October 1 skins will automatically have their skin date updated as if they were new (our way of thanking authors for doing them).

Also, anyone with Apprentice access or higher can also help authors and the community out by submitting missing skins too (screenshot will still be needed). Here's how you do that: Go to the file library and you will see which skins are missing by the special missing skin thumbnail. There will be a [Help Update] link there. We'll be keeping track of who helps the most and sending out T-shirts for Christmas to some of the top helpers as a token of our and the community's appreciation. Note: Do NOT try to submit missing skins using the usual "upload" mechanism. Do it from the Help Update system or from the link you (as an author) received in email.

By working together, we may actually be able to have reasonably up to date libraries by Thanksgiving!

Thank you for all your help! Have fun!