Hair pulling
Monday, January 5, 2004 by Frogboy | Discussion: WinCustomize News
This is just unbelievable. I am sure many fo you, like us, have dealt with plenty of machines that run fine 24/7 over years and years.
Today WinCustomize lost another server to total hardware failure. This is the THIRD server in 3 months. This one in some ways was worse and the timing was particularly bad.
The SQL database was on that machine. Luckily, because of our previous experiences, we have backups. But because of an additional glitch, the auto-backup didn't work which means we lost all the database changes since our last manual backup which was just before we left on the holidays. Net result is that we lost any uploads between the time we went away for Christmas (22nd?) and yesterday.
You can imagine how frustrated we are. Oh, we have a pretty good idea the commonolity on this (all 3 servers were quite very new, from the same provider with the same OS). But we won't get into public finger pointing.
It's just incredible though. We have el-cheapo servers here that have run 24/7 for the past 5 years without any problems.
We'll have the libraries restored with what we have. If authors who have recently submitted stuff could resubmit their stuff we appreciate it and apologize for the inconvenience. Ironically, we have the files (the crash in November wiped out the physical files but the database was fine). I'm just glad we keep backups.
Thanks for your patience and support. Unfortunatley, we are now down a main server again. This one was our fastest/newest (and most expensive) machine unfortunately.
If anyone needs me, I'll be over here pulling my hair out. 
 |
|
Page 2 of 3 | |
|
|---|
Reply #22 Monday, January 5, 2004 4:20 PM
1. Is the servers owned by service provider? if that's the case they should replace with another new one and provide service. If the hardware is owned by Stardock, then that should have 7x24 service for at least 1-3 years, e.g. from Dell.
2. Backup - have no idea how large is the current skin library, if it is talking about 50-100GB, believe it take 2-3 hours to have full backup every day using current DDS, DLT or LTO technology. We have some DB servers that we need to have 2 full backup daily as the data is quite critical to us. Obviously the skin library is critical to both you and your customers.
3. Consider 7x24 support on software support as well. I think you can may have some statistics that how many times the problem happen to the skin library during weekend, midnight, holidays? I understand this increase workload to your team, but I believe everyone would like this system running 7x24.
Reply #23 Monday, January 5, 2004 4:30 PM
Hope everything's back soon good luck. 
Reply #24 Monday, January 5, 2004 4:33 PM
good luck guys, hope you get everything straightened out soon!
Reply #25 Monday, January 5, 2004 4:42 PM
Yea, we own the servers. They're all quite new. They're on warranty but the problem is downtime. Being on warranty still means sending them back or having the repair guy come in and take care of the system which takes time.
I'm just beside myself in frustration because we've always had good luck with servers and yet here we are with 3 in just a few months.
We should be back up by the end of the day today but things will be slow. The server that died was our main server. The one ultra fast machine that was there. Even before this happened, we were consolidated to fewer machines than our design was for.
With the subscription drive, we will be in better shape soon -- we'll have new servers. It's just a matter of time of getting them all there.
Reply #26 Monday, January 5, 2004 4:47 PM
Reply #27 Monday, January 5, 2004 5:00 PM
If you need, you can go here http://www.hairclub.com/google/
Reply #28 Monday, January 5, 2004 5:03 PM
Besides bringing new servers, below is some of my comments:
1. Consider to have dedicated SQL server, currently I observe that the skin library is not available due to several issues (1. web server issue - can't handle further request 2. DB issue 3. hardware issue).
I believe majority of weekend downtime (not counting these three special downtime) is due to web front-end issue. You can still provide the DB access to other web servers front-end even one particular web server is down.
2. Consider to seperate several web application on different servers, or using 'application pool' feature of IIS6 (from Windows Server 2003) - I understand you are using IIS5/W2K? This can prevent the situation that the 'skin library' web will bring down other application as well (e.g. customer page, stardock central...etc)
Also there has many performance tuning aress on both web servers and DB servers.
3. Hmm...errr... any plan to migrate the front-end to ASP.NET? Believe compiled code is faster than ASP engine that trying to render the ASP code everytime?
4. This may be off-topic, but currently the logon to Stardock Central still need to validate the account in three servers (1. sdcentral.stardock.com 2. wincustomize.com 3. galciv.com). I prefer that we can still access to Stardock Central even the latter two websites is not available.
Reply #29 Monday, January 5, 2004 5:07 PM
We'll be here patiently waiting for your return. Its a shame this had to happen. I'm sure WC will pull out of this better than ever.
Reply #30 Monday, January 5, 2004 5:15 PM
Hope you'll don't get another major trouble and things will be as much secure as possible with the new servers ... 
Reply #31 Monday, January 5, 2004 5:22 PM
And by this day and age I thought reliability would be taken for granted.
Reply #32 Monday, January 5, 2004 5:28 PM
hope things go well with the repair, and thanks for all the hard work to those dealing with resolving the server problems.Reply #33 Monday, January 5, 2004 5:34 PM
Good luck!
Reply #34 Monday, January 5, 2004 5:36 PM
I would label them loud and clear - Get their attention - You don't get millions of hits for nothing
Reply #35 Monday, January 5, 2004 5:36 PM
Reply #36 Monday, January 5, 2004 5:42 PM
Well, I cannot offer my help, being a poor uni student and having no contacts to IT pros at all, but I want to tell u that I feel your pain either way! Good wishes from Germany! Get WinCust back up as soon as ya can, coz it rocks! Keep it up, Frogboy!!
Reply #37 Monday, January 5, 2004 5:51 PM
Grumble, mutter, mutter, grouch.....
Still....'twas a little forced holiday.....
Good luck with resolving everything, guys....
Reply #38 Monday, January 5, 2004 5:57 PM
Reply #39 Monday, January 5, 2004 6:17 PM
Reply #40 Monday, January 5, 2004 6:51 PM
Yup. I think that's the cheapest solution. Don't allow any high-quality art on the site and riddle the Stardock apps with bugs. Presto! Everything becomes manageable again!
Jeff
Please login to comment and/or vote for this skin.
Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:
- Richer content, access to many features that are disabled for guests like commenting on the forums and downloading skins.
- Access to a great community, with a massive database of many, many areas of interest.
- Access to contests & subscription offers like exclusive emails.
- It's simple, and FREE!
|
|
Page 2 of 3 | |
|
|---|
Reply #21 Monday, January 5, 2004 4:04 PM
TrueKaiser would be looking for someone elese for my server needs.""
this might sound ironic but i would look for alternitives to the current os and hardware..