"Scheduled Maintenance" - Behind the Scenes
steve
November 26, 2007, 12:03 PM posted in General DiscussionMaybe I should have used the term we have been using internally for the last few months - Project Iceberg. The idea was to upgrade all the systems that are 'below the surface'. In theory the magnitude of this upgrade was supposed to be largely invisible to users.
Of course, the reality is that we've caused a fair bit of unintended disruption. I apologize for this. So let me explain a little more about what has been going on. Here are some of the goals we set for Iceberg.
1) Database Unification: This allows all future services to be tightly integrated. For example, now you can access ChinesePod content via 88Groups without having to login again. There will be lots of additional benefits from this unification in the future.
2) Extra Content: we've had a lot of requests from users who want to be able to subscribe to extra content. By streamlining the way our internal systems handle lesson publication, we've made it possible to do exactly that. In the future, you'll be able to Bookmark these episodes just like our other podcasts too.
3) Modularity: We've re-designed the site with modularity in mind (the exercises in particular went through a big change). The benefits of this won't be immediately obvious, but we want it to be possible for lessons to be studied 'elsewhere', be that in 88Groups, or Facebook, or on your mobile phone. This is part of 'learning on your terms'. A more modular system also gives us a better upgrade path for making future changes less disruptive.
4) Group Learning: In addition to these changes, we developed 88Groups.com: a step towards helping teachers remix ChinesePod materials to create and manage their own learning groups.
In the end, Iceberg wasn't as invisible as intended. There are still some glitches, but we've solved many of the problems created, and the rest will be solved soon, along with some fixes in response to longstanding user requests ( i.e. id3 tagging on "The Fix").
Here are a few updates on places people have had problems:
a) Vocabulary
The upgrade required us to migrate all your vocab lists into the new database. There are well over a million entries, and this is taking much longer than expected, and because it is server-intensive we have put a throttle on it so as to keep system resources free for site use. However, we anticipate it will be complete by the end of the week.
b) Extra Content.
The publication process involved remigrating all of our Extra content. You may have noticed that some was missing recently. It is all there now, along with the migration of all of the old comments.
c) Grammar Guide.
While we loved the content in our grammar guide, one of the weaknesses we noticed was that it was static: an online textbook. In order to help users make better use of the guide, we are going through some backend work reformatting the data in preparation for rolling out a new and improved grammar guide over the next few months. This is probably the biggest negative consequence of the change, but we believe it will improve the useability and usefulness of the guide in the long run, and hope you'll bear with us.
d) Archived Lesson Issues
The upgrade re-imported all of our our lessons into a unified database. As each legacy system worked slightly differently, there may be slight issues that affect different generations of lessons. We're conducting a site-wide sweep at the moment to catalogue any issues with missing or irregular lesson content, in order to get them fixed. If you spot any, please let us know.
Once again, I apologize for the service disruptions, and also for the patchy communications as we dealt with the issues. I guess we are still learning how to develop the site without compromising service levels. Thanks for your patience and understanding though. We are working hard for you.
Regards
Steve
RonInDC
November 30, 2007, 03:39 AMAdding vocab from the CPod dictionary is still broken.
bazza
November 26, 2007, 01:43 PMProject Iceberg, cool name. ;)
goulnik
November 26, 2007, 02:27 PMcool name with uncool connotations (according to Wenlin / ABC dictionary) : 冰山 bīngshān n. ①iceberg <座> (zuò) ②individual/group not to be relied on for long
jamestheron
November 26, 2007, 02:54 PMVery interesting choice of a project name. Now, when project iceberg and project titanic meet...
phil
November 26, 2007, 02:56 PMStrange, if I link to this conversation from Bloglines then Bazza appears as the author! But from within Cpod Steve is the author.
steve
November 26, 2007, 03:24 PMHi Kyle, The vocab you recently added is indeed missing. We've backed it up, and once we are sure the pre-Iceberg migration is completed smoothly, we will migrate across your post-Iceberg saves. Jamestheron, Yep - bad association. Once we'd named the project it was too late to change, but it did feel like tempting fate at times. Goulniky, It definitely felt a few degrees warmer around here last week!
steve
November 26, 2007, 03:29 PMOn the subject of project names, V3 was almost dubbed 'The Long March'. I had a soft spot for that name, but in the end the snappier 'V3' won the day. We are also working on a project called 'Hydrogen', and another called 'Anthill'. I'm afraid the names don't give much away though...
sparechange
November 26, 2007, 03:44 PMCool. Thanks for the update, steve. You now have my permission to sleep. ;-)
sparechange
November 26, 2007, 03:47 PMThe grammar guide was static? Seems like that would have been a nightmare to maintain...
goulnik
November 26, 2007, 01:22 PMthanks to let us know the 内幕 (nèimù). one day, a business school professor and a bunch of graduate students will swarm the CPod offices and rewrite the Unusual but True Story of the ManyPod Migration. Meanwhile, don't underestimate the effects of global warming...
azerdocmom
November 26, 2007, 10:07 PMThank you Steve for letting us in behind the scenes! We know you guys work hard for us all the time, but we feel extra loved when you become this intimate : ) I for one don't understand all the technical lingo, but have trusted all along in the integrity of the CPod team. That eventually, all the bugs would be worked out. It just takes some time. Rome was not built in one day, so they say. BIG HUGS to all you guys who probably haven't seen the back of your eyelids very much this past week! (OK, now, I can't help myself...) Eat well, eat lots of fruits and vegetables, drink lots and lots of fluids and wash your hands compulsively! And, don't touch things with your bare hands that collect viruses, like bathrooms fixtures and door knobs! Take in lots of vitamin C!
mrdtait
November 26, 2007, 10:46 PMDo ChinesePod have any plans on making any web services available for us to consume in our applications?
segeda
November 28, 2007, 01:11 PMUnderstood....thanks for the explanation and the hard work of you and your colleagues. (I've been in IT for 12 years now...routing/switching/firewalls/vpn....general networking really...so I understand that people tend to "notice" what goes wrong...never really seeing what goes well). Hang in there.
Kyle
November 30, 2007, 12:47 AM*starting to wonder if project Iceberg will ever thaw...* Going on two weeks without being able to study vocabulary has brought on withdraw fits....
John
November 30, 2007, 02:15 AMKyle, The vocab migration is now complete. Is your vocabulary still missing?
Kyle
November 30, 2007, 02:49 AM!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Yes, it's all still missing! If it's all unsalvageable the only choice I have is to re-input all 1400+ entries. And many of those entries were "spontaneous", meaning I don't have them recorded elsewhere, meaning their lost for good. Let me know what I need to do so I can get to work ASAP. Project Iceberg is the bane of my existence!!!!!!!!
phil
November 30, 2007, 03:28 AMJohn My vocab list has 22 items (as it has had for several days). There were originally around 120 I think though my last download from a few months back has 58. Not too important for me, just for feedback.
RonInDC
November 30, 2007, 03:35 AMI lost between 80 and 120 words, but not sure. I can't edit any tags except those that haven't been tagged at all.
AuntySue
November 26, 2007, 07:25 PMBut if the choice was between being the iceberg or being the Titanic, I'm barracking for the iceberg! :-) It makes sense now. I had thought it was a couple of small changes that mushroomed into a major uncontrolled stuffup with a domino effect, but it seems it was the opposite, a lot of simultaneous major planned changes that basically worked but woke up some computer gremlins along the way, which is no surprise really given the magnitude. Big difference. You guys seem to know what you're doing, and now we know too. Steve, thanks for bringing us into the project, and good luck!
Kyle
November 26, 2007, 01:15 PMNice to see what the man (people?) behind the curtain has been up to. Don't know about the others, but about 90% of my CPod usage revolves around vocabulary and its related functions. So, it goes without saying that I haven't been getting any studying done over the last week. Glad to hear that we should be back to normal (hopefully) by the end of the week. Also, and maybe this is just a glitch, but all of the vocab I've added SINCE "Project Iceberg" was instated seems to be have been deleted within the last few hours. Strange, because it was there this morning....