Recently we launched a consumer based web-app on a high-traffic channel. It was a 4-day, staying up late till 3-am nerve-wracking experience, with highs and lows.
More so that try to analyze what went wrong, and what went well, I would like to focus more on the journey, the emotions and try to share with everyone, what it feels like launching a product.
We were tasked to build the web-app two months ago, on a "moving target" basis. That means, there was an end goal, but it wasn't clear what steps had to be taken to get there, due to ever-changing requirements - the goals change every day, sometimes twice a day.
The high level architecture looks like this - there are three separate systems:
We were tasked to build C, and connect them to A and B.
So we started out the development, breezing through the first two months with moderate stress (which includes the daily milestone changes). Everyone thought the product looked good, but it wasn't tested against real users yet.
3 days before launch, we started getting concerns about the stability of all 3 systems. If any one system goes down, it means a big service disruption to end users. We started thinking about full-on stress tests and how to simulate a real user load on the front page.
I cracked up a simple load test, hammering our web-app with 5000 requests per second for 36 minutes. It was the most expensive load test I ever had to make.
(Whew), the results looked good. We had a failure rate of about 0.02%, a negligable amount. But the loading time still wasn't good enough. We had to start embarking on using an ajax-based loading system, which promised to cut the loading time by 50%. The thing is, there was no time to complete the ajax loading system by launch day, because we had other issues to handle: more requirements we just piling up on launch day itself.
Come launch day, we went ahead without ajax loading, and it turned out ok. However, a bigger issue occured: there was some miscommunication about a certain mechanic for System C. It caused an overflow of free items given out to end users. About 900 were able to acquire some real items for free within the first 10 minutes of launch. We ended up taking some heat and monetary loss. Ouch.
Day 2 was all disaster management, the first 8 hours. We had to build custom patch, and root out the end users who were deemed worthy of the real items. A lot of customer support work :(
On the 2nd half of Day 2, we scrambled to fix the overflow issue. First test looked promising, but we couldn't plug the patch into the live version yet, until the next evening. In the meantime, we had to fix other logic issues in the end-consumer templates.
Day 3 of launch, the overflow still managed to happen within the first 10 seconds of a mini-product going live. We figured out that simultaneous writes to the database caused contentions. The server couldn't take writes of more than 5 per second. We spent the next 8 hours building a sharding counter for our systems, to ensure more efficient writes. 3 am woes. Plus there was more logic errors in the end-user templates (causing some users to view unwanted UI elements). Mini victory: we finally managed to complete the ajax loading system in time for a midnight patch.
Day 4 looked more promising. We had a item come up at 6pm, with 100 people being able to acquire it. Within 2 minutes, it was gone: with zero overflow. Yay! But wait, yesterday's UI fixes didn't show. Help! Luckily we found a simple fix on the backend, and the issue was solved within 5 minutes. What a scare!
At the end of Day 4, everyone was weary from the product launch. Because everything finally stabilized (quasi), the team finally tasted some success, and it was fulfilling.
I let everyone leave by 8pm. We'll see what happens tomorrow (fingers crossed).
|Social Blackjack: Card Game with Friends||02-13-2017|
|Wasteland Warriors: a real time multiplayer game||11-28-2016|
|We built a Google Chromecast game||02-02-2016|
|Rising Cost Per Install In The App Economy||08-24-2015|
|New HTML5 Game: Monsters and Cake||07-24-2015|
|New HTML5 Game: Street Fight||05-15-2015|
|New HTML5 Game: Taxi Pickup||04-18-2015|
|Merchandising done right||02-28-2015|
|Superbowl Ad Recap : Clash of Clans||02-10-2015|
|The Old Game Flow Design||01-29-2015|
|New HTML5 Game: Guess The Celebrity||01-20-2015|
|New HTML5 Game: Leave Me Alone||01-12-2015|
|HTML5 Endless Runner: History and Design Thinking||12-14-2014|
|New HTML5 Game: Snowball Office Fight||11-15-2014|
|Thinking HTML5 with castle walls and moats||11-07-2014|
|Thoughts on Supercell videos||11-05-2014|
|New HTML5 Game: Hidden Objects Pirate Adventures||11-03-2014|
|Would a brand spend $20,000 to acquire 10,000 users?||10-09-2014|
|New HTML5 Game: Banana Jungle||10-08-2014|
|New HTML5 Game: Snowball Fight||10-08-2014|
|New HTML5 Game: Totem Volcano||09-12-2014|
|New HTML5 Game: Shark Attack||07-22-2014|
|Good old times with OpenRA||05-24-2014|
|Announcement: We acquired Artists&Clients||05-16-2014|
|New HTML5 game: Pop Star Dentist 2||05-06-2014|
|Indie Reality Check: Guidelines||03-23-2014|
|New HTML5 game: Slots Beach||01-27-2014|
|HTML5 Game Development Workshop||01-21-2014|
|How we can all learn from Jay-Z||12-22-2013|
|Product launch: Triumphs and Tribulations||12-12-2013|
|3 AM days and ways to avoid them||12-11-2013|
|When the errors keep coming, what do you do?||12-05-2013|
|What a great producer does||12-03-2013|
|Mini-Detachment Experiment 1: Stop reading tech news||11-29-2013|
|New game: Monster Mash||11-23-2013|
|Gmail is winning on mobile||11-20-2013|
|Why software matters||11-06-2013|
|Serendipity as a game mechanic||11-05-2013|
|A blast from the past: Buenos Aires||10-28-2013|
|Upcoming event worth attending||10-28-2013|
|New game: Sector 49||10-25-2013|
|New game: Vampire Dress Up||10-03-2013|
|New game: Kitten Bounce||09-25-2013|
|New game: Ships and Monsters||09-24-2013|
|iOS7 Mobile Safari: Big Branding Opportunity||09-21-2013|
|New game: Mouse and Cheese||09-19-2013|
|New game: Whats My Brand?||09-19-2013|
|New game: Royal Princess Dress Up||09-18-2013|