Recently we launched a consumer based web-app on a high-traffic channel. It was a 4-day, staying up late till 3-am nerve-wracking experience, with highs and lows.
More so that try to analyze what went wrong, and what went well, I would like to focus more on the journey, the emotions and try to share with everyone, what it feels like launching a product.
We were tasked to build the web-app two months ago, on a “moving target” basis. That means, there was an end goal, but it wasn’t clear what steps had to be taken to get there, due to ever-changing requirements - the goals change every day, sometimes twice a day.
The high level architecture looks like this - there are three separate systems:
We were tasked to build C, and connect them to A and B.
So we started out the development, breezing through the first two months with moderate stress (which includes the daily milestone changes). Everyone thought the product looked good, but it wasn’t tested against real users yet.
3 days before launch, we started getting concerns about the stability of all 3 systems. If any one system goes down, it means a big service disruption to end users. We started thinking about full-on stress tests and how to simulate a real user load on the front page.
I cracked up a simple load test, hammering our web-app with 5000 requests per second for 36 minutes. It was the most expensive load test I ever had to make.
(Whew), the results looked good. We had a failure rate of about 0.02%, a negligable amount. But the loading time still wasn’t good enough. We had to start embarking on using an ajax-based loading system, which promised to cut the loading time by 50%. The thing is, there was no time to complete the ajax loading system by launch day, because we had other issues to handle: more requirements we just piling up on launch day itself.
Come launch day, we went ahead without ajax loading, and it turned out ok. However, a bigger issue occured: there was some miscommunication about a certain mechanic for System C. It caused an overflow of free items given out to end users. About 900 were able to acquire some real items for free within the first 10 minutes of launch. We ended up taking some heat and monetary loss. Ouch.
Day 2 was all disaster management, the first 8 hours. We had to build custom patch, and root out the end users who were deemed worthy of the real items. A lot of customer support work :(
On the 2nd half of Day 2, we scrambled to fix the overflow issue. First test looked promising, but we couldn’t plug the patch into the live version yet, until the next evening. In the meantime, we had to fix other logic issues in the end-consumer templates.
Day 3 of launch, the overflow still managed to happen within the first 10 seconds of a mini-product going live. We figured out that simultaneous writes to the database caused contentions. The server couldn’t take writes of more than 5 per second. We spent the next 8 hours building a sharding counter for our systems, to ensure more efficient writes. 3 am woes. Plus there was more logic errors in the end-user templates (causing some users to view unwanted UI elements). Mini victory: we finally managed to complete the ajax loading system in time for a midnight patch.
Day 4 looked more promising. We had a item come up at 6pm, with 100 people being able to acquire it. Within 2 minutes, it was gone: with zero overflow. Yay! But wait, yesterday’s UI fixes didn’t show. Help! Luckily we found a simple fix on the backend, and the issue was solved within 5 minutes. What a scare!
At the end of Day 4, everyone was weary from the product launch. Because everything finally stabilized (quasi), the team finally tasted some success, and it was fulfilling.
I let everyone leave by 8pm. We’ll see what happens tomorrow (fingers crossed).
MarketJS is a company that leverages HTML5 to provide cross-platform solutions to clients. We have been serving clients since 2012.
MarketJS offers a variety of products and services:
We can be reached at support at marketjs dot com.
"We found MarketJS very easy to work with, and were impressed with how quickly they delivered and responded to our questions."
Rebecca MacDonald, VP of Marketing, Active Navigation
"MarketJS was very easy to work with, and they were always responsive to our questions and requests"
Jeff Robertson, Vice President of Technology & Operations, Carbon8
"MarketJS consistently delivers quality games on time and on budget. They are a great partner."
Scott Sheppard, President and CTO, MoZeus
"We had a great experience working with MarketJS. They were able to move fast on a tight timeline project and deliver exactly what we required with record time and quality. Definitely will be working with them on future projects to help bring our client’s digital needs to life."
Lauren Boutette, Account Director, Lumency
"MarketJS went above and beyond to deliver our projects tailored to our needs. Very swift and quick response and turnaround time. Highly recommended."
Yazan Al Tamimi, Account Manager, Socialize Agency
"Thanks so much for the work your team did. MarketJS was able to work with us to create a custom solution that exceeded our expectations."
Mike Davis, Sales Manager, ComQi
"We are really satisfied with MarketJS. They are quick to help with any troubleshooting and implementation."
Liliya H., Content Specialist, Telecoming
"It was a great pleasure working with MarketJS, they really helped thinking along with our needs. The delivery of the content was always quick and every custom request we had was possible to develop."
Willem van Ditzhuijzen, Business Development, CreativeClicks
"We are extremely happy with the content and service which MarketJS has provided us."
Matt McPherson, Director / Strategic Accounts , Wildtangent
"MarketJS provided excellent support and responded well to ad-hoc requests."
Seah Li-Wei, Project Manager, Spoon Creative
"MarketJS collaborated with our team really well and delivered quality work."
Kevin MacDonald, CTO, TeachTown
"MarketJS has been outstanding; they were patient and very helpful in delivering quality service that matches our custom game requirements."
Marlon Santos, Researcher and Senior Software Engineer, University of Sydney
"We've worked with MarketJS on a number of games now and always find them to be easy and efficient to work with."
Ben Gilbert, Project Manager, Ichi Worldwide
"We are very pleased working with MarketJS, it was far above our expectations in terms of deliverables."
Didier Seevraz, Digital Media Specialist, Lottotech
"MarketJS was fast, responsive and very customer focused. It was a pleasure to work with them."
Bryan Wursten, Senior Digital Editor, Boy Scouts of America
Our team shall be in touch within 12 hours.