Serendipitously named Maynard Webb just stepped into one of the most closely watched positions on the Internet. As the newly appointed chief of eBay technologies, Webb, 43, will take charge of eBay's tech operations, strategy and engineering staff. But his first task is to shore up a system that has shown signs of serious growing pains. Over the summer the auction site suffered a string of outages -- including a disastrous 21-hour shutdown in June -- that enraged users and sent eBay shares tumbling more than 60 percent from their one-year high of $209.25. But the week after eBay announced Webb's appointment on Aug. 9, the stock gained 23 percent.
How did Webb add more than $2 billion to a company's market valuation before taking any action? Well, it may not have been Webb alone -- but his reputation as a seasoned, methodical technology infrastructure and operations expert certainly didn't hurt; he is just the type of taskmaster eBay needs. Webb built an integrated IT system that included enterprise resource planning and e-commerce capabilities for Bay Networks, and orchestrated a shift for disk-drive maker Quantum from a mainframe, closed system to a client-server architecture. Prior to his top tech position at eBay, Webb was chief information officer for PC maker Gateway, where he oversaw e-commerce and other Internet operations.
Webb is well aware of eBay's technology woes, and has plans to fix them. Rather than pointing the finger at hardware and application vendors such as Sun Microsystems or Oracle, though, the unassuming new technology chief hammers away at eBay's need to build out its back-end systems with the same vigor the company originally devoted to designing its front-end applications. It's a situation many Web companies, rushing to launch their own online services, may find uncomfortably familiar.
EBay has attributed its numerous outages this summer variously to hardware, software or network problems. What's going on?
We've had a number of low-level systems problems, no two of them the same. The front-end application itself, the stuff that's real brain surgery, is actually scaling very well and has lots of headroom. But we haven't implemented all of the operations excellence measures we need in the way of hardware redundancy and operational procedures. That has caused us some of the gotchas.
What types of redundancies and procedures are you talking about?
The two longest outages we've ever had, back in June, were tied to issues in the database server. To prevent that from happening again we've purchased another set of Sun Starfire E10000s, essentially replicating our production environment with another whole production environment. We are in what we call warm back-up state right now, where we can switch over to a second database if the main servers fail, and be operational again within a couple of hours. But we're still working toward a high-availability, fully fault-tolerant back-up, where the switch-over is almost instantaneous. That will be in place in the fourth quarter.
Is it possible to create a full duplicate of eBay's system, running at all times, that can automatically take over if the main system goes down?
We essentially have that at the front-end applications server layer. Even if your session and interaction fails, you can switch over to any of the 120 other Windows NT servers were running, and it's transparent to you. What I just described in terms of warm back-up and high availability are very much like having another full back-end, too. The tough part is keeping those two systems totally in sync when you have the speed and the amount of transactions we have on eBay. We have to go through some heroics to make sure we keep both systems totally tuned and ready, but fundamentally that's where we're headed.
Why weren't these steps taken ahead of time, rather than after eBay suffered these outages?
Even though we certainly have stumbled, it is by no means an indictment of the folks who made this site the wild success that it is. I think a lot of the energy and focus was put into developing features, functions and the right kind of application architecture for the front end. Ebay didn't do as stellar a job of building the underlying infrastructure as it needed to. Then things just grew faster on the applications side than the infrastructure was able to keep up with.
Frankly, if they'd done a lot of work on making sure the site could handle anything, and not done that applications work, I don't think they would have had the success they've had. And if the application itself was in trouble and breaking all the time, our cycles to recover would be a lot longer than just figuring out the right kind of redundancies to implement.
EBay already has two other technology executives, CIO Bob Quinn and CTO Mark Ryan. Why did the company need to bring in another technologist?
EBay is a technology company, period. The end. Our product is based entirely on making sure we have the underpinnings of a great system and a great series of processes. In a traditional computing environment you know where your inputs are coming from and you can control them. But in the Web world, anybody can touch you from anywhere. It's just some real hard work to get all these layers of technology synchronized and scaleable. I believe the skill-set they wanted from me, more than anything else, is my "been there, done it" experience with operational excellence in a high-volume, traditional environment, along with a knowledge of what's happening in the Web world.
How much more complicated is it to manage something like eBay's system, with its millions of auctions and users and hundreds of unique features, compared to a more standard e-commerce site such as Gateway?
Gateway's system actually deals with far more complexity than eBay's, in that it must be able to configure a PC that has 2000 different parts on the fly, and then actually produce and ship out 25,000 of those orders a day. It has to tie together manufacturing systems, shop-floor automation, supply-chain management and all that good stuff. Asking your Web system to handle a product directly configurable to each customer's specifications, and do that flawlessly, is not trivial by any stretch of the imagination.
The difference with eBay is that you get simpler transactions, but the volume of interactions is much higher and they come in bursts. The system has to work in real time with no tolerance for error. Just take the example of purchasing a PC online. The Web system has to be full time and ready to go when you enter your order, but then what are you interested in? You want to track your order through the system and know when you're going to get it. Those iterations of customer updates are infrequent, because there are only so many stages in the shipping process. You can have batch processes to handle those interactions on a regular basis and they look like they're working in real time.
But with an auction, you want to know every second when somebody has outbid you. Those have to be real-time interactions, and the volume of them is what separates us from the rest of the Internet pack in terms of complexity.
What attracted you to this new position?
I had a great job at Gateway, and they're an Internet savvy company with a whole heck of lot more complex issues to solve. But eBay is an absolute market leader, crafting out an entirely new business model that I just fundamentally believe in, where people get to decide how world is going to work from a commerce standpoint. To be a part of the company at this stage, and to clearly contribute to taking one of its most critical hurdles to success off the table was too great an opportunity to pass up.
Can eBay or any other Web company ever guarantee 100-percent reliability?
If anyone ever said their site was 100 percent reliable I wouldn't trust them. That's nirvana. But the answer is yes, if you don't grow and don't scale. If you stop your growth and you spend an unbelievable amount of money you can make anything almost perfect. The tricky equation is how to deal with growth and money and time elements, and get them all working in harmony. You've got a whole host of things inside that can happen everyday.
I won't guarantee you eBay will never go down again. The worst thing we can do is give people a false sense of security. But I will guarantee that were putting a system in place that will make for longer periods of time between outages, and shorter outages when they do happen. We've got to build in the redundancy and the reliability to handle the Internet's unique volume and interaction characteristics, and that's an art rather than a science today.
So, have you bought anything on eBay yet?
Absolutely. I collect sports memorabilia, but I haven't found a ton of that because every time I go searching I end up with my kids looking for Beanie Babies. In fact, one of the ways we got our kids excited about moving up here from San Diego was by bribing them: Instead of an allowance they're going to get a certain amount of money to spend each month on eBay. So I expect to be looking for a lot more Beanie Babies.
Shares