Comments - More 9's and fewer 0's please - TechHui2024-03-29T00:36:07Zhttp://www.techhui.com/profiles/comment/feed?attachedTo=1702911%3ABlogPost%3A109906&xn_auth=noDespite the bad luck, I think…tag:www.techhui.com,2012-07-07:1702911:Comment:1103892012-07-07T16:38:09.306ZCameron Souzahttp://www.techhui.com/profile/CameronSouza
<p>Despite the bad luck, I think Amazon's ability to recover quickly was impressive. They are still <em>much</em> better than nearly all on-site solutions I've seen.</p>
<p>Despite the bad luck, I think Amazon's ability to recover quickly was impressive. They are still <em>much</em> better than nearly all on-site solutions I've seen.</p> Alex, my intention is not to…tag:www.techhui.com,2012-07-06:1702911:Comment:1103842012-07-06T23:23:22.982ZPaul Graydonhttp://www.techhui.com/profile/Paul76
<p>Alex, my intention is not to be hard on NoSQL, it was more just an observation from the fallout of last Friday.</p>
<p>There are all sorts of good uses for NoSQL stores, and they're being used in the right way all over the place. RDBMSs should never be seen as the be-all and end-all of data storage, like they were for a while.</p>
<p>The problem is people keep using them wrong, too, and they seem to get all vocal and rant a lot when they suddenly get hurt. There was a whole bunch of that…</p>
<p>Alex, my intention is not to be hard on NoSQL, it was more just an observation from the fallout of last Friday.</p>
<p>There are all sorts of good uses for NoSQL stores, and they're being used in the right way all over the place. RDBMSs should never be seen as the be-all and end-all of data storage, like they were for a while.</p>
<p>The problem is people keep using them wrong, too, and they seem to get all vocal and rant a lot when they suddenly get hurt. There was a whole bunch of that on Friday & Saturday.</p>
<p>MongoDB is the biggest target because it seems to have picked up an almost cultish following. People have been switching to it when their RDBMS is slow, instead of either tuning their RDBMS or considering if they're using the RDBMS incorrectly. It's quick and simple to set up MongoDB, particularly if you don't bother to learn much about it, or learn how to use it correctly.</p>
<p>Worse there has been an almost ongoing e-peen measuring contest as to how fast they're able to insert data (there is an awesome BLACKHOLE engine in MySQL that's arguably faster ;)), rather than paying attention to the important things too like reliability. Folks like those at Basho are doing great stuff with Riak for its scalability and reliability. It's dead simple to scale a Riak cluster, especially in comparison to typical RDBMS setups.</p>
<p>Like the same argument I'll make in response to people who insist on the stupid Windows/OS X/Linux/Whatever. Use the right tool for the right job :)</p> Joseph++. Its nothing to do w…tag:www.techhui.com,2012-07-06:1702911:Comment:1105422012-07-06T20:32:21.558ZCalin Westhttp://www.techhui.com/profile/CalinWest
<p>Joseph++. Its nothing to do with SQL vs. no-SQL. Its ACID vs. non-ACID. There is a place for ACID and a place for performance optimized non-ACID databases (logging, caching, reporting, etc.) There is no golden hammer.</p>
<p>Joseph++. Its nothing to do with SQL vs. no-SQL. Its ACID vs. non-ACID. There is a place for ACID and a place for performance optimized non-ACID databases (logging, caching, reporting, etc.) There is no golden hammer.</p> Hello Alex, good to hear from…tag:www.techhui.com,2012-07-06:1702911:Comment:1102632012-07-06T08:06:30.960ZJoseph Luihttp://www.techhui.com/profile/JosephLui448
<p>Hello Alex, good to hear from you. I'm not going to split hairs about what NoSQL has come to mean, but there are certainly ACID-compliant data stores with no SQL involved. What I think Paul was trying to say (and I agree with) is that the pendulum swung a bit too far away from SQL recently. Reliability with your datastore is so important that with most applications you want to start with it first, and relax it if you need to based on requirements, not the other way around. After something…</p>
<p>Hello Alex, good to hear from you. I'm not going to split hairs about what NoSQL has come to mean, but there are certainly ACID-compliant data stores with no SQL involved. What I think Paul was trying to say (and I agree with) is that the pendulum swung a bit too far away from SQL recently. Reliability with your datastore is so important that with most applications you want to start with it first, and relax it if you need to based on requirements, not the other way around. After something like a power outage, you want those guarantees afforded to you by ACID-compliance, and not a contaminated DB with countless half-committed transactions. That being said, probably neither my original post nor Paul's comments were meant to primarily be about ACID databases. So yes we should bring the focus back to how a large provider like Amazon AWS should stay vigilant minimizing single points of failure and historically demonstrated bottlenecks, plus maximizing fault tolerance, disaster recovery, transparency, and overall QoS.</p> Also, I think you guys are ma…tag:www.techhui.com,2012-07-06:1702911:Comment:1104482012-07-06T05:20:42.809ZAlex Salkeverhttp://www.techhui.com/profile/AlexSalkever
<p>Also, I think you guys are maybe being a bit too hard on NoSQL. It opens up a lot of possibilities for scalable architectures that would simply not work on ACID DBs without a massive infrastructure investment / or some really fancy coding. There are some HUGE telcos running Riak and Mongo, for example, in critical applications or business analytics.</p>
<p>That said, the next gen of NewSQL platforms could be insanely good and offer both ACID and the type of speed and easy scalability that…</p>
<p>Also, I think you guys are maybe being a bit too hard on NoSQL. It opens up a lot of possibilities for scalable architectures that would simply not work on ACID DBs without a massive infrastructure investment / or some really fancy coding. There are some HUGE telcos running Riak and Mongo, for example, in critical applications or business analytics.</p>
<p>That said, the next gen of NewSQL platforms could be insanely good and offer both ACID and the type of speed and easy scalability that NoSQL offers.</p> Joseph, if you dig deep into…tag:www.techhui.com,2012-07-06:1702911:Comment:1099332012-07-06T05:05:57.981ZAlex Salkeverhttp://www.techhui.com/profile/AlexSalkever
<p>Joseph, if you dig deep into the AWS explanation to customers, you see that the outage also caused software problems on AWS not related to the power outage. Granted, shutting down a massive chunk of their infrastructure caused the problem, but folks who wanted to move their instances to other zones could not do so because they had to go through key components on EBS hosted in Amazon East Zone. I haven't seen it covered. I asked reporters about it and they did not get an answer out. So this…</p>
<p>Joseph, if you dig deep into the AWS explanation to customers, you see that the outage also caused software problems on AWS not related to the power outage. Granted, shutting down a massive chunk of their infrastructure caused the problem, but folks who wanted to move their instances to other zones could not do so because they had to go through key components on EBS hosted in Amazon East Zone. I haven't seen it covered. I asked reporters about it and they did not get an answer out. So this may also be less about ACID and DBs like Mongo and more about layers of abstraction that fail poorly, as well as system architectures that have persistent single points of failure. </p> Paul Graydon: ...(prime among…tag:www.techhui.com,2012-07-03:1702911:Comment:1099092012-07-03T03:23:36.947ZDaniel Leuckhttp://www.techhui.com/profile/dleuck
<blockquote><p>Paul Graydon: ...(prime amongst most of the anguish seems to be MongoDB) aren't all they're cracked up to be.</p>
</blockquote>
<p>Very true. MongoDB has its place in areas such as logging and caching, but right now, for many developers, its a golden hammer and everything looks like a nail.</p>
<p></p>
<blockquote><p>Paul Graydon: ...(prime amongst most of the anguish seems to be MongoDB) aren't all they're cracked up to be.</p>
</blockquote>
<p>Very true. MongoDB has its place in areas such as logging and caching, but right now, for many developers, its a golden hammer and everything looks like a nail.</p>
<p></p> Ah cool, one non-EC2 Amazon s…tag:www.techhui.com,2012-07-02:1702911:Comment:1102342012-07-02T06:27:46.907ZJoseph Luihttp://www.techhui.com/profile/JosephLui448
<p>Ah cool, one non-EC2 Amazon service that made me happy through the storm was RDS. I checked a Multi-AZ instance during that time and saw that it indeed had automatically flipped its IP to the replicant.</p>
<p>Your last point is a good one. Infrastructure reliability has two basic dimensions: uptime and data integrity--of course ongoing data integrity, but especially post-disaster data integrity. The only common acceptable use I can think of off the top of my head for a non-ACIDic database…</p>
<p>Ah cool, one non-EC2 Amazon service that made me happy through the storm was RDS. I checked a Multi-AZ instance during that time and saw that it indeed had automatically flipped its IP to the replicant.</p>
<p>Your last point is a good one. Infrastructure reliability has two basic dimensions: uptime and data integrity--of course ongoing data integrity, but especially post-disaster data integrity. The only common acceptable use I can think of off the top of my head for a non-ACIDic database is as a performance optimizing cache, where cache misses automatically cascade to the underlying ACIDic persisted store. That and casual games which involve a lot of users and no real money.</p> Netflix are probably one of t…tag:www.techhui.com,2012-07-01:1702911:Comment:1100172012-07-01T22:52:27.777ZPaul Graydonhttp://www.techhui.com/profile/Paul76
<p>Netflix are probably one of the biggest poster childs of the Amazon EC2 infrastructure. Their entire infrastructure runs on it.If you're interested in cloud/Amazon stuff from an operations perspective, follow <a href="https://twitter.com/#" target="_blank">https://twitter.com/#</a>!/adrianco and keep track of their tech blog: <a href="http://techblog.netflix.com/" target="_blank">http://techblog.netflix.com/</a></p>
<p></p>
<p>Amongst the things that they do to ensure up-time is run a piece…</p>
<p>Netflix are probably one of the biggest poster childs of the Amazon EC2 infrastructure. Their entire infrastructure runs on it.If you're interested in cloud/Amazon stuff from an operations perspective, follow <a href="https://twitter.com/#" target="_blank">https://twitter.com/#</a>!/adrianco and keep track of their tech blog: <a href="http://techblog.netflix.com/" target="_blank">http://techblog.netflix.com/</a></p>
<p></p>
<p>Amongst the things that they do to ensure up-time is run a piece of software called Chaos Monkey, which will randomly shut down parts of their production infrastructure to help them identify any single points of failure. Their infrastructure is designed to continue to operate even when components die, such as search or the recommendation engine. They have extensive monitoring and automated systems in place to automatically handle any failure condition.</p>
<p></p>
<p>Good chunks of Netflix went down hard on Friday night. Those in the other AZs were okay, but ye<em>t again</em> Amazon's back-plane proved incapable of handling the incoming requests. Netflix were sending requests to AWS to change the IPs on the Elastic Load Balance service to remove those in the troubled zone but found it to be stuck, completely unresponsive.</p>
<p>With almost every major problem that's happened in the last few years in their infrastructure, Amazon has been unable to cope with the in-bound provisioning or reconfiguration requests and similar, to the point where the back-plane has become completely unresponsive. Worse they've made no attempt to provide any real reassurance of intent to increase the capacity.</p>
<p>Amongst some of the debate back and forth on twitter was discussions about relying on Amazon's non-EC2 services. It's nice and attractive to be able to do your DNS (Route 53), Load balancing (ELB) and persistent storage (EBS) with them, but if you can't change them whenever, wherever, regardless of the circumstances you're a little screwed :)</p>
<p></p>
<p>What I found a little amusing in an admittedly smug and arrogant way was the number of people who suddenly discovered the importance of ACID with their databases, and why some of these newer DB stores (prime amongst most of the anguish seems to be MongoDB) aren't all they're cracked up to be. Speed is one thing, data reliability is ultimately more important though.</p>