Stuff The Internet Says On Scalability For March 3rd, 2017

Hey, it’s HighScalability time:

Only 235 trillion miles away. Engage. ( NASA )

If you like this sort of  Stuff then please support me  on Patreon .

  • $5 billion : Netflix spend on new content; $1 billion : Netflix spend on tech; 10% : bounced BBC users for every additional second page load;  $3.5 billion : Priceline Group ad spend;  12.6 million : hours streamed by Pornhub per day; 1 billion : hours streamed by YouTube per day;  38,000 BC : auroch carving; 5% : decrease in US TV sets;
  • Quotable Quotes:
    • Fahim ul Haq : Rule 1: Reading High Scalability a night before your interview does not make you an expert in Distributed Systems.
    • @Pinboard : Root cause of outage: S3 is actually hosted on Google Cloud Storage, and today Google Cloud Storage migrated to AWS
    • Matthew Green : ransomware currently is using only a tiny fraction of the capabilities available to it. Secure execution technologies in particular represent a giant footgun just waiting to go off if manufacturers get things only a little bit wrong.
    • dsr_ : This [S3 outage] is analogous to “we needed to fsck, and nobody realized how long that would take”.
    • tptacek : Uber isn’t the driver’s employer. Uber is a vendor to the driver. The driver is complaining that its vendor made commitments, on which the driver depended, and then reneged. The driver might be right or might be wrong, but in no discussion with a vendor in the history of the Fortune 500 has it ever been OK for the vendor to accuse their customer of “not taking responsibility for their own shit”.
    • @felixsalmon : Hours of video served per day: Facebook: 100 million Netflix: 116 million YouTube: 1 billion
    • @Geek_Manager : “Everybody wants to write reusable code. Nobody wants to reuse anyone else’s code.” @eryno #leaddev
    • @ellenhuet : a private South Bay high school 1) having a growth fund and 2) being early in Snap is the most silicon valley thing ever
    • @_ginger_kid : I speak from experience as a cash strapped startup CTO. Would love to be multi region, just cannot justify it. V hard.
    • @Objective_Neo : SpaceX, $12 billion valuation: Launches 70m rockets into space and lands them safely. Snapchat, $20 billion valuation: Rainbow Filters.
    • @neil_conway : (2/4): MTTR (repair time) is AT LEAST as important as MTBF in determining service uptime and often easier to improve.
    • John Hagel : we’re likely to see a new category of gig work emerge – let’s call it “creative opportunity targeting.”…we anticipate that more and more of the workforce will be pulled into this arena of creative gig workgroups
    • Seyi Fabode : The constraint is that the broker model, even with new technology, is not value additive. 
    • Robert Kolker : From his experience with the Gary police, Hargrove learned the first big lesson of data: If it’s bad news, not everyone wants to see the numbers
    • gamache : A piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You’re signing up for the oldest hardware and the most frequent outages.
    • Dan Sperber : we each have a great many mental devices that contribute to our cognition. There are many subsystems. Not two, but dozens or hundreds or thousands of little mechanisms that are highly specialized and interact in our brain. Nobody doubts that something like this is the case with visual perception. I want to argue that it’s also the case for the so-called central systems, for reasoning, for inference in general.
    • Joaquin Quiñonero Candela : Facebook today cannot exist without AI. Every time you use Facebook or Instagram or Messenger, you may not realize it, but your experiences are being powered by AI.
    • alicebob : Sometimes keeping things simple is worth more than keeping things globally available.
    • Sveta Smirnova : Both MySQL and PostgreSQL on a machine with a large number of CPU cores hit CPU resources limits before disk speed can start affecting performance.
    • @jamesiry : Using many $100,000’s of compute, Google collided a known weak hash. Meanwhile one botched memcpy leaked the Internet’s passwords.
    • @david4096 : teaching engineers to say no is cheaper than Haskell
    • @cgvendrell : #AI will be dictated by Google. They’re 1 order of magnitude ahead, they understood key = chip level of stack (TPU) + training data @chamath
    • @antirez : There are tons of more tests to do, but the radix trees could replace most hash tables inside Redis in the future: faster & smaller.
    • DHH : So it remains mostly our fault. Our choice, our dollars. Every purchase a vote for an ever more dysfunctional future. We will spend our way into the abyss.
    • @jamesurquhart : This is why I write about data stream processing and serverless—lessons I learned at @SOASTAInc about the value of real time and BizOps.
    • twakefield : The brilliance of open sourcing Borg (aka Kubernetes) is evident in times like these. We[0] are seeing more and more SaaS companies abstract away their dependencies on AWS or any particular cloud provider with Kubernetes.
    • flak : password hashes aren’t broken by cryptanalysis. They’re rendered irrelevant by time (hardware advancements). What was once expensive is now cheap, what was once slow is now fast. The amount of work hasn’t been reduced, but the difficulty of performing it has.
    • @darkuncle : biz decisions again … gotta weigh cost/frequency of AWS single-region downtime vs. cost/complexity of multi-region & GSLB.
    • @nantonius : Reducing network latencies are a key enabler for moving away from monolith towards serverless. @adrianco:
    • tbrowbdidnso : These companies that all run their own hardware exclusively are telling everyone that it’s stupid to run your own hardware… Why are we listening?
    • jasonhoyt : “People make mistakes all the time…the problem was that our systems that were designed to recognize and correct human error failed us.” 
    • @chuhnk : Bob: Service Discovery is a SPOF. You should build async services. Me: How do you receive messages? Bob: A message bus Me: …
    • @JoeEmison : These articles on serverless remind me of articles on NoSQL from a few years ago. FaaS may have low adoption b/c of the req’d architectures.
    • @Jason : We have 30-60% open rates for  emails vs 1% for app downloads!
    • @adrianco : Split brain syndrome: half your brain thinks message buses are reliable. Other half is wondering how to recover from split brain syndrome.
    • @dbrady : The older I get, the less I care about making tech decisions right and the more I care about retaining the ability to change a wrong one.
    • @littleidea : “Automation code, like unit test code, dies when the maintaining team isn’t obsessive about keeping the code in sync with the codebase.”
    • @adulau : I don’t ask for bug bounties, fame, cash or even tshirt. I just want a good security point of contact to fix the issues.
    • StorageMojo : most of the SSD vendors don’t make AFAs [all flash arrays]. They have little to lose by pushing NVMe/PCIe SSDs for broad adoption.
    • cookiecaper : I mean, that’s not really AWS’s problem, is it? Outages happen. If you have a mission-critical service like health care, you really shouldn’t write systems with single points of failure like this, especially not systems that depend on something consumer-grade like S3.
    • plgeek : To me his main point is there is a spectrum of what you might consider evidence/proof. However, in Software Engineering their have been low standards set, and it’s really not acceptable to continue with low standards. He is not saying the only sort of acceptable evidence is a double blind study.
    • n00b101 : I asked an Intel chip designer about this and his opinion was that asynchronous processors are a “fantasy.” His reasoning was that an asynchronous chip would still need to synchronize data communication within the chip. Apparently global clock synchronization accounts for about 20% of the power usage of a synchronous chip. In the asynchronous case, if you had to synchronize every communication, then the cost of communication is doubled.
  • Anti-virus software uses fingerprinting as a detection technique. Surprise, nature got there first.  Update: CRISPR . Bacteria grab pieces of DNA from viri and store them. This lets them recognize a virus later. When a virus enters a bacteria the bacteria will send out enzymes to combat the invader. Usually the bacteria dies. Sometimes the bacteria wins. The bacteria sends out enzymes to find stray viruses and cut the enemy DNA into small pieces. Those enzymes take the little bits of DNA and splice them into the bacteria’s own DNA. DNA is used as a memory device. Next time the virus shows up the bacteria creates molecular assassins that contain a copy of the virus DNA. If there’s a pattern match then kill it. The protein looks something like a clam shell. It has a copy of the virus DNA. Whenever it bumps into some virus DNA it pulls apart the DNA, unzips it, reads it, if it’s not the right one it moves on. If the RNA has the same sequence then molecular blades come out and chop. Like smart scissors. This is CRISPR.
  • Videos from microXchg 2017 are  now available
  • A natural disaster occurred. S3 went down. Were you happy with how your infrastructure responded? @spire was.  Mitigating an AWS Instance Failure with the Magic of Kubernetes : “Kubernetes immediately detected what was happening. It created replacement pods on other instances we had running in different availability zones, bringing them back into service as they became available. All of this happened automatically and without any service disruption, there was zero downtime. If we hadn’t been paying attention, we likely wouldn’t have noticed everything Kubernetes did behind the scenes to keep our systems up and running.” How do you make this happen?: Distribute nodes across multiple AZs; Nodes should have capacity to handle at least one node failure; Use at least 2 pods per deployment; Use readiness and liveness probes.

Don’t miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)…