This was my first CodeMesh edition. I had been told to attend several times by my former colleague Félix López and it was really good. Probably the conference with highest technical level I’ve ever attended so far and as such I made lots of notes that you can see right now.
Concurrency + Distribution = Availability + Scalability
This talk was a really useful step by step recipe to distributed systems with a final summary in the form of a 10 bullet points list:
- Split up your system’s functionality into manageable, stand-alone nodes.
- Decide what distributed architectural pattern you are going to use.
- Decide what network protocols your nodes, node families and clusters will use when communicating with each other.
- Define your node interfaces, state and data model.
- For every interface function in your nodes, you need to pick a retry strategy.
- For all your data and state, pick your sharing strategy across node families, clusters and types, taking into consideration the needs of your retry strategy.
- Reiterate through steps 1 to 6 until you have the trade-offs which suit your specification.
- Design you cluster blueprint, looking at node ratios for scaling up and down.
- Identify where to apply back-pressure and load regulation.
- Define your Ops & Management approach, defining system and business alarms, logs and metrics.
You definitely want to have this list close when designing distributed systems, don’t you?
From irrational configuration system to functional data store
This talk was an interesting story about how they migrated a highly complicated and inflexible Oracle data store into a functional one based on Tries. With the migration they achieved, apart from a much more simple and manageable system, an impressive read latency speedup (from more than 15 seconds to less than 10 ms). Two take aways from this talk:
Target your response times below 10 ms.
It is always good to have reference values, this is a good one.
Build tools so that your juniors can work on it.
- You can hire juniors
- You can develop seniors (not only hire them).
Contemporary approach to data at scale
This talk was a really nice journey through all concepts, problems and available solutions for having data at
- Have your indexes in RAM
- Use indexed batches (Cassandra SSTables)
- Brute force approach: Columnar. Save one file per row and compress.
- Consistent hashing: Agents know where the data is stored. Very efficient and scalable.
- Broadcast: Simpler but may cause network overhead when getting bigger.
Architectures (click to enlarge):
Find the slides at benstopford.com
This talk was specially interesting as the usage of mobile sensors to classify user’s activity is something we’re working on at MyDrive as well therefore very interesting takeaways for us here:
- Use OpenCV library for classification.
- Classify data on the phone.
And two ideas to improve the product that are potentially applicable for us at MyDrive as well:
- Have a ML model per user.
- Implement live sessions instead of submitting all the data to the backend at the end.
Beyond Lists: High Performance Data Structures
This talk was a very good and dense one, therefore I’m really looking forward the video to be released, but until then…
Immutability + memory affinity for high speed.
And three new data structures to look at:
Boot my (secure)->(portable) clouds
This talk was really interesting as well because we at MyDrive, as pretty much everyone out there I think, we rely on cloud system providers as Amazon AWS, for our deployments.
The takeaways from this talk are the Cloud computing ASAP principles and three lessons:
- Automate everything
- Separate config from code
- API driven clouds & tools
- Prefer modular, open source tools
Lesson 1: There’s NO single common cloud API, therefore use Terraform
This is something we are already doing, but I also made a note on checking Packer out. A tool to manage base OS images (we currently rely on Ubuntu’s AMIs)
Lesson 2: Decouple infrastructure and configuration management tools using automated configuration management tools, such as Chef, Ansible, …
Again, here we’re doing good at MyDrive using Chef, but also an interesting note on checking FreeIPA out to manage user accounts was made.
Lesson 3: Don’t roll your own security tools, use Vault to manage secrets and sensitive information.
This is again a tool to be checked out, but we, at MyDrive, are already following the rule using Citadel, a tool to store secrets in Amazon S3.
OpenCredo‘s guys have already compiled this whole talk in their own Boot my secure government cloud blog post. Read recommended!!
Reactive Design Patterns
This talk was a good review of concepts and patterns from The Reactive Manifesto
- Responsiveness: The system responds in a timely manner.
- Elasticity: Capability of scaling both up and down, both to meet requirements and keep costs controlled.
- Resilience: The system responds in the presence of failures.
- Message driven: This means that the system is decoupled with a share nothing architecture.
And also some architecture patterns for it:
- Single Responsibility Principle:
- A component must do only one thing and do it completely.
- In order to achieve it, keep splitting responsibilities into nodes until you have single components, but not any further. Avoid trivial components.
- Circuit breaker Pattern:
- Protect services by breaking connection to failing components
- Request – Response Pattern:
- Include a return address in the message in order to receive the response.
- Saga Pattern:
- Divide long-lived and distributed transactions into quick local ones with compensating transactions for failure cases.
- Partition your data so that every operation remains local.
- Example: Bank transfer avoiding lock of both accounts:
- T1: Transfer money from X to a local working account.
- C1: Compensate failure by transferring money back to X.
- T2: Transfer money from local working account to Y.
Well, this was a very interesting closing event, having the chance to hear people with so many years of experience and knowledge as the panelists was a pleasure. I was specially excited to listen to Tony Hoare talking about his story and a little bit of the future.
The panelists were:
As you can imagine I can only advice you to watch the video, but there were two specially interesting ideas:
- A very big improvement for computer science would be to fix the so problematic mutable shared state
- The tools is where the action is: