Posts

Pitfalls in Kryo Serialization

Image
For one of our projects we serialize information coming from IoT devices into a Kafka topic, which is later processed by several application that aggregate the information for different use cases (e.g. notify user when room temperature has drop below a threshold). In the Java ecosystem we have several tools to perform this serialization: Apache Thrift , Apache Avro , or Google Protocol Buffers to name a few. In this project, though, we decided to use Kryo , which offers another set of tradeoffs than the aforementioned libraries. Along the years we ran into several pitfalls, my hope is that with this blogpost you can avoid making the same mistakes. You can check the companion code to this blogpost here . EDIT: Added pitfall #4 and guidelines Introduction According to Kryo’s GitHub page: Kryo is a fast and efficient binary object graph serialization framework for Java. The goals of the project are high speed, low size, and an easy to use API. The project is useful any time objec
Post a Comment
Read more

Airports at Scale

Image
Assessing NoSQL Databases In a previous post, I explained the assessment we were using at Lunatech, detailing a solution using a common tech stack, Java with Spring Boot and PostgreSQL. An idea mentioned was that it would’ve been possible to also develop a solution using a NoSQL database, since we did receive such solutions, this got me thinking: could every NoSQL family easily solve each requirement? What would be the advantages of using them over SQL? And, more importantly, do they fit in the problem and domain? In this series of posts I want to explore these questions, where each post focuses on each family. Assessment You can take a look a the assessment in the previous post , but as a quick refresher, we had to implement four requirements: Query Option will ask the user for the country name or code and print the airports & runways at each airport. Bonus point, allow fuzzy search.
 Report: 10 countries with highest number of airports (with count) and countries with lowest numbe
Post a Comment
Read more

Senior: To be, or not to be

Image
During a recent interview we touched the concept of seniority, which is a concept that we also discussed at length with several colleagues, and I thought it would be a good time to share my opinion on the subject. SPOILER ALERT: Seniors don’t exist . When reviewing a resume we always wonder, Is she/he a senior or a junior, or maybe something in between? It’s almost inevitable to put tags or labels, we as humans are obsessed with categorical concepts. It’s our way of abstracting the world, how we make sense of it. And as much as I understand the need for such constructs in our daily life, and even if I don’t agree with deconstructing everything, I beg to wonder: What is a senior? How does a person become one? What are its differences with, say, a mid-level? Let’s start with an analogy, I got into weightlifting a few years ago, nothing really serious, just wanted to get in shape. A great beginner program is SL5x5, which yields a linear progression, meaning you lift heavier weights in a l
Post a Comment
Read more

How to Manage External Dependencies

A few months ago we deployed the awesome Scala Steward at one of our current projects, and even if I can’t recommend it enough I have to admit the process of getting all our dependencies up to date was somewhat painful. In this article I would like to share our experience and some tips or guidelines on how to manage external dependencies. These are by no means best practices but just my opinion on the subject (and would like to see how they stand months or years from now). Note: I would like to thank Chris Kipp ( @ckipp01 ) and Martin Kok for providing the proofreading of this article. The image comes from " Epicurious HTML graph " by Noah Sussman is licensed under CC BY 2.0   Project setup Our project is composed of multiple teams with different experience levels, goals, and users. Each team is in charge of several applications or repositories, although ownership is sometimes shared, and like any other project we develop internal libraries to guide or impose a certain cul
Post a Comment
Read more

How to make your own DIY budget gym

Image
Where there is a will there's a way . With the current COVID-19 situation gyms were closed for several months, so as a lot of other people, I’d to find my way around exercising and keeping in shape. My particular situation when all this started was that I was resuming my training after a trip to Argentina, which lasted 3 weeks. During that time I did mostly bodyweight exercises following a similar template as Convict Conditioning . When I came back to NL, gyms closed after one session, and after 2 years of hard work I didn’t want to go back to square one. In this article I’ll describe some of DIY projects and exercises that I did during the COVID-19 lockdown, and also share that you can get a decent workout if you're willing to be flexible. Disclaimer: This article includes DIY equipment, and thus is not meant for heavy duty. You won’t find here the same range as you’ve on a gym either. The resources provided here are meant to be used as guidelines or templates, YMMV. And, I&#
4 comments
Read more

Time range queries with Cassandra and Akka Streams

Image
Apache Cassandra and Akka Streams, a match made in heaven. In this blogpost I hope to explain how the two seamlessly work together by using a real-life example. For a client we'd to use Cassandra as our data store to serve REST requests, one special requirement was that it had to support sorted time range queries over the entire data set or for a specific data producer (in our case, an IoT device recording temperature and atmospheric pressure). Cassandra data modelling is different from Relational modelling, in the former we design our tables around the queries we want to perform disregarding any data normalisation (duplication is in fact encouraged) whereas in the latter we normalise each table, avoid duplication like the plague, and let the DMBS handle how to efficiently query a table. From this fact we must also consider: 1 Queries should target few partitions, only one if possible. Data should be evenly spread across partitions. Now onto the problem presented, let
Post a Comment
Read more

Setting HypriotOS on a Raspberry PI 4 PicoCluster

A few weeks ago I ordered a PicoCluster for an upcoming learning project and blogpost series, to my surprise the entire setup wasn't straightforward. The bundled application set couldn't initially see each other node and neither connect to my Wi-Fi network, and after fiddling with the config I gave up on the default image. Having WiFi and Ethernet is important in the setup because I want download and run docker images, and each node can see each other through LAN. Some weeks before buying the cluster I took an Akka Training at Lunatech  where we saw how a similar cluster was used to show Akka Cluster, and having heard of HypriotOS and how easily it could run docker containers I decided to give a shot. Flashing The first step involves flashing the SD cards with Hypriot image, this guys provide a flash tool with convenient configuration parameters. For this step I'm using release 2.4.0, I'd to flash a card several times until I got the WLAN config correctly, mayb
Post a Comment
Read more