At SaleCycle, microservices process data from thousands of web sites in real time. Microservices are great but life becomes difficult when we need to change a service’s API. With 5000 chunky pieces of data flowing into the system every second we can’t just swap the new API in, the existing consumers will break, we’ll lose data, and probably money (I don’t deal with the finances).
How do we sort large data sets when we can’t guarantee that the whole data set will fit into memory? Can we do it in a performant way whilst guaranteeing an upper bound on our memory usage?
One of the great things about deploying in to AWS is the ease of which you can create infrastructure. Whether you’re adding components via the console or using CloudFormation, the speed of which you can define and create the architecture for a solution is one of the fundamental reasons to move from an on premise solution into the Public Cloud.
Here at SaleCycle a large portion of our user base is internal staff. This provides us with an excellent opportunity for easy access into the day to day life of our most frequent users. It also removes any room for excuses for a poor user experience!
Moonfire is an open source Java library for crafting Druid.io queries.
It provides strongly typed query models and result models, enabling you to build up a query and serialise it to the required JSON and to then deserialise Druid’s response into an easily usable POJO.
Good health monitoring is key when using Kinesis Streams for real-time (or close to real-time) processing. Tracking key metrics in CloudWatch during development can illuminate bottlenecks before they become a problem in production. Tracking these metrics in production can give you an early warning before your system is slowed down by a bottleneck.