Guillem Solé & Àngel Vilardell – Kafka and ksqlDB at The Hotels Network

Security

Discover how The Hotels Network leveraged Kafka and ksqlDB to transform their data processing, achieving 25-30% faster query times, real-time analysis, and scalable insights.

Key takeaways

KSQL significantly improved the performance of The Hotels Network’s data processing, reducing query times by 25-30%.
The company moved from a batch processing model to a streaming model using Kafka and ksqlDB, which allowed for real-time data processing and analysis.
The use of ksqlDB enabled easier querying and analysis of large datasets, with a single SQL query able to process millions of events.
TinyVid was chosen for its ability to materialize states and perform complex aggregations, such as counting visits to a website.
Red Panda was used to connect to and query the Kafka topic, allowing for efficient extraction of data.
The company had to add ACLs to control access to the TinyVid cluster, as well as ensure that the cluster was properly secured.
The use of Kafka Streams was considered, but the company chose ksqlDB for its ease of use and performance.
The company faced challenges with memory and knowledge of the technology, but was able to overcome these issues with the help of Red Panda.
The company’s data processing pipeline is now more efficient, with fewer issues and faster query times.
The use of ksqlDB and TinyVid has enabled the company to process large datasets in real-time, providing more accurate and timely insights.
The company’s data engineers were able to learn and adapt to the new technology, resolving issues and optimizing the pipeline.
The company’s data processing pipeline is now more scalable, with the ability to handle large amounts of data and multiple concurrent queries.
The use of ksqlDB and TinyVid has enabled the company to integrate with other tools and technologies, such as Datadog and ClickHouse.

Guillem Solé & Àngel Vilardell – Kafka and ksqlDB at The Hotels Network

More talks