Java, How Fast Can You Parse 1 Billion Rows of Weather Data? • Roy van Rijn • GOTO 2024

Java developer Roy van Rijn presents his GOTO 2024 talk on achieving fast parsing of vast weather data sets, exploring techniques such as memory-mapped files, branchless programming, and native compilation.

Key takeaways

Java’s garbage collection can slow down processing, especially when dealing with large amounts of data.
Using a memory-mapped file can significantly speed up processing.
Branchless programming can be used to optimize code and reduce branches, making it faster.
Using Unsafe class can allow for low-level memory access and manipulation, making it possible to create a hash table without using objects.
Native compilation and binary compilation can further optimize code.
Multiplying every bit by 10 can be used to quickly parse temperature values.
Using a single long value to store the temperature can reduce memory allocation and improve performance.
SWAR (Single Write Array Read) method can be used to optimize code and reduce overhead.
Using JSON-LD for cross-platform communication can be more efficient than SOAP.
Observing the behavior of the JVM can be helpful in understanding how it works.
Using a console app and having it print out the results can be a way to view and test the output.
Debugging and testing is important to ensure the program is working correctly.
Hash collisions can occur and result in slower performance.
The JVM (Java Virtual Machine) is a layer between the Java code and the underlying computer hardware.
Compiler optimization and predictive vectorization can also improve performance.
Using a for loop instead of recursion can sometimes be more efficient.
Using parallel processing can take advantage of multi-core processors.
Java’s Byte Buffer can be used to store and retrieve bytes efficiently.
The JVM can optimize code, but this is not always the case.
The GraalVM is a just-in-time compiler that can run native code.
The Oracle GraalVM is a fixed-size array implementation.
The register allocation refers to how the JVM assigns memory for variables.
The JVM is a layer between the Java code and the underlying computer hardware.
Compiler optimization can improve performance.
The garbage collector should clear up the memory after use.
Un stunned and in agreement with you, you’re my colleague, they’re my colleagues.

Java, How Fast Can You Parse 1 Billion Rows of Weather Data? • Roy van Rijn • GOTO 2024

More talks