We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Francesc Alted - Btune: Making Compression Better | PyData Global 2023
Discover how Bitune, a tool using genetic algorithms, optimizes compression parameters for improved compression speed and ratio, and learn how to balance tradeoffs for your specific use case.
- Bitune: a tool for finding optimal compression parameters based on genetic algorithms.
- Compression time: can be optimized by using chunk-based compression and shuffling data.
- Compression ratio: can be improved by using high-level compression algorithms like Blur Scale Z.
- Neural network models: can be used for automatic guessing of best compression parameters.
- Tradeoff: balancing compression speed and compression ratio can be achieved using the balance mode in Bitune.
- Platform dependence: compression performance can vary greatly depending on the machine and operating system used.
- Memory usage: compressing data in memory can be faster and more efficient than compressing data on disk.
- Variable compression: using different compression algorithms and parameters for different datasets or use cases.
- Genetic algorithm: can be used to fine-tune compression parameters for optimal performance.
- Cache efficiency: compressing data into chunks that fit in caches can improve compression speed.
- Z-standard: a high-level compression algorithm that achieves high compression ratios at the cost of speed.
- LZ4: a fast compression algorithm that is suitable for real-time compression applications.
- Bit shuffle: a filtering technique that can be used to improve compression ratios.
- Chunk-based compression: compressing data in chunks can improve compression speed and reduce memory usage.
- Split mode: can be used to split data into smaller chunks for improved compression speed and reduced memory usage.
- Inference: can be used to predict the best compression parameters for a given dataset.
- Performance mode: can be set to either compression or decompression to optimize for speed or compression ratio.
- Environmental variables: can be used to set compression parameters and modes in Bitune.