We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
lychee - writing a fast, async link checker in Rust - Matthias Endler - Rust Linz August 2023
Learn how to build a fast and scalable link checker in Rust using Lychee, an open-source library that supports asynchronous requests and concurrent parsing, with insights from the talk's presenter, Matthias Endler.
- Writing a link checker in Rust is faster and more scalable than other languages.
- Lychee is an open-source link checker written in Rust, which supports asynchronous requests and concurrent parsing.
- Writing a link checker is important for detecting broken links and preserving knowledge online.
- There are various challenges in writing a link checker, including handling redirects, parsing HTML, and dealing with different types of links (e.g., UI links, email links).
- Lychee uses the Servo browser engine and supports various formats, including HTML, XML, and JSON.
- The library is designed to be fast, scalable, and easy to use, with a focus on solving the problem of broken links.
- Rust is a good choice for writing a link checker due to its performance, memory safety, and concurrency features.
- Some issues with link checkers include bot detection, link rot, and time-consuming parsing.
- Lychee uses a token bucket algorithm to prevent overwhelming servers with requests.
- The library supports multiple formats for output, including JSON and CSV.
- Lychee is used by Google, AWS, Azure, and other companies for their link checking needs.
- The talk title “So you can write a link checker in a weekend” is a joke, and the talk shows how to build a link checker in less than 100 lines of Rust code.