Hooncheol Shin The easiest way to collaborate on Jupyter | JupyterCon 2023

Hooncheol Shin

Discover the simplest way to collaborate on Jupyter notebooks with Link Git, enhancing reproducibility, conflict resolution, and standardization for data scientists and machine learning engineers.

Key takeaways
  • Jupyter is a widely used tool for data scientists, but its flexibility leads to diverse user behavior and makes it challenging to handle notebook differences caused by commits.
  • Version control of Jupyter files is difficult due to the JSON format and usage patterns.
  • Collaborating on Jupyter can be problematic, with issues like conflicts, difficulty in reproducing results, and lack of standardization.
  • Introducing pipelines in Jupyter can improve reproducibility and collaboration by assigning identities to cells and allowing for caching of execution results.
  • Link Git is a tool that helps resolve conflicts in the Jupyter editor and provides a way to collaborate seamlessly on Jupyter.
  • The easiest way to collaborate on Jupyter is by using Link Git, which offers features like pipeline building, caching, and remote execution.
  • Machine learning engineers also collaborate extensively on their code and results, and Link Git can help improve their workflow.
  • The goal is to contribute to a more collaborative environment on the Jupyter platform.
  • The speaker suggests using Jupyter only for prototyping machine learning projects, as it doesn’t provide a robust framework for improving code reusability and manageability.
  • The community has been experimenting with text-based notebooks for quite some time, and there is a pull request to enable a text-based format for Jupyter notebooks.
  • Link is a pipeline building tool that provides various features like remote execution, hyperparameter optimization, and parallel execution.
  • The speaker recommends installing Link with the pip install mrx-link command.
  • Link Git is available with the pip install mrx-link command and offers features like pipeline building, caching, and remote execution.
  • The speaker’s company, Makina Rocks, has developed ML products like Runway and Lync, and they are constantly thinking about how to improve collaboration with the realm of MLOps.
  • The speaker’s goal is to propose a way to collaborate seamlessly on Jupyter.

More talks

Stephan Sahm - Jolin.io – Reactive Notebooks for Python Stephan Sahm RailsConf 2023 - 10x your teamwork through pair programming by Michael Milewski, Selena Small Seanina Small Anita Sarma - Keynote: Effective Mentoring Strategies for an Inclusive Community | SciPy 2024 Anita Sarma [VDBUH2024] - Alex Pavăl - Innovation takes courage Alex Pavăl Luciano Resende, Felipe Barros Elyra an AI development workspace based on Jupyter Notebooks Luciano Resende FP & Scala: The Killer Combo for Machine Learning • Marek Kolodziej • YOW! 2015 Marek Kolodziej Right and Responsibilities of a Delivery Team (Sandro Mancuso) Sandro Mancuso Sebastian Ruder | Scaling NLP to the World's Languages | Rise of AI Conference 2022 Sebastian Ruder