Leo Klarner

leo.klarner [at] stats.ox.ac.uk

I’m a final-year PhD student in the Oxford Protein Informatics (OPIG) and CompStats & Machine Learning (OxCSML) groups. My research focuses on developing more effective deep learning algorithms that are able to leverage scientific prior knowledge and constraints to improve their robustness and generalisation in practical drug discovery settings. Specifically, I’ve worked on using the compositional structure of drug-like molecules to improve the OOD generalisation of molecular property predictors (ICML 2023), as well as generative modelling under domain-informed constraints (NeurIPS 2023).

I feel strongly about understanding the data I work with and have spent many weeks in ChEMBL and PubChem. I am also interested in creating better and more useful datasets and benchmarks (ICML 2023, AI for Chemistry Best Poster Award) and am a main contributor of the open-source Gauche package (NeurIPS 2023).

This work requires familiarity with both modern deep learning research and the realities of practical drug discovery, which I have been fortunate to pick up from my supervisors Yee Whye Teh (Oxford/DeepMind), Charlotte Deane (Oxford) and Garrett Morris (Oxford), as well as Torsten Schindler and Michael Reutlinger (Roche). My work is funded by a Clarendon Scholarship (Oxford’s flagship academic merit scholarship for graduate students) and additional partnership awards from Brasenose College, Oxford and Roche.

Before starting my PhD, I completed a BSc. in Interdisciplinary Sciences (chemistry, biology and CS) at ETH Zürich. During this time, I had the opportunity to design and synthesise antimicrobial peptides with Prof Gisbert Schneider and engineer bacteria for targeted cancer therapy with Prof Simone Schürle-Finke.

Publications

2023

Metropolis Sampling for Constrained Diffusion Models

Nic Fishman, Leo Klarner, Emile Mathieu, Michael Hutchinson, and Valentin De Bortoli

Advances in Neural Information Processing Systems (NeurIPS), 2023

Abs PDF Bib

Denoising diffusion models have recently emerged as the predominant paradigm for generative modelling on image domains. In addition, their extension to Riemannian manifolds has facilitated a range of applications across the natural sciences. While many of these problems stand to benefit from the ability to specify arbitrary, domain-informed constraints, this setting is not covered by the existing (Riemannian) diffusion model methodology. Recent work has attempted to address this issue by constructing novel noising processes based on the reflected Brownian motion and logarithmic barrier methods. However, the associated samplers are either computationally burdensome or only apply to convex subsets of Euclidean space. In this paper, we introduce an alternative, simple noising scheme based on Metropolis sampling that affords substantial gains in computational efficiency and empirical performance compared to the earlier samplers. Of independent interest, we prove that this new process corresponds to a valid discretisation of the reflected Brownian motion. We demonstrate the scalability and flexibility of our approach on a range of problem settings with convex and non-convex constraints, including applications from geospatial modelling, robotics and protein design.
@inproceedings{fishman2023metropolis, title = {Metropolis Sampling for Constrained Diffusion Models}, author = {Fishman, Nic and Klarner, Leo and Mathieu, Emile and Hutchinson, Michael and Bortoli, Valentin De}, author_show = {Nic Fishman and Leo Klarner and Emile Mathieu and Michael Hutchinson and Valentin De Bortoli}, booktitle = {Advances in Neural Information Processing Systems 37}, booktitle_show = {Advances in Neural Information Processing Systems}, booktitle_abbr = {NeurIPS}, year = {2023}, url = {https://openreview.net/pdf?id=jzseUq55eP}, pdf = {https://openreview.net/pdf?id=jzseUq55eP}, bibtex_show = {true}, selected = {true} }
GAUCHE: A Library for Gaussian Processes in Chemistry

Ryan-Rhys Griffiths, Leo Klarner, Henry Moss, Aditya Ravuri, Sang T. Truong, Yuanqi Du, Samuel Don Stanton, Gary Tom, Bojana Ranković, Arian Rokkum Jamasb ... Alpha Lee, Bingqing Cheng, Alan Aspuru-Guzik, Philippe Schwaller, and Jian Tang

Advances in Neural Information Processing Systems (NeurIPS), 2023

Abs PDF Code Bib

We introduce GAUCHE, an open-source library for GAUssian processes in CHEmistry. Gaussian processes have long been a cornerstone of probabilistic machine learning, affording particular advantages for uncertainty quantification and Bayesian optimisation. Extending Gaussian processes to molecular representations, however, necessitates kernels defined over structured inputs such as graphs, strings and bit vectors. By providing such kernels in a modular, robust and easy-to-use framework, we seek to enable expert chemists and materials scientists to make use of state-of-the-art black-box optimization techniques. Motivated by scenarios frequently encountered in practice, we showcase applications for GAUCHE in molecular discovery, chemical reaction optimisation and protein design. The codebase is made available at https://github.com/leojklarner/gauche.
@inproceedings{griffiths2023gauche, title = {{GAUCHE}: A Library for Gaussian Processes in Chemistry}, author = {Griffiths, Ryan-Rhys and Klarner, Leo and Moss, Henry and Ravuri, Aditya and Truong, Sang T. and Du, Yuanqi and Stanton, Samuel Don and Tom, Gary and Rankovi{\'c}, Bojana and Lee, Arian Rokkum Jamasb ... Alpha and Cheng, Bingqing and Aspuru-Guzik, Alan and Schwaller, Philippe and Tang, Jian}, author_show = {Ryan-Rhys Griffiths and Leo Klarner and Henry Moss and Aditya Ravuri and Sang T. Truong and Yuanqi Du and Samuel Don Stanton and Gary Tom and Bojana Rankovi{\'c} and Arian Rokkum Jamasb ... Alpha Lee and Bingqing Cheng and Alan Aspuru-Guzik and Philippe Schwaller and Jian Tang}, booktitle = {Advances in Neural Information Processing Systems 37}, booktitle_show = {Advances in Neural Information Processing Systems}, booktitle_abbr = {NeurIPS}, year = {2023}, url = {https://openreview.net/pdf?id=vzrA6uqOis}, pdf = {https://openreview.net/pdf?id=vzrA6uqOis}, code = {https://github.com/leojklarner/gauche}, bibtex_show = {true}, selected = {true} }
Drug Discovery under Covariate Shift with Domain-Informed Prior Distributions over Functions

Leo Klarner, Tim G. J. Rudner, Michael Reutlinger, Torsten Schindler, Garrett M. Morris, Charlotte Deane, and Yee Whye Teh

International Conference on Machine Learning (ICML), 2023

Abs PDF Code Bib

Accelerating the discovery of novel and more effective therapeutics is an important pharmaceutical problem in which deep learning is playing an increasingly significant role. However, real-world drug discovery tasks are often characterized by a scarcity of labeled data and significant covariate shift—a setting that poses a challenge to standard deep learning methods. In this paper, we present Q-SAVI, a probabilistic model able to address these challenges by encoding explicit prior knowledge of the data-generating process into a prior distribution over functions, presenting researchers with a transparent and probabilistically principled way to encode data-driven modeling preferences. Building on a novel, gold-standard bioactivity dataset that facilitates a meaningful comparison of models in an extrapolative regime, we explore different approaches to induce data shift and construct a challenging evaluation setup. We then demonstrate that using Q-SAVI to integrate contextualized prior knowledge of drug-like chemical space into the modeling process affords substantial gains in predictive accuracy and calibration, outperforming a broad range of state-of-the-art self-supervised pre-training and domain adaptation techniques.
@inproceedings{klarner2023qsavi, title = {{D}rug {D}iscovery {u}nder {C}ovariate {S}hift {w}ith {D}omain-{I}nformed {P}rior {D}istributions {o}ver {F}unctions}, author = {Klarner, Leo and Rudner, Tim G. J. and Reutlinger, Michael and Schindler, Torsten and Morris, Garrett M. and Deane, Charlotte and Teh, Yee Whye}, author_show = {Leo Klarner and Tim G. J. Rudner and Michael Reutlinger and Torsten Schindler and Garrett M. Morris and Charlotte Deane and Yee Whye Teh}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, booktitle_show = {International Conference on Machine Learning}, booktitle_abbr = {ICML}, year = {2023}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, url = {https://proceedings.mlr.press/v202/klarner23a.html}, pdf = {https://proceedings.mlr.press/v202/klarner23a.html}, bibtex_show = {true}, preview = {klarner2023qsavi.png}, code = {https://github.com/leojklarner/Q-SAVI}, selected = {true} }
Diffusion Models for Constrained Domains

Nic Fishman, Leo Klarner, Valentin De Bortoli, Emile Mathieu, and Michael Hutchinson

Transactions on Machine Learning Research (TMLR), 2023

Abs PDF Bib

Denoising diffusion models are a novel class of generative algorithms that achieve state-of-the-art performance across a range of domains, including image generation and text-to-image tasks. Building on this success, diffusion models have recently been extended to the Riemannian manifold setting, broadening their applicability to a range of problems from the natural and engineering sciences. However, these Riemannian diffusion models are built on the assumption that their forward and backward processes are well-defined for all times, preventing them from being applied to an important set of tasks that consider manifolds defined via a set of inequality constraints. In this work, we introduce a principled framework to bridge this gap. We present two distinct noising processes based on (i) the logarithmic barrier metric and (ii) the reflected Brownian motion induced by the constraints. As existing diffusion model techniques cannot be applied in this setting, we proceed to derive new tools to define such models in our framework. We then empirically demonstrate the scalability and flexibility of our methods on a number of synthetic and real-world tasks, including applications from robotics and protein design.
@inproceedings{fishman2023diffusion, title = {Diffusion Models for Constrained Domains}, author = {Fishman, Nic and Klarner, Leo and Bortoli, Valentin De and Mathieu, Emile and Hutchinson, Michael}, author_show = {Nic Fishman and Leo Klarner and Valentin De Bortoli and Emile Mathieu and Michael Hutchinson}, booktitle = {Transactions on Machine Learning Research}, booktitle_show = {Transactions on Machine Learning Research}, booktitle_abbr = {TMLR}, year = {2023}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, url = {https://openreview.net/pdf?id=xuWTFQ4VGO}, pdf = {https://openreview.net/pdf?id=xuWTFQ4VGO}, bibtex_show = {true}, selected = {true} }

2022

Bias in the Benchmark: Systematic experimental errors in bioactivity databases confound
multi-task and meta-learning algorithms

Leo Klarner, Michael Reutlinger, Torsten Schindler, Charlotte Deane, and Garrett Morris

2nd ICML AI for Science Workshop, 2022

Best Poster Award at 5th AI for Chemistry Conference, 2022

Abs PDF Bib

There is considerable interest in employing deep learning algorithms to predict pharmaceutically relevant properties of small molecules. To overcome the issues inherent in this low-data regime, researchers are increasingly exploring multi-task and meta-learning algorithms that leverage sets of related biochemical and toxicological assays to learn robust and generalisable representations. However, we show that the data from which commonly used multi-task benchmarks are derived often exhibits systematic experimental errors that lead to confounding statistical dependencies across tasks. Representation learning models that aim to acquire an inductive bias in this domain risk compounding these biases and may overfit to patterns that are counterproductive to many downstream applications of interest. We investigate to what extent these issues are reflected in the molecular embeddings learned by multi-task graph neural networks and discuss methods to address this pathology.
@inproceedings{klarner2022bias, title = {Bias in the Benchmark: Systematic experimental errors in bioactivity databases confound<br>multi-task and meta-learning algorithms}, author = {Klarner, Leo and Reutlinger, Michael and Schindler, Torsten and Deane, Charlotte and Morris, Garrett}, author_show = {Leo Klarner and Michael Reutlinger and Torsten Schindler and Charlotte Deane and Garrett Morris}, url = {https://openreview.net/pdf?id=Gc5oq8sr6A3}, pdf = {https://openreview.net/pdf?id=Gc5oq8sr6A3}, booktitle_show = {2nd ICML AI for Science Workshop}, year = {2022}, bibtex_show = {true}, selected = {true}, note = {Best Poster Award at 5th AI for Chemistry Conference, 2022} }