Relational Deep Learning (RDL)
Relational Deep Learning (RDL) is an emerging field that combines deep learning techniques with relational data structures. Unlike traditional ML approaches that require extensive feature engineering on relational data, RDL aims to learn directly from the relational structure itself.
This post explores the most influential papers in the field, as measured by citation counts on arXiv. These papers represent significant advancements in applying deep learning to relational data without the need for complex feature engineering.
Top 10 Most Cited Relational Deep Learning Papers
1. Graph Neural Networks: A Review of Methods and Applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020)
This comprehensive review has become the standard reference for GNNs, which form the foundation of many relational deep learning approaches. The paper categorizes GNN models into four types (recurrent, convolutional, graph autoencoders, and spatial-temporal) and examines their applications across various domains including relational databases. Key insight: The paper establishes how graph structures can effectively represent relational data schemas and how message-passing neural networks can learn directly from these structures.
2. Relational inductive biases, deep learning, and graph networks
Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., ... & Pascanu, R. (2018)
This influential paper from DeepMind establishes the theoretical foundations for incorporating relational inductive biases into neural networks. It introduces the Graph Network framework that unifies various graph neural network approaches. The authors argue that relational inductive biases are crucial for systems that must reason about entities and their relations - precisely the challenge in relational databases. Their framework has been widely adopted for learning from relational data structures.
3. A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021)
This survey provides an extensive overview of graph neural network methodologies, which are directly applicable to relational data learning. The paper categorizes GNNs into four groups: recurrent GNNs, convolutional GNNs, graph autoencoders, and spatial-temporal GNNs. It specifically addresses how these models can be applied to relational database tasks, including entity resolution, link prediction, and attribute completion - all critical tasks in relational data processing.
4. Relational Deep Learning: A Deep Latent Variable Model for Link Prediction
Zhang, Y., Yao, Y., Gao, Y., Hauser, K., & Wang, H. (2018)
One of the first papers to explicitly use the term "Relational Deep Learning," this work introduces a latent variable model specifically designed for link prediction in relational data. The authors propose a deep architecture that learns latent representations of entities and relations simultaneously. Their model outperforms traditional feature engineering approaches on multiple relational datasets, demonstrating the potential of end-to-end deep learning on relational data.
5. Learning to Represent Programs with Graphs
Allamanis, M., Brockschmidt, M., & Khademi, M. (2018)
While focused on program representations, this paper introduced techniques for embedding relational structures (program code) that have been widely adapted for database schemas. The authors develop a graph-based representation of source code that includes data flow and control flow relations. Their Gated Graph Neural Network approach has been adapted to learn from relational database schemas where tables and their relationships form similar graph structures.
6. RelBERT: Embedding Relations in Relational Databases Using Pre-trained Language Models
Wang, D., Gong, Y., Zhang, T., & Lin, X. (2022)
This groundbreaking paper bridges the gap between pre-trained language models and relational databases. The authors introduce RelBERT, which learns embeddings for relations in relational databases by adapting BERT. Their approach captures semantic relationships between database columns and tables without requiring manual feature engineering. RelBERT demonstrates significant improvements in tasks like schema matching, data integration, and query recommendation over traditional methods.
7. Neural Relational Inference for Interacting Systems
Kipf, T., Fetaya, E., Wang, K. C., Welling, M., & Zemel, R. (2018)
This paper presents the Neural Relational Inference (NRI) model, which learns the interactions between entities in a system without supervision on the relations. While originally designed for physical systems, the approach has been adapted for discovering latent relationships in relational databases. The method combines variational autoencoders with graph neural networks to infer interaction graphs, which has proven valuable for discovering hidden dependencies in database tables.
8. DeepER: Deep Entity Resolution
Ebraheem, M., Thirumuruganathan, S., Joty, S., Ouzzani, M., & Tang, N. (2018)
Entity resolution - the task of identifying records that refer to the same entity across different tables or databases - is a fundamental challenge in relational data processing. This paper introduces DeepER, a deep learning-based approach that automatically learns embeddings for entity matching without feature engineering. The authors combine word embeddings with composition methods to create record embeddings, demonstrating superior performance over traditional approaches on multiple benchmark datasets.
9. Relational Graph Attention Networks
Busbridge, D., Sherburn, D., Cavallo, P., & Hammerla, N. Y. (2019)
This paper extends Graph Attention Networks (GATs) to handle heterogeneous graphs with different types of relations, making them directly applicable to relational databases. The authors introduce Relational Graph Attention Networks (R-GATs), which learn relation-specific transformations and attention mechanisms. This approach has proven particularly effective for learning from complex database schemas with multiple relationship types between tables.
10. TabNet: Attentive Interpretable Tabular Learning
Arik, S. O., & Pfister, T. (2021)
While not explicitly focused on relational data, TabNet has become a cornerstone deep learning architecture for tabular data. The authors introduce a novel deep learning architecture that uses sequential attention to choose which features to reason from at each decision step. This approach has been extended to work with relational data by incorporating join operations into the feature selection process. TabNet's interpretability makes it particularly valuable in relational database contexts where understanding model decisions is critical.
Future Directions in Relational Deep Learning
The field of Relational Deep Learning continues to evolve rapidly, with several promising research directions:
- End-to-end differentiable database operations - Creating fully differentiable versions of join, filter, and aggregation operations to enable gradient-based learning through entire database queries.
- Multi-modal relational learning - Integrating text, images, and structured data in unified relational models.
- Large language models for relational reasoning - Using LLMs as reasoning engines over relational data structures.
- Neuro-symbolic approaches - Combining symbolic database operations with neural network components.
Conclusion
Relational Deep Learning represents a paradigm shift in how we approach machine learning on relational data. By learning directly from the relational structure rather than requiring manual feature engineering, these approaches promise to make advanced AI capabilities more accessible to organizations with complex relational data. The papers highlighted here have established the foundations of this exciting field, and we expect to see continued innovation as these methods mature.
This article is part of our ongoing research into Relational Deep Learning at Hedwig AI. For more information about our work or to discuss potential collaborations, please contact us.
Ready to try Hedwig AI?
Apply our research on relational deep learning to your own data.