Schedule

(01/17) Introduction

• Slides

(01/22) Python

• Slides
• Reading: McKinney, Chs. 1 & 2
• Notebook: Download, View

(01/24) Python

• Slides
• Reading: McKinney, Chs. 3
• Notebook: Download, View

(01/29) Relational Databases

• Slides
• References:
  • Database Systems Concepts, Silberschatz, Korth, and Sudarshan

(01/31) Relational Databases

• Slides
• References:
  • Database Systems Concepts, Silberschatz, Korth, and Sudarshan

(02/05) Structured Data

• Slides
• Reading: McKinney, Chs. 4

(02/07) Pandas

• Slides
• Reading: McKinney, Chs. 5 & 6
• Notebook: Download, View

(02/12) Data

• Slides
• Notebook: Download, View

(02/14) Data Wrangling

• Slides
• Reading: Kandel et al.
• References:
  • Original Wrangler Demo
  • Potter’s wheel: An interactive data cleaning system, V. Raman and J. M. Hellerstein, 2001.
  • Proactive Wrangling: Mixed-Initiative End-User Programming of Data Transformation Scripts, P. J. Guo et al., 2011
  • Predictive Interaction for Data Transformation, J. Heer et al., 2015
  • Trifacta Wrangler
  • Self-Service Data Preparation: Research to Practice, J. M. Hellerstein et al., 2018

(02/19) Data Wrangling

• No Lecture
• Complete Pandas Wrangling Courselet (right-click and download)

• Reading:
  • Auto-Transform: Learning-to-Transform by Patterns
• References:
  • Foofah: Transforming Data By Example, Z. Jin et al., 2017
  • Spreadsheet Table Transformations from Examples, W. R. Harris and S. Gulwani, 2011.
  • Transform-Data-by-Example (TDE): An Extensible Search Engine for Data Transformations, Y. He et al., 2018

(02/21) Data Cleaning

• No Lecture
• Complete Data Cleaning Courselet (right-click and download)

• Reading:
  • HoloClean: Holistic Data Repairs with Probabilistic Inference, T. Rekatsinas et al., 2017
• References:
  • SampleClean: Fast and Reliable Analytics on Dirty Data, S. Krishnan et al., 2015
  • Data Cleaning: Overview and Emerging Challenges, X. Chu et al., 2016
  • Python for Data Analysis, Ch. 7

(02/26) Data Wrangling & Data Cleaning

• Slides
• Reading:
  • Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks, C. Yan and Y. He, 2020
• References:
  • Wrex: A Unifed Programming-by-Example Interaction for Synthesizing Readable Code for Data Scientists
  • Spreadsheet Data Manipulation Using Examples, S. Gulwani et al., 2012
  • Tidy Data, H. Wickham, 2014
  • Component-Based Synthesis of Table Consolidation and Transformation Tasks from Examples, Y. Feng et al., 2017
  • Python for Data Analysis, Ch. 8

(03/04) Data Transfomation & Integration

• Slides
• Reading:
  • Data Integration: The Current Status and the Way Forward, M. Stonebraker and I. Ilyas, 2018
• References:
  • Principles of Data Integration, AH Doan et al., 2012
  • Data Fusion—Resolving Data Conflicts in Integration, X. L. Dong and F. Naumann
  • Data Integration and Machine Learning: A Natural Synergy, X. L. Dong and T. Rekatsinas, 2018

(03/06) Data Fusion

• Slides
• Reading:
  • Integrating Conflicting Data: The Role of Source Dependence, Dong et al., 2009
• References:
  • Data Fusion, J. Bleiholder and F. Naumann, 2008
  • Entity Resolution, L. Getoor and A. Machanavajjhala, 2012
  • Record Linkage, P. Christen, 2019
  • Data Fusion—Resolving Data Conflicts in Integration, X. L. Dong and F. Naumann
  • Slides: Integrating Conflicting Data:The Role of Source Dependence, X. L. Dong, 2009

(03/18) Scalable Databases

• Slides
• Reading:
  • NoSQL Database Systems: A Survey and Decision Guidance, F. Gessert et al., 2017.
• References:
  • Cassandra - A Decentralized Structured Storage System, A. Lakshman and P. Malik, 2009.
  • Introduction to Apache Cassandra, R. Stupp
  • Introduction to Cassandra, G. Atil
  • A Survey on NoSQL Stores, A. Davoudian et al., 2020

(03/20) Scalable Databases

• Slides
• Reading:
  • What’s Really New with NewSQL?, A. Pavlo and M. Aslett, 2016
• References:
  • NewSQL, A. Pavlo, 2012.
  • The Official Ten-Year Retrospective of NewSQL, A. Pavlo, 2021.
  • Spanner: Google’s Globally-Distributed Database, J. C. Corbett et al., 2012.
  • F1: A Distributed SQL Database That Scales, J. Shute et al., 2013.
  • Spanner, TrueTime & The CAP Theorem, E. Brewer, 2017.
  • A Critique of the CAP Theorem, M. Kleppmann, 2015
  • Is Scalable OLTP in the Cloud a Solved Problem?, T. Ziegler et al., 2022

(03/25) Scalable Dataframes

• Slides
• Reading:
  • Towards Scalable Dataframe Systems, D. Petersohn et al., 2020
• References:
  • Scaling up your pandas workflows with Modin, D. Petersohn, 2022.
  • Spark SQL: Relational Data Processing in Spark, M. Armbrust et al., 2015
  • Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System, D. Petersohn et al., 2022
  • Blazing fast dataframes in Python with Polars, J. L. C. Rodríguez, 2022
  • Ibis Overview
  • Polars
  • Modin
  • Vaex
  • Dask

(03/27) Scalable Dataframes

• Slides
• Reading:
  • Magpie: Python at Speed and Scale using Cloud Backends, A. Jindal et al., 2021
• References:
  • ConnectorX: accelerating data loading from databases to dataframes, X. Wang et al., 2022
  • Magpie CIDR Slides, A. Jindal, 2021
  • An Opinionated Introduction to Polars, N. Kreiling, 2023
  • Handling Large Data with Polars, E. Bacher.

(04/01) Time Series Data

• Slides
• Reading:
  • Gorilla: a fast, scalable, in-memory time series database, T. Pelkonen et al., 2015
• References:
  • Time Series Data Encoding for Efficient Storage: A Comparative Analysis in Apache IoTDB, J. Xiao et al., 2021
  • Time Series Management Systems: A Survey, S. K. Jensen et al., 2019

(04/03) Graph Data

• Slides
• Reading:
  • The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing, S. Sahu et al., 2017
• References:
  • Introduction to Neo4j and Graph Databases, M. D. Allen, 2019.
  • Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries (preprint), M. Besta et al., 2022.
  • Survey of Graph Database Models, R. Angles and C. Gutierrez, 2008.
  • An Introduction to Graph Data Management, R. Angles and C. Gutierrez, 2017
  • Graph Databases, D. Lembo and R. Rosati, 2015
  • Introduction to Graph Databases, M. De Marzi, 2012
  • The Future Is Big Graphs: A Community View on Graph Processing Systems, S. Sakr et al., 2021
  • The (sorry) State of Graph Database Systems, P. Boncz, 2022

(04/10) Databases and Visualization

• Slides
• Reading:
  • Mosaic: An Architecture for Scalable & Interoperable Data Views, J. Heer and D. Moritz, 2024
• References:
  • imMens: Real-time Visual Querying of Big Data, Z. Liu et al., 2013
  • A Structured Review of Data Management Technology for Interactive Visualization and Analysis, L. Battle and C. Scheidegger, 2021
  • Scalable Visualization, J. Heer, 2020

(04/15) Spatial Data

• Slides
• Reading:
  • Beast: Scalable Exploratory Analytics on Spatio-temporal Data, A. Eldawy et al., 2021
• References:
  • Big Spatial Data Management, A. Eldawy, 2020
  • Data Cubes, J. Han, M. Kamber, and J. Pei, 2011.
  • Nanocubes for Real-Time Exploration of Spatiotemporal Datasets, L. Lins et al., 2013.
  • TopKube: A Rank-Aware Data Cube for Real-Time Exploration of Spatiotemporal Datasets, F. Miranda et al., 2017.
  • Dynamic prefetching of data tiles for interactive visualization, L. Battle et al., 2016.

(04/17) Data Curation

• Slides
• Reading:
  • The FAIR Guiding Principles for scientific data management and stewardship, M. D. Wilkinson et al., 2016.
• References:
  • Big Data, Little Data, No Data: Scholarship in the Networked World, (eBook available through NIU Library) C. L. Borgman, 2015.
  • Research Data Infrastructure: A Problem of Governance, C. L. Borgman, 2022.

(04/22) Provenance

• Slides
• Reading:
  • Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science, A. Chapman et al., 2020
• References:
  • Provenance for Computational Tasks: A Survey, J. Freire et al., 2008
  • Provenance in Databases: Why, How, and Where, J. Cheney et al., 2007
  • Provenance in Databases, A. Amarilli, 2019.
  • Capturing and querying fine-grained provenance of preprocessing pipelines in data science, P. Missier, 2023.
• Notebook: Download, View

(04/24) Reproducibility

• Slides
• Reading:
  • Repeatability in computer systems research, C. Collberg and T. A. Proebsting, 2016.
• References:
  • Repeatability and Benefaction in Computer Systems Research, C. Collberg et al., 2015.
  • Reproducible Research in Computational Science, R. D. Peng, 2011.
  • Ten Simple Rules for Reproducible Computational Research, G. K. Sandve et al., 2013.
  • Computational Reproducibility: State-of-the-Art, Challenges, and Database Research Opportunities, J. Freire et al., 2012.
  • A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks, J. F. Pimentel, 2019.

(04/29) Databases and Machine Learning

• Slides
• Reading:
  • SageDB: A Learned Database System, T. Kraska et al., 2019
• References:
  • Learning and Self-Designing Data Structures (SIGMOD Tutorial), S. Idreos and T. Kraska: Part 1 and Part 2
  • Benchmarking Learned Indexes, R. Marcus et al., 2021
  • Tsunami: A Learned Multi-dimensional Index for Correlated Data and Skewed Workloads, J. Ding et al., 2021
  • Bao: Making Learned Query Optimization Practical, R. Marcus et al., 2021

(05/01) Review

• Slides