- Instructor
- Jiayun (Peter) Wang
pjwang@gatech.edu - Teaching assistant
- Wentao Zhou
wentao@gatech.edu - Meetings
- Monday / Wednesday, 11:00 am – 12:15 pm
- Location
- Skiles 202
- Office hours
- Wednesday after class
- CRN
- 35844 · 3 credit hours
- Canvas
- TBA
Overview
This course studies how machine learning systems represent, integrate, and generate information across heterogeneous modalities, including text, images, video, time series, and scientific measurements. Topics include multimodal Transformers and modality-specific encoders; self-supervised representation learning and cross-modal alignment; autoregressive generation, diffusion, continuous and discrete flow matching; and robust learning with incomplete or noisy observations. We also introduce geometric and physical inductive biases, neural operators, and applications in science, medicine, and agentic systems, including computer-use and embodied agents.
The course begins with instructor-led foundations and then runs primarily as a graduate-led research paper seminar, complemented by a semester-long team research project. There are no standalone graded programming assignments and no written exams.
Prerequisites
Machine learning or deep learning (e.g., CS 7643 or equivalent), linear algebra, probability, and basic Python/PyTorch. Prior coursework in medical imaging, physics, or robotics is not required.
Learning objectives
By the end of the course, students should be able to:
- Explain Transformer attention, tokenization, positional embeddings, and modality-specific representations for text, images, video, and other signals.
- Compare self-supervised representation, alignment, fusion, and missing-modality approaches.
- Explain autoregressive, diffusion/score-based, and continuous/discrete flow-based generation, including conditioning and evaluation.
- Identify when geometric/physical priors, neural operators, and inverse-problem formulations are useful.
- Understand the basic agent loop, harness/tools, memory, self-improvement, and how multimodal inputs support computer-use or robotic agents.
- Design, execute, and clearly present a reproducible team research project, with rigorous research synthesis.
Format
- Weeks 1–3: instructor-led foundations. Core concepts, expectations, and how to critically read and present papers. Students sign up for presentation topics and form project teams. Monday, Jan 25 is the final instructor-led session; Wednesday, Jan 27 is the first student-led paper session.
- Weeks 4–13: student-led paper sessions on both Monday and Wednesday, except Wednesday, Mar 3 (midterm project presentations). Students explain the methods, derivations, evidence, and related work; the instructor moderates, adds context or corrections, and synthesizes key lessons. Unfilled slots, although unlikely, will be taught by the instructor or TA.
- Readings: each student-led session assigns exactly two papers, one foundational and one frontier. Presenters study and explain both; everyone else prepares the comparison for discussion.
- Presentations: one or two students lead each session, and every student presents at least once. Presenters must explicitly compare the two papers rather than deliver two disconnected summaries.
Paper session structure (75 minutes)
- 10 minBig-picture overview — problem, motivation, history, and where the two papers fit
- 15 minFoundational paper — core idea and technical method
- 25 minFrontier paper — what is new, method, experiments, limitations
- 20 minComparison and class discussion
- 5 minInstructor synthesis
This is a suggested allocation, not a rigid slide-by-slide schedule.
Schedule
Paired readings are preliminary and may change as the course progresses; final selections and presenters will be posted on Canvas. For each paper session, the first paper is foundational and the second is a frontier or later development.
| Date | Topic | Foundational paper | Frontier paper |
|---|---|---|---|
| Week 1Foundations: Transformers and modality encoders | |||
| Mon Jan 11 | LectureCourse introduction, attention, Transformer basics | ||
| Wed Jan 13 | LectureImage/video/signal encoders, tokenization, positional and spatiotemporal embeddings; project examples | ||
| Week 2Multimodal inputs and representation foundations | |||
| Mon Jan 18 | No class (MLK Day) | ||
| Wed Jan 20 | LecturePaired/unpaired multimodal data, representation objectives, how to critique and present research papersPaper sign-ups open | ||
| Week 3Representation learning and alignment | |||
| Mon Jan 25 | LectureSelf-supervised, masked/contrastive objectives and cross-modal alignmentFinal instructor-led session | ||
| Wed Jan 27 | PapersCross-modal alignmentFirst student paper session; team, topic and presentation preferences due | CLIP | ImageBind |
| Week 4Cross-modal fusion, robustness, and generative foundations | |||
| Mon Feb 1 | PapersFusion and missing modalities | MulT | Predicting Missing Modalities with Foundation Models |
| Wed Feb 3 | PapersLatent-variable multimodal modelingProject proposal due Fri Feb 5 | Auto-Encoding Variational Bayes | Multimodal Generative Models for Scalable Weakly-Supervised Learning |
| Week 5Autoregressive and token-based generation | |||
| Mon Feb 8 | PapersTokenization and autoregressive generation | VQ-VAE | Taming Transformers |
| Wed Feb 10 | PapersHierarchical image generation | Taming Transformers | VAR |
| Week 6Diffusion and score-based generative modeling | |||
| Mon Feb 15 | PapersDiffusion objectives and backbones | DDPM | Diffusion Transformers (DiT) |
| Wed Feb 17 | PapersLatent diffusion at scale | Latent Diffusion Models | SDXL |
| Week 7Continuous flow matching and stochastic interpolants | |||
| Mon Feb 22 | PapersContinuous-time generative models | Neural Ordinary Differential Equations | Flow Matching for Generative Modeling |
| Wed Feb 24 | PapersInterpolants and rectified flow | Stochastic Interpolants | Scaling Rectified Flow Transformers |
| Week 8Discrete generative models and project midterm | |||
| Mon Mar 1 | PapersDiscrete diffusion and discrete flow | D3PM | Flow Matching with General Discrete Paths |
| Wed Mar 3 | ProjectMidterm project presentations (entire session)Written midterm report due Fri Mar 5 | ||
| Mar 8–12Spring Break | |||
| Mon Mar 8 | No class | ||
| Wed Mar 10 | No class | ||
| Week 9Conditional generation and inverse problems | |||
| Mon Mar 15 | PapersConditional guidance and control | Classifier-Free Diffusion Guidance | ControlNet |
| Wed Mar 17 | PapersInverse problems and posterior sampling | Denoising Diffusion Restoration Models (DDRM) | Diffusion Posterior Sampling |
| Week 10Unified multimodal generation and world models | |||
| Mon Mar 22 | PapersUnified image-text generation | Chameleon | Transfusion |
| Wed Mar 24 | PapersWorld models and action-conditioned generationContrast latent predictive world models with generative interactive environments: model-based planning, need for action data, and long-horizon consistency | World Models | Genie: Generative Interactive Environments |
| Week 11Inductive biases: geometry and physical structure | |||
| Mon Mar 29 | PapersPhysical inductive biases | Physics-Informed Deep Learning | Physics-Informed Diffusion Models |
| Wed Mar 31 | PapersGeometric inductive biases | Tensor Field Networks | Equivariant Diffusion for Molecule Generation in 3D |
| Week 12Scientific and biomedical ML: neural operators and applications | |||
| Mon Apr 5 | PapersNeural operators for scientific fields | Fourier Neural Operator | Geo-FNO |
| Wed Apr 7 | PapersScientific and biomedical multimodal models | Towards Generalist Biomedical AI | Capabilities of Gemini Models in Medicine (Med-Gemini) |
| Week 13Multimodal agents: foundations, self-improvement, and embodied applications | |||
| Mon Apr 12 | PapersAgent loop, tools, and self-improvement | ReAct | MARS |
| Wed Apr 14 | PapersComputer-use agents and embodied agentsRobotic vision-language-action agents (e.g., OpenVLA) covered in the overview; final-project logistics | OSWorld | UI-TARS |
| Week 14Final research project presentations | |||
| Mon Apr 19 | ProjectFinal presentations, session I | ||
| Wed Apr 21 | ProjectFinal presentations, session II | ||
| Week 15Seminar synthesis and project report guidance | |||
| Mon Apr 26 | LectureCourse synthesis, overflow final presentation (if any), optional Q&A on the final report | ||
| Wed Apr 28 | Reading Day, no class | ||
| FinalsFinal project report | |||
| Apr 29 – May 6 | ProjectFinal report and reproducibility package due at the official final-assessment slot (TBD on Canvas) | ||
Grading
| Component | Weight | Expectations |
|---|---|---|
| Participation and discussion | 15% | Prepared engagement, critical questions, constructive peer feedback (not attendance alone) |
| Research paper critiques | 20% | Six short analytical reviews of assigned paper pairs (~1 page each): foundational contribution, frontier advance, evidence, limitations |
| Paper presentation | 15% | Big-picture overview, foundational paper, frontier paper, and a direct comparison (usually presented in pairs); evaluated on depth, synthesis, clarity and discussion leadership |
| Semester team project | 50% | Proposal 5%; midterm presentation and report 5%; final presentation 10%; final report and reproducibility 30% |
| Total | 100% | No separate programming assignments; no written exams |
Project
All students complete a team research project, normally with 3–4 students per team (two by permission). Every team should state a focused question and a credible baseline, show experimental evidence, and document reproducibility. Negative findings are acceptable if well investigated. Students are expected to demonstrate research-level synthesis, critical analysis, and strong baseline and ablation design. Project code is not a separate programming assignment.
Milestones
| Milestone | Date | Weight | Deliverable |
|---|---|---|---|
| Team formation and topic | Wed Jan 27 | Ungraded | Team roster (normally 3–4), question/topic, preliminary dataset and model plan |
| Project proposal | Fri Feb 5 | 5% | 1–2 pages: motivation, hypothesis, related work, datasets, baselines, evaluation plan, realistic compute and risks |
| Midterm project presentation | Wed Mar 3, in class | 2.5% | Working baseline, preliminary evidence, obstacles, next experiments; about 5 minutes plus 1–2 minutes of questions per team |
| Midterm progress report | Fri Mar 5 | 2.5% | ~2–3 pages: method, reproducible preliminary results, error analysis, remaining plan |
| Draft results check-in | Fri Apr 16 | Ungraded | Optional project clinic: plots or tables and presentation outline |
| Final project presentation | Mon Apr 19 / Wed Apr 21 | 10% | One assigned in-class slot per team: method, quantitative evidence, limitations, team contributions |
| Final report and reproducibility | Official final-assessment slot (TBD) | 30% | Conference-style report: comparisons/ablations or error analysis, code/data setup, limitations, contribution statement |
Possible projects
- Robust representation with missing modalities
- Autoregressive vs. discrete diffusion for image/text tokens
- Conditional diffusion/flow for medical measurements or scientific inverse problems
- Physics-aware scientific signal modeling
- Tool-using multimodal agents
- Self-improvement strategies with held-out evaluation
Policies
- Slack days: each student has 5 slack days in total for all written assignments (paper reviews and project reports).
- Final assessment: the final project report is the alternative final assessment, due at the course's official final-assessment time.
- Academic integrity: critiques and presentations should critically analyze the evidence and be students' own work. All external datasets, figures, code, pretrained models, and substantive AI assistance must be credited or disclosed. Students are responsible for understanding and explaining submitted results.
- The official grading scale, attendance, late submission, accessibility, integrity, and accommodations policies will be posted on Canvas.