rohit-db / dais-2023

Data & AI Summit Major Announcements

Demos & Code Examples available in this repo (WIP): 
- Data Engineering 
    - Delta Lake 3.0
    - Spark English SDK
- Data Governance
    - Lakehouse Federation 
- Data Warehousing
    - Streaming Tables and Materialized Views

Generative AI

• LakehouseIQ: The knowledge engine of the Lakehouse that learns what makes your business unique to arm everyone with accurate answers.
• Databricks Assistant: A context-aware collaborator powered by LakehouseIQ that uses natural language to generate reports, generate and explain code, and answer data and code-related questions.
• Lakehouse AI: Quickly, cost-effectively and securely build Generative AI applications.
  • Vector Index: Easily create auto-updating vector search indexes from data in Unity Catalog
  • Model Serving: GPU-enabled, real-time inference of LLMs at up to 10X lower latency and reduced costs
  • Curated Open Source Models, backed by optimized Model Serving for high performance
  • MLflow 2.5: Manage your end-to-end LLM Operations (LLMOps) effectively and reliably

Unity Catalog

• Lakehouse Federation: Discover, query and govern your data no matter where it lives
• AI Governance: Feature Store, Model Registry and Volumes in Unity Catalog
• Lakehouse Monitoring and Observability: Monitor quality and integrity for all your data and AI assets. Billing, audit, lineage and security info as tables for enhanced observability Data Sharing and Collaboration
• Databricks Marketplace: An open marketplace for all your data, analytics, and AI models. Now Generally Available.
• Lakehouse Apps: The most secure way to build, distribute, and run innovative data and AI applications directly on the Databricks Lakehouse.
• Databricks Clean Rooms: Privacy-safe collaboration. Available now in Private Preview on AWS.
• Delta Sharing: Databricks is expanding the Delta Sharing ecosystem with new partners, including Cloudflare, Dell, Oracle and Twilio.

Data Engineering

• Delta Lake 3.0: UniForm provides automatic translation to Apache Iceberg or Apache Hudi to eliminate lakehouse platform incompatibility
• Workflows: Serverless Workflows and enhanced control flow
• Project Lightspeed: Improvements in Performance, enhanced ecosystem and more
• Spark English SDK

Data Warehousing

• Streaming tables provide incremental ingest from cloud storage and message queues.
• Materialized views are automatically and incrementally updated as new data arrives.

Blogs
Introducing LakehouseIQ: The AI-Powered Engine that Uniquely Understands Your Business Lakehouse AI: A Data-Centric Approach to Building Generative AI Applications Introducing Lakehouse Federation Capabilities in Unity Catalog
What’s new with Unity Catalog at Data and AI Summit 2023
What’s New with Data Sharing and Collaboration on the Lakehouse Streaming Tables & Materialized views Project Lightspeed