microsoft/data-formulator 0.6

Data Formulator 0.6: Live Data

on GitHub

⚡ ⚡ Real-time insights from live data

• Connect to URLs and databases with automatic refresh intervals.
• Visualizations update automatically as your data changes to provide you live insights.

Demo: Track International Space Station Position Speed Live (unmute for audio intro)

How to work with live data

When loading data from URLs and databases, enable watch mode so it automatically loads new data when updates are available.

Explore data as you typically do, using agent mode or interactive mode. And enjoy a set of live visualizations that provides you real-time insights as data gets updated. When you create a report from the visualization, you also benefit from the live updates in the report.

To make it easier to play with live features, we created a few examples in the Load Data from URLs section, so you can play with live data easily:

• Stock data (based on yfinance) - Track stock prices with daily or intraday updates, including historical data and financial metrics
• International Space Station (ISS) positions, based on OpenNotify - Real-time tracking of ISS location (updates every 30 seconds)
• Earthquakes, based on USGS - Monitor significant earthquakes worldwide with minute-by-minute updates
• Weather data, based on Open Meteo - Track current weather conditions and forecasts for major US cities (updates daily or hourly)
• Live sales feed (simulated) - Real-time e-commerce sales data for testing and demonstrations (updates every 5 seconds)

🎨 UI Updates

• Unified UI for data loading: Streamlined interface for loading data from various sources
• Direct drag-and-drop fields: Drag fields directly from the data table to update visualization designs

For Developers

Key Code Changes

Frontend:

• New useDataRefresh hook (src/app/useDataRefresh.tsx) manages automatic refresh intervals for stream and database sources
• Extended DataSourceConfig interface with autoRefresh and refreshIntervalSeconds fields
• New Redux action updateTableSourceRefreshSettings for managing refresh settings
• UI components for watch mode configuration in data upload dialogs and table cards
• Removed Concept Shelf component; concepts are now directly shown in the data table for drag-and-drop operations.

Backend:

• New /data-loader/refresh-table endpoint for refreshing database tables with content hash tracking
• Demo stream routes in demo_stream_routes.py for live data examples (stocks, ISS, earthquakes, weather, sales)

Architecture:

• Auto-refresh supports both stream (URL) and database source types
• Frontend controls refresh timing; backend handles data fetching
• Content hash comparison optimizes updates by detecting actual data changes
• No breaking changes; all new fields are optional