MarquezProject/marquez 0.50.0

Marquez 0.50.0

on GitHub

Added

• Web: New Data Observability dashboard for stats on OpenLineage events (24hrs, past 7.days); views are also available for sources, datasets, and jobs; new job list view has also been introduced displaying the latest N runs (and duration) for a given job #2913 @phixMe
• Web: 404 page #2890 @phixMe
• Web: Display parent job (if present) in job panel #2868 @phixMe
• Web: Allow override of web.port via WEB_PORT environment variable #2838 @bidlako
• Web: Allow nullable columns for schema in dataset panel (use N/A) #2896 @phixMe
• Web: Better feedback when lineage events are loading #2916 @NisargChokshi45
• API: Job object will now return Job.latestRuns (latest N runs) and Job.latestRun (last run to execute) #2901 @phixMe
• API: Use io.openlineage.server.* pkg and class Metadata (utility class for OpenLineage.RunEvent) #2853 @wslulciuc
• API: Use TIMESTAMPTZ for timestamps in database; supports Data Observability dashboard with timezone of user #2924 @wslulciuc
• API: Set current_run_uuid in table jobs optimizing query for JobDao.findAll() #2929 @wslulciuc
• API: New GET /api/v1/jobs #2930 @wslulciuc
• CLI: New cmd args for cli.MetadataCommand #2923 @wslulciuc
  • --jobs: limits OL jobs up to N (default: 5)
  • --runs-per-job: limits OL run executions per job up to N (default: 10)
  • --runs-active: limits OL run executions marked as active (='RUNNING') up to N
  • --max-run-fails-per-job: maximum OL run fails per job (default: 2)
  • --min-run-duration: minimum OL run duration (in seconds) per execution (default: 300)
  • --run-start-time: specifies the OL run start time in UTC ISO ('YYYY-MM-DDTHH:MM:SSZ'); used for the initial OL run, with subsequent runs starting relative to the initial start time. (default: 2024-10-15T01:00:11.080828Z)
  • --run-end-time: specifies the OL run end time in UTC ISO ('YYYY-MM-DDTHH:MM:SSZ'); used for the initial OL run, with subsequent runs ending relative to the initial end time. (default: 2024-10-15T01:07:25.080828Z)

Fixed

• Web: Better rendering of long text #2942 @phixMe
• Web: Display full runID and check icon when copied #2940 #2941 @wslulciuc @phixMe
• Web: Use DatasetVersionAPI to display latest schema and remove extra job facets API call in dataset panel #2938 @phixMe
• Web: Use DatasetAPI for data quality assertions in dataset panel #2937 @phixMe
• Web: Fill-in job node in lineage graph with correct color for JobEvents #2934 @phixMe
• Web: Fill-in job node in lineage graph with correct color for run states RUNNING, COMPLETED, etc #2897 @phixMe
• API: Pagination for DatasetVersion.findAll(); not all dataset versions were returned for GET /api/v1/namespaces/{namespace}/datasets/{dataset}/versions #2944 @inanalper
• API: null namespace and dataset name in view dataset_view for old versions; use table dataset_versions instead in column lineage query #2881 @sophiely
• API: Missing DELETE CASCADE on table job_facets #2878 @mattwparas
• API: Ensure Job.latestRun in Job object is set for runs in a RUNNING state; before Job.latestRun was set only for a run in a done state (COMPLETED / FAILED) #2933 @phixMe
• CLI: Repurpose cmd db-migrate to run all pending database migrations, no longer coupling migrations with HTTP server startup #2936 @davidjgoss
• Chart: Missing common labels for deployment.replicas #2877 @alaturqua