Release notes - Hopsworks - Version 1.1.0
Bug
- [HOPSWORKS-1214] - Airflow HopsworksLaunchOperator does not handle job state App Master Failed
- [HOPSWORKS-1245] - [ePipe] Fixes for xattrs handling to be work with the changes introduced in HOPS-1434 and HOPS-1451
- [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
- [HOPSWORKS-1296] - [ePipe] log replay fails on xattr op
- [HOPSWORKS-1305] - Fix broken Anaconda export yml
- [HOPSWORKS-1318] - Featurestore should throw an error if you try to create the same featuregroup twice
- [HOPSWORKS-1323] - [ndb-chef] Fix bug in NDB cluster-init.sh
- [HOPSWORKS-1325] - Online Feature store tests fail when project name contains uppercase letters
- [HOPSWORKS-1342] - Add parameter for s3 encryption type in core-site.xml
- [HOPSWORKS-1353] - Flink timer should handle case when services are not installed
- [HOPSWORKS-1385] - HA elastic does not have a synchronization point between starting nodes and creating indices
- [HOPSWORKS-1398] - User activation date is irrational
- [HOPSWORKS-1408] - AirflowManager kills Hopsworks deployments if Airflow directory does not exist
- [HOPSWORKS-1412] - Fix create featuregroup with partition keys
- [HOPSWORKS-1413] - AWS S3 does not accept empty Authorization header when downloading from a public bucket
- [HOPSWORKS-1425] - Update stack to handle hopsfs paths starting by hopsfs instead of hdfs
- [HOPSWORKS-1426] - Avoid using 100% containers for yarn containers
- [HOPSWORKS-1428] - hops_airflow fails to install pymssql 3.0.3, pin it to 2.1.4
- [HOPSWORKS-1431] - Better handling of Jupyter/Git failures
- [HOPSWORKS-1432] - Elasticsearch node should always use its private ip
- [HOPSWORKS-1434] - Scale jvm -Xmx for Spark and Jupyter based on allocated memory
- [HOPSWORKS-1442] - Update statistics job shows app file as undefined
- [HOPSWORKS-1443] - [featurestore] Spark returning Infinity values for mean statistics of string type features
- [HOPSWORKS-1451] - Storage connector ignored when creating a training dataset from the UI
- [HOPSWORKS-1456] - [hops-hadoop-chef] Hopsworks security related properties are not templated correctly
- [HOPSWORKS-1457] - Correct systemd scripts
- [HOPSWORKS-1458] - spark should not bring any hops jar with it
- [HOPSWORKS-1461] - The JWT signing key should be decoded first before using it to sign jwt
- [HOPSWORKS-1463] - Hops ndb version not taken in account when downloading dal
- [HOPSWORKS-1464] - fix chef for HOPS-1542
- [HOPSWORKS-1484] - Shared feature stores do not appear in the UI
- [HOPSWORKS-1485] - Sparkmagic upgrades not working
- [HOPSWORKS-1488] - Deep Learning tour project doesn't create Kafka inferenceschema
- [HOPSWORKS-1489] - Increase default Heap size and max shards per node for Elasticsearch
- [HOPSWORKS-1490] - Handle Kibana failures in Hopsworks Kibana client
- [HOPSWORKS-1496] - Elastic::default does not template correctly hosts in elasticsearch.yml
- [HOPSWORKS-1497] - hops-util-py incorrectly sets Kafka config
- [HOPSWORKS-1498] - hops-util-py can't find attributes in KafkaTopicDTO
- [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
- [HOPSWORKS-1508] - [kagent-chef] Update kibana x.509 certificate when host certificate is rotated
- [HOPSWORKS-1510] - Remove the code to delete index patterns since we delete kibana project index
- [HOPSWORKS-1516] - [ePipe] handle the case if XAttrAddAll is received for a deleted inode
- [HOPSWORKS-1518] - [livy-chef] Create root Hops directory
- [HOPSWORKS-1519] - The subject of ELK JWT token should be always lowercase
- [HOPSWORKS-1521] - Dataset browser refresh button not working
- [HOPSWORKS-1523] - Kibana opendistro_security plugin is not idempotent
- [HOPSWORKS-1524] - [hops-hadoop-chef] Hops services startup should be dependent on Glassfish
- [HOPSWORKS-1526] - Can't update job when doing insert into featuregroup
- [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string
Sub-task
- [HOPSWORKS-295] - Datasets
- [HOPSWORKS-1241] - Refactor Kafka API
- [HOPSWORKS-1378] - [featurestore] DDL enable composite key
- [HOPSWORKS-1388] - Remove Text/Blobs from hopsworks database tables with FK
New Feature
- [HOPSWORKS-1109] - Experiments and models v2
- [HOPSWORKS-1334] - [hopsworks-cloud-sdk] SQL query for creating training datasets
- [HOPSWORKS-1338] - [featurestore] Configurable in-bucket path for exporting training datasets to S3
- [HOPSWORKS-1340] - Airflow DAG UI composer
- [HOPSWORKS-1362] - Endpoint to securely get a project user's x.509 credentials
- [HOPSWORKS-1363] - [featurestore] feature provenance for training datasets
- [HOPSWORKS-1372] - Upgrade the ELK stack to 7.2 OSS and use the OpenDistro Security Plugin
- [HOPSWORKS-1377] - [featurestore][queryplanner] composite key support
- [HOPSWORKS-1422] - Custom metadata to featuregroups
- [HOPSWORKS-1459] - Tracking file changes - provenance - epipe
- [HOPSWORKS-1467] - File/App Provenance/Lineage
- [HOPSWORKS-1476] - Add support for elasticsearch spark connector
Improvement
- [HOPSWORKS-1248] - [ePipe] Adapt the new dataset tracking introduced in HOPS-1483
- [HOPSWORKS-1254] - [ePipe] handle elastic search failures
- [HOPSWORKS-1310] - Refactor Kafka API - ACLs
- [HOPSWORKS-1324] - [ePipe] Adapt to the new dataset table schema
- [HOPSWORKS-1332] - [featurestore] improve queryplanner docs/examples
- [HOPSWORKS-1356] - Enable paraller executions for jobs
- [HOPSWORKS-1380] - [featurestore] Configure Feature Group statistics during creation
- [HOPSWORKS-1384] - Refactor Kafka API - Schemas
- [HOPSWORKS-1391] - Extend Pandas and Numpy Helper Classes
- [HOPSWORKS-1395] - Hopsworks CA REST endpoint to download Hops CRL instead of Glassfish serving the file
- [HOPSWORKS-1404] - Prometheus should handle the case in which recipes are missing.
- [HOPSWORKS-1415] - Admin should be able to reset users' passwords
- [HOPSWORKS-1418] - Bump PyTorch, Numpy, Torchvision versions
- [HOPSWORKS-1423] - [cloud-sdk] Import feature group job not failing in client when storage connector does not exist
- [HOPSWORKS-1424] - [featurestore] Handle upper/lower case for feature group names
- [HOPSWORKS-1435] - [featurestore] Unify feature store entity max length validation in backend
- [HOPSWORKS-1436] - Accept DB name when setting up Hive connection in hops-util-py
- [HOPSWORKS-1440] - Recipes should be able to update Hive/Spark jars without updating the distribution
- [HOPSWORKS-1465] - Make feature store documentation easier to find
- [HOPSWORKS-1469] - REST API v2 - Admin - Users
- [HOPSWORKS-1517] - Double // in download url for Kubenetes artifacts
Task
- [HOPSWORKS-1027] - Add support for multiple elasticsearch nodes (HA)
- [HOPSWORKS-1194] - Make tour state simpler to manage
- [HOPSWORKS-1250] - Bump ePipe version to 0.10
- [HOPSWORKS-1303] - Refactor Kafka API - Topics
- [HOPSWORKS-1331] - Inconsistent "upgrade" flags across Chef cookbooks
- [HOPSWORKS-1376] - add missing kafka ssl property to hops-util*
- [HOPSWORKS-1389] - Improve Jupyter configuration page look and feel
- [HOPSWORKS-1433] - Explicitly start JupyterLab or Jupyter classic from Hopsworks
- [HOPSWORKS-1445] - Bump Anaconda's python to version 3
- [HOPSWORKS-1446] - Disable request hostname verify in localhost mode
- [HOPSWORKS-1448] - Move Hopsworks Feature store controllers in a separate package
- [HOPSWORKS-1466] - metaEnabled changed from boolean to MetaStatus enum
- [HOPSWORKS-1474] - Blacklist cloud conda environment
- [HOPSWORKS-1475] - Bump Hive version in Spark
- [HOPSWORKS-1477] - Remove exception when creating project with online feature store
- [HOPSWORKS-1479] - Enable all Jupyter kernels by default
- [HOPSWORKS-1480] - Do not install development libraries in Airflow environment
- [HOPSWORKS-1491] - Bump maggy version to 0.4.0
- [HOPSWORKS-1499] - Bump Hops version to 2.8.2.9
- [HOPSWORKS-1502] - Bump Hive version to 3.0.0.3
- [HOPSWORKS-1505] - Beam default Python env should be 3.6
- [HOPSWORKS-1535] - Hopsworks variables maintain Public and Private endpoint for Hive