The Feature Store is a data management system for managing machine learning features, including the feature engineering code and the feature data. The Feature Store helps ensure that features used during training and serving are consistent and that features are documented and reused within Enterprises.
Hopsworks cannot currently be deployed on a Kubernetes cluster. You can, however, connect Enterprise Hopsworks to an existing Kubernetes cluster, where you can run Python Jobs, Jupyter notebooks, and serve models (KFServing). Hopsworks Community does not support Kubernetes integration.
Hopsworks can be deployed as a self-managed cluster on virtual machines (VMs) inside an existing GCP project. You can either run the Hopsworks Cloud Installer tool that uses the GCP command-line utility to create VMs and install Hopsworks or you can create the VMs yourself and then run the Hopsworks Installer tool.
Hopsworks can be deployed as either:
(1) a managed platform inside your organisation’s existing cloud account at www.hopsworks.ai. You need to connect your Azure account by creating a service principal.
(2) a self-managed cluster on virtual machines (VMs) inside your organisation’s existing Azure subscription. You can either run the Hopsworks Cloud Installer tool that uses the AZ command-line utility to create VMs and install Hopsworks or you can create the VMs, resource group, virtual network, private DNS zone, yourself and then run the Hopsworks Installer tool.
Hopsworks can be deployed as either:
(1) a managed platform inside your organisation’s existing cloud account at www.hopsworks.ai. You need to connect your AWS cloud account by creating a cross-account role.
(2) a self-managed cluster on virtual machines (VMs) inside your organisation’s existing cloud account. You create EC2 VMs and run the Hopsworks Installer Tool.
Hopsworks Feature Store supports delta lake as a data source
Hopsworks Feature Store supports JDBC as a data source.
We support the following databases:
• MySQL
• Microsoft SQL Server
• PostgreSQL
• MongoDB
• Db2
• Redis
Hopsworks Feature Store supports Snowflake as a data source
Hopsworks Feature Store supports Azure Data Lake and Azure Block Storage as data sources.
Hopsworks Feature Store supports Spark as a data source.
We support the following databases:
• Elasticsearch
• Cassandra
You can do your feature engineering natively into the Hopsworks platform.
Hopsworks supports feature engineering from Azure ML. Contact us for more information.
Hopsworks supports feature engineering on Kubeflow. Contact us for more information.
Hopsworks supports other python environment, such as Jupyter.
Hopsworks supports other Spark environment, such as Cloudera.
If you don’t already have a Data Science platform, Hopsworks can be used to train and serve models, providing end-to-end security and governance, in a notebook friendly, Python-first platform.
Databricks users can use the Hopsworks Feature Store as a standalone service to manage their features for model training and model serving.
AWS Sagemaker users can use the Hopsworks Feature Store as a standalone service to manage their features for model training and model serving.
Azure ML users can use the Hopsworks Feature Store as a standalone service to manage their features for model training and model serving.
Dataiku users can use the Hopsworks Feature Store as a standalone service to manage their features for model training and model serving.
Domino users can use the Hopsworks Feature Store as a standalone service to manage their features for model training and model serving.
Kubeflow users can use the Hopsworks Feature Store as a standalone service to manage their features for model training and model serving.
Jupyter users can use the Hopsworks Feature Store as a standalone service to manager their features for model training and model serving.
The online Feature Store stores the latest values of features and supports low latency access to features that should be highly available.
• Hopsworks Online Feature Store is built on NDB, the world’s highest throughput low latency key-value store that happens to be a full database with 99.999% high availability;
• Highly available across availability zones within a region,
• It can be replicated asynchronously between regions.
The offline Feature Store supports exploratory data analysis and large volumes of data for model training and for use by batch (analytical) applications for model scoring.
• Hopsworks Offline Feature Store is built on HopsFS to provide higher performance than S3 and Azure Block Storage (ABS) with no increase in cost (data is stored in a S3/ABS bucket in your account) while remaining open, providing a HDFS API and open file formats like Parquet, Hudi, Delta Lake, and ORC;
• It can be configured to be highly available across availability zones within a region;
• It can be replicated asynchronously between regions.
We provide comprehensive API documentation and examples, enabling you to create, version and control your features using Python, Scala, or Java.
• Other Feature Stores provide a SQL API for you to transform your data into features and to retrieve features from the Feature Store. Hopsworks Feature Store is Python-Friendly, providing a Pandas-like API, making complex operations simple, such as joining features together to create training data;
• Our API has also undergone revision, to apply what we have learnt about supporting production Feature Stores, so we support versioning both feature schemas and feature values (time-travel). Versioning is key to enable developers to update feature definitions without breaking existing feature pipelines.
The more features that are made available, the more features will be reused directly in many models.
Feature statistics help to quickly monitor, identify and adjust problems with models performance.
Correct feature pipelines, with no training/inference feature engineering skew affecting model performance.
Access control, custom metadata, and end-to-end security requirements assuring that all actions are audited, from ingesting to using features.
The Feature Store's benefits are broad, and while there is some overlap, it may vary depending on the users; Data scientists, Data engineers or ML engineers. You may look at the direct benefits for your onw usage below;
