Domino Model Monitor vs Pulsar

Domino Model Monitor evaluates production machine-learning models and alerts when degradation warrants expert review. It monitors data drift and model quality from a central dashboard showing status, summaries, customized alerts, and monitoring results. Automated prediction-data pipelines and metric generation support ongoing checks. When remediation is needed, teams can recreate a model’s development environment, retrain with production data, and redeploy a version. Models built in Python, R, SAS, MATLAB, TensorFlow, or DataRobot can be registered. The Elastic Monitoring Engine is designed to scale and ingest data from Amazon S3 and Hadoop-compatible storage, including Azure Blob Storage, Azure Data Lake, Google Cloud Storage, and HDFS. Domino supports API, web, and self-hosted use, including cloud VPCs, on-premises systems, and air-gapped environments. Pricing is on request, with Premium, Domino Cloud, and Enterprise plans listed.

Pulsar is a free, open-source, self-hosted system for monitoring machine learning models in production. It tracks model drift, concept drift, and performance by comparing incoming behavior with training data used as a baseline. An SDK and containerised ingestion service collect model inputs and outputs into a time series database, supporting historical and real-time data. The current database integration is InfluxDB, and the project is designed for large ingestion volumes without degraded performance. A scheduler checks drift and can trigger standard or customised alerts when metrics reach thresholds. Grafana displays drift measures across selected time ranges and user-defined features. Available drift tests include Kullback-Leibler divergence, Kolmogorov-Smirnov, Levene, and Wasserstein; performance measures include accuracy, recall, F1 score, Brier score, mean squared error, and R2. Deployment logos cover Flask, FastAPI, TensorFlow Extended, Seldon, AWS, Azure, Google Cloud, and Grafana.