Apache SINGA vs NVIDIA Triton Inference Server

Apache SINGA is a free distributed deep-learning library for training deep-learning and machine-learning models. Its listed training mode is local, with data-parallel training across multiple GPUs on one node or across multiple nodes. GPU packages use CUDA and cuDNN, and Docker images are available for Nvidia GPUs. Installation options include pip, Docker, Conda, and source builds; pip documentation covers Python 3.9, 3.10, and 3.11. SINGA supports Python and C++, loads ONNX models, and can save models defined through its APIs in ONNX format. Optimizers listed include momentum-based stochastic gradient descent, Adam, RMSProp, and AdaGrad. Models can be queried in an RDBMS. The repository and Google Colab provide domain-specific models, including healthcare and science examples. It runs on Linux, macOS, Windows, and self-hosted environments, with on-premises deployment and Apache License 2.0 licensing.

NVIDIA Triton Inference Server is open-source software for deploying and serving AI models on GPU- or CPU-based infrastructure. It supports TensorRT, PyTorch, ONNX, OpenVINO, Python and RAPIDS FIL, with model formats including TensorRT Plan, ONNX, TensorFlow GraphDef and SavedModel, PyTorch TorchScript and PyTorch 2.0. Triton serves real-time, batched, ensemble and audio/video streaming workloads. Dynamic batching and concurrent model execution are among its performance features. Requests can use HTTP/REST, gRPC or the C API, with a Java API for in-process use. Triton integrates with Kubernetes for scaling and Prometheus for monitoring. Deployment options include NVIDIA GPUs, other accelerators, x86 and ARM CPUs, cloud, data centers, edge and embedded systems. NVIDIA lists Linux containers for x86 and Arm, plus Windows and Jetson JetPack binaries. Development code and containers are free; production options include a cloud marketplace plan billed pay as you go and a subscription listed at $4,500 a year per GPU. NVIDIA says deployers are responsible for security, and Triton does not sandbox arbitrary model or backend code.