ONNX Runtime vs tinygrad

ONNX Runtime is a free engine for machine-learning inference and training within existing technology stacks. It can run models from frameworks including PyTorch, TensorFlow/Keras, TFLite, and scikit-learn. For inference, it optimizes latency, throughput, memory use, and binary size through graph transformations, accelerator-aware partitioning, and optimized computation kernels. Its Execution Providers connect ONNX models to hardware acceleration libraries for CPUs, GPUs, FPGAs, and specialized NPUs. Listed options include NVIDIA CUDA and TensorRT, Intel OpenVINO, DirectML, Qualcomm QNN, Android NNAPI, Apple CoreML, and WebGPU. Deployment spans cloud servers, edge and mobile devices, and browsers; its web runtime supports browser models and its mobile runtime supports Android and iOS. Supported languages include Python, C#, C++, Java, JavaScript, and Rust. Developers can create smaller web or mobile packages by including only the operators and opsets their models need. The open-source runtime is available on Android, iOS, Linux, macOS, self-hosted environments, web, and Windows.

tinygrad is a deep learning framework for neural network training, bringing together a tensor library, compiler, JIT, and training tools. It supports forward and backward automatic differentiation, and tensor operations run lazily when a tensor is realized. Neural network classes, optimizers, and state loading and saving are included. TinyJit captures and replays kernels in a decorated function, and tensors can be distributed across multiple GPUs. Listed accelerators include OpenCL, CPU, Metal, CUDA, AMD, NV, Qualcomm, and WebGPU. The project describes use in openpilot to run a driving model on a Snapdragon 845 GPU. It supports Python and local training, with multiple deployment targets and safetensors or PyTorch weights through model-specific loaders. Installation instructions cover pip, though the project recommends installing from source. tinygrad is free under the MIT license. Its API is described as stable for a while, though the project is not yet version 1.0.