Theta Tech AI: Enhancing Healthcare AI Systems with Neptune

• Analytics & Modeling - Machine Learning
• Infrastructure as a Service (IaaS) - Public Cloud

• Education
• Equipment & Machinery

• Product Research & Development
• Quality Assurance

• Experimentation Automation
• Virtual Training

• System Integration
• Training

Theta Tech AI, a company that builds customized artificial intelligence algorithms and front-end user interfaces for large-scale healthcare AI systems, faced several challenges in developing generalizable medical AI systems. The team had to manage thousands of experiments for large-scale parallel training workflows, which were run on GPU servers in AWS. However, they found that AWS CloudWatch Logs, their initial choice for monitoring the jobs, was inadequate for managing experiment logs. The team was unable to get experiment-relevant metrics from AWS CloudWatch Logs, debug problems with training jobs and experiments, integrate Optuna for hyperparameter optimization, and communicate the results of ML models to clients effectively.

Theta Tech AI is a company that specializes in building customized artificial intelligence algorithms and front-end user interfaces for large-scale healthcare AI systems. Their main objective is to build 'hospitals in the cloud' powered by AI. Their products include image and signal-processing tools that detect anomalies indicating health risks. The team comprises seven engineers who focus on developing generalizable medical AI systems representative of the real world. These systems are deployed in hospitals to help healthcare providers increase clinical effectiveness and efficiency. The team works with 1D ECG signals, 2D X-rays, or 3D magnetic resonance imaging (MRI) medical and biological datasets.

To overcome these challenges, Theta Tech AI adopted Neptune, an experiment tracking solution that could interact with Optuna to track hyperparameters and offer collaborative features. Neptune met the team's criteria for an ideal solution, including integration with open-source tools, real-time support, easy-to-interpret visualizations, and ease of development. Neptune helped the team track thousands of training jobs running on AWS at scale, offered seamless Neptune-Optuna integration, provided an interactive real-time dashboard for Optuna, and offered a grouping and filtering feature valuable for organizing experiments. The team found Neptune easy to set up and integrate with the existing stack without provisioning a separate infrastructure.

• The adoption of Neptune significantly improved Theta Tech AI's operational efficiency. The team could finally track and view only the metrics and files necessary for their research projects. Neptune removed the CloudWatch Logs barrier of leveraging external visualization tools like Grafana and provided secure and collaborative options. It also enforced experiment lineage, making it simple for the team to review earlier experiments, troubleshoot them, and reproduce their results. The Neptune-Optuna integrated dashboard gave them insights into how well the hyperparameters performed and offered all the information about the performance of each model. The team could share dashboards with coworkers and clients participating in the project, facilitating communication and cooperation. Neptune also improved how the team utilized compute resources for their training and data processing jobs.

• Neptune can scale to track all the jobs operating on different compute clusters when they ran thousands of training runs at scale on AWS.

• Neptune provides relevant dashboards and an interactive user interface to monitor their training jobs and hardware utilization.

• Neptune provides a wide range of integrations and support for open-source tools used in the industry, making it simple to set up and run using tools like fastai and Optuna.