Why Should You Use Cloud Inference (Inference as a Service) in 2024?

Figure 1. Popularity of the keyword “cloud inference” on Google search engine worldwide in the past 5 years.

Deep learning models have emerged as powerful tools in areas like speech recognition and image classification, even surpassing human performance in tasks such as object classification and real-time strategy games.¹ However, their effectiveness hinges on the availability of substantial training datasets and inference, necessitating considerable computational resources. Cloud inference presents itself as a viable solution to meet these demanding requirements.

This article delves into the concept of cloud inference, contrasts it with on-device (at source) inference, and explores the various benefits and challenges associated with cloud-based inference systems.

What is inference?

Figure 2. Visualization of how inference works.

Inference is the phase where a model trained is used to make predictions or decisions based on new data. The new data is passed through the trained model, which starts to use the learned weights to make predictions (run inference) without further adjustment.

For example, if your trained model consists of cat images and you feed a new image to the model, it should be able to tell you if there’s a cat in the image or not. However, there comes a challenge; this process involves billions of parameters, requiring high computational resources.² This is where cloud inference comes in.

What is cloud inference/inference as a service?

Meta has boosted its infrastructure capacity by 250% to accommodate the increasing demand for machine learning (ML) inference.³ Requirements of the inference process led organizations to look for alternatives.

Cloud inference refers to the deployment of machine learning models on cloud platforms, allowing users to access powerful computational resources remotely. This service is particularly beneficial for applications requiring significant computational power or specialized hardware, like GPUs or TPUs. These are often expensive and difficult to maintain locally.

Cloud inference vs. inference at source

Source: SoftmaxAI⁴

Figure 3. Visualization of how and where edge vs cloud inference differs.

The key difference between inference as a service and at source lies in their deployment models. Inference at source, or edge inference, happens directly on the device where data is generated (like an IoT device). In contrast, inference as a service relies on cloud-based resources, where data is sent to the cloud for processing and then returned to the source.

Advantages of cloud inference

1- It is time sensitive

Cloud-based solutions often offer lower latency due to the high-performance compute capabilities available in the cloud. This is an advantage for applications where real-time results are needed (e.g., fraud detection).

2- It works anywhere

Cloud-based solutions are inherently global, offering the advantage of location independence and the ability to operate across different geographical regions.

3- It has a strong battery life

By offloading heavy computational tasks to cloud servers, there is a significant reduction in the power consumption of local devices.

Key challenges of cloud inference

1- Data leaks

Consider the operation of recommendation systems; they are developed through training on substantial volumes of data, much of which is personal in nature. When inference is conducted utilizing this trained dataset and the process is executed in a cloud environment, there arises a potential risk that the data could be accessed by unauthorized parties.

2- Attacks

Models used in cloud inference are susceptible to various forms of cyberattacks. For instance, most current cloud inference attacks are targeted at images.⁵ These attacks can compromise the integrity and confidentiality of sensitive data.

Strategies to prevent attacks

To safeguard against attacks in cloud-based inference systems, the following strategies can be implemented:

1. Cryptographic methods: They are used to prevent unauthorized replication of training data and access to the model. An example would be encrypting the cloud-based models. Encryption acts as a barrier, ensuring that even if an attacker gains access to the model, they cannot easily understand or replicate the training data.
2. Controlled noise: You can minimize the risk of information leakage from the model’s output by adding designed noise to the output vector of the model. This will make it more difficult for attackers to extract meaningful data.
3. Prevent overfitting: Overfitting makes a model too tailored to the training data, potentially revealing sensitive information. By randomly removing some connections, or edges, during training, the model becomes less prone to overfitting.

3- Costs

Another major challenge in cloud inference systems is efficiently deploying machine learning models over large areas while meeting the demands for latency and accuracy in a cost-effective way. These systems are resource-heavy, both in terms of computation (for instance, up to 192.2 TFLOPs for BERT-Large) and bandwidth (as high-resolution images may be needed). Consequently, the costs can escalate significantly with a large number of inference queries.

4- Network delays

Despite aiming for low latency, cloud inference can be hindered by bandwidth limitations and network delays. These challenges are especially pronounced for those handling high-resolution data, where quick and efficient data processing is crucial.

If you need assistance in choosing between edge vs cloud inference, don’t hesitate to contact us:

External Links

• 1. Joshi, Praveen; Hasanuzzaman, Mohammed; Thapa, Chandra; Afli, Haithem; Scully, Ted (2023). “Enabling All In-Edge Deep Learning: A Literature Review“. IEEE. Retrieved January 31,2024.
• 2. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., … & Fiedel, N. (2023). “Palm: Scaling language modeling with pathways.” Journal of Machine Learning Research, 24(240), 1-113. Retrieved January 31,2024.
• 3. Li, B., Samsi, S., Gadepally, V., & Tiwari, D. (2023). “Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service.” In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1-15). Retrieved January 31,2024.
• 4. “Edge AI vs Cloud AI” SoftmaxAI. Retrieved January 31,2024.
• 5. Gong, X., Chen, Y., Wang, Q., Wang, M., & Li, S. (2022). “Private data inference attacks against cloud: Model, technologies, and research directions.” IEEE Communications Magazine, 60(9), 46-52. Retrieved January 31,2024.