Hello everyone! I’m Rohith Kumar Chandragiri, a Machine Learning Engineer at Money Forward. Over the last four years, our team has been dedicated to building state-of-the-art ML systems using diverse open-source tools, hosting large-scale models with millions of parameters for our products. In this tech blog, I will introduce the Ray Serve framework and build upon the concepts from my previous article.

The Challenge: The Monolith Bottleneck

Let’s dive into the ongoing journey of deploying multiple ML models into production.

When we first deployed, we wrapped a heavyweight model in a single FastAPI service. That worked great for about six months. However, as traffic increased, the system began to choke, forcing us to rethink our architecture. We realized our next-generation ML platform needed to do at least the following things:

• Decouple our CPU, memory, and GPU workloads
• Scale those components independently
• Implement dynamic batching for the GPUs
• Process massive batch jobs asynchronously so our synchronous APIs wouldn’t degrade—while also keeping our GPU bills in check.

Any software engineer looking at those requirements would immediately suggest a standard Pub/Sub architecture via a Queue or maybe a distributed system. It was our first thought, too. But going the Pub/Sub route means operational bloat. You end up managing a web of N queues, N microservices, and N Dead Letter Queues. If a process drops, finding the root cause across all those moving parts is a massive headache and trying to spin all of that up for local development is a nightmare we wanted to avoid.

And this is when we found Ray — Ray Serve

Ray Serve

Simply put, Ray Serve is a modern model serving framework which solves the problems of hosting complex inference pipelines with efficient resource utilization, while simplifying the underlying distributed infrastructure.

The images above illustrate Ray Serve’s core capabilities. Figure A demonstrates a static graph where every request fans out through the same set of models, while Figure B shows a dynamic one where only some branches run for a given request. Beyond simply making it easier to host these complex pipelines, Ray Serve maximizes their efficiency. It enables batch processing for branched models and allows every component of the pipeline to scale independently.

Ray Cluster

A Ray Cluster operates as a unified service consisting of two main components: a single Head Node that manages the cluster, schedules tasks, and oversees the overall lifecycle, alongside multiple Worker Nodes that execute application code, run ML models, and process data. To utilize Ray Serve as your inference engine, your foundational infrastructure must already be provisioned and fully operational. At minimum you need a Kubernetes cluster backed by cloud compute nodes — EC2 on AWS, or the equivalent on GKE, AKS, or on-prem — preferably with a node autoscaler so capacity follows demand. Everything past that depends on your environment: our production setup also runs Redis and Istio, but neither is required by Ray Serve itself.

Here is the current status of the cluster showing the active pods, services, and Ray resources:

NAME                                                     READY   STATUS      RESTARTS   AGE
pod/ml-app-api-5gv8m-cpu-group-worker-tzhzs              1/1     Running     0          11h
pod/ml-app-api-5gv8m-gpu-group-worker-dft24              1/1     Running     0          159m
pod/ml-app-api-5gv8m-head-72pp9                          2/2     Running     0          11h

NAME                                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                         AGE
service/ml-app-api-5gv8m-head-svc       ClusterIP   None           <none>        10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP   11d
service/ml-app-api-head-svc             ClusterIP   None           <none>        10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP   279d
service/ml-app-api-serve-svc            ClusterIP   172.20.44.51   <none>        8000/TCP                                        279d

NAME                                     DESIRED WORKERS   AVAILABLE WORKERS   CPUS   MEMORY   GPUS   STATUS   AGE
raycluster.ray.io/ml-app-api-5gv8m       3                 3                   6     40Gi     1      ready    11d

NAME                               SERVICE STATUS   NUM SERVE ENDPOINTS
rayservice.ray.io/ml-app-api       Running          3

We can observe that the Ray head node and worker nodes are actually Kubernetes pods.

Bridging Infrastructure and Code

In a traditional microservices migration, taking advantage of this Kubernetes cluster would be a massive DevOps hurdle. If we wanted to split a monolithic application into three separate services, we would typically need to write three different Dockerfiles, set up three CI/CD pipelines, and manage complex Kubernetes deployment YAMLs and networking rules to get them to talk to each other.

This is where Ray Serve truly shines. Instead of drowning in DevOps configurations, Ray Serve maps your Python code directly to that underlying infrastructure. You define your microservices and their routing logic purely in Python, and Ray automatically handles distributing those components across the worker nodes we just saw.

Let’s look at exactly how little code it takes to make that happen.

Rewriting From Monolithic to Ray

I will assume my application follows the branching pipeline in Figure A. I have three models, model_a, model_b, model_c and some business logic. In a regular monolithic application, we can create the application using singleton classes for model a, model b, model c and there will be a main process file which will handle the flow and finally return the response.

Simple Python FastAPI

from fastapi import FastAPI
from pydantic import BaseModel

# 1. Define the FastAPI app and Request schema
app = FastAPI()

class PipelineRequest(BaseModel):
    x: str  # Adjust type based on your actual input

# 2. Define standard Python classes for the models
class ModelA:
    def forward(self, x):
        return f"A({x})"

class ModelB:
    def forward(self, x):
        return f"B({x})"

class ModelC:
    def forward(self, x):
        return f"C({x})"

# 3. Instantiate the models (loaded once into memory when the server starts)
model_a = ModelA()
model_b = ModelB()
model_c = ModelC()

# 4. Define the final processing function
def process_data(x2, x3):
    return f"Combined: {x2} | {x3}"

# 5. Define the API endpoint
@app.post("/forward")
async def run_pipeline(payload: PipelineRequest):
    x = payload.x

    # Step 1: Forward through Model A
    x1 = model_a.forward(x)

    # Step 2: Forward through Model B and Model C sequentially
    x2 = model_b.forward(x1)
    x3 = model_c.forward(x1)

    # Step 3: Final processing step
    x4 = process_data(x2, x3)

    return {"result": x4}

Using Ray Serve

# ray[serve]==2.56.1
# The DeploymentHandle API below is not available in early 2.x releases.
import asyncio
from fastapi import FastAPI
from pydantic import BaseModel
from ray import serve
from ray.serve.handle import DeploymentHandle

# 1. Define the FastAPI app and Request schema
app = FastAPI()

class PipelineRequest(BaseModel):
    x: str  # Adjust type based on your actual input (e.g., list, dict)

# 2. Define the generic process function
def process_data(x2, x3):
    return f"Combined: {x2} | {x3}"

# 3. Define the Models as Ray Serve Deployments
@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_gpus": 0.5, "num_cpus": 1}
)
class ModelA:
    async def forward(self, x):
        return f"A({x})"

@serve.deployment
class ModelB:
    async def forward(self, x):
        return f"B({x})"

@serve.deployment
class ModelC:
    async def forward(self, x):
        return f"C({x})"

# 4. Define the Ingress Deployment that orchestrates the graph
@serve.deployment
@serve.ingress(app)
class PipelineIngress:
    def __init__(
        self,
        a_handle: DeploymentHandle,
        b_handle: DeploymentHandle,
        c_handle: DeploymentHandle
    ):
        # Inject the deployment handles during initialization
        self.a = a_handle
        self.b = b_handle
        self.c = c_handle

    @app.post("/forward")
    async def run_pipeline(self, payload: PipelineRequest):
        x = payload.x

        # Step 1: Forward through Model A
        # Use .remote() to call methods on the deployment handle
        x1 = await self.a.forward.remote(x)

        # Step 2: Forward through Model B and Model C CONCURRENTLY
        # Because they both depend on x1 but not each other, they can run in parallel
        task_x2 = self.b.forward.remote(x1)
        task_x3 = self.c.forward.remote(x1)

        # Await both results
        x2, x3 = await asyncio.gather(task_x2, task_x3)

        # Step 3: Final synchronous processing step
        x4 = process_data(x2, x3)

        return {"result": x4}

# 5. Bind the deployments together into a Ray Serve Application
# This is what you actually deploy via the CLI or Python API
pipeline_app = PipelineIngress.bind(
    ModelA.bind(),
    ModelB.bind(),
    ModelC.bind()
)

In the standard monolithic approach, ModelA, ModelB, and ModelC are instantiated as singleton classes bound to a single Python process and its memory space. This creates massive bottlenecks: if Model A is computationally heavy and needs to scale to handle more traffic, you are forced to replicate the entire monolithic application, wasting expensive resources on Model B and Model C. Furthermore, because they share a single process, Python’s Global Interpreter Lock (GIL) often prevents them from executing in true parallel, causing them to block one another.

Ray Serve completely flips this paradigm. By simply adding the @serve.deployment decorator, those standard Python classes are transformed into independently scalable microservices. Instead of running everything in a single constrained process, the architecture relies on Deployment Handles and the .bind() method to wire these models into a distributed computational graph.

This provides massive advantages over singletons:

1. Hardware Isolation & Independent Scaling: You can explicitly allocate GPUs to Model A and CPUs to Model B. If Model A becomes a bottleneck, you can spin up 10 replicas of it across your cluster while leaving Model B at a single replica.

@serve.deployment( 
    num_replicas=2, 
    ray_actor_options={"num_gpus": 0.5, "num_cpus": 1} 
)
class ModelA:
    async def forward(self, x):
        return f"A({x})"

Note: two replicas at num_gpus: 0.5 add up to one full GPU, so both land on the same card. These fractions are purely logical accounting in Ray’s scheduler — it tracks how much of each GPU it has handed out and packs actors accordingly. There is no MIG or vGPU partitioning involved, and no hardware-level isolation between the two replicas: they share the card’s memory, so it is on you to make sure both models actually fit.

2. True Parallelism: By using .remote() for asynchronous calls, the processing is dispatched to separate actors (processes) on the cluster. In our code, task_x2 and task_x3 aren’t just running concurrently—they are running in true parallel on different hardware, completely bypassing the GIL and slashing latency.
3. Seamless API Integration: Finally, @serve.ingress embeds the FastAPI application directly into Ray Serve, giving you FastAPI’s elegant HTTP routing backed by Ray’s powerful distributed execution engine.
4. Dynamic Batching: Handling traffic spikes in a monolith often requires building complex message queues to group incoming requests for efficient GPU processing. With Ray Serve, applying a simple @serve.batch decorator automatically groups concurrent individual API requests into a single vectorized batch. This maximizes your GPU throughput without requiring you to write any custom queueing or polling logic.

Note: this ModelB is a drop-in replacement for the one in the pipeline above — PipelineIngress still calls self.b.forward.remote(x1) and needs no changes at all. All that changes is that forward now hands the request to a batched method instead of doing the work itself.

from typing import List
from ray import serve

@serve.deployment
class ModelB:
    # Groups requests waiting up to 0.1s, up to a max of 16 items
    @serve.batch(max_batch_size=16, batch_wait_timeout_s=0.1)
    async def process_batch(self, inputs: List[str]) -> List[str]:
        # Process multiple inputs in one optimized operation
        return [f"B_batched({x})" for x in inputs]

    async def forward(self, x: str) -> str:
        # Individual requests are automatically paused and grouped here
        return await self.process_batch(x)

5. The Same Graph Runs on Your Laptop: Remember the local development nightmare we wanted to avoid? There is no broker to stand up and no cluster to fake. serve run serve_app:pipeline_app starts a local Ray instance and serves the identical graph on localhost:8000, so you can develop against the real routing logic before it ever reaches Kubernetes.

Conclusion

Looking back, transitioning from our monolithic setup to Ray Serve fundamentally changed how we handle machine learning inference. We no longer have expensive hardware sitting idle; by decoupling our CPU and GPU services, we can now scale them independently to clear long-running bottlenecks without overprovisioning. Instead of losing months building complex Kafka or SQS message queues to handle traffic spikes, we get dynamic batching effortlessly out of the box.

The clearest win showed up in throughput. Our workload is a vision transformer served on a single g4dn.xlarge instance — one NVIDIA T4. Hosted as a plain FastAPI service, that setup handled about 0.4 RPS. With Ray Serve’s dynamic batching, the same model on the same GPU now sustains about 1.5 RPS—nearly 4x the throughput without adding hardware. GPU slicing helped as well: we can declare fractional resources and run more than one model on the same card, instead of dedicating a full GPU to a single process.

In the end, the win was not just higher throughput. Ray Serve let us break a monolith into independently scalable services—without taking on the operational tax of N queues, N microservices, and N deployment pipelines. We still write Python. Ray handles the distributed infrastructure.