github-starred/Made-With-ML
2
0
Fork 0
mirror of https://github.com/GokuMohandas/Made-With-ML.git synced 2026-03-09 07:12:37 -05:00
Code Issues 21 Packages Projects Releases Wiki Activity

Serving issue with /predict/ POST request #61

New Issue
Closed
opened 2025-11-02 00:02:48 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2025-11-02 00:02:48 -05:00
Owner
Copy Link

Originally created by @transiteration on GitHub (Jan 14, 2024).

I have this code running on jupyter notebook:

@serve.deployment(num_replicas="1", ray_actor_options={"num_cpus": 8, "num_gpus": 0})
@serve.ingress(app)
class ModelDeployment:
    def __init__(self, run_id: str, threshold: int = 0.9):
        self.run_id = run_id
        self.threshold = threshold
        mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)
        checkpoint = predict.get_best_checkpoint(run_id=run_id)
        self.predictor = predict.TorchPredictor.from_checkpoint(checkpoint)

    @app.get("/")
    def _index(self) -> Dict:
        response = {
            "message": HTTPStatus.OK.phrase,
            "status-code": HTTPStatus.OK,
            "data": {}
        }
        return response

    @app.post("/evaluate/")
    async def _evaluate(self, request: Request) -> Dict:
        data = await request.json()
        results = evaluate.evaluate(run_id=self.run_id, dataset_loc=data.get("dataset"))
        return {"results": results}

    @app.post("/predict/")
    async def _predict(self, request: Request) -> Dict:
        data = await request.json()
        sample_ds = ray.data.from_items([{"title": data.get("title", ""),
                                  "description": data.get("description", ""), 
                                  "tag": "other"}])    
        results = predict.predict_proba(ds=sample_ds, predictor=self.predictor)
        
        for i, result in enumerate(results):
            pred = result["prediction"]
            prob = result["probabilities"]
            if prob[pred] < self.threshold:
                results[i]["prediction"] = "other"

        return {"results": results}

When I send a POST request from Postman with a json file:

{
    "title": "Transfer learning with transformers",
    "description": "no title",
}

to /predict/, I get this error.
I checked /evaluate/, results of the metrics are outputed. I can't figure it out where is the problem...

(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/starlette/routing.py", line 746, in __call__ [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(ServeReplica:default:ModelDeployment pid=53855)     raise e from None
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/ray/serve/_private/replica.py", line 895, in call_user_method [repeated 2x across cluster]
(ServeReplica:default:ModelDeployment pid=53855) ray.exceptions.RayTaskError: ray::ServeReplica:default:ModelDeployment() (pid=53855, ip=172.20.10.2)
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/fastapi/encoders.py", line 330, in jsonable_encoder [repeated 7x across cluster]
(ServeReplica:default:ModelDeployment pid=53855)     data = dict(obj)
(ServeReplica:default:ModelDeployment pid=53855) TypeError: 'numpy.float32' object is not iterable
(ServeReplica:default:ModelDeployment pid=53855) ray::ServeReplica:default:ModelDeployment() (pid=53855, ip=172.20.10.2) [repeated 6x across cluster]
(ServeReplica:default:ModelDeployment pid=53855) During handling of the above exception, another exception occurred:
(ServeReplica:default:ModelDeployment pid=53855)     data = vars(obj)
(ServeReplica:default:ModelDeployment pid=53855) TypeError: vars() argument must have __dict__ attribute
(ServeReplica:default:ModelDeployment pid=53855) The above exception was the direct cause of the following exception:
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/ray/serve/_private/utils.py", line 165, in wrap_to_ray_error
(ServeReplica:default:ModelDeployment pid=53855)     raise exception
(ServeReplica:default:ModelDeployment pid=53855)     result = await method_to_call(*request_args, **request_kwargs)
(ServeReplica:default:ModelDeployment pid=53855)     await self._asgi_app(
(ServeReplica:default:ModelDeployment pid=53855)     await super().__call__(scope, receive, send)
(ServeReplica:default:ModelDeployment pid=53855)     await self.middleware_stack(scope, receive, send)
(ServeReplica:default:ModelDeployment pid=53855)     raise exc [repeated 3x across cluster]
(ServeReplica:default:ModelDeployment pid=53855)     await self.app(scope, receive, _send)
(ServeReplica:default:ModelDeployment pid=53855)     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app [repeated 4x across cluster]
(ServeReplica:default:ModelDeployment pid=53855)     await app(scope, receive, sender) [repeated 2x across cluster]
(ServeReplica:default:ModelDeployment pid=53855)     await route.handle(scope, receive, send)
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
(ServeReplica:default:ModelDeployment pid=53855)     await self.app(scope, receive, send)
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/fastapi/routing.py", line 315, in app [repeated 3x across cluster]
(ServeReplica:default:ModelDeployment pid=53855)     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(ServeReplica:default:ModelDeployment pid=53855)     response = await func(request)
(ServeReplica:default:ModelDeployment pid=53855)     content = await serialize_response(
(ServeReplica:default:ModelDeployment pid=53855)   File "/home/miras/envs/v1/lib/python3.10/site-packages/fastapi/routing.py", line 170, in serialize_response
(ServeReplica:default:ModelDeployment pid=53855)     return jsonable_encoder(
(ServeReplica:default:ModelDeployment pid=53855)     encoded_value = jsonable_encoder( [repeated 3x across cluster]
(ServeReplica:default:ModelDeployment pid=53855)     jsonable_encoder(
(ServeReplica:default:ModelDeployment pid=53855)     raise ValueError(errors) from e
(ServeReplica:default:ModelDeployment pid=53855) ValueError: [TypeError("'numpy.float32' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]
Originally created by @transiteration on GitHub (Jan 14, 2024). I have this code running on jupyter notebook: ``` @serve.deployment(num_replicas="1", ray_actor_options={"num_cpus": 8, "num_gpus": 0}) @serve.ingress(app) class ModelDeployment: def __init__(self, run_id: str, threshold: int = 0.9): self.run_id = run_id self.threshold = threshold mlflow.set_tracking_uri(MLFLOW_TRACKING_URI) checkpoint = predict.get_best_checkpoint(run_id=run_id) self.predictor = predict.TorchPredictor.from_checkpoint(checkpoint) @app.get("/") def _index(self) -> Dict: response = { "message": HTTPStatus.OK.phrase, "status-code": HTTPStatus.OK, "data": {} } return response @app.post("/evaluate/") async def _evaluate(self, request: Request) -> Dict: data = await request.json() results = evaluate.evaluate(run_id=self.run_id, dataset_loc=data.get("dataset")) return {"results": results} @app.post("/predict/") async def _predict(self, request: Request) -> Dict: data = await request.json() sample_ds = ray.data.from_items([{"title": data.get("title", ""), "description": data.get("description", ""), "tag": "other"}]) results = predict.predict_proba(ds=sample_ds, predictor=self.predictor) for i, result in enumerate(results): pred = result["prediction"] prob = result["probabilities"] if prob[pred] < self.threshold: results[i]["prediction"] = "other" return {"results": results} ``` When I send a POST request from Postman with a json file: ``` { "title": "Transfer learning with transformers", "description": "no title", } ``` to ```/predict/```, I get this error. I checked ```/evaluate/```, results of the metrics are outputed. I can't figure it out where is the problem... ```((ServeReplica:default:ModelDeployment pid=53855) Traceback (most recent call last): (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/starlette/routing.py", line 746, in __call__ [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.) (ServeReplica:default:ModelDeployment pid=53855) raise e from None (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/ray/serve/_private/replica.py", line 895, in call_user_method [repeated 2x across cluster] (ServeReplica:default:ModelDeployment pid=53855) ray.exceptions.RayTaskError: ray::ServeReplica:default:ModelDeployment() (pid=53855, ip=172.20.10.2) (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/fastapi/encoders.py", line 330, in jsonable_encoder [repeated 7x across cluster] (ServeReplica:default:ModelDeployment pid=53855) data = dict(obj) (ServeReplica:default:ModelDeployment pid=53855) TypeError: 'numpy.float32' object is not iterable (ServeReplica:default:ModelDeployment pid=53855) ray::ServeReplica:default:ModelDeployment() (pid=53855, ip=172.20.10.2) [repeated 6x across cluster] (ServeReplica:default:ModelDeployment pid=53855) During handling of the above exception, another exception occurred: (ServeReplica:default:ModelDeployment pid=53855) data = vars(obj) (ServeReplica:default:ModelDeployment pid=53855) TypeError: vars() argument must have __dict__ attribute (ServeReplica:default:ModelDeployment pid=53855) The above exception was the direct cause of the following exception: (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/ray/serve/_private/utils.py", line 165, in wrap_to_ray_error (ServeReplica:default:ModelDeployment pid=53855) raise exception (ServeReplica:default:ModelDeployment pid=53855) result = await method_to_call(*request_args, **request_kwargs) (ServeReplica:default:ModelDeployment pid=53855) await self._asgi_app( (ServeReplica:default:ModelDeployment pid=53855) await super().__call__(scope, receive, send) (ServeReplica:default:ModelDeployment pid=53855) await self.middleware_stack(scope, receive, send) (ServeReplica:default:ModelDeployment pid=53855) raise exc [repeated 3x across cluster] (ServeReplica:default:ModelDeployment pid=53855) await self.app(scope, receive, _send) (ServeReplica:default:ModelDeployment pid=53855) await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app [repeated 4x across cluster] (ServeReplica:default:ModelDeployment pid=53855) await app(scope, receive, sender) [repeated 2x across cluster] (ServeReplica:default:ModelDeployment pid=53855) await route.handle(scope, receive, send) (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle (ServeReplica:default:ModelDeployment pid=53855) await self.app(scope, receive, send) (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/fastapi/routing.py", line 315, in app [repeated 3x across cluster] (ServeReplica:default:ModelDeployment pid=53855) await wrap_app_handling_exceptions(app, request)(scope, receive, send) (ServeReplica:default:ModelDeployment pid=53855) response = await func(request) (ServeReplica:default:ModelDeployment pid=53855) content = await serialize_response( (ServeReplica:default:ModelDeployment pid=53855) File "/home/miras/envs/v1/lib/python3.10/site-packages/fastapi/routing.py", line 170, in serialize_response (ServeReplica:default:ModelDeployment pid=53855) return jsonable_encoder( (ServeReplica:default:ModelDeployment pid=53855) encoded_value = jsonable_encoder( [repeated 3x across cluster] (ServeReplica:default:ModelDeployment pid=53855) jsonable_encoder( (ServeReplica:default:ModelDeployment pid=53855) raise ValueError(errors) from e (ServeReplica:default:ModelDeployment pid=53855) ValueError: [TypeError("'numpy.float32' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]
GiteaMirror commented 2025-11-02 00:02:49 -05:00
Author
Owner
Copy Link

@transiteration commented on GitHub (Jan 16, 2024):

I solved the problem by using the predict.py function predict(run_id, title, description).

@transiteration commented on GitHub (Jan 16, 2024): I solved the problem by using the predict.py function predict(run_id, title, description).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/Made-With-ML#61
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?