Internal Server Error

[LPUoO]

Dear @ksugar,

I am trying to train a model with Cellsparse-api, in Qupath. I created annotations with the SAM-api, which I assigned to the Foreground and Background classes with no issues but when I want to run Cellsparce>StarDist>training I get an internal server error.

This happens on a single-plane greyscale image. I've set up SAM and cellsparse on different ports.

Below is the error I get in my console:

INFO:     Started server process [28052]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:62580 - "POST /stardist/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 435, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\fastapi\routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\fastapi\routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\cellsparse_api\main.py", line 144, in stardist
    return run(
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\cellsparse_api\main.py", line 42, in run
    img = normalize(decode_image(b64img), 0, 100, axis=(0, 1))
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\cellsparse_api\main.py", line 30, in decode_image
    return np.array(Image.open(io.BytesIO(base64.b64decode(b64data))))
  File "C:\Users\LPUoO\anaconda3\envs\cellsparse-api\lib\site-packages\PIL\Image.py", line 3280, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x00000158029F9B70>

Am I making a mistake?

Thanks a lot!

[LPUoO]

I notice now that similar errors may have been reported in the two issues below:
cellsparse-api/issues/2
qupath-extension-cellsparse/issues/3

[ksugar]

Hi @LPUoO, thank you for reporting the issue.
Did you try it with a multi-channel image? Unfortunately, Cellsparse currently supports only grayscale images.

Thank you @FabianaMoresi, for reporting the issue!
It seems that the error happened when it tried to decode the image.
If you use an image with multi-channels, this error may happen.
I will work on solving it as soon as possible and will let you know once it's done.
For the moment, only grayscale images will work properly. If you want to test how it works, please try it with the grayscale images.

Originally posted by @ksugar in #2 (comment)

I will work on supporting it as soon as possible.

[LPUoO]

Did you try it with a multi-channel image?

No, this happens on single-plane greyscale images.

Thanks @ksugar !

[AlexHego]

Dear @ksugar,
First a big thank you for your work. I install SAM and Cellsparse with GPU. But I recieve error from Cellsparse
SAM have the port 8000
Cellspare port 18000
I train a model with Cellpose but I can't load it
I tried to train a model Stardist or use inference but I recieve the 500 Internal server Error

2023-07-11 12:40:41.509574: W tensorflow/core/common_runtime/bfc_allocator.cc:491] <allocator contains no memory>
INFO:     127.0.0.1:49816 - "POST /stardist/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 435, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\fastapi\applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\fastapi\routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\fastapi\routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\cellsparse_api\main.py", line 144, in stardist
    return run(
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\cellsparse_api\main.py", line 47, in run
    pred = runner._eval([img], modelname)[0]
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\cellsparse\runners\stardist_runner.py", line 106, in _eval
    model = StarDist2D(self.config, name=description, basedir=self.basedir)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\stardist\models\model2d.py", line 296, in __init__
    super().__init__(config, name=name, basedir=basedir)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\stardist\models\base.py", line 223, in __init__
    super().__init__(config=config, name=name, basedir=basedir)
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\csbdeep\models\base_model.py", line 111, in __init__
    self.keras_model = self._build()
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\stardist\models\model2d.py", line 312, in _build
    pooled_img = Conv2D(self.config.unet_n_filter_base, self.config.unet_kernel_size,
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\Alex\miniconda3\envs\cellsparse-api\lib\site-packages\keras\backend.py", line 2100, in random_uniform
    return tf.random.stateless_uniform(
tensorflow.python.framework.errors_impl.FailedPreconditionError: {{function_node __wrapped____EagerConst_device_/job:localhost/replica:0/task:0/device:GPU:0}} Failed to allocate scratch buffer for device 0

[OmarAshkar]

This is related

Node: 'model/conv2d/Relu' Fused conv implementation does not support grouped convolutions for now. [[{{node model/conv2d/Relu}}]] [Op:__inference_train_function_5859]

I think a parameter for the number of channels should be picked up from QuPath somehow.