RFC: po/bitmap vectorisation - prerequisit for AI masking

[TurboGit]

This is a first attempt at creating path forms from a raster mask. The goal is that this work will be used by the AI based masking if someone want to work on it.

There is still an offset issue, but the core is working. I have added a [vectorize] button on the external raster mask module.

Click on [vectorize]:

From there is it easy to reuse those new forms on any module as any other masks:

[TurboGit]

@jenshannoschwalm : I'll need your help on this. No matter what I try on some images I have a displacement of the drawn masks compared to the raster mask. So I'll need help from you pixelpipe expert.

[jenshannoschwalm]

No matter what I try on some images I have a displacement of the drawn masks compared to the raster mask. So I'll need help from you pixelpipe expert.

First, congrats and big thanks for getting into this, i think we will all love it.

Having tested yet but my first idea on this question, iirc the standard masks are defined on whole image data (before cropped in rawprepare that must be so as changing the crops for rawprepare should keep masks as they are) and it seems to me you currently define on data provided after rawprepare.

btw, the croppings are available in the image struct as crop_x and crop_y

[TurboGit]

@jenshannoschwalm : It seems that rawprepare transform/backtransform are not correct. All those offset are seen when there is crop in rawprepare. If you reset all to 0 then the drawn masks are perfectly aligned.

[jenshannoschwalm]

It seems that rawprepare transform/backtransform are not correct.

Currently not my thinking, they work fine for all distortions/masks right now.

What i meant is, when you process the provided raster image with the new algorithm all coordinates are inside the range provided by the data of that file but not as we have for rawprepare. Like here:

    // starting point
    float pts[2] = { start.x, start.y };
    dt_dev_distort_backtransform(darktable.develop, pts, 1);

for start.x/y you'd have to add some rawprepare crop-x/y data before doing the backtransform. no?

[zisoft]

To fix the CI/nightly fail on macOS you need to add

brew 'potrace'

to ./.ci/Brewfile

[TurboGit]

@zisoft : Done !

[TurboGit]

@jenshannoschwalm

for start.x/y you'd have to add some rawprepare crop-x/y data before doing the backtransform. no?

My brain hurts tonight :) How can I get rawprepare offset here? Isn't this the role of dt_dev_distort_backtransform()?

[jenshannoschwalm]

Isn't this the role of dt_dev_distort_backtransform()?

Arg, i think i may have hurted you without good reason :-) I think we have the distort_plus() variants where you can specify the iop order (leaving out rawprepare :-)

[zisoft]

oh, and please add

potrace \

to the hbDependencies list in ./packaging/macosx/1_install_hb_dependencies.sh (around line 40) :)

So we have all the necessary stuff in this PR.

[TurboGit]

Arg, i think i may have hurted you without good reason :-) I think we have the distort_plus() variants where you can specify the iop order (leaving out rawprepare :-)

My head hurts even more now :) I have evidence that if in rawprepare the 4 crop sliders the mask forms are properly created.

But as soon as there is some crop we have an offset:

So I don't think we want to ignore rawprepare.

[jenshannoschwalm]

Could you provide the test files you are using so a) we can compare and b) i will try my best on this topic and you have a wine or so?

[TurboGit]

@jenshannoschwalm : The tests files:

https://drive.google.com/file/d/17uUarrVT1R3t0yEXPxlp4e7B2LV-rMQU/view?usp=sharing

• Open mire1.cr2
• Add mask-3.pfm in the external raster file module
• Click on [vectorize]
• See the 3 forms created into the mask manager
• Select one and see the offset

Now before clicking on [vectorize] go to rawprepare module, in the crop section move all sliders to 0. Create again the forms and see that they are all properly aligned.

Part of the issue is that the mask is not in the size of the raw we probably need to scale x and y independently and we also need to take into account the x, y offset from rawprepare to align the raw and the mask. Now all I have tried has failed :)

[TurboGit]

I have pushed a new version with may wip commits. The last modification is for the ras2forms() routine to return forms in the mask space and then to rewrite the points depending on the image. This is better as in this case the ras2form() could be reused in some other contexts.

[jenshannoschwalm]

About _trans_pts() i think it should be pretty simple as below, btw there is similar code in dt_masks_legacy_params_v2_to_v3_transform()

  const dt_image_t *const image = &(self->dev->image_storage);

  const float iwidth = image->width;
  const float iheight = image->height;
  const float pwidth = image->p_width;
  const float pheight = image->p_height;
  const float cx = image->crop_x;
  const float cy = image->crop_y;

  const float xscale = pwidth / (float)cd->width;
  const float yscale = pheight / (float)cd->height;

  float posx = p[0] * xscale;
  float posy = p[1] * yscale;

  posx += cx;
  posy += cy;

  posx /= iwidth ;
  posy /= iheight ;
  dt_print(DT_DEBUG_ALWAYS, "insert raster point (%d/%d) in raster %dx%d image %dx%d cx=%d cy=%d scales %.2f %.2f as maskpoint (%.4f/%.4f)",
    (int)p[0], (int)p[1],
    cd->width, cd->height,
    (int)pwidth, (int)pheight, (int)cx, (int)cy, xscale, yscale, posx, posy);
  p[0] = posx;
  p[1] = posy;

yet - it doesn't work. Why - just check that crop_x/y are not correct here :-) At least that's one point.

It seems to me that rawprepare isn't doing it correctly (code in commit params seems to be wrong) but not sure if that's the whole story. Will investigate further ...

[TurboGit]

@jenshannoschwalm :

yet - it doesn't work. Why - just check that crop_x/y are not correct here :-) At least that's one point.

Works ok on 4 test images on my side with 2 different masks.

[TurboGit]

@jenshannoschwalm : Thanks a lot for your help ! I have just pushed a cleaned up version using the code you've propose.

[jenshannoschwalm]

There are two issues for me,

1. Dt crashing if file was not set on master before
2. If you change the rawpeepare crops, thats not taken into account. A minor issue i think.

[TurboGit]

1. Dt crashing if file was not set on master before

Should be fixed in current PR. If there is no file loaded the vectorize button is set inactive now.

[TurboGit]

And now Windows is fixed !

[TurboGit]

I plan also on displaying some information to the user after the vectorization:

• If no mask found
• Number of forms created

[jenshannoschwalm]

Tested again and working great!

While being here:

1. We miss orientation handling as has been reported
2. Other files than .pfm as in Raster Mask: PNG support and image folder #20170
3. imho this would also be the place for a "present image to outside algorithm, get result file from there and convert by new algo" workflow (without using rastermask folder)

[TurboGit]

Point 1. Can you tell me more about the orientation issue? I've probably missed it.

Point 3. Either that or with a new module or a Lua script, but clearly yes we don't need in this case a raster mask visible. My initial idea was to add a button on the Mask Manager to extract the subjects from the edited images and create the corresponding path forms. But all this will be more thinking when we know what kind of techno we want to integrate.

[jenshannoschwalm]

Point 1. Can you tell me more about the orientation issue? I've probably missed it.

We import the external rastermasks "as they are", likely generated from other software. If you export an image to be used for other algorithms, a portrait photo will be a portrait and likely treated as such by that software. But in the pipe position of the "external rastermask" we don't know and don't care yet and that mask will be also be rotated/flipped/...