Modality dispatch by mzweilin · Pull Request #136 · IntelLabs/MART

mzweilin · 2023-04-18T17:56:39Z

What does this PR do?

Implement modality_dispatch() in the singledispatch manner.
Add modality_dispatch() in Perturber for Initializer and Projector.
- Remove list-dispatch in Initializer and Projector.
Make Enforcer and Composer modality-aware with modality_dispatch()
- GradientModifier is different because of the way we handle trainable parameters.
Annotate modality-aware params in Optimizer for GradientModifier.
Add a modality-aware adversary example.

Benign:

CUDA_VISIBLE_DEVICES=0 \
python -m mart \
experiment=ArmoryCarlaOverObjDet_TorchvisionFasterRCNN \
trainer=gpu \
fit=false \
+trainer.limit_test_batches=1 \
+model.load_state_dict.losses_and_detections.model=/home/weilinxu/coder/GARD-with-MART/oscar/model_zoo/carla_rgb_weights_eval6.pt

│      test_metrics/map_50       │       0.6349384784698486

Adversarial:

CUDA_VISIBLE_DEVICES=0 \
python -m mart \
experiment=ArmoryCarlaOverObjDet_TorchvisionFasterRCNN \
trainer=gpu \
fit=false \
+trainer.limit_test_batches=1 \
+model.load_state_dict.losses_and_detections.model=/home/weilinxu/coder/GARD-with-MART/oscar/model_zoo/carla_rgb_weights_eval6.pt \
+attack@model.modules.input_adv_test=object_detection_rgb_mask_adversary \
+model.test_sequence.seq005=input_adv_test \
model.test_sequence.seq010.preprocessor=["input_adv_test"]

│      test_metrics/map_50       │       0.4633878767490387       │

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

pytest
CUDA_VISIBLE_DEVICES=0,1 python -m mart experiment=CIFAR10_CNN_Adv trainer=ddp trainer.precision=16 trainer.devices=2 model.optimizer.lr=0.2 trainer.max_steps=2925 datamodule.ims_per_batch=256 datamodule.world_size=2 reports 70% (14 sec/epoch).

Before submitting

The title is self-explanatory and the description concisely explains the PR
My PR does only one thing, instead of bundling different changes together
I list all the breaking changes introduced by this pull request
I have commented my code
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have run pre-commit hooks with pre-commit run -a command without errors

Did you have fun?

Make sure you had fun coding 🙃

dxoigmn

See comments.

dxoigmn · 2023-04-19T20:39:44Z

mart/utils/modality_dispatch.py

+    data: torch.Tensor | dict[str, torch.Tensor] | Iterable[Any],
+    *,
+    input: torch.Tensor | dict[str, torch.Tensor] | Iterable[Any],
+    target: torch.Tensor | Iterable[Any] | None,


These types should match whatever is in Enforcer above.

Um, shall we make some changes since it is a recursive function?

Huh? Whatever enforcer takes should be the same things this function takes? If you believe that, then why are the type annotations different?

Because modality_dispatch() is a recursive function that would see sub-components of the original input at some depth.

But so does Enforcer? I can pass a tensor, a iterable[tensor] or iterable[dict[str, tensor]] to Enforcer. I don't get why the function being recursive matters. The type annotations tell you what the function accepts.

I made it the same as in Enforcer.

Well, it is now different from Enforcer, because other modules such as Initializer also use modality_dispatch() and they may have different types of arguments.

dxoigmn · 2023-04-19T20:40:37Z

mart/attack/enforcer.py

-        input_adv: torch.Tensor | Iterable[torch.Tensor],
+        input_adv: torch.Tensor | Iterable[torch.Tensor] | Iterable[dict[str, torch.Tensor]],
        *,
-        input: torch.Tensor | Iterable[torch.Tensor],
-        target: torch.Tensor | Iterable[torch.Tensor | dict[str, Any]],
+        input: torch.Tensor | Iterable[torch.Tensor] | Iterable[dict[str, torch.Tensor]],
+        target: torch.Tensor | Iterable[torch.Tensor] | Iterable[dict[str, Any]],


Can you be more specific than Any with the types?

Fixed. I think it can be a tensor or a string (file name for example).

I changed it back to Any because it doesn't really matter. Other modules that use modality_dispatch() also annotate that as Any.

mart/attack/enforcer.py

mart/utils/modality_dispatch.py

dxoigmn · 2023-04-19T20:43:19Z

mart/utils/modality_dispatch.py

+    if isinstance(input, torch.Tensor):
+        if isinstance(modality_func, dict):
+            # A dictionary of Callable indexed by modality.
+            return modality_func[modality](data, input=input, target=target)
+        else:
+            # A Callable with modality=? as a keyword argument.
+            return modality_func(data, input=input, target=target, modality=modality)


I think it's clear to explicit check the type of each input instead of having nested if statements. else should trigger an exception.

I made it a singledispatch function.

dxoigmn · 2023-04-19T20:45:05Z

mart/utils/modality_dispatch.py

+    The function returns an object that is homomorphic to input and data.
+    """
+
+    assert type(data) == type(input)


Note that this will break for universal perturbation idea, since I want to apply the same perturbation to every input. instead, you should dispatch on types below and raise an NotImplementedError if you don't support that combination of types.

The new singledispatch implementation does not have the assertion.

dxoigmn

Did we ever discuss the merits of making Composer and GradientModifier modality aware instea of making Adversary modality aware? What if I want to generate a multi-modal universal attack?

dxoigmn · 2023-06-22T18:48:06Z

mart/attack/composer.py

-        perturbation: torch.Tensor | Iterable[torch.Tensor],
+        perturbation: torch.Tensor,
        *,
-        input: torch.Tensor | Iterable[torch.Tensor],
-        target: torch.Tensor | Iterable[torch.Tensor] | Iterable[dict[str, Any]],
+        input: torch.Tensor,
+        target: torch.Tensor | dict[str, Any],


These changes make me think this change is a regression? Why do we not accept Iterable[torch.Tensor] anymore?

OK. I reverted changes to Composer and created a new Modality(Composer).

dxoigmn · 2023-06-22T21:33:44Z

Do you think we should wait to merge this until we change MART to bound inputs between [0, 1] instead of [0, 255]? See #70.

…er to consume modality info.

dxoigmn and others added 9 commits April 14, 2023 14:25

Replace tuple with Iterable[torch.Tensor]

56014e6

Fix tests

1c47cc0

Cleanup

70cc36a

Make GradientModifier accept Iterable[torch.Tensor]

53ee7f4

Pull the modality_dispatch code from PR 115.

3f399fa

Add a constant DEFAULT_MODALITY.

fe87864

Add modality aware enforcer.

ad1f372

Type annotation.

5acc632

Type annotation.

8861436

mzweilin requested a review from dxoigmn April 18, 2023 17:57

dxoigmn requested changes Apr 19, 2023

View reviewed changes

Base automatically changed from iterable_instead_of_tuple to main May 16, 2023 15:57

mzweilin added 10 commits June 12, 2023 11:47

Merge branch 'main' into modality_dispatch

9c03718

Make a single-level if-else in modality_dispatch().

e46151d

Remove unused keys early.

7bb3321

Merge branch 'main' into modality_dispatch

ca92767

Make it fancy with singledispatch.

20ffada

Rename back to Enforcer.enforce().

e77236f

Comment.

3737c98

Comment.

3833c34

Loosen data type requirement.

f08510b

Modality-aware adversary.

4038610

mzweilin marked this pull request as draft June 21, 2023 03:35

mzweilin added 7 commits June 20, 2023 20:49

Backward compatible with exisiting non-modality configs of adversary.

7f47ab6

Fix test.

a121c80

Type annotation for modality-aware componenets.

c508266

Make a new name ModalityParameterDict for modality-aware parameters.

2676954

Fix function arguments and type annotations.

e386276

Make modality an optional keyword argument.

7f100d9

Fix type annotation.

c523907

mzweilin marked this pull request as ready for review June 21, 2023 23:12

mzweilin added 7 commits June 21, 2023 16:12

Fix type annotation.

8866c53

Simplify composer, initializerr and projector with modality_dispatch.

5611b3a

Simplify type annotaiton with modality_dispatch().

3fbd048

Update type annotation.

b675d3f

Make explicit function arguments from modality_dispatch().

2492232

Fix test.

430ed3f

Simplify type annotation.

7bfb9cb

mzweilin requested a review from dxoigmn June 22, 2023 17:50

dxoigmn reviewed Jun 22, 2023

View reviewed changes

mzweilin added 8 commits June 23, 2023 11:39

Revert changes in Composer and make a new Modality(Composer).

3607491

Merge branch 'main' into modality_dispatch

8c2a676

Add Modality(GradientModifier) and change the usage of GradientModifi…

086bda8

…er to consume modality info.

Fix test on gradient modifier.

c57465f

Cleanup.

bff59bc

Merge branch 'main' into modality_dispatch

ec648be

Merge branch 'main' into modality_dispatch

ef3a055

Keep modality-wise params for weights for later iterations.

a06db23

Conversation

mzweilin commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of change

Testing

Before submitting

Did you have fun?

Uh oh!

dxoigmn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dxoigmn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dxoigmn commented Jun 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mzweilin commented Apr 18, 2023 •

edited

Loading