Compact proof. by cheme · Pull Request #295 · paritytech/cumulus

cheme · 2021-01-18T15:11:28Z

This PR depends on :
paritytech/substrate#8574

This PR adds code proof compression (the state proof).

roots/state.

This reverts commit aa5c568.

project.

cheme · 2021-01-28T09:34:18Z

I switch this PR to also extract ':code' from proof.

The idea is that, with cumulus setting, validation code is the same as runtime code, so we have access to it while validating.

Therefore the code is removed from proof when compressing and added back when decompressing (in a lazy maner by some validation host function).

Notice that on polka size, code is pretty hacky, probably running
let validation_code = runtime_api_request( ctx, descriptor.relay_parent, RuntimeApiRequest::ValidationCode( descriptor.para_id, assumption, code_tx, ), code_rx, ).await?;
would be less impacting (still need to have these parameters attached to validation data.

Also design could be more generic (maybe this can be applied to other key than ':code'):

host function targetting known content: something like 'known_storage_key' returning 'Option<Option<Vec>'.
trie decompress could use a lazy trait that take key as input and trigger on ALL encoded 0 length content.
(currently we need to know skip key when decompressing)
This requires to escape the compressed 0 length content to some known byte sequence (so it changes the encoding
in the rather rare case of empty content).

Regarding test done:
At this point only the fact that integration test is running (I did see root mismatch when I inserted &[] as code before linking the host function).

So clearly there is more work to do (polkadot and alternative design), but this seems pretty ok, so this question is as relevant as either.

bkchr

2 small nitpicks

pallets/parachain-system/src/validate_block/implementation.rs

bkchr · 2021-06-07T17:31:16Z

pallets/parachain-system/src/validate_block/tests.rs

@@ -134,6 +143,7 @@ fn validate_block_no_extra_extrinsics() {
 		witness,
 		validation_data,
 	} = build_block_with_witness(&client, vec![], parent_head.clone());


Why isn't this function directly returning the CompactProof?

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

bkchr · 2021-06-07T17:38:14Z

pallets/parachain-system/src/validate_block/implementation.rs

+	let root = match sp_trie::decode_compact::<sp_trie::Layout<HashFor<B>>, _, _>(
+		&mut db,
+		storage_proof.iter_compact_encoded_nodes(),
+		Some(parent_head.state_root()),
+	) {


Ahh and didn't you added a method to the compactproof for uncompressing it?

We would have to rebuild the db afterward, here we directly got the db with the right content.

pallets/parachain-system/src/validate_block/implementation.rs

bkchr · 2021-06-07T17:54:50Z

bot merge

ghost · 2021-06-07T17:54:54Z

Waiting for commit status.

ghost · 2021-06-07T18:05:57Z

Merge aborted: Checks failed for 23afc08

* compact, need to be made optional and look into/compress child trie roots/state. * proto with child trie support * Missing set_offchain_storage overload. * right name * Ignore offchain indexing in validation function. * patch trie-db * decompress from iter * use compressed proof * remove wasm blob from proof (no inject plugged yet) * change lock * update trie * change in toml * Revert "change in toml" This reverts commit aa5c568. * use patch to branches * i * i:wq * switch branch * ii * ok, needed to patch the runtime by putting substrate patch in polkadot project. * test passing with this conf * actual lazy code fetch * patch issue * Code reorg * restore commented tests. * update deps. * remove polka patch * fixes * remove patch * revert cargo.lock * cargo update -p sp-trie polkadot-service * fix collator test (using parent state root). * Update pallets/parachain-system/src/validate_block/implementation.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Remove encode_witness test function. * Update pallets/parachain-system/src/validate_block/implementation.rs * Fix compilation Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <info@kchr.de>

This reverts commit d935b81.

cheme added 2 commits January 18, 2021 08:54

compact, need to be made optional and look into/compress child trie

5b29ae2

roots/state.

proto with child trie support

5d38bc6

cheme added the A3-inprogress label Jan 18, 2021

cheme changed the title ~~Should we compress proof?~~ Should we compress trie storage proof? Jan 18, 2021

cheme requested a review from bkchr January 18, 2021 15:16

cheme added 19 commits January 19, 2021 09:57

Missing set_offchain_storage overload.

9aa6397

right name

c4b932f

Ignore offchain indexing in validation function.

c04b767

patch trie-db

1c32a57

Merge branch 'ec-compress-proof' into ec-compress-proof-no-code

de5e8a0

decompress from iter

62c793d

use compressed proof

939f2b1

remove wasm blob from proof (no inject plugged yet)

7e14a81

change lock

a9d8d16

update trie

d73edc9

change in toml

aa5c568

Revert "change in toml"

762e2c3

This reverts commit aa5c568.

use patch to branches

719d44d

i

05b69e7

i:wq

8b72401

switch branch

9087c4c

ii

ad9a67b

ok, needed to patch the runtime by putting substrate patch in polkadot

c2b4459

project.

test passing with this conf

aa9d53d

actual lazy code fetch

d300493

cheme mentioned this pull request Jan 30, 2021

Allow skipping value for compact proof paritytech/trie#126

Open

cheme added 3 commits February 1, 2021 17:02

patch issue

9555c0d

Code reorg

4e41b12

restore commented tests.

44e8208

cheme changed the title ~~Compact proof and skipped wasm code.~~ Compact proof. Apr 9, 2021

fixes

39a12ec

cheme mentioned this pull request Apr 9, 2021

Compact proof utilities in sp_trie. paritytech/substrate#8574

Merged

cheme added 6 commits June 7, 2021 11:26

remove patch

6b47847

Merge branch 'master' into ec-compress-proof

50aa22f

revert cargo.lock

9e269ee

Merge branch 'master' into ec-compress-proof

4bcf210

cargo update -p sp-trie polkadot-service

2ced602

fix collator test (using parent state root).

b44c5f9

bkchr approved these changes Jun 7, 2021

View reviewed changes

Update pallets/parachain-system/src/validate_block/implementation.rs

47eb7b9

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

bkchr reviewed Jun 7, 2021

View reviewed changes

Remove encode_witness test function.

d205a7c

bkchr reviewed Jun 7, 2021

View reviewed changes

pallets/parachain-system/src/validate_block/implementation.rs Outdated Show resolved Hide resolved

Update pallets/parachain-system/src/validate_block/implementation.rs

23afc08

Fix compilation

16b7a4b

bkchr merged commit d935b81 into master Jun 8, 2021

bkchr deleted the ec-compress-proof branch June 8, 2021 09:51

bkchr added a commit that referenced this pull request Jun 23, 2021

Revert "Compact proof. (#295)"

e2850ce

This reverts commit d935b81.

bkchr added a commit that referenced this pull request Jun 30, 2021

Revert "Compact proof. (#295)"

8786309

This reverts commit d935b81.

JoshOrndorff pushed a commit to moonbeam-foundation/cumulus that referenced this pull request Jun 30, 2021

Revert "Compact proof. (paritytech#295)"

38dc027

This reverts commit d935b81.

andresilva added a commit that referenced this pull request Jul 15, 2021

Revert "Compact proof. (#295)"

2a50fc1

This reverts commit d935b81.

bkchr added a commit that referenced this pull request Jul 26, 2021

Revert "Compact proof. (#295)"

0f1caf3

This reverts commit d935b81.

yrong pushed a commit to yrong/cumulus that referenced this pull request Aug 11, 2021

Revert "Compact proof. (paritytech#295)"

fda39cb

This reverts commit d935b81.

yrong pushed a commit to yrong/cumulus that referenced this pull request Sep 3, 2021

Revert "Compact proof. (paritytech#295)"

04262f9

This reverts commit d935b81.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compact proof.#295

Compact proof.#295
bkchr merged 39 commits intomasterfrom
ec-compress-proof

cheme commented Jan 18, 2021 •

edited

Loading

Uh oh!

cheme commented Jan 28, 2021

Uh oh!

bkchr left a comment

Uh oh!

Uh oh!

bkchr Jun 7, 2021

Uh oh!

cheme Jun 7, 2021

Uh oh!

bkchr Jun 7, 2021

Uh oh!

cheme Jun 7, 2021

Uh oh!

bkchr Jun 7, 2021

Uh oh!

Uh oh!

bkchr commented Jun 7, 2021

Uh oh!

ghost commented Jun 7, 2021

Uh oh!

ghost commented Jun 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cheme commented Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cheme commented Jan 28, 2021

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bkchr Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

cheme Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

bkchr Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

cheme Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

bkchr Jun 7, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bkchr commented Jun 7, 2021

Uh oh!

ghost commented Jun 7, 2021

Uh oh!

ghost commented Jun 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cheme commented Jan 18, 2021 •

edited

Loading