From 71469687036d16dbf9241dd7f6dc9a3d3174e705 Mon Sep 17 00:00:00 2001 From: Andrew Chow Date: Wed, 26 Oct 2022 11:12:28 -0400 Subject: [PATCH 01/16] Merge bitcoin/bitcoin#25957: wallet: fast rescan with BIP157 block filters for descriptor wallets 0582932260e7de4e8aba01d63e7c8a9ddb9c3685 test: add test for fast rescan using block filters (top-up detection) (Sebastian Falbesoner) ca48a4694f73e5be8f971ae482ebc2cce4caef44 rpc: doc: mention rescan speedup using `blockfilterindex=1` in affected wallet RPCs (Sebastian Falbesoner) 3449880b499d54bfbcf6caeed52851ce55259ed7 wallet: fast rescan: show log message for every non-skipped block (Sebastian Falbesoner) 935c6c4b234bbb0565cda6f58ee298048856acae wallet: take use of `FastWalletRescanFilter` (Sebastian Falbesoner) 70b35139040a2351c845a1cec1dafd2fbcd16e93 wallet: add `FastWalletRescanFilter` class for speeding up rescans (Sebastian Falbesoner) c051026586fb269584bcba41de8a4a90280f5a7e wallet: add method for retrieving the end range for a ScriptPubKeyMan (Sebastian Falbesoner) 845279132b494f03b84d689c666fdcfad37f5a42 wallet: support fetching scriptPubKeys with minimum descriptor range index (Sebastian Falbesoner) 088e38d3bbea9694b319bc34e0d2e70d210c38b4 add chain interface methods for using BIP 157 block filters (Sebastian Falbesoner) Pull request description: ## Description This PR is another take of using BIP 157 block filters (enabled by `-blockfilterindex=1`) for faster wallet rescans and is a modern revival of #15845. For reviewers new to this topic I can highly recommend to read the corresponding PR review club (https://bitcoincore.reviews/15845). The basic idea is to skip blocks for deeper inspection (i.e. looking at every single tx for matches) if our block filter doesn't match any of the block's spent or created UTXOs are relevant for our wallet. Note that there can be false-positives (see https://bitcoincore.reviews/15845#l-199 for a PR review club discussion about false-positive rates), but no false-negatives, i.e. it is safe to skip blocks if the filter doesn't match; if the filter *does* match even though there are no wallet-relevant txs in the block, no harm is done, only a little more time is spent extra. In contrast to #15845, this solution only supports descriptor wallets, which are way more widespread now than back in the time >3 years ago. With that approach, we don't have to ever derive the relevant scriptPubKeys ourselves from keys before populating the filter, and can instead shift the full responsibility to that to the `DescriptorScriptPubKeyMan` which already takes care of that automatically. Compared to legacy wallets, the `IsMine` logic for descriptor wallets is as trivial as checking if a scriptPubKey is included in the ScriptPubKeyMan's set of scriptPubKeys (`m_map_script_pub_keys`): https://github.com/bitcoin/bitcoin/blob/e191fac4f3c37820f0618f72f0a8e8b524531ab8/src/wallet/scriptpubkeyman.cpp#L1703-L1710 One of the unaddressed issues of #15845 was that [the filter was only created once outside the loop](https://github.com/bitcoin/bitcoin/pull/15845#discussion_r343265997) and as such didn't take into account possible top-ups that have happened. This is solved here by keeping a state of ranged `DescriptorScriptPubKeyMan`'s descriptor end ranges and check at each iteration whether that range has increased since last time. If yes, we update the filter with all scriptPubKeys that have been added since the last filter update with a range index equal or higher than the last end range. Note that finding new scriptPubKeys could be made more efficient than linearly iterating through the whole `m_script_pub_keys` map (e.g. by introducing a bidirectional map), but this would mean introducing additional complexity and state and it's probably not worth it at this time, considering that the performance gain is already significant. Output scripts from non-ranged `DescriptorScriptPubKeyMan`s (i.e. ones with a fixed set of output scripts that is never extended) are added only once when the filter is created first. ## Benchmark results Obviously, the speed-up indirectly correlates with the wallet tx frequency in the scanned range: the more blocks contain wallet-related transactions, the less blocks can be skipped due to block filter detection. In a [simple benchmark](https://github.com/theStack/bitcoin/blob/fast_rescan_functional_test_benchmark/test/functional/pr25957_benchmark.py), a regtest chain with 1008 blocks (corresponding to 1 week) is mined with 20000 scriptPubKeys contained (25 txs * 800 outputs) each. The blocks each have a weight of ~2500000 WUs and hence are about 62.5% full. A global constant `WALLET_TX_BLOCK_FREQUENCY` defines how often wallet-related txs are included in a block. The created descriptor wallet (default setting of `keypool=1000`, we have 8*1000 = 8000 scriptPubKeys at the start) is backuped via the `backupwallet` RPC before the mining starts and imported via `restorewallet` RPC after. The measured time for taking this import process (which involves a rescan) once with block filters (`-blockfilterindex=1`) and once without block filters (`-blockfilterindex=0`) yield the relevant result numbers for the benchmark. The following table lists the results, sorted from worst-case (all blocks contain wallte-relevant txs, 0% can be skipped) to best-case (no blocks contain walltet-relevant txs, 100% can be skipped) where the frequencies have been picked arbitrarily: wallet-related tx frequency; 1 tx per... | ratio of irrelevant blocks | w/o filters | with filters | speed gain --------------------------------------------|-----------------------------|-------------|--------------|------------- ~ 10 minutes (every block) | 0% | 56.806s | 63.554s | ~0.9x ~ 20 minutes (every 2nd block) | 50% (1/2) | 58.896s | 36.076s | ~1.6x ~ 30 minutes (every 3rd block) | 66.67% (2/3) | 56.781s | 25.430s | ~2.2x ~ 1 hour (every 6th block) | 83.33% (5/6) | 58.193s | 15.786s | ~3.7x ~ 6 hours (every 36th block) | 97.22% (35/36) | 57.500s | 6.935s | ~8.3x ~ 1 day (every 144th block) | 99.31% (143/144) | 68.881s | 6.107s | ~11.3x (no txs) | 100% | 58.529s | 5.630s | ~10.4x Since even the (rather unrealistic) worst-case scenario of having wallet-related txs in _every_ block of the rescan range obviously doesn't take significantly longer, I'd argue it's reasonable to always take advantage of block filters if they are available and there's no need to provide an option for the user. Feedback about the general approach (but also about details like naming, where I struggled a lot) would be greatly appreciated. Thanks fly out to furszy for discussing this subject and patiently answering basic question about descriptor wallets! ACKs for top commit: achow101: ACK 0582932260e7de4e8aba01d63e7c8a9ddb9c3685 Sjors: re-utACK 0582932260e7de4e8aba01d63e7c8a9ddb9c3685 aureleoules: ACK 0582932260e7de4e8aba01d63e7c8a9ddb9c3685 - minor changes, documentation and updated test since last review w0xlt: re-ACK https://github.com/bitcoin/bitcoin/pull/25957/commits/0582932260e7de4e8aba01d63e7c8a9ddb9c3685 Tree-SHA512: 3289ba6e4572726e915d19f3e8b251d12a4cec8c96d041589956c484b5575e3708b14f6e1e121b05fe98aff1c8724de4564a5a9123f876967d33343cbef242e1 --- src/interfaces/chain.h | 9 ++ src/logging.cpp | 3 + src/logging.h | 1 + src/node/interfaces.cpp | 16 +++ src/wallet/rpc/backup.cpp | 7 +- src/wallet/rpc/transactions.cpp | 4 +- src/wallet/scriptpubkeyman.cpp | 14 ++- src/wallet/scriptpubkeyman.h | 2 + src/wallet/wallet.cpp | 140 ++++++++++++++++++++------ test/functional/test_runner.py | 1 + test/functional/wallet_fast_rescan.py | 102 +++++++++++++++++++ 11 files changed, 265 insertions(+), 34 deletions(-) create mode 100755 test/functional/wallet_fast_rescan.py diff --git a/src/interfaces/chain.h b/src/interfaces/chain.h index 19ebf9d6f3bf..e0c5560b0f4f 100644 --- a/src/interfaces/chain.h +++ b/src/interfaces/chain.h @@ -5,6 +5,7 @@ #ifndef BITCOIN_INTERFACES_CHAIN_H #define BITCOIN_INTERFACES_CHAIN_H +#include #include // For CTransactionRef #include // For util::SettingsValue @@ -139,6 +140,13 @@ class Chain //! or one of its ancestors. virtual std::optional findLocatorFork(const CBlockLocator& locator) = 0; + //! Returns whether a block filter index is available. + virtual bool hasBlockFilterIndex(BlockFilterType filter_type) = 0; + + //! Returns whether any of the elements match the block via a BIP 157 block filter + //! or std::nullopt if the block filter for this block couldn't be found. + virtual std::optional blockFilterMatchesAny(BlockFilterType filter_type, const uint256& block_hash, const GCSFilter::ElementSet& filter_set) = 0; + //! Check if transaction is locked by InstantSendManager virtual bool isInstantSendLockedTx(const uint256& hash) = 0; @@ -147,6 +155,7 @@ class Chain //! Return list of MN Collateral from outputs virtual std::vector listMNCollaterials(const std::vector>& outputs) = 0; + //! Return whether node has the block and optionally return block metadata //! or contents. virtual bool findBlock(const uint256& hash, const FoundBlock& block={}) = 0; diff --git a/src/logging.cpp b/src/logging.cpp index 3cf1b12a433e..98dcf1e2c69d 100644 --- a/src/logging.cpp +++ b/src/logging.cpp @@ -179,6 +179,7 @@ const CLogCategoryDesc LogCategories[] = #endif {BCLog::BLOCKSTORE, "blockstorage"}, {BCLog::TXRECONCILIATION, "txreconciliation"}, + {BCLog::SCAN, "scan"}, {BCLog::ALL, "1"}, {BCLog::ALL, "all"}, @@ -294,6 +295,8 @@ std::string LogCategoryToStr(BCLog::LogFlags category) return "blockstorage"; case BCLog::LogFlags::TXRECONCILIATION: return "txreconciliation"; + case BCLog::LogFlags::SCAN: + return "scan"; /* Start Dash */ case BCLog::LogFlags::CHAINLOCKS: return "chainlocks"; diff --git a/src/logging.h b/src/logging.h index ff7f98223585..d0b2069319f0 100644 --- a/src/logging.h +++ b/src/logging.h @@ -68,6 +68,7 @@ namespace BCLog { #endif BLOCKSTORE = (1 << 26), TXRECONCILIATION = (1 << 27), + SCAN = (1 << 28), //Start Dash CHAINLOCKS = ((uint64_t)1 << 32), diff --git a/src/node/interfaces.cpp b/src/node/interfaces.cpp index d3e6283eb5a9..61f2f198b7f4 100644 --- a/src/node/interfaces.cpp +++ b/src/node/interfaces.cpp @@ -6,6 +6,7 @@ #include #include +#include #include #include #include @@ -18,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -1067,6 +1069,20 @@ class ChainImpl : public Chain } return std::nullopt; } + bool hasBlockFilterIndex(BlockFilterType filter_type) override + { + return GetBlockFilterIndex(filter_type) != nullptr; + } + std::optional blockFilterMatchesAny(BlockFilterType filter_type, const uint256& block_hash, const GCSFilter::ElementSet& filter_set) override + { + const BlockFilterIndex* block_filter_index{GetBlockFilterIndex(filter_type)}; + if (!block_filter_index) return std::nullopt; + + BlockFilter filter; + const CBlockIndex* index{WITH_LOCK(::cs_main, return chainman().m_blockman.LookupBlockIndex(block_hash))}; + if (index == nullptr || !block_filter_index->LookupFilter(index, filter)) return std::nullopt; + return filter.GetFilter().MatchAny(filter_set); + } bool isInstantSendLockedTx(const uint256& hash) override { if (m_node.llmq_ctx == nullptr || m_node.llmq_ctx->isman == nullptr) return false; diff --git a/src/wallet/rpc/backup.cpp b/src/wallet/rpc/backup.cpp index 25829a801b1e..a30b98b52fb8 100644 --- a/src/wallet/rpc/backup.cpp +++ b/src/wallet/rpc/backup.cpp @@ -1824,7 +1824,8 @@ RPCHelpMan importdescriptors() { return RPCHelpMan{"importdescriptors", "\nImport descriptors. This will trigger a rescan of the blockchain based on the earliest timestamp of all descriptors being imported. Requires a new wallet backup.\n" "\nNote: This call can take over an hour to complete if using an early timestamp; during that time, other rpc calls\n" - "may report that the imported keys, addresses or scripts exist but related transactions are still missing.\n", + "may report that the imported keys, addresses or scripts exist but related transactions are still missing.\n" + "The rescan is significantly faster if block filters are available (using startup option \"-blockfilterindex=1\").\n", { {"requests", RPCArg::Type::ARR, RPCArg::Optional::NO, "Data to be imported", { @@ -2155,7 +2156,9 @@ RPCHelpMan restorewallet() { return RPCHelpMan{ "restorewallet", - "\nRestore and loads a wallet from backup.\n", + "\nRestore and loads a wallet from backup.\n" + "\nThe rescan is significantly faster if a descriptor wallet is restored" + "\nand block filters are available (using startup option \"-blockfilterindex=1\").\n", { {"wallet_name", RPCArg::Type::STR, RPCArg::Optional::NO, "The name that will be applied to the restored wallet"}, {"backup_file", RPCArg::Type::STR, RPCArg::Optional::NO, "The backup file that will be used to restore the wallet."}, diff --git a/src/wallet/rpc/transactions.cpp b/src/wallet/rpc/transactions.cpp index 05be4fcdf800..c141d382e067 100644 --- a/src/wallet/rpc/transactions.cpp +++ b/src/wallet/rpc/transactions.cpp @@ -847,7 +847,9 @@ RPCHelpMan rescanblockchain() { return RPCHelpMan{"rescanblockchain", "\nRescan the local blockchain for wallet related transactions.\n" - "Note: Use \"getwalletinfo\" to query the scanning progress.\n", + "Note: Use \"getwalletinfo\" to query the scanning progress.\n" + "The rescan is significantly faster when used on a descriptor wallet\n" + "and block filters are available (using startup option \"-blockfilterindex=1\").\n", { {"start_height", RPCArg::Type::NUM, RPCArg::Default{0}, "block height where the rescan should start"}, {"stop_height", RPCArg::Type::NUM, RPCArg::Optional::OMITTED_NAMED_ARG, "the last block height that should be scanned. If none is provided it will rescan up to the tip at return time of this call."}, diff --git a/src/wallet/scriptpubkeyman.cpp b/src/wallet/scriptpubkeyman.cpp index 9615bbf7e0f3..221e3826722c 100644 --- a/src/wallet/scriptpubkeyman.cpp +++ b/src/wallet/scriptpubkeyman.cpp @@ -2514,17 +2514,27 @@ WalletDescriptor DescriptorScriptPubKeyMan::GetWalletDescriptor() const } std::vector DescriptorScriptPubKeyMan::GetScriptPubKeys() const +{ + return GetScriptPubKeys(0); +} + +std::vector DescriptorScriptPubKeyMan::GetScriptPubKeys(int32_t minimum_index) const { LOCK(cs_desc_man); std::vector script_pub_keys; script_pub_keys.reserve(m_map_script_pub_keys.size()); - for (auto const& script_pub_key: m_map_script_pub_keys) { - script_pub_keys.push_back(script_pub_key.first); + for (auto const& [script_pub_key, index] : m_map_script_pub_keys) { + if (index >= minimum_index) script_pub_keys.push_back(script_pub_key); } return script_pub_keys; } +int32_t DescriptorScriptPubKeyMan::GetEndRange() const +{ + return m_max_cached_index + 1; +} + bool DescriptorScriptPubKeyMan::GetDescriptorString(std::string& out, const bool priv) const { LOCK(cs_desc_man); diff --git a/src/wallet/scriptpubkeyman.h b/src/wallet/scriptpubkeyman.h index 631b442e9ef4..060e0cb2720e 100644 --- a/src/wallet/scriptpubkeyman.h +++ b/src/wallet/scriptpubkeyman.h @@ -641,6 +641,8 @@ class DescriptorScriptPubKeyMan : public ScriptPubKeyMan WalletDescriptor GetWalletDescriptor() const EXCLUSIVE_LOCKS_REQUIRED(cs_desc_man); std::vector GetScriptPubKeys() const; + std::vector GetScriptPubKeys(int32_t minimum_index) const; + int32_t GetEndRange() const; bool GetDescriptorString(std::string& out, const bool priv) const; bool GetMnemonicString(SecureString& mnemonic_out, SecureString& mnemonic_passphrase_out) const; diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp index 3132c36766d6..02b07f4184d6 100644 --- a/src/wallet/wallet.cpp +++ b/src/wallet/wallet.cpp @@ -6,6 +6,7 @@ #include +#include #include #include #include @@ -280,6 +281,64 @@ std::shared_ptr LoadWalletInternal(WalletContext& context, const std::s return nullptr; } } + +class FastWalletRescanFilter +{ +public: + FastWalletRescanFilter(const CWallet& wallet) : m_wallet(wallet) + { + // fast rescanning via block filters is only supported by descriptor wallets right now + assert(!m_wallet.IsLegacy()); + + // create initial filter with scripts from all ScriptPubKeyMans + for (auto spkm : m_wallet.GetAllScriptPubKeyMans()) { + auto desc_spkm{dynamic_cast(spkm)}; + assert(desc_spkm != nullptr); + AddScriptPubKeys(desc_spkm); + // save each range descriptor's end for possible future filter updates + if (desc_spkm->IsHDEnabled()) { + m_last_range_ends.emplace(desc_spkm->GetID(), desc_spkm->GetEndRange()); + } + } + } + + void UpdateIfNeeded() + { + // repopulate filter with new scripts if top-up has happened since last iteration + for (const auto& [desc_spkm_id, last_range_end] : m_last_range_ends) { + auto desc_spkm{dynamic_cast(m_wallet.GetScriptPubKeyMan(desc_spkm_id))}; + assert(desc_spkm != nullptr); + int32_t current_range_end{desc_spkm->GetEndRange()}; + if (current_range_end > last_range_end) { + AddScriptPubKeys(desc_spkm, last_range_end); + m_last_range_ends.at(desc_spkm->GetID()) = current_range_end; + } + } + } + + std::optional MatchesBlock(const uint256& block_hash) const + { + return m_wallet.chain().blockFilterMatchesAny(BlockFilterType::BASIC_FILTER, block_hash, m_filter_set); + } + +private: + const CWallet& m_wallet; + /** Map for keeping track of each range descriptor's last seen end range. + * This information is used to detect whether new addresses were derived + * (that is, if the current end range is larger than the saved end range) + * after processing a block and hence a filter set update is needed to + * take possible keypool top-ups into account. + */ + std::map m_last_range_ends; + GCSFilter::ElementSet m_filter_set; + + void AddScriptPubKeys(const DescriptorScriptPubKeyMan* desc_spkm, int32_t last_range_end = 0) + { + for (const auto& script_pub_key : desc_spkm->GetScriptPubKeys(last_range_end)) { + m_filter_set.emplace(script_pub_key.begin(), script_pub_key.end()); + } + } +}; } // namespace std::shared_ptr LoadWallet(WalletContext& context, const std::string& name, std::optional load_on_start, const DatabaseOptions& options, DatabaseStatus& status, bilingual_str& error, std::vector& warnings) @@ -1835,7 +1894,11 @@ CWallet::ScanResult CWallet::ScanForWalletTransactions(const uint256& start_bloc uint256 block_hash = start_block; ScanResult result; - WalletLogPrintf("Rescan started from block %s...\n", start_block.ToString()); + std::unique_ptr fast_rescan_filter; + if (!IsLegacy() && chain().hasBlockFilterIndex(BlockFilterType::BASIC_FILTER)) fast_rescan_filter = std::make_unique(*this); + + WalletLogPrintf("Rescan started from block %s... (%s)\n", start_block.ToString(), + fast_rescan_filter ? "fast variant using block filters" : "slow variant inspecting all blocks"); ShowProgress(strprintf("%s " + _("Rescanning…").translated, GetDisplayName()), 0); // show rescan progress in GUI as dialog or on splashscreen, if -rescan on startup uint256 tip_hash = WITH_LOCK(cs_wallet, return GetLastBlockHash()); @@ -1862,9 +1925,22 @@ CWallet::ScanResult CWallet::ScanForWalletTransactions(const uint256& start_bloc WalletLogPrintf("Still rescanning. At block %d. Progress=%f\n", block_height, progress_current); } - // Read block data - CBlock block; - chain().findBlock(block_hash, FoundBlock().data(block)); + bool fetch_block{true}; + if (fast_rescan_filter) { + fast_rescan_filter->UpdateIfNeeded(); + auto matches_block{fast_rescan_filter->MatchesBlock(block_hash)}; + if (matches_block.has_value()) { + if (*matches_block) { + LogPrint(BCLog::SCAN, "Fast rescan: inspect block %d [%s] (filter matched)\n", block_height, block_hash.ToString()); + } else { + result.last_scanned_block = block_hash; + result.last_scanned_height = block_height; + fetch_block = false; + } + } else { + LogPrint(BCLog::SCAN, "Fast rescan: inspect block %d [%s] (WARNING: block filter not found!)\n", block_height, block_hash.ToString()); + } + } // Find next block separately from reading data above, because reading // is slow and there might be a reorg while it is read. @@ -1873,35 +1949,41 @@ CWallet::ScanResult CWallet::ScanForWalletTransactions(const uint256& start_bloc uint256 next_block_hash; chain().findBlock(block_hash, FoundBlock().inActiveChain(block_still_active).nextBlock(FoundBlock().inActiveChain(next_block).hash(next_block_hash))); - if (!block.IsNull()) { - LOCK(cs_wallet); - if (!block_still_active) { - // Abort scan if current block is no longer active, to prevent - // marking transactions as coming from the wrong block. - result.last_failed_block = block_hash; - result.status = ScanResult::FAILURE; - break; - } - for (size_t posInBlock = 0; posInBlock < block.vtx.size(); ++posInBlock) { - SyncTransaction(block.vtx[posInBlock], TxStateConfirmed{block_hash, block_height, static_cast(posInBlock)}, batch, fUpdate, /*rescanning_old_block=*/true); - } - // scan succeeded, record block as most recent successfully scanned - result.last_scanned_block = block_hash; - result.last_scanned_height = block_height; + if (fetch_block) { + // Read block data + CBlock block; + chain().findBlock(block_hash, FoundBlock().data(block)); + + if (!block.IsNull()) { + LOCK(cs_wallet); + if (!block_still_active) { + // Abort scan if current block is no longer active, to prevent + // marking transactions as coming from the wrong block. + result.last_failed_block = block_hash; + result.status = ScanResult::FAILURE; + break; + } + for (size_t posInBlock = 0; posInBlock < block.vtx.size(); ++posInBlock) { + SyncTransaction(block.vtx[posInBlock], TxStateConfirmed{block_hash, block_height, static_cast(posInBlock)}, batch, fUpdate, /*rescanning_old_block=*/true); + } + // scan succeeded, record block as most recent successfully scanned + result.last_scanned_block = block_hash; + result.last_scanned_height = block_height; - if (save_progress && next_interval) { - CBlockLocator loc = m_chain->getActiveChainLocator(block_hash); + if (save_progress && next_interval) { + CBlockLocator loc = m_chain->getActiveChainLocator(block_hash); - if (!loc.IsNull()) { - WalletLogPrintf("Saving scan progress %d.\n", block_height); - WalletBatch batch(GetDatabase()); - batch.WriteBestBlock(loc); + if (!loc.IsNull()) { + WalletLogPrintf("Saving scan progress %d.\n", block_height); + WalletBatch batch(GetDatabase()); + batch.WriteBestBlock(loc); + } } + } else { + // could not scan block, keep scanning but record this block as the most recent failure + result.last_failed_block = block_hash; + result.status = ScanResult::FAILURE; } - } else { - // could not scan block, keep scanning but record this block as the most recent failure - result.last_failed_block = block_hash; - result.status = ScanResult::FAILURE; } if (max_height && block_height >= *max_height) { break; diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py index fcbc405d2d9b..bb7fe5cf5a16 100755 --- a/test/functional/test_runner.py +++ b/test/functional/test_runner.py @@ -166,6 +166,7 @@ 'rpc_quorum.py', 'wallet_keypool_topup.py --legacy-wallet', 'wallet_keypool_topup.py --descriptors', + 'wallet_fast_rescan.py --descriptors', 'feature_fee_estimation.py', 'interface_zmq_dash.py --legacy-wallet', 'interface_zmq.py', diff --git a/test/functional/wallet_fast_rescan.py b/test/functional/wallet_fast_rescan.py new file mode 100755 index 000000000000..84729278b212 --- /dev/null +++ b/test/functional/wallet_fast_rescan.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +# Copyright (c) 2022 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test that fast rescan using block filters for descriptor wallets detects + top-ups correctly and finds the same transactions than the slow variant.""" +import os +from typing import List + +from test_framework.descriptors import descsum_create +from test_framework.test_framework import BitcoinTestFramework +from test_framework.test_node import TestNode +from test_framework.util import assert_equal +from test_framework.wallet import MiniWallet +from test_framework.wallet_util import get_generate_key + + +KEYPOOL_SIZE = 100 # smaller than default size to speed-up test +NUM_DESCRIPTORS = 3 # number of descriptors (2 default ranged ones + 1 fixed non-ranged one) +NUM_BLOCKS = 6 # number of blocks to mine + + +class WalletFastRescanTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 1 + self.extra_args = [[f'-keypool={KEYPOOL_SIZE}', '-blockfilterindex=1']] + + def skip_test_if_missing_module(self): + self.skip_if_no_wallet() + self.skip_if_no_sqlite() + + def get_wallet_txids(self, node: TestNode, wallet_name: str) -> List[str]: + w = node.get_wallet_rpc(wallet_name) + txs = w.listtransactions('*', 1000000) + return [tx['txid'] for tx in txs] + + def run_test(self): + node = self.nodes[0] + wallet = MiniWallet(node) + wallet.rescan_utxos() + + self.log.info("Create descriptor wallet with backup") + WALLET_BACKUP_FILENAME = os.path.join(node.datadir, 'wallet.bak') + node.createwallet(wallet_name='topup_test', descriptors=True) + w = node.get_wallet_rpc('topup_test') + fixed_key = get_generate_key() + print(w.importdescriptors([{"desc": descsum_create(f"wpkh({fixed_key.privkey})"), "timestamp": "now"}])) + descriptors = w.listdescriptors()['descriptors'] + assert_equal(len(descriptors), NUM_DESCRIPTORS) + w.backupwallet(WALLET_BACKUP_FILENAME) + + self.log.info(f"Create txs sending to end range address of each descriptor, triggering top-ups") + for i in range(NUM_BLOCKS): + self.log.info(f"Block {i+1}/{NUM_BLOCKS}") + for desc_info in w.listdescriptors()['descriptors']: + if 'range' in desc_info: + start_range, end_range = desc_info['range'] + addr = w.deriveaddresses(desc_info['desc'], [end_range, end_range])[0] + spk = bytes.fromhex(w.getaddressinfo(addr)['scriptPubKey']) + self.log.info(f"-> range [{start_range},{end_range}], last address {addr}") + else: + spk = bytes.fromhex(fixed_key.p2wpkh_script) + self.log.info(f"-> fixed non-range descriptor address {fixed_key.p2wpkh_addr}") + wallet.send_to(from_node=node, scriptPubKey=spk, amount=10000) + self.generate(node, 1) + + self.log.info("Import wallet backup with block filter index") + with node.assert_debug_log(['fast variant using block filters']): + node.restorewallet('rescan_fast', WALLET_BACKUP_FILENAME) + txids_fast = self.get_wallet_txids(node, 'rescan_fast') + + self.log.info("Import non-active descriptors with block filter index") + node.createwallet(wallet_name='rescan_fast_nonactive', descriptors=True, disable_private_keys=True, blank=True) + with node.assert_debug_log(['fast variant using block filters']): + w = node.get_wallet_rpc('rescan_fast_nonactive') + w.importdescriptors([{"desc": descriptor['desc'], "timestamp": 0} for descriptor in descriptors]) + txids_fast_nonactive = self.get_wallet_txids(node, 'rescan_fast_nonactive') + + self.restart_node(0, [f'-keypool={KEYPOOL_SIZE}', '-blockfilterindex=0']) + self.log.info("Import wallet backup w/o block filter index") + with node.assert_debug_log(['slow variant inspecting all blocks']): + node.restorewallet("rescan_slow", WALLET_BACKUP_FILENAME) + txids_slow = self.get_wallet_txids(node, 'rescan_slow') + + self.log.info("Import non-active descriptors w/o block filter index") + node.createwallet(wallet_name='rescan_slow_nonactive', descriptors=True, disable_private_keys=True, blank=True) + with node.assert_debug_log(['slow variant inspecting all blocks']): + w = node.get_wallet_rpc('rescan_slow_nonactive') + w.importdescriptors([{"desc": descriptor['desc'], "timestamp": 0} for descriptor in descriptors]) + txids_slow_nonactive = self.get_wallet_txids(node, 'rescan_slow_nonactive') + + self.log.info("Verify that all rescans found the same txs in slow and fast variants") + assert_equal(len(txids_slow), NUM_DESCRIPTORS * NUM_BLOCKS) + assert_equal(len(txids_fast), NUM_DESCRIPTORS * NUM_BLOCKS) + assert_equal(len(txids_slow_nonactive), NUM_DESCRIPTORS * NUM_BLOCKS) + assert_equal(len(txids_fast_nonactive), NUM_DESCRIPTORS * NUM_BLOCKS) + assert_equal(sorted(txids_slow), sorted(txids_fast)) + assert_equal(sorted(txids_slow_nonactive), sorted(txids_fast_nonactive)) + + +if __name__ == '__main__': + WalletFastRescanTest().main() From 2e1722a7a3c7626f9ddd9144fb1ba967ef0feab2 Mon Sep 17 00:00:00 2001 From: fanquake Date: Thu, 19 Jan 2023 13:40:11 +0000 Subject: [PATCH 02/16] Merge bitcoin/bitcoin#26920: doc: add release note for #25957 (fast wallet rescan) 783288334c98844ffce1f8ffa0e64a9fdb0c1c13 doc: add release note for #25957 (fast wallet rescan) (Sebastian Falbesoner) Pull request description: This PR adds a missing release note for #25957. ACKs for top commit: Sjors: ACK 783288334c98844ffce1f8ffa0e64a9fdb0c1c13 Tree-SHA512: 817aa3d27b3f839de3975ace7c8ec59bcc4dbe4b5628bf64153e503cd143599d8923bd7e181ad5b196dacf1a9078347825bc40d4de5c6e2df9ed12e752217094 --- doc/release-notes-25957.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/release-notes-25957.md diff --git a/doc/release-notes-25957.md b/doc/release-notes-25957.md new file mode 100644 index 000000000000..c71afa2c2e17 --- /dev/null +++ b/doc/release-notes-25957.md @@ -0,0 +1,9 @@ +Wallet +------ + +- Rescans for descriptor wallets are now significantly faster if compact + block filters (BIP158) are available. Since those are not constructed + by default, the configuration option "-blockfilterindex=1" has to be + provided to take advantage of the optimization. This improves the + performance of the RPC calls `rescanblockchain`, `importdescriptors` + and `restorewallet`. (#25957) From c5d2173ed66d7a07b3d6b1bd9a1a9d5292ce900e Mon Sep 17 00:00:00 2001 From: Andrew Chow Date: Thu, 20 Oct 2022 10:59:31 -0400 Subject: [PATCH 03/16] Merge bitcoin/bitcoin#26158: bench: add "priority level" to the benchmark framework BACKPORT NOTE: missing changes are in: - src/bench/addrman.cpp - src/bench/verify_script.cpp - src/bench/chacha_poly_aead.cpp - src/bench/descriptors.cpp - src/bench/chacha20.cpp - src/bench/crypto_hash.cpp But backport is not partial because adding these files without these changes will cause compilation error 3e9d0bea8deb61596c91ead997e9db83f5b0ff68 build: only run high priority benchmarks in 'make check' (furszy) 466b54bd4ab8227ff8c066a027a92791366a81c1 bench: surround main() execution with try/catch (furszy) 3da7cd2a762077fa81dc40832d556d8a3fd53674 bench: explicitly make all current benchmarks "high" priority (furszy) 05b8c76232dedf938740e8034c725ac16d32974a bench: add "priority level" to the benchmark framework (furszy) f1593780b8e3b6adefee08b10d270c5c329f91fe bench: place benchmark implementation inside benchmark namespace (furszy) Pull request description: This is from today's meeting, a simple "priority level" for the benchmark framework. Will allow us to run certain benchmarks while skip non-prioritized ones in `make check`. By default, `bench_bitcoin` will run all the benchmarks. `make check`will only run the high priority ones, and have marked all the existent benchmarks as "high priority" to retain the current behavior. Could test it by modifying any benchmark priority to something different from "high", and run `bench_bitcoin -priority-level=high` and/or `bench_bitcoin -priority-level=medium,low` (the first command will skip the modified bench while the second one will include it). Note: the second commit could be avoided by having a default arg value for the priority level but.. an explicit set in every `BENCHMARK` macro call makes it less error-prone. ACKs for top commit: kouloumos: re-ACK 3e9d0bea8deb61596c91ead997e9db83f5b0ff68 achow101: ACK 3e9d0bea8deb61596c91ead997e9db83f5b0ff68 theStack: re-ACK 3e9d0bea8deb61596c91ead997e9db83f5b0ff68 stickies-v: re-ACK https://github.com/bitcoin/bitcoin/commit/3e9d0bea8deb61596c91ead997e9db83f5b0ff68 Tree-SHA512: ece59bf424c5fc1db335f84caa507476fb8ad8c6151880f1f8289562e17023aae5b5e7de03e8cbba6337bf09215f9be331e9ef51c791c43bce43f7446813b054 fixup bench --- src/bench/addrman.cpp | 12 +-- src/bench/base58.cpp | 6 +- src/bench/bech32.cpp | 4 +- src/bench/bench.h | 4 +- src/bench/bip324_ecdh.cpp | 2 +- src/bench/block_assemble.cpp | 2 +- src/bench/bls.cpp | 28 ++--- src/bench/bls_dkg.cpp | 8 +- src/bench/bls_pubkey_agg.cpp | 20 ++-- src/bench/ccoins_caching.cpp | 2 +- src/bench/chacha20.cpp | 12 +-- src/bench/checkblock.cpp | 4 +- src/bench/checkqueue.cpp | 2 +- src/bench/coin_selection.cpp | 4 +- src/bench/crypto_hash.cpp | 64 +++++------ src/bench/duplicate_inputs.cpp | 2 +- src/bench/ecdsa.cpp | 6 +- src/bench/ellswift.cpp | 2 +- src/bench/examples.cpp | 2 +- src/bench/gcs_filter.cpp | 10 +- src/bench/hashpadding.cpp | 4 +- src/bench/load_external.cpp | 2 +- src/bench/lockedpool.cpp | 2 +- src/bench/logging.cpp | 10 +- src/bench/mempool_eviction.cpp | 2 +- src/bench/mempool_stress.cpp | 4 +- src/bench/merkle_root.cpp | 2 +- src/bench/peer_eviction.cpp | 12 +-- src/bench/poly1305.cpp | 6 +- src/bench/pool.cpp | 4 +- src/bench/pow_hash.cpp | 190 ++++++++++++++++----------------- src/bench/prevector.cpp | 8 +- src/bench/rollingbloom.cpp | 4 +- src/bench/rpc_blockchain.cpp | 4 +- src/bench/rpc_mempool.cpp | 2 +- src/bench/strencodings.cpp | 2 +- src/bench/string_cast.cpp | 12 +-- src/bench/util_time.cpp | 8 +- src/bench/verify_script.cpp | 2 +- src/bench/wallet_balance.cpp | 8 +- src/bench/wallet_create.cpp | 4 +- src/bench/wallet_loading.cpp | 4 +- 42 files changed, 246 insertions(+), 246 deletions(-) diff --git a/src/bench/addrman.cpp b/src/bench/addrman.cpp index b17e508e36da..e6eeb0301107 100644 --- a/src/bench/addrman.cpp +++ b/src/bench/addrman.cpp @@ -167,9 +167,9 @@ static void AddrManAddThenGood(benchmark::Bench& bench) }); } -BENCHMARK(AddrManAdd); -BENCHMARK(AddrManSelect); -BENCHMARK(AddrManSelectFromAlmostEmpty); -BENCHMARK(AddrManSelectByNetwork); -BENCHMARK(AddrManGetAddr); -BENCHMARK(AddrManAddThenGood); +BENCHMARK(AddrManAdd, benchmark::PriorityLevel::HIGH); +BENCHMARK(AddrManSelect, benchmark::PriorityLevel::HIGH); +BENCHMARK(AddrManSelectFromAlmostEmpty, benchmark::PriorityLevel::HIGH); +BENCHMARK(AddrManSelectByNetwork, benchmark::PriorityLevel::HIGH); +BENCHMARK(AddrManGetAddr, benchmark::PriorityLevel::HIGH); +BENCHMARK(AddrManAddThenGood, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/base58.cpp b/src/bench/base58.cpp index 6f6b4e3bfa9c..3d08b7201b71 100644 --- a/src/bench/base58.cpp +++ b/src/bench/base58.cpp @@ -50,6 +50,6 @@ static void Base58Decode(benchmark::Bench& bench) } -BENCHMARK(Base58Encode); -BENCHMARK(Base58CheckEncode); -BENCHMARK(Base58Decode); +BENCHMARK(Base58Encode, benchmark::PriorityLevel::HIGH); +BENCHMARK(Base58CheckEncode, benchmark::PriorityLevel::HIGH); +BENCHMARK(Base58Decode, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/bech32.cpp b/src/bench/bech32.cpp index 7af4b3fea390..59bafb65fd7a 100644 --- a/src/bench/bech32.cpp +++ b/src/bench/bech32.cpp @@ -32,5 +32,5 @@ static void Bech32Decode(benchmark::Bench& bench) } -BENCHMARK(Bech32Encode); -BENCHMARK(Bech32Decode); +BENCHMARK(Bech32Encode, benchmark::PriorityLevel::HIGH); +BENCHMARK(Bech32Decode, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/bench.h b/src/bench/bench.h index 1f412c9aecbe..63e1bf67e21e 100644 --- a/src/bench/bench.h +++ b/src/bench/bench.h @@ -76,7 +76,7 @@ class BenchRunner } // namespace benchmark // BENCHMARK(foo) expands to: benchmark::BenchRunner bench_11foo("foo", foo, priority_level); -#define BENCHMARK(n) \ - benchmark::BenchRunner PASTE2(bench_, PASTE2(__LINE__, n))(STRINGIZE(n), n, benchmark::PriorityLevel::HIGH); +#define BENCHMARK(n, priority_level) \ + benchmark::BenchRunner PASTE2(bench_, PASTE2(__LINE__, n))(STRINGIZE(n), n, priority_level); #endif // BITCOIN_BENCH_BENCH_H diff --git a/src/bench/bip324_ecdh.cpp b/src/bench/bip324_ecdh.cpp index b3ac57f8b287..659da0f08e60 100644 --- a/src/bench/bip324_ecdh.cpp +++ b/src/bench/bip324_ecdh.cpp @@ -48,4 +48,4 @@ static void BIP324_ECDH(benchmark::Bench& bench) ECC_Stop(); } -BENCHMARK(BIP324_ECDH); +BENCHMARK(BIP324_ECDH, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/block_assemble.cpp b/src/bench/block_assemble.cpp index 01832712b706..c34e7b3c7b42 100644 --- a/src/bench/block_assemble.cpp +++ b/src/bench/block_assemble.cpp @@ -52,4 +52,4 @@ static void AssembleBlock(benchmark::Bench& bench) }); } -BENCHMARK(AssembleBlock); +BENCHMARK(AssembleBlock, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/bls.cpp b/src/bench/bls.cpp index e6206d241d9f..af6fc16b6928 100644 --- a/src/bench/bls.cpp +++ b/src/bench/bls.cpp @@ -358,17 +358,17 @@ static void BLS_Verify_BatchedParallel(benchmark::Bench& bench) blsWorker.Stop(); } -BENCHMARK(BLS_PubKeyAggregate_Normal) -BENCHMARK(BLS_SecKeyAggregate_Normal) -BENCHMARK(BLS_SignatureAggregate_Normal) -BENCHMARK(BLS_Sign_Normal) -BENCHMARK(BLS_Verify_Normal) -BENCHMARK(BLS_Verify_LargeBlock100) -BENCHMARK(BLS_Verify_LargeBlock1000) -BENCHMARK(BLS_Verify_LargeBlockSelfAggregated100) -BENCHMARK(BLS_Verify_LargeBlockSelfAggregated1000) -BENCHMARK(BLS_Verify_LargeAggregatedBlock100) -BENCHMARK(BLS_Verify_LargeAggregatedBlock1000) -BENCHMARK(BLS_Verify_LargeAggregatedBlock1000PreVerified) -BENCHMARK(BLS_Verify_Batched) -BENCHMARK(BLS_Verify_BatchedParallel) +BENCHMARK(BLS_PubKeyAggregate_Normal, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_SecKeyAggregate_Normal, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_SignatureAggregate_Normal, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Sign_Normal, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_Normal, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeBlock100, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeBlock1000, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeBlockSelfAggregated100, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeBlockSelfAggregated1000, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeAggregatedBlock100, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeAggregatedBlock1000, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_LargeAggregatedBlock1000PreVerified, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_Batched, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_Verify_BatchedParallel, benchmark::PriorityLevel::HIGH) diff --git a/src/bench/bls_dkg.cpp b/src/bench/bls_dkg.cpp index 14e83810a100..c2f45045418c 100644 --- a/src/bench/bls_dkg.cpp +++ b/src/bench/bls_dkg.cpp @@ -136,7 +136,7 @@ static void BLSDKG_GenerateContributions(benchmark::Bench& bench, uint32_t epoch { \ BLSDKG_GenerateContributions(bench, epoch_iters, quorumSize); \ } \ - BENCHMARK(BLSDKG_GenerateContributions_##name##_##quorumSize) + BENCHMARK(BLSDKG_GenerateContributions_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) static void BLSDKG_InitDKG(benchmark::Bench& bench, uint32_t epoch_iters, int quorumSize) { @@ -154,7 +154,7 @@ static void BLSDKG_InitDKG(benchmark::Bench& bench, uint32_t epoch_iters, int qu { \ BLSDKG_InitDKG(bench, epoch_iters, quorumSize); \ } \ - BENCHMARK(BLSDKG_InitDKG_##name##_##quorumSize) + BENCHMARK(BLSDKG_InitDKG_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) #define BENCH_BuildQuorumVerificationVectors(name, quorumSize, epoch_iters) \ static void BLSDKG_BuildQuorumVerificationVectors_##name##_##quorumSize(benchmark::Bench& bench) \ @@ -169,7 +169,7 @@ static void BLSDKG_InitDKG(benchmark::Bench& bench, uint32_t epoch_iters, int qu ptr->Bench_BuildQuorumVerificationVectors(bench, epoch_iters); \ ptr.reset(); \ } \ - BENCHMARK(BLSDKG_BuildQuorumVerificationVectors_##name##_##quorumSize) + BENCHMARK(BLSDKG_BuildQuorumVerificationVectors_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) #define BENCH_VerifyContributionShares(name, quorumSize, invalidCount, aggregated, epoch_iters) \ static void BLSDKG_VerifyContributionShares_##name##_##quorumSize(benchmark::Bench& bench) \ @@ -184,7 +184,7 @@ static void BLSDKG_InitDKG(benchmark::Bench& bench, uint32_t epoch_iters, int qu ptr->Bench_VerifyContributionShares(bench, invalidCount, aggregated, epoch_iters); \ ptr.reset(); \ } \ - BENCHMARK(BLSDKG_VerifyContributionShares_##name##_##quorumSize) + BENCHMARK(BLSDKG_VerifyContributionShares_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) BENCH_GenerateContributions(simple, 50, 50); BENCH_GenerateContributions(simple, 100, 5); diff --git a/src/bench/bls_pubkey_agg.cpp b/src/bench/bls_pubkey_agg.cpp index ab732e93806a..d0982be725df 100644 --- a/src/bench/bls_pubkey_agg.cpp +++ b/src/bench/bls_pubkey_agg.cpp @@ -87,13 +87,13 @@ static void BLS_PubKeyAggregate_Iterative_200(benchmark::Bench& bench) { BLS_Pub static void BLS_PubKeyAggregate_Batch_200(benchmark::Bench& bench) { BLS_PubKeyAggregate_Batch(200, bench); } // Register all benchmarks -BENCHMARK(BLS_PubKeyAggregate_Iterative_5) -BENCHMARK(BLS_PubKeyAggregate_Batch_5) -BENCHMARK(BLS_PubKeyAggregate_Iterative_25) -BENCHMARK(BLS_PubKeyAggregate_Batch_25) -BENCHMARK(BLS_PubKeyAggregate_Iterative_50) -BENCHMARK(BLS_PubKeyAggregate_Batch_50) -BENCHMARK(BLS_PubKeyAggregate_Iterative_100) -BENCHMARK(BLS_PubKeyAggregate_Batch_100) -BENCHMARK(BLS_PubKeyAggregate_Iterative_200) -BENCHMARK(BLS_PubKeyAggregate_Batch_200) +BENCHMARK(BLS_PubKeyAggregate_Iterative_5, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Batch_5, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Iterative_25, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Batch_25, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Iterative_50, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Batch_50, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Iterative_100, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Batch_100, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Iterative_200, benchmark::PriorityLevel::HIGH) +BENCHMARK(BLS_PubKeyAggregate_Batch_200, benchmark::PriorityLevel::HIGH) diff --git a/src/bench/ccoins_caching.cpp b/src/bench/ccoins_caching.cpp index 727665dce58f..ad4d5e131231 100644 --- a/src/bench/ccoins_caching.cpp +++ b/src/bench/ccoins_caching.cpp @@ -50,4 +50,4 @@ static void CCoinsCaching(benchmark::Bench& bench) ECC_Stop(); } -BENCHMARK(CCoinsCaching); +BENCHMARK(CCoinsCaching, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/chacha20.cpp b/src/bench/chacha20.cpp index 42aceeea24b2..a577a2756f34 100644 --- a/src/bench/chacha20.cpp +++ b/src/bench/chacha20.cpp @@ -66,9 +66,9 @@ static void FSCHACHA20POLY1305_1MB(benchmark::Bench& bench) FSCHACHA20POLY1305(bench, BUFFER_SIZE_LARGE); } -BENCHMARK(CHACHA20_64BYTES); -BENCHMARK(CHACHA20_256BYTES); -BENCHMARK(CHACHA20_1MB); -BENCHMARK(FSCHACHA20POLY1305_64BYTES); -BENCHMARK(FSCHACHA20POLY1305_256BYTES); -BENCHMARK(FSCHACHA20POLY1305_1MB); +BENCHMARK(CHACHA20_64BYTES, benchmark::PriorityLevel::HIGH); +BENCHMARK(CHACHA20_256BYTES, benchmark::PriorityLevel::HIGH); +BENCHMARK(CHACHA20_1MB, benchmark::PriorityLevel::HIGH); +BENCHMARK(FSCHACHA20POLY1305_64BYTES, benchmark::PriorityLevel::HIGH); +BENCHMARK(FSCHACHA20POLY1305_256BYTES, benchmark::PriorityLevel::HIGH); +BENCHMARK(FSCHACHA20POLY1305_1MB, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/checkblock.cpp b/src/bench/checkblock.cpp index b5e866d3a1b7..48bdb5418a62 100644 --- a/src/bench/checkblock.cpp +++ b/src/bench/checkblock.cpp @@ -56,5 +56,5 @@ static void DeserializeAndCheckBlockTest(benchmark::Bench& bench) }); } -BENCHMARK(DeserializeBlockTest); -BENCHMARK(DeserializeAndCheckBlockTest); +BENCHMARK(DeserializeBlockTest, benchmark::PriorityLevel::HIGH); +BENCHMARK(DeserializeAndCheckBlockTest, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/checkqueue.cpp b/src/bench/checkqueue.cpp index d861faac243f..8591bb958b0b 100644 --- a/src/bench/checkqueue.cpp +++ b/src/bench/checkqueue.cpp @@ -70,4 +70,4 @@ static void CCheckQueueSpeedPrevectorJob(benchmark::Bench& bench) queue.StopWorkerThreads(); ECC_Stop(); } -BENCHMARK(CCheckQueueSpeedPrevectorJob); +BENCHMARK(CCheckQueueSpeedPrevectorJob, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/coin_selection.cpp b/src/bench/coin_selection.cpp index 7b44f5425bdc..aeaa8e9264e8 100644 --- a/src/bench/coin_selection.cpp +++ b/src/bench/coin_selection.cpp @@ -118,5 +118,5 @@ static void BnBExhaustion(benchmark::Bench& bench) }); } -BENCHMARK(CoinSelection); -BENCHMARK(BnBExhaustion); +BENCHMARK(CoinSelection, benchmark::PriorityLevel::HIGH); +BENCHMARK(BnBExhaustion, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/crypto_hash.cpp b/src/bench/crypto_hash.cpp index 9db0703c6d1e..9e8d7d8bd0a5 100644 --- a/src/bench/crypto_hash.cpp +++ b/src/bench/crypto_hash.cpp @@ -258,31 +258,31 @@ static void MuHashPrecompute(benchmark::Bench& bench) }); } -BENCHMARK(BenchRIPEMD160); -BENCHMARK(SHA1); -BENCHMARK(SHA256_STANDARD); -BENCHMARK(SHA256_SSE4); -BENCHMARK(SHA256_AVX2); -BENCHMARK(SHA256_SHANI); -BENCHMARK(SHA512); -BENCHMARK(SHA3_256_1M); - -BENCHMARK(SHA256_32b_STANDARD); -BENCHMARK(SHA256_32b_SSE4); -BENCHMARK(SHA256_32b_AVX2); -BENCHMARK(SHA256_32b_SHANI); -BENCHMARK(SipHash_32b); -BENCHMARK(SHA256D64_1024_STANDARD); -BENCHMARK(SHA256D64_1024_SSE4); -BENCHMARK(SHA256D64_1024_AVX2); -BENCHMARK(SHA256D64_1024_SHANI); -BENCHMARK(FastRandom_32bit); -BENCHMARK(FastRandom_1bit); - -BENCHMARK(MuHash); -BENCHMARK(MuHashMul); -BENCHMARK(MuHashDiv); -BENCHMARK(MuHashPrecompute); +BENCHMARK(BenchRIPEMD160, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA1, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_STANDARD, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_SSE4, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_AVX2, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_SHANI, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA512, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA3_256_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(SHA256_32b_STANDARD, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_32b_SSE4, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_32b_AVX2, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256_32b_SHANI, benchmark::PriorityLevel::HIGH); +BENCHMARK(SipHash_32b, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256D64_1024_STANDARD, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256D64_1024_SSE4, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256D64_1024_AVX2, benchmark::PriorityLevel::HIGH); +BENCHMARK(SHA256D64_1024_SHANI, benchmark::PriorityLevel::HIGH); +BENCHMARK(FastRandom_32bit, benchmark::PriorityLevel::HIGH); +BENCHMARK(FastRandom_1bit, benchmark::PriorityLevel::HIGH); + +BENCHMARK(MuHash, benchmark::PriorityLevel::HIGH); +BENCHMARK(MuHashMul, benchmark::PriorityLevel::HIGH); +BENCHMARK(MuHashDiv, benchmark::PriorityLevel::HIGH); +BENCHMARK(MuHashPrecompute, benchmark::PriorityLevel::HIGH); /* --------------------------- Dash-specific tests start here --------------------------- */ @@ -350,11 +350,11 @@ static void DSHA256_2048b_single(benchmark::Bench& bench) }); } -BENCHMARK(DSHA256_1M); +BENCHMARK(DSHA256_1M, benchmark::PriorityLevel::HIGH); -BENCHMARK(DSHA256_0032b_single); -BENCHMARK(DSHA256_0080b_single); -BENCHMARK(DSHA256_0128b_single); -BENCHMARK(DSHA256_0512b_single); -BENCHMARK(DSHA256_1024b_single); -BENCHMARK(DSHA256_2048b_single); +BENCHMARK(DSHA256_0032b_single, benchmark::PriorityLevel::HIGH); +BENCHMARK(DSHA256_0080b_single, benchmark::PriorityLevel::HIGH); +BENCHMARK(DSHA256_0128b_single, benchmark::PriorityLevel::HIGH); +BENCHMARK(DSHA256_0512b_single, benchmark::PriorityLevel::HIGH); +BENCHMARK(DSHA256_1024b_single, benchmark::PriorityLevel::HIGH); +BENCHMARK(DSHA256_2048b_single, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/duplicate_inputs.cpp b/src/bench/duplicate_inputs.cpp index fbb24445be4d..6ce1552cee35 100644 --- a/src/bench/duplicate_inputs.cpp +++ b/src/bench/duplicate_inputs.cpp @@ -61,4 +61,4 @@ static void DuplicateInputs(benchmark::Bench& bench) }); } -BENCHMARK(DuplicateInputs); +BENCHMARK(DuplicateInputs, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/ecdsa.cpp b/src/bench/ecdsa.cpp index 0b63d4621ebc..91640ffda7ef 100644 --- a/src/bench/ecdsa.cpp +++ b/src/bench/ecdsa.cpp @@ -84,6 +84,6 @@ static void ECDSAVerify_LargeBlock(benchmark::Bench& bench) ECC_Stop(); } -BENCHMARK(ECDSASign) -BENCHMARK(ECDSAVerify) -BENCHMARK(ECDSAVerify_LargeBlock) +BENCHMARK(ECDSASign, benchmark::PriorityLevel::HIGH) +BENCHMARK(ECDSAVerify, benchmark::PriorityLevel::HIGH) +BENCHMARK(ECDSAVerify_LargeBlock, benchmark::PriorityLevel::HIGH) diff --git a/src/bench/ellswift.cpp b/src/bench/ellswift.cpp index 3cf9ae489d17..f0348421b38e 100644 --- a/src/bench/ellswift.cpp +++ b/src/bench/ellswift.cpp @@ -28,4 +28,4 @@ static void EllSwiftCreate(benchmark::Bench& bench) ECC_Stop(); } -BENCHMARK(EllSwiftCreate); +BENCHMARK(EllSwiftCreate, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/examples.cpp b/src/bench/examples.cpp index 72a9922e94e5..abef69cc42df 100644 --- a/src/bench/examples.cpp +++ b/src/bench/examples.cpp @@ -18,4 +18,4 @@ static void Trig(benchmark::Bench& bench) }); } -BENCHMARK(Trig); +BENCHMARK(Trig, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/gcs_filter.cpp b/src/bench/gcs_filter.cpp index c9bccbc50233..5be1e3165c67 100644 --- a/src/bench/gcs_filter.cpp +++ b/src/bench/gcs_filter.cpp @@ -81,8 +81,8 @@ static void GCSFilterMatch(benchmark::Bench& bench) filter.Match(GCSFilter::Element()); }); } -BENCHMARK(GCSBlockFilterGetHash); -BENCHMARK(GCSFilterConstruct); -BENCHMARK(GCSFilterDecode); -BENCHMARK(GCSFilterDecodeSkipCheck); -BENCHMARK(GCSFilterMatch); +BENCHMARK(GCSBlockFilterGetHash, benchmark::PriorityLevel::HIGH); +BENCHMARK(GCSFilterConstruct, benchmark::PriorityLevel::HIGH); +BENCHMARK(GCSFilterDecode, benchmark::PriorityLevel::HIGH); +BENCHMARK(GCSFilterDecodeSkipCheck, benchmark::PriorityLevel::HIGH); +BENCHMARK(GCSFilterMatch, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/hashpadding.cpp b/src/bench/hashpadding.cpp index 753c8c288119..ac5aeebe51d3 100644 --- a/src/bench/hashpadding.cpp +++ b/src/bench/hashpadding.cpp @@ -26,7 +26,7 @@ static void PrePadded(benchmark::Bench& bench) }); } -BENCHMARK(PrePadded); +BENCHMARK(PrePadded, benchmark::PriorityLevel::HIGH); static void RegularPadded(benchmark::Bench& bench) { @@ -44,4 +44,4 @@ static void RegularPadded(benchmark::Bench& bench) }); } -BENCHMARK(RegularPadded); +BENCHMARK(RegularPadded, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/load_external.cpp b/src/bench/load_external.cpp index 2d691484a3bd..150654a095ca 100644 --- a/src/bench/load_external.cpp +++ b/src/bench/load_external.cpp @@ -60,4 +60,4 @@ static void LoadExternalBlockFile(benchmark::Bench& bench) fs::remove(blkfile); } -BENCHMARK(LoadExternalBlockFile); +BENCHMARK(LoadExternalBlockFile, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/lockedpool.cpp b/src/bench/lockedpool.cpp index b6d8824abae5..ac8262654c1f 100644 --- a/src/bench/lockedpool.cpp +++ b/src/bench/lockedpool.cpp @@ -39,4 +39,4 @@ static void BenchLockedPool(benchmark::Bench& bench) addr.clear(); } -BENCHMARK(BenchLockedPool); +BENCHMARK(BenchLockedPool, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/logging.cpp b/src/bench/logging.cpp index d28777df9e20..49a9e59893ab 100644 --- a/src/bench/logging.cpp +++ b/src/bench/logging.cpp @@ -41,8 +41,8 @@ static void LoggingNoFile(benchmark::Bench& bench) }); } -BENCHMARK(LoggingYoThreadNames); -BENCHMARK(LoggingNoThreadNames); -BENCHMARK(LoggingYoCategory); -BENCHMARK(LoggingNoCategory); -BENCHMARK(LoggingNoFile); +BENCHMARK(LoggingYoThreadNames, benchmark::PriorityLevel::HIGH); +BENCHMARK(LoggingNoThreadNames, benchmark::PriorityLevel::HIGH); +BENCHMARK(LoggingYoCategory, benchmark::PriorityLevel::HIGH); +BENCHMARK(LoggingNoCategory, benchmark::PriorityLevel::HIGH); +BENCHMARK(LoggingNoFile, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/mempool_eviction.cpp b/src/bench/mempool_eviction.cpp index b9cfdf3914f9..a1fea6f7419f 100644 --- a/src/bench/mempool_eviction.cpp +++ b/src/bench/mempool_eviction.cpp @@ -122,4 +122,4 @@ static void MempoolEviction(benchmark::Bench& bench) }); } -BENCHMARK(MempoolEviction); +BENCHMARK(MempoolEviction, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/mempool_stress.cpp b/src/bench/mempool_stress.cpp index 4c35b6592fcf..0cf4e83ef8ce 100644 --- a/src/bench/mempool_stress.cpp +++ b/src/bench/mempool_stress.cpp @@ -112,5 +112,5 @@ static void MempoolCheck(benchmark::Bench& bench) }); } -BENCHMARK(ComplexMemPool); -BENCHMARK(MempoolCheck); +BENCHMARK(ComplexMemPool, benchmark::PriorityLevel::HIGH); +BENCHMARK(MempoolCheck, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/merkle_root.cpp b/src/bench/merkle_root.cpp index ba6629b9f0a5..4140d67bc7ab 100644 --- a/src/bench/merkle_root.cpp +++ b/src/bench/merkle_root.cpp @@ -23,4 +23,4 @@ static void MerkleRoot(benchmark::Bench& bench) }); } -BENCHMARK(MerkleRoot); +BENCHMARK(MerkleRoot, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/peer_eviction.cpp b/src/bench/peer_eviction.cpp index f05f5e8f6464..c3e3670de735 100644 --- a/src/bench/peer_eviction.cpp +++ b/src/bench/peer_eviction.cpp @@ -141,15 +141,15 @@ static void EvictionProtection3Networks250Candidates(benchmark::Bench& bench) // - 250 candidates is the number of peers reported by operators of busy nodes // No disadvantaged networks, with 250 eviction candidates. -BENCHMARK(EvictionProtection0Networks250Candidates); +BENCHMARK(EvictionProtection0Networks250Candidates, benchmark::PriorityLevel::HIGH); // 1 disadvantaged network (Tor) with 250 eviction candidates. -BENCHMARK(EvictionProtection1Networks250Candidates); +BENCHMARK(EvictionProtection1Networks250Candidates, benchmark::PriorityLevel::HIGH); // 2 disadvantaged networks (I2P, Tor) with 250 eviction candidates. -BENCHMARK(EvictionProtection2Networks250Candidates); +BENCHMARK(EvictionProtection2Networks250Candidates, benchmark::PriorityLevel::HIGH); // 3 disadvantaged networks (I2P/localhost/Tor) with 50/100/250 eviction candidates. -BENCHMARK(EvictionProtection3Networks050Candidates); -BENCHMARK(EvictionProtection3Networks100Candidates); -BENCHMARK(EvictionProtection3Networks250Candidates); +BENCHMARK(EvictionProtection3Networks050Candidates, benchmark::PriorityLevel::HIGH); +BENCHMARK(EvictionProtection3Networks100Candidates, benchmark::PriorityLevel::HIGH); +BENCHMARK(EvictionProtection3Networks250Candidates, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/poly1305.cpp b/src/bench/poly1305.cpp index 746f2f62e894..4719db8cc993 100644 --- a/src/bench/poly1305.cpp +++ b/src/bench/poly1305.cpp @@ -39,6 +39,6 @@ static void POLY1305_1MB(benchmark::Bench& bench) POLY1305(bench, BUFFER_SIZE_LARGE); } -BENCHMARK(POLY1305_64BYTES); -BENCHMARK(POLY1305_256BYTES); -BENCHMARK(POLY1305_1MB); +BENCHMARK(POLY1305_64BYTES, benchmark::PriorityLevel::HIGH); +BENCHMARK(POLY1305_256BYTES, benchmark::PriorityLevel::HIGH); +BENCHMARK(POLY1305_1MB, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/pool.cpp b/src/bench/pool.cpp index 0ce3b952f870..b2a5f8debf9d 100644 --- a/src/bench/pool.cpp +++ b/src/bench/pool.cpp @@ -45,5 +45,5 @@ static void PoolAllocator_StdUnorderedMapWithPoolResource(benchmark::Bench& benc BenchFillClearMap(bench, map); } -BENCHMARK(PoolAllocator_StdUnorderedMap); -BENCHMARK(PoolAllocator_StdUnorderedMapWithPoolResource); +BENCHMARK(PoolAllocator_StdUnorderedMap, benchmark::PriorityLevel::HIGH); +BENCHMARK(PoolAllocator_StdUnorderedMapWithPoolResource, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/pow_hash.cpp b/src/bench/pow_hash.cpp index 79859966ac95..a4feedbb1544 100644 --- a/src/bench/pow_hash.cpp +++ b/src/bench/pow_hash.cpp @@ -262,98 +262,98 @@ static void Pow_Skein512_1024b(benchmark::Bench& bench) { return Pow_Skein512(be static void Pow_Skein512_2048b(benchmark::Bench& bench) { return Pow_Skein512(bench, 2048); } static void Pow_Skein512_1M(benchmark::Bench& bench) { return Pow_Skein512(bench, BUFFER_SIZE); } -BENCHMARK(Pow_X11_0032b); -BENCHMARK(Pow_X11_0080b); -BENCHMARK(Pow_X11_0128b); -BENCHMARK(Pow_X11_0512b); -BENCHMARK(Pow_X11_1024b); -BENCHMARK(Pow_X11_2048b); -BENCHMARK(Pow_X11_1M); - -BENCHMARK(Pow_Blake512_0032b); -BENCHMARK(Pow_Blake512_0080b); -BENCHMARK(Pow_Blake512_0128b); -BENCHMARK(Pow_Blake512_0512b); -BENCHMARK(Pow_Blake512_1024b); -BENCHMARK(Pow_Blake512_2048b); -BENCHMARK(Pow_Blake512_1M); - -BENCHMARK(Pow_Bmw512_0032b); -BENCHMARK(Pow_Bmw512_0080b); -BENCHMARK(Pow_Bmw512_0128b); -BENCHMARK(Pow_Bmw512_0512b); -BENCHMARK(Pow_Bmw512_1024b); -BENCHMARK(Pow_Bmw512_2048b); -BENCHMARK(Pow_Bmw512_1M); - -BENCHMARK(Pow_Cubehash512_0032b); -BENCHMARK(Pow_Cubehash512_0080b); -BENCHMARK(Pow_Cubehash512_0128b); -BENCHMARK(Pow_Cubehash512_0512b); -BENCHMARK(Pow_Cubehash512_1024b); -BENCHMARK(Pow_Cubehash512_2048b); -BENCHMARK(Pow_Cubehash512_1M); - -BENCHMARK(Pow_Echo512_0032b); -BENCHMARK(Pow_Echo512_0080b); -BENCHMARK(Pow_Echo512_0128b); -BENCHMARK(Pow_Echo512_0512b); -BENCHMARK(Pow_Echo512_1024b); -BENCHMARK(Pow_Echo512_2048b); -BENCHMARK(Pow_Echo512_1M); - -BENCHMARK(Pow_Groestl512_0032b); -BENCHMARK(Pow_Groestl512_0080b); -BENCHMARK(Pow_Groestl512_0128b); -BENCHMARK(Pow_Groestl512_0512b); -BENCHMARK(Pow_Groestl512_1024b); -BENCHMARK(Pow_Groestl512_2048b); -BENCHMARK(Pow_Groestl512_1M); - -BENCHMARK(Pow_Jh512_0032b); -BENCHMARK(Pow_Jh512_0080b); -BENCHMARK(Pow_Jh512_0128b); -BENCHMARK(Pow_Jh512_0512b); -BENCHMARK(Pow_Jh512_1024b); -BENCHMARK(Pow_Jh512_2048b); -BENCHMARK(Pow_Jh512_1M); - -BENCHMARK(Pow_Keccak512_0032b); -BENCHMARK(Pow_Keccak512_0080b); -BENCHMARK(Pow_Keccak512_0128b); -BENCHMARK(Pow_Keccak512_0512b); -BENCHMARK(Pow_Keccak512_1024b); -BENCHMARK(Pow_Keccak512_2048b); -BENCHMARK(Pow_Keccak512_1M); - -BENCHMARK(Pow_Luffa512_0032b); -BENCHMARK(Pow_Luffa512_0080b); -BENCHMARK(Pow_Luffa512_0128b); -BENCHMARK(Pow_Luffa512_0512b); -BENCHMARK(Pow_Luffa512_1024b); -BENCHMARK(Pow_Luffa512_2048b); -BENCHMARK(Pow_Luffa512_1M); - -BENCHMARK(Pow_Shavite512_0032b); -BENCHMARK(Pow_Shavite512_0080b); -BENCHMARK(Pow_Shavite512_0128b); -BENCHMARK(Pow_Shavite512_0512b); -BENCHMARK(Pow_Shavite512_1024b); -BENCHMARK(Pow_Shavite512_2048b); -BENCHMARK(Pow_Shavite512_1M); - -BENCHMARK(Pow_Simd512_0032b); -BENCHMARK(Pow_Simd512_0080b); -BENCHMARK(Pow_Simd512_0128b); -BENCHMARK(Pow_Simd512_0512b); -BENCHMARK(Pow_Simd512_1024b); -BENCHMARK(Pow_Simd512_2048b); -BENCHMARK(Pow_Simd512_1M); - -BENCHMARK(Pow_Skein512_0032b); -BENCHMARK(Pow_Skein512_0080b); -BENCHMARK(Pow_Skein512_0128b); -BENCHMARK(Pow_Skein512_0512b); -BENCHMARK(Pow_Skein512_1024b); -BENCHMARK(Pow_Skein512_2048b); -BENCHMARK(Pow_Skein512_1M); +BENCHMARK(Pow_X11_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_X11_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_X11_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_X11_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_X11_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_X11_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_X11_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Blake512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Blake512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Blake512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Blake512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Blake512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Blake512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Blake512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Bmw512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Bmw512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Bmw512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Bmw512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Bmw512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Bmw512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Bmw512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Cubehash512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Cubehash512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Cubehash512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Cubehash512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Cubehash512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Cubehash512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Cubehash512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Echo512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Echo512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Echo512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Echo512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Echo512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Echo512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Echo512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Groestl512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Groestl512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Groestl512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Groestl512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Groestl512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Groestl512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Groestl512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Jh512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Jh512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Jh512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Jh512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Jh512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Jh512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Jh512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Keccak512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Keccak512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Keccak512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Keccak512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Keccak512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Keccak512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Keccak512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Luffa512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Luffa512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Luffa512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Luffa512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Luffa512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Luffa512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Luffa512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Shavite512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Shavite512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Shavite512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Shavite512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Shavite512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Shavite512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Shavite512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Simd512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Simd512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Simd512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Simd512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Simd512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Simd512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Simd512_1M, benchmark::PriorityLevel::HIGH); + +BENCHMARK(Pow_Skein512_0032b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Skein512_0080b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Skein512_0128b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Skein512_0512b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Skein512_1024b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Skein512_2048b, benchmark::PriorityLevel::HIGH); +BENCHMARK(Pow_Skein512_1M, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/prevector.cpp b/src/bench/prevector.cpp index 8de8e05c067f..ac06cb01da3f 100644 --- a/src/bench/prevector.cpp +++ b/src/bench/prevector.cpp @@ -138,19 +138,19 @@ static void PrevectorFillVectorIndirect(benchmark::Bench& bench) { \ Prevector##name(bench); \ } \ - BENCHMARK(Prevector##name##Nontrivial); \ + BENCHMARK(Prevector##name##Nontrivial, benchmark::PriorityLevel::HIGH); \ static void Prevector##name##Trivial(benchmark::Bench& bench) \ { \ Prevector##name(bench); \ } \ - BENCHMARK(Prevector##name##Trivial); + BENCHMARK(Prevector##name##Trivial, benchmark::PriorityLevel::HIGH); PREVECTOR_TEST(Clear) PREVECTOR_TEST(Destructor) PREVECTOR_TEST(Resize) PREVECTOR_TEST(Deserialize) -BENCHMARK(PrevectorAssign) -BENCHMARK(PrevectorAssignTo) +BENCHMARK(PrevectorAssign, benchmark::PriorityLevel::HIGH) +BENCHMARK(PrevectorAssignTo, benchmark::PriorityLevel::HIGH) PREVECTOR_TEST(FillVectorDirect) PREVECTOR_TEST(FillVectorIndirect) diff --git a/src/bench/rollingbloom.cpp b/src/bench/rollingbloom.cpp index 8f05e3bad01d..865d99f9e84f 100644 --- a/src/bench/rollingbloom.cpp +++ b/src/bench/rollingbloom.cpp @@ -32,5 +32,5 @@ static void RollingBloomReset(benchmark::Bench& bench) }); } -BENCHMARK(RollingBloom); -BENCHMARK(RollingBloomReset); +BENCHMARK(RollingBloom, benchmark::PriorityLevel::HIGH); +BENCHMARK(RollingBloomReset, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/rpc_blockchain.cpp b/src/bench/rpc_blockchain.cpp index 9ed1fffb2f78..d593aced02f6 100644 --- a/src/bench/rpc_blockchain.cpp +++ b/src/bench/rpc_blockchain.cpp @@ -50,7 +50,7 @@ static void BlockToJsonVerbose(benchmark::Bench& bench) }); } -BENCHMARK(BlockToJsonVerbose); +BENCHMARK(BlockToJsonVerbose, benchmark::PriorityLevel::HIGH); static void BlockToJsonVerboseWrite(benchmark::Bench& bench) { @@ -63,4 +63,4 @@ static void BlockToJsonVerboseWrite(benchmark::Bench& bench) }); } -BENCHMARK(BlockToJsonVerboseWrite); +BENCHMARK(BlockToJsonVerboseWrite, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/rpc_mempool.cpp b/src/bench/rpc_mempool.cpp index ef989b286d20..dbfd187c07b0 100644 --- a/src/bench/rpc_mempool.cpp +++ b/src/bench/rpc_mempool.cpp @@ -39,4 +39,4 @@ static void RpcMempool(benchmark::Bench& bench) }); } -BENCHMARK(RpcMempool); +BENCHMARK(RpcMempool, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/strencodings.cpp b/src/bench/strencodings.cpp index 4be7cd6d6f68..00354617a51a 100644 --- a/src/bench/strencodings.cpp +++ b/src/bench/strencodings.cpp @@ -15,4 +15,4 @@ static void HexStrBench(benchmark::Bench& bench) }); } -BENCHMARK(HexStrBench); +BENCHMARK(HexStrBench, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/string_cast.cpp b/src/bench/string_cast.cpp index d358e2a7ced8..e63ac4308a37 100644 --- a/src/bench/string_cast.cpp +++ b/src/bench/string_cast.cpp @@ -67,9 +67,9 @@ static void strings_2_strptintf(benchmark::Bench& bench) }); } -BENCHMARK(int_atoi); -BENCHMARK(strings_1_numberToString); -BENCHMARK(strings_1_tostring); -BENCHMARK(strings_2_multi_numberToString); -BENCHMARK(strings_2_multi_tostring); -BENCHMARK(strings_2_strptintf); +BENCHMARK(int_atoi, benchmark::PriorityLevel::HIGH); +BENCHMARK(strings_1_numberToString, benchmark::PriorityLevel::HIGH); +BENCHMARK(strings_1_tostring, benchmark::PriorityLevel::HIGH); +BENCHMARK(strings_2_multi_numberToString, benchmark::PriorityLevel::HIGH); +BENCHMARK(strings_2_multi_tostring, benchmark::PriorityLevel::HIGH); +BENCHMARK(strings_2_strptintf, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/util_time.cpp b/src/bench/util_time.cpp index c45584988d9a..f12ce3b3f541 100644 --- a/src/bench/util_time.cpp +++ b/src/bench/util_time.cpp @@ -36,7 +36,7 @@ static void BenchTimeMillisSys(benchmark::Bench& bench) }); } -BENCHMARK(BenchTimeDeprecated); -BENCHMARK(BenchTimeMillis); -BENCHMARK(BenchTimeMillisSys); -BENCHMARK(BenchTimeMock); +BENCHMARK(BenchTimeDeprecated, benchmark::PriorityLevel::HIGH); +BENCHMARK(BenchTimeMillis, benchmark::PriorityLevel::HIGH); +BENCHMARK(BenchTimeMillisSys, benchmark::PriorityLevel::HIGH); +BENCHMARK(BenchTimeMock, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/verify_script.cpp b/src/bench/verify_script.cpp index 9fb95e9116fd..7b3d9a12004d 100644 --- a/src/bench/verify_script.cpp +++ b/src/bench/verify_script.cpp @@ -30,4 +30,4 @@ static void VerifyNestedIfScript(benchmark::Bench& bench) { } -BENCHMARK(VerifyNestedIfScript); +BENCHMARK(VerifyNestedIfScript, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/wallet_balance.cpp b/src/bench/wallet_balance.cpp index 7c8db6187d55..36a95f5765a0 100644 --- a/src/bench/wallet_balance.cpp +++ b/src/bench/wallet_balance.cpp @@ -55,7 +55,7 @@ static void WalletBalanceClean(benchmark::Bench& bench) {WalletBalance(bench, /* static void WalletBalanceMine(benchmark::Bench& bench) { WalletBalance(bench, /*set_dirty=*/false, /*add_mine=*/true, 16000); } static void WalletBalanceWatch(benchmark::Bench& bench) { WalletBalance(bench, /*set_dirty=*/false, /*add_mine=*/false, 8000); } -BENCHMARK(WalletBalanceDirty); -BENCHMARK(WalletBalanceClean); -BENCHMARK(WalletBalanceMine); -BENCHMARK(WalletBalanceWatch); +BENCHMARK(WalletBalanceDirty, benchmark::PriorityLevel::HIGH); +BENCHMARK(WalletBalanceClean, benchmark::PriorityLevel::HIGH); +BENCHMARK(WalletBalanceMine, benchmark::PriorityLevel::HIGH); +BENCHMARK(WalletBalanceWatch, benchmark::PriorityLevel::HIGH); diff --git a/src/bench/wallet_create.cpp b/src/bench/wallet_create.cpp index 046dc29a804f..f900a7e90edb 100644 --- a/src/bench/wallet_create.cpp +++ b/src/bench/wallet_create.cpp @@ -49,8 +49,8 @@ static void WalletCreatePlain(benchmark::Bench& bench) { WalletCreate(bench, /*e static void WalletCreateEncrypted(benchmark::Bench& bench) { WalletCreate(bench, /*encrypted=*/true); } #ifdef USE_SQLITE -BENCHMARK(WalletCreatePlain); -BENCHMARK(WalletCreateEncrypted); +BENCHMARK(WalletCreatePlain, benchmark::PriorityLevel::HIGH); +BENCHMARK(WalletCreateEncrypted, benchmark::PriorityLevel::HIGH); #endif } // namespace wallet diff --git a/src/bench/wallet_loading.cpp b/src/bench/wallet_loading.cpp index 1a2bd7eceb32..a81d79936e5b 100644 --- a/src/bench/wallet_loading.cpp +++ b/src/bench/wallet_loading.cpp @@ -118,10 +118,10 @@ static void WalletLoading(benchmark::Bench& bench, bool legacy_wallet) #ifdef USE_BDB static void WalletLoadingLegacy(benchmark::Bench& bench) { WalletLoading(bench, /*legacy_wallet=*/true); } -BENCHMARK(WalletLoadingLegacy); +BENCHMARK(WalletLoadingLegacy, benchmark::PriorityLevel::HIGH); #endif #ifdef USE_SQLITE static void WalletLoadingDescriptors(benchmark::Bench& bench) { WalletLoading(bench, /*legacy_wallet=*/false); } -BENCHMARK(WalletLoadingDescriptors); +BENCHMARK(WalletLoadingDescriptors, benchmark::PriorityLevel::HIGH); #endif From b70edda3e52bd50113ad9583a36d87612e67b280 Mon Sep 17 00:00:00 2001 From: Konstantin Akimov Date: Wed, 31 Dec 2025 15:15:32 +0700 Subject: [PATCH 04/16] fmt: apply clang-format for blsl_dkg.cpp --- src/bench/bls_dkg.cpp | 64 +++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/src/bench/bls_dkg.cpp b/src/bench/bls_dkg.cpp index c2f45045418c..251f68a5beb4 100644 --- a/src/bench/bls_dkg.cpp +++ b/src/bench/bls_dkg.cpp @@ -131,11 +131,11 @@ static void BLSDKG_GenerateContributions(benchmark::Bench& bench, uint32_t epoch blsWorker.Stop(); } -#define BENCH_GenerateContributions(name, quorumSize, epoch_iters) \ +#define BENCH_GenerateContributions(name, quorumSize, epoch_iters) \ static void BLSDKG_GenerateContributions_##name##_##quorumSize(benchmark::Bench& bench) \ - { \ - BLSDKG_GenerateContributions(bench, epoch_iters, quorumSize); \ - } \ + { \ + BLSDKG_GenerateContributions(bench, epoch_iters, quorumSize); \ + } \ BENCHMARK(BLSDKG_GenerateContributions_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) static void BLSDKG_InitDKG(benchmark::Bench& bench, uint32_t epoch_iters, int quorumSize) @@ -149,41 +149,41 @@ static void BLSDKG_InitDKG(benchmark::Bench& bench, uint32_t epoch_iters, int qu }); } -#define BENCH_InitDKG(name, quorumSize, epoch_iters) \ +#define BENCH_InitDKG(name, quorumSize, epoch_iters) \ static void BLSDKG_InitDKG_##name##_##quorumSize(benchmark::Bench& bench) \ - { \ - BLSDKG_InitDKG(bench, epoch_iters, quorumSize); \ - } \ + { \ + BLSDKG_InitDKG(bench, epoch_iters, quorumSize); \ + } \ BENCHMARK(BLSDKG_InitDKG_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) -#define BENCH_BuildQuorumVerificationVectors(name, quorumSize, epoch_iters) \ +#define BENCH_BuildQuorumVerificationVectors(name, quorumSize, epoch_iters) \ static void BLSDKG_BuildQuorumVerificationVectors_##name##_##quorumSize(benchmark::Bench& bench) \ - { \ - if (!bench.output()) { \ - std::unique_ptr ptr = std::make_unique(1); \ - ptr->Bench_BuildQuorumVerificationVectors(bench, 1); \ - ptr.reset(); \ - return; \ - } \ - std::unique_ptr ptr = std::make_unique(quorumSize); \ - ptr->Bench_BuildQuorumVerificationVectors(bench, epoch_iters); \ - ptr.reset(); \ - } \ + { \ + if (!bench.output()) { \ + std::unique_ptr ptr = std::make_unique(1); \ + ptr->Bench_BuildQuorumVerificationVectors(bench, 1); \ + ptr.reset(); \ + return; \ + } \ + std::unique_ptr ptr = std::make_unique(quorumSize); \ + ptr->Bench_BuildQuorumVerificationVectors(bench, epoch_iters); \ + ptr.reset(); \ + } \ BENCHMARK(BLSDKG_BuildQuorumVerificationVectors_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) #define BENCH_VerifyContributionShares(name, quorumSize, invalidCount, aggregated, epoch_iters) \ - static void BLSDKG_VerifyContributionShares_##name##_##quorumSize(benchmark::Bench& bench) \ - { \ - if (!bench.output()) { \ - std::unique_ptr ptr = std::make_unique(1); \ - ptr->Bench_VerifyContributionShares(bench, invalidCount, aggregated, 1); \ - ptr.reset(); \ - return; \ - } \ - std::unique_ptr ptr = std::make_unique(quorumSize); \ - ptr->Bench_VerifyContributionShares(bench, invalidCount, aggregated, epoch_iters); \ - ptr.reset(); \ - } \ + static void BLSDKG_VerifyContributionShares_##name##_##quorumSize(benchmark::Bench& bench) \ + { \ + if (!bench.output()) { \ + std::unique_ptr ptr = std::make_unique(1); \ + ptr->Bench_VerifyContributionShares(bench, invalidCount, aggregated, 1); \ + ptr.reset(); \ + return; \ + } \ + std::unique_ptr ptr = std::make_unique(quorumSize); \ + ptr->Bench_VerifyContributionShares(bench, invalidCount, aggregated, epoch_iters); \ + ptr.reset(); \ + } \ BENCHMARK(BLSDKG_VerifyContributionShares_##name##_##quorumSize, benchmark::PriorityLevel::HIGH) BENCH_GenerateContributions(simple, 50, 50); From ad46699208231ba36364a0170a787b464ba3c37b Mon Sep 17 00:00:00 2001 From: MacroFake Date: Thu, 27 Oct 2022 16:15:12 +0200 Subject: [PATCH 05/16] Merge bitcoin/bitcoin#26388: ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task da168934741b776bce07d5503ca2344d300723b3 ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task (Hennadii Stepanov) 702836530ffa351e863b1b1300fd2e559a14ef23 ci: Make `getopt` path architecture agnostic (Hennadii Stepanov) Pull request description: The "macOS native" CI task always uses the recent OS image. This PR updates it up to the recent macOS release. Cirrus Labs [stopped](https://github.com/bitcoin/bitcoin/pull/25160#issuecomment-1162829773) updating macOS images for `x86_64`, therefore, an `arm64` image been used. Also `make test-security-check` has been dropped as it ["isn't even expected to pass"](https://github.com/bitcoin/bitcoin/issues/26386#issuecomment-1290318628) on `arm64` in CI. ACKs for top commit: Sjors: utACK da16893 Tree-SHA512: 36785d33b7f11b3cdbc53bcfbf97d88bf821fad248c825982dd9f8e3413809a4ef11190eaf950e60fdf479b62ff66920c35d9ea42d534723f015742eec7e19b6 --- ..._native_x86_64.sh => 00_setup_env_mac_native_arm64.sh} | 8 +++----- ci/test/04_install.sh | 2 +- 2 files changed, 4 insertions(+), 6 deletions(-) rename ci/test/{00_setup_env_mac_native_x86_64.sh => 00_setup_env_mac_native_arm64.sh} (68%) diff --git a/ci/test/00_setup_env_mac_native_x86_64.sh b/ci/test/00_setup_env_mac_native_arm64.sh similarity index 68% rename from ci/test/00_setup_env_mac_native_x86_64.sh rename to ci/test/00_setup_env_mac_native_arm64.sh index b20efb2bde37..aef3112924b3 100755 --- a/ci/test/00_setup_env_mac_native_x86_64.sh +++ b/ci/test/00_setup_env_mac_native_arm64.sh @@ -7,13 +7,11 @@ export LC_ALL=C.UTF-8 export CONTAINER_NAME=ci_macos -export HOST=x86_64-apple-darwin -export PIP_PACKAGES="zmq lief" +export HOST=arm64-apple-darwin +export PIP_PACKAGES="zmq" export GOAL="install" -export BITCOIN_CONFIG="--with-gui --enable-reduce-exports --disable-miner" +export BITCOIN_CONFIG="--with-gui --with-miniupnpc --with-natpmp --enable-reduce-exports --disable-miner" export CI_OS_NAME="macos" export NO_DEPENDS=1 export OSX_SDK="" export CCACHE_MAXSIZE=300M - -export RUN_SECURITY_TESTS="true" diff --git a/ci/test/04_install.sh b/ci/test/04_install.sh index bf14526ccad3..e300a36e7d4b 100755 --- a/ci/test/04_install.sh +++ b/ci/test/04_install.sh @@ -89,7 +89,7 @@ if [ -n "$PIP_PACKAGES" ]; then if [ "$CI_OS_NAME" == "macos" ]; then sudo -H pip3 install --upgrade pip # shellcheck disable=SC2086 - IN_GETOPT_BIN="/usr/local/opt/gnu-getopt/bin/getopt" ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES + IN_GETOPT_BIN="$(brew --prefix gnu-getopt)/bin/getopt" ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES else # shellcheck disable=SC2086 ${CI_RETRY_EXE} CI_EXEC pip3 install --user $PIP_PACKAGES From 3fa8c3d6484fabc69ad8cadc6bc50eb1c83ca374 Mon Sep 17 00:00:00 2001 From: Andrew Chow Date: Thu, 27 Oct 2022 13:08:40 -0400 Subject: [PATCH 06/16] Merge bitcoin/bitcoin#26349: rpc: make `address` field optional `list{transactions, sinceblock}` response eb679a7896ce00e322972a011b023661766923b9 rpc: make `address` field optional (w0xlt) Pull request description: Close https://github.com/bitcoin/bitcoin/issues/26338. This PR makes optional the `address` field in the response of `listtransactions` and `listsinceblock` RPC. And adds two tests that fail on master, but not on this branch. ACKs for top commit: achow101: ACK eb679a7896ce00e322972a011b023661766923b9 aureleoules: ACK eb679a7896ce00e322972a011b023661766923b9 Tree-SHA512: b267439626e2ec3134ae790c849949a4c40ef0cebd20092e8187be3db0a61941b2da10bbbba92ca880b8369f46c1aaa806d057eaa5159325f65cbec7cb33c52f --- src/wallet/rpc/transactions.cpp | 6 +++--- test/functional/wallet_listsinceblock.py | 16 ++++++++++++++++ test/functional/wallet_listtransactions.py | 12 ++++++++++++ 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/src/wallet/rpc/transactions.cpp b/src/wallet/rpc/transactions.cpp index c141d382e067..889d81d7ece0 100644 --- a/src/wallet/rpc/transactions.cpp +++ b/src/wallet/rpc/transactions.cpp @@ -457,8 +457,8 @@ RPCHelpMan listtransactions() {RPCResult::Type::OBJ, "", "", Cat(Cat>( { {RPCResult::Type::BOOL, "involvesWatchonly", /*optional=*/true, "Only returns true if imported addresses were involved in transaction"}, - {RPCResult::Type::STR, "address", "The Dash address of the transaction. Not present for\n" - "move transactions (category = move)."}, + {RPCResult::Type::STR, "address", /*optional=*/true, "The Dash address of the transaction. Not present for\n" + "move transactions (category = move) or if the output does not have an address, e.g. OP_RETURN null data."}, {RPCResult::Type::STR, "category", "The transaction category.\n" "\"send\" Transactions sent.\n" "\"coinjoin\" Transactions sent using CoinJoin funds.\n" @@ -572,7 +572,7 @@ RPCHelpMan listsinceblock() {RPCResult::Type::OBJ, "", "", Cat(Cat>( { {RPCResult::Type::BOOL, "involvesWatchonly", /*optional=*/true, "Only returns true if imported addresses were involved in transaction"}, - {RPCResult::Type::STR, "address", "The Dash address of the transaction."}, + {RPCResult::Type::STR, "address", /*optional=*/true, "The Dash address of the transaction (not returned if the output does not have an address, e.g. OP_RETURN null data)."}, {RPCResult::Type::STR, "category", "The transaction category.\n" "\"send\" Transactions sent.\n" "\"coinjoin\" Transactions sent using CoinJoin funds.\n" diff --git a/test/functional/wallet_listsinceblock.py b/test/functional/wallet_listsinceblock.py index 4a4167f11155..e0e8581db81f 100755 --- a/test/functional/wallet_listsinceblock.py +++ b/test/functional/wallet_listsinceblock.py @@ -41,6 +41,7 @@ def run_test(self): self.test_double_send() self.double_spends_filtered() self.test_targetconfirmations() + self.test_op_return() def test_no_blockhash(self): self.log.info("Test no blockhash") @@ -399,5 +400,20 @@ def double_spends_filtered(self): assert_equal(original_found, False) assert_equal(double_found, False) + + def test_op_return(self): + """Test if OP_RETURN outputs will be displayed correctly.""" + block_hash = self.nodes[2].getbestblockhash() + + raw_tx = self.nodes[2].createrawtransaction([], [{'data': 'aa'}]) + funded_tx = self.nodes[2].fundrawtransaction(raw_tx) + signed_tx = self.nodes[2].signrawtransactionwithwallet(funded_tx['hex']) + tx_id = self.nodes[2].sendrawtransaction(signed_tx['hex']) + + op_ret_tx = [tx for tx in self.nodes[2].listsinceblock(blockhash=block_hash)["transactions"] if tx['txid'] == tx_id][0] + + assert 'address' not in op_ret_tx + + if __name__ == '__main__': ListSinceBlockTest().main() diff --git a/test/functional/wallet_listtransactions.py b/test/functional/wallet_listtransactions.py index c6d3c2812352..e89533a82896 100755 --- a/test/functional/wallet_listtransactions.py +++ b/test/functional/wallet_listtransactions.py @@ -104,6 +104,7 @@ def run_test(self): self.run_externally_generated_address_test() self.run_invalid_parameters_test() + self.test_op_return() def run_externally_generated_address_test(self): """Test behavior when receiving address is not in the address book.""" @@ -166,6 +167,17 @@ def run_invalid_parameters_test(self): assert_raises_rpc_error(-8, "Negative count", self.nodes[0].listtransactions, count=-1) assert_raises_rpc_error(-8, "Negative from", self.nodes[0].listtransactions, skip=-1) + def test_op_return(self): + """Test if OP_RETURN outputs will be displayed correctly.""" + raw_tx = self.nodes[0].createrawtransaction([], [{'data': 'aa'}]) + funded_tx = self.nodes[0].fundrawtransaction(raw_tx) + signed_tx = self.nodes[0].signrawtransactionwithwallet(funded_tx['hex']) + tx_id = self.nodes[0].sendrawtransaction(signed_tx['hex']) + + op_ret_tx = [tx for tx in self.nodes[0].listtransactions() if tx['txid'] == tx_id][0] + + assert 'address' not in op_ret_tx + if __name__ == '__main__': ListTransactionsTest().main() From f04448b92837bafc591d420d69d41b50e7cbdf80 Mon Sep 17 00:00:00 2001 From: MacroFake Date: Sat, 29 Oct 2022 11:14:06 +0200 Subject: [PATCH 07/16] Merge bitcoin/bitcoin#26404: test: fix intermittent failure in rpc_getblockfrompeer.py 8a9f1e4d18e2b94509548af2aa3ad14185793f73 test: fix intermittent failure in rpc_getblockfrompeer.py (Martin Zumsande) Pull request description: Fixes an intermittent failure in `rpc_getblockfrompeer.py` observed in https://cirrus-ci.com/task/6610115527704576 by adding a sync to make sure the node has processed the header we sent it before we query it for the corresponding block. Fixes #26412 ACKs for top commit: jonatack: ACK 8a9f1e4d18e2b94509548af2aa3ad14185793f73 Tree-SHA512: f6188ab3cfd863034e44e9806d0d99a8781462bec94141501aefc71589153481ffb144e126326ab81807c2b2c93de7f4aac5d09dbcf26c4512a176e06fc6e5f8 --- test/functional/rpc_getblockfrompeer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/functional/rpc_getblockfrompeer.py b/test/functional/rpc_getblockfrompeer.py index 6b460336e698..3bed07485f31 100755 --- a/test/functional/rpc_getblockfrompeer.py +++ b/test/functional/rpc_getblockfrompeer.py @@ -92,7 +92,7 @@ def run_test(self): # Connect a P2PInterface to the pruning node and have it submit only the header of the # block that the pruning node has not seen node1_interface = self.nodes[1].add_p2p_connection(P2PInterface()) - node1_interface.send_message(msg_headers([block])) + node1_interface.send_and_ping(msg_headers([block])) # Get the peer id of the P2PInterface from the pruning node node1_peers = self.nodes[1].getpeerinfo() From 582e31b90da841bebdc828e597a8c8491c1f4289 Mon Sep 17 00:00:00 2001 From: MacroFake Date: Mon, 31 Oct 2022 08:42:47 +0100 Subject: [PATCH 08/16] Merge bitcoin/bitcoin#26424: doc: correct deriveaddresses RPC name 0f38524c31da4cf69d8e904569fe56292e4325b9 doc: correct deriveaddresses RPC name (Bitcoin Hodler) Pull request description: There never was a `deriveaddress` RPC, from what I can tell. It was always called `deriveaddresses` (plural). ACKs for top commit: theStack: ACK 0f38524c31da4cf69d8e904569fe56292e4325b9 Zero-1729: ACK 0f38524c31da4cf69d8e904569fe56292e4325b9 Tree-SHA512: 3f405f5479a0d39cf150fd80b4d854ffe4eef718a358202c619e34a08d98c1252b82fc70d106cdf2215dc5a50c6f6cd5e26fe7ed87156f6b08f8e97d963affb7 --- doc/descriptors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/descriptors.md b/doc/descriptors.md index fce4f9a323a4..0e5a06c22f45 100644 --- a/doc/descriptors.md +++ b/doc/descriptors.md @@ -241,6 +241,6 @@ ones. For larger numbers of errors, or other types of errors, there is a roughly 1 in a trillion chance of not detecting the errors. All RPCs in Dash Core will include the checksum in their output. Only -certain RPCs require checksums on input, including `deriveaddress` and +certain RPCs require checksums on input, including `deriveaddresses` and `importmulti`. The checksum for a descriptor without one can be computed using the `getdescriptorinfo` RPC. From 9134f2a80f881cbb5a2741b7881d53ca4d047c4d Mon Sep 17 00:00:00 2001 From: fanquake Date: Tue, 1 Nov 2022 10:00:32 +0000 Subject: [PATCH 09/16] Merge bitcoin/bitcoin#26294: build: move util/url to common/url 3a0b352c63db543833e0f919a004cf2c5093fe9c refactor: move url.h/cpp from lib util to lib common (fanquake) 058eb69ce47c17205a41fc667ce66429f09a332a build: add missing event cflags to libbitcoin_util (fanquake) Pull request description: Move `util/url` to `common/url`. Also add missing `event_*` flags to `libbitcoin_util`. #26293 + the commit dropping boost cppflags from `libbitcoin_util` shows this issue. i.e: ```bash CXX util/libbitcoin_util_a-url.o util/url.cpp:7:10: fatal error: 'event2/http.h' file not found #include ^~~~~~~~~~~~~~~ 1 error generated. ``` ACKs for top commit: hebasto: ACK 3a0b352c63db543833e0f919a004cf2c5093fe9c ryanofsky: Code review ACK 3a0b352c63db543833e0f919a004cf2c5093fe9c Tree-SHA512: 600a76fd334267a02d332df9b67891a38d3fd7f5baf8a82b2447879b3bc65eab2552d2c081c0a5f1ec927bf80df7fc1f0cbbdda4cb76994b46dadf260b8e1cb3 --- ci/dash/lint-tidy.sh | 2 +- src/Makefile.am | 12 +++++++----- src/bitcoin-cli.cpp | 2 +- src/bitcoin-wallet.cpp | 2 +- src/bitcoind.cpp | 2 +- src/{util => common}/url.cpp | 2 +- src/{util => common}/url.h | 6 +++--- src/qt/main.cpp | 2 +- src/test/fuzz/string.cpp | 2 +- src/test/util/setup_common.cpp | 2 +- src/wallet/rpc/util.cpp | 2 +- src/wallet/rpc/wallet.cpp | 2 +- 12 files changed, 20 insertions(+), 18 deletions(-) rename src/{util => common}/url.cpp (95%) rename src/{util => common}/url.h (79%) diff --git a/ci/dash/lint-tidy.sh b/ci/dash/lint-tidy.sh index e8153a62d844..814861423689 100755 --- a/ci/dash/lint-tidy.sh +++ b/ci/dash/lint-tidy.sh @@ -83,6 +83,7 @@ echo "==========================" cd "${BASE_ROOT_DIR}/build-ci/dashcore-${BUILD_TARGET}" iwyu_tool.py \ + "src/common/url.cpp" \ "src/compat" \ "src/dbwrapper.cpp" \ "src/init" \ @@ -110,7 +111,6 @@ iwyu_tool.py \ "src/util/string.cpp" \ "src/util/strencodings.cpp" \ "src/util/syserror.cpp" \ - "src/util/url.cpp" \ "src/zmq" \ -p . "${MAKEJOBS}" \ -- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_ROOT_DIR}/contrib/devtools/iwyu/bitcoin.core.imp" \ diff --git a/src/Makefile.am b/src/Makefile.am index a7a0819ce088..c108d1f929af 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -187,6 +187,7 @@ BITCOIN_CORE_H = \ coins.h \ common/bloom.h \ common/run_command.h \ + common/url.h \ compat/assumptions.h \ compat/byteswap.h \ compat/compat.h \ @@ -430,7 +431,6 @@ BITCOIN_CORE_H = \ util/translation.h \ util/types.h \ util/ui_change_type.h \ - util/url.h \ util/vector.h \ util/wpipe.h \ validation.h \ @@ -956,6 +956,11 @@ libbitcoin_common_a_SOURCES = \ script/standard.cpp \ warnings.cpp \ $(BITCOIN_CORE_H) + +if USE_LIBEVENT +libbitcoin_common_a_CPPFLAGS += $(EVENT_CFLAGS) +libbitcoin_common_a_SOURCES += common/url.cpp +endif # # util # @@ -1012,10 +1017,6 @@ libbitcoin_util_a_SOURCES = \ util/tokenpipe.cpp \ util/wpipe.cpp \ $(BITCOIN_CORE_H) - -if USE_LIBEVENT -libbitcoin_util_a_SOURCES += util/url.cpp -endif # # cli # @@ -1080,6 +1081,7 @@ endif dash_cli_LDADD = \ $(LIBBITCOIN_CLI) \ $(LIBUNIVALUE) \ + $(LIBBITCOIN_COMMON) \ $(LIBBITCOIN_UTIL) \ $(LIBBITCOIN_CRYPTO) dash_cli_LDADD += $(BACKTRACE_LIBS) $(EVENT_LIBS) $(GMP_LIBS) diff --git a/src/bitcoin-cli.cpp b/src/bitcoin-cli.cpp index c8018e7b9aa7..549a54da9b1f 100644 --- a/src/bitcoin-cli.cpp +++ b/src/bitcoin-cli.cpp @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -23,7 +24,6 @@ #include #include #include -#include #include #include diff --git a/src/bitcoin-wallet.cpp b/src/bitcoin-wallet.cpp index cb6876d20da1..21669542cd79 100644 --- a/src/bitcoin-wallet.cpp +++ b/src/bitcoin-wallet.cpp @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -18,7 +19,6 @@ #include #include #include -#include #include #include diff --git a/src/bitcoind.cpp b/src/bitcoind.cpp index 2bc239b915e5..18153e371b01 100644 --- a/src/bitcoind.cpp +++ b/src/bitcoind.cpp @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -26,7 +27,6 @@ #include #include #include -#include #include #include diff --git a/src/util/url.cpp b/src/common/url.cpp similarity index 95% rename from src/util/url.cpp rename to src/common/url.cpp index ea9323e66696..5200d55096a7 100644 --- a/src/util/url.cpp +++ b/src/common/url.cpp @@ -2,7 +2,7 @@ // Distributed under the MIT software license, see the accompanying // file COPYING or http://www.opensource.org/licenses/mit-license.php. -#include +#include #include diff --git a/src/util/url.h b/src/common/url.h similarity index 79% rename from src/util/url.h rename to src/common/url.h index 5a7b11fa0414..7bbd8b60de3c 100644 --- a/src/util/url.h +++ b/src/common/url.h @@ -2,8 +2,8 @@ // Distributed under the MIT software license, see the accompanying // file COPYING or http://www.opensource.org/licenses/mit-license.php. -#ifndef BITCOIN_UTIL_URL_H -#define BITCOIN_UTIL_URL_H +#ifndef BITCOIN_COMMON_URL_H +#define BITCOIN_COMMON_URL_H #include @@ -11,4 +11,4 @@ using UrlDecodeFn = std::string(const std::string& url_encoded); UrlDecodeFn urlDecode; extern UrlDecodeFn* const URL_DECODE; -#endif // BITCOIN_UTIL_URL_H +#endif // BITCOIN_COMMON_URL_H diff --git a/src/qt/main.cpp b/src/qt/main.cpp index eca9e9d88f3a..19a8b3685548 100644 --- a/src/qt/main.cpp +++ b/src/qt/main.cpp @@ -4,9 +4,9 @@ #include +#include #include #include -#include #include diff --git a/src/test/fuzz/string.cpp b/src/test/fuzz/string.cpp index 80d6625e6700..bca20f1f5891 100644 --- a/src/test/fuzz/string.cpp +++ b/src/test/fuzz/string.cpp @@ -4,6 +4,7 @@ #include #include +#include #include #include #include @@ -25,7 +26,6 @@ #include #include #include -#include #include #include diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp index cd999067b278..c40569744420 100644 --- a/src/test/util/setup_common.cpp +++ b/src/test/util/setup_common.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -43,7 +44,6 @@ #include #include #include -#include #include #include #include diff --git a/src/wallet/rpc/util.cpp b/src/wallet/rpc/util.cpp index 2ab5fb449b42..bb5e7da70daf 100644 --- a/src/wallet/rpc/util.cpp +++ b/src/wallet/rpc/util.cpp @@ -4,9 +4,9 @@ #include +#include #include #include -#include #include #include diff --git a/src/wallet/rpc/wallet.cpp b/src/wallet/rpc/wallet.cpp index 7fc43fa5ef2a..3ec1d5fa3304 100644 --- a/src/wallet/rpc/wallet.cpp +++ b/src/wallet/rpc/wallet.cpp @@ -5,6 +5,7 @@ // file COPYING or http://www.opensource.org/licenses/mit-license.php. #include +#include #include #include #include @@ -15,7 +16,6 @@ #include #include #include -#include #include #include #include From 2cc1ce74cdb3c4faa9fc8fe05453069e5811c9d8 Mon Sep 17 00:00:00 2001 From: fanquake Date: Thu, 30 Jun 2022 15:41:36 +0100 Subject: [PATCH 10/16] Merge bitcoin/bitcoin#24836: add RPC (-regtest only) for testing package policy e866f0d0666664885d4c15c79bf59cc59975887a [functional test] submitrawpackage RPC (glozow) fa076515b07ac4b10b2134e323bf4f56be5996a8 [rpc] add new submitpackage RPC (glozow) Pull request description: It would be nice for LN/wallet/app devs to test out package policy, package RBF, etc., but the only interface to do so right now is through unit tests. This PR adds a `-regtest` only RPC interface so people can test by submitting raw transaction data. It is regtest-only, as it would be unsafe/confusing to create an actual mainnet interface while package relay doesn't exist. Note that the functional tests are there to ensure the RPC interface is working properly; they aren't for testing policy itself. See src/test/txpackage_tests.cpp. ACKs for top commit: t-bast: Tested ACK against eclair https://github.com/bitcoin/bitcoin/pull/24836/commits/e866f0d0666664885d4c15c79bf59cc59975887a ariard: Code Review ACK e866f0d0 instagibbs: code review ACK e866f0d0666664885d4c15c79bf59cc59975887a Tree-SHA512: 824a26b10d2240e0fd85e5dd25bf499ee3dd9ba8ef4f522533998fcf767ddded9f001f7a005fe3ab07ec95e696448484e26599803e6034ed2733125c8c376c84 --- src/rpc/client.cpp | 1 + src/rpc/mempool.cpp | 131 +++++++++++++++++++++++++++++- src/test/fuzz/rpc.cpp | 1 + src/util/error.cpp | 2 + src/util/error.h | 1 + test/functional/rpc_packages.py | 136 +++++++++++++++++++++++++++++++- test/functional/test_runner.py | 2 +- 7 files changed, 268 insertions(+), 6 deletions(-) diff --git a/src/rpc/client.cpp b/src/rpc/client.cpp index 636ca29d6163..5c529c26669d 100644 --- a/src/rpc/client.cpp +++ b/src/rpc/client.cpp @@ -139,6 +139,7 @@ static const CRPCConvertParam vRPCConvertParams[] = { "sendrawtransaction", 3, "bypasslimits" }, { "testmempoolaccept", 0, "rawtxs" }, { "testmempoolaccept", 1, "maxfeerate" }, + { "submitpackage", 0, "package" }, { "combinerawtransaction", 0, "txs" }, { "fundrawtransaction", 1, "options" }, { "walletcreatefundedpsbt", 0, "inputs" }, diff --git a/src/rpc/mempool.cpp b/src/rpc/mempool.cpp index adaf7eee581a..9ab6a9917389 100644 --- a/src/rpc/mempool.cpp +++ b/src/rpc/mempool.cpp @@ -5,6 +5,7 @@ #include +#include #include #include #include @@ -236,7 +237,7 @@ static RPCHelpMan testmempoolaccept() static std::vector MempoolEntryDescription() { return { - RPCResult{RPCResult::Type::NUM, "vsize", "virtual transaction size. This can be different from actual serialized size for high-sigop transactions."}, + RPCResult{RPCResult::Type::NUM, "vsize", "Transaction size."}, RPCResult{RPCResult::Type::STR_AMOUNT, "fee", /*optional=*/true, "transaction fee, denominated in " + CURRENCY_UNIT + " (DEPRECATED, returned only if config option -deprecatedrpc=fees is passed)"}, RPCResult{RPCResult::Type::STR_AMOUNT, "modifiedfee", /*optional=*/true, @@ -765,6 +766,133 @@ static RPCHelpMan savemempool() }; } +static RPCHelpMan submitpackage() +{ + return RPCHelpMan{"submitpackage", + "Submit a package of raw transactions (serialized, hex-encoded) to local node (-regtest only).\n" + "The package will be validated according to consensus and mempool policy rules. If all transactions pass, they will be accepted to mempool.\n" + "This RPC is experimental and the interface may be unstable. Refer to doc/policy/packages.md for documentation on package policies.\n" + "Warning: until package relay is in use, successful submission does not mean the transaction will propagate to other nodes on the network.\n" + "Currently, each transaction is broadcasted individually after submission, which means they must meet other nodes' feerate requirements alone.\n" + , + { + {"package", RPCArg::Type::ARR, RPCArg::Optional::NO, "An array of raw transactions.", + { + {"rawtx", RPCArg::Type::STR_HEX, RPCArg::Optional::OMITTED, ""}, + }, + }, + }, + RPCResult{ + RPCResult::Type::OBJ, "", "", + { + {RPCResult::Type::OBJ_DYN, "tx-results", "transaction results keyed by txid", + { + {RPCResult::Type::OBJ, "txid", "transaction txid", { + {RPCResult::Type::STR_HEX, "txid", "The transaction hash in hex"}, + {RPCResult::Type::NUM, "size", "Size of transaction in bytes"}, + {RPCResult::Type::OBJ, "fees", "Transaction fees", { + {RPCResult::Type::STR_AMOUNT, "base", "transaction fee in " + CURRENCY_UNIT}, + }}, + }} + }}, + {RPCResult::Type::STR_AMOUNT, "package-feerate", /*optional=*/true, "package feerate used for feerate checks in " + CURRENCY_UNIT + " per KvB. Excludes transactions which were deduplicated or accepted individually."}, + }, + }, + RPCExamples{ + HelpExampleCli("testmempoolaccept", "[rawtx1, rawtx2]") + + HelpExampleCli("submitpackage", "[rawtx1, rawtx2]") + }, + [&](const RPCHelpMan& self, const JSONRPCRequest& request) -> UniValue + { + if (!Params().IsMockableChain()) { + throw std::runtime_error("submitpackage is for regression testing (-regtest mode) only"); + } + RPCTypeCheck(request.params, { + UniValue::VARR, + }); + const UniValue raw_transactions = request.params[0].get_array(); + if (raw_transactions.size() < 1 || raw_transactions.size() > MAX_PACKAGE_COUNT) { + throw JSONRPCError(RPC_INVALID_PARAMETER, + "Array must contain between 1 and " + ToString(MAX_PACKAGE_COUNT) + " transactions."); + } + + std::vector txns; + txns.reserve(raw_transactions.size()); + for (const auto& rawtx : raw_transactions.getValues()) { + CMutableTransaction mtx; + if (!DecodeHexTx(mtx, rawtx.get_str())) { + throw JSONRPCError(RPC_DESERIALIZATION_ERROR, + "TX decode failed: " + rawtx.get_str() + " Make sure the tx has at least one input."); + } + txns.emplace_back(MakeTransactionRef(std::move(mtx))); + } + + NodeContext& node = EnsureAnyNodeContext(request.context); + CTxMemPool& mempool = EnsureMemPool(node); + CChainState& chainstate = EnsureChainman(node).ActiveChainstate(); + const auto package_result = WITH_LOCK(::cs_main, return ProcessNewPackage(chainstate, mempool, txns, /*test_accept=*/ false)); + + // First catch any errors. + switch(package_result.m_state.GetResult()) { + case PackageValidationResult::PCKG_RESULT_UNSET: break; + case PackageValidationResult::PCKG_POLICY: + { + throw JSONRPCTransactionError(TransactionError::INVALID_PACKAGE, + package_result.m_state.GetRejectReason()); + } + case PackageValidationResult::PCKG_MEMPOOL_ERROR: + { + throw JSONRPCTransactionError(TransactionError::MEMPOOL_ERROR, + package_result.m_state.GetRejectReason()); + } + case PackageValidationResult::PCKG_TX: + { + for (const auto& tx : txns) { + auto it = package_result.m_tx_results.find(tx->GetHash()); + if (it != package_result.m_tx_results.end() && it->second.m_state.IsInvalid()) { + throw JSONRPCTransactionError(TransactionError::MEMPOOL_REJECTED, + strprintf("%s failed: %s", tx->GetHash().ToString(), it->second.m_state.GetRejectReason())); + } + } + // If a PCKG_TX error was returned, there must have been an invalid transaction. + NONFATAL_UNREACHABLE(); + } + } + for (const auto& tx : txns) { + size_t num_submitted{0}; + bilingual_str err_string; + const auto err = BroadcastTransaction(node, tx, err_string, 0, true, true); + if (err != TransactionError::OK) { + throw JSONRPCTransactionError(err, + strprintf("transaction broadcast failed: %s (all transactions were submitted, %d transactions were broadcast successfully)", + err_string.original, num_submitted)); + } + } + UniValue rpc_result{UniValue::VOBJ}; + UniValue tx_result_map{UniValue::VOBJ}; + for (const auto& tx : txns) { + auto it = package_result.m_tx_results.find(tx->GetHash()); + CHECK_NONFATAL(it != package_result.m_tx_results.end()); + UniValue result_inner{UniValue::VOBJ}; + result_inner.pushKV("txid", tx->GetHash().GetHex()); + if (it->second.m_result_type == MempoolAcceptResult::ResultType::VALID || + it->second.m_result_type == MempoolAcceptResult::ResultType::MEMPOOL_ENTRY) { + result_inner.pushKV("size", int64_t{it->second.m_vsize.value()}); + UniValue fees(UniValue::VOBJ); + fees.pushKV("base", ValueFromAmount(it->second.m_base_fees.value())); + result_inner.pushKV("fees", fees); + } + tx_result_map.pushKV(tx->GetHash().GetHex(), result_inner); + } + rpc_result.pushKV("tx-results", tx_result_map); + if (package_result.m_package_feerate.has_value()) { + rpc_result.pushKV("package-feerate", ValueFromAmount(package_result.m_package_feerate.value().GetFeePerK())); + } + return rpc_result; + }, + }; +} + void RegisterMempoolRPCCommands(CRPCTable& t) { static const CRPCCommand commands[]{ @@ -777,6 +905,7 @@ void RegisterMempoolRPCCommands(CRPCTable& t) {"blockchain", &getmempoolinfo}, {"blockchain", &getrawmempool}, {"blockchain", &savemempool}, + {"hidden", &submitpackage}, }; for (const auto& c : commands) { t.appendCommand(c.name, &c); diff --git a/src/test/fuzz/rpc.cpp b/src/test/fuzz/rpc.cpp index b95b93425e8e..d234b27bb1db 100644 --- a/src/test/fuzz/rpc.cpp +++ b/src/test/fuzz/rpc.cpp @@ -160,6 +160,7 @@ const std::vector RPC_COMMANDS_SAFE_FOR_FUZZING{ "signrawtransactionwithkey", "submitblock", "submitheader", + "submitpackage", "syncwithvalidationinterfacequeue", "testmempoolaccept", "uptime", diff --git a/src/util/error.cpp b/src/util/error.cpp index 9a6002209e3e..390cb6c11b38 100644 --- a/src/util/error.cpp +++ b/src/util/error.cpp @@ -37,6 +37,8 @@ bilingual_str TransactionErrorString(const TransactionError err) return Untranslated("External signer not found"); case TransactionError::EXTERNAL_SIGNER_FAILED: return Untranslated("External signer failed to sign"); + case TransactionError::INVALID_PACKAGE: + return Untranslated("Transaction rejected due to invalid package"); // no default case, so the compiler can warn about missing cases } assert(false); diff --git a/src/util/error.h b/src/util/error.h index 42f19cf3cb21..27916501f09c 100644 --- a/src/util/error.h +++ b/src/util/error.h @@ -32,6 +32,7 @@ enum class TransactionError { MAX_FEE_EXCEEDED, EXTERNAL_SIGNER_NOT_FOUND, EXTERNAL_SIGNER_FAILED, + INVALID_PACKAGE, }; bilingual_str TransactionErrorString(const TransactionError error); diff --git a/test/functional/rpc_packages.py b/test/functional/rpc_packages.py index 740a8c02750f..d0e070000a1a 100755 --- a/test/functional/rpc_packages.py +++ b/test/functional/rpc_packages.py @@ -12,16 +12,20 @@ from test_framework.messages import ( tx_from_hex, ) +from test_framework.p2p import P2PTxInvStore from test_framework.script import ( CScript, OP_TRUE, ) from test_framework.util import ( assert_equal, + assert_fee_amount, + assert_raises_rpc_error, ) from test_framework.wallet import ( create_child_with_parents, create_raw_chain, + DEFAULT_FEE, make_chain, ) @@ -48,7 +52,7 @@ def run_test(self): self.address = node.get_deterministic_priv_key().address self.coins = [] # The last 100 coinbase transactions are premature - for b in self.generatetoaddress(node, 200, self.address)[:100]: + for b in self.generatetoaddress(node, 220, self.address)[:-100]: coinbase = node.getblock(blockhash=b, verbosity=2)["tx"][0] self.coins.append({ "txid": coinbase["txid"], @@ -78,7 +82,7 @@ def run_test(self): self.test_multiple_children() self.test_multiple_parents() self.test_conflicting() - + self.test_submitpackage() def test_independent(self): self.log.info("Test multiple independent transactions in a package") @@ -129,8 +133,7 @@ def test_independent(self): def test_chain(self): node = self.nodes[0] - first_coin = self.coins.pop() - (chain_hex, chain_txns) = create_raw_chain(node, first_coin, self.address, self.privkeys) + (chain_hex, chain_txns) = create_raw_chain(node, self.coins.pop(), self.address, self.privkeys) self.log.info("Check that testmempoolaccept requires packages to be sorted by dependency") assert_equal(node.testmempoolaccept(rawtxs=chain_hex[::-1]), [{"txid": tx.rehash(), "package-error": "package-not-sorted"} for tx in chain_txns[::-1]]) @@ -263,5 +266,130 @@ def test_conflicting(self): {"txid": tx2.rehash(), "package-error": "conflict-in-package"} ]) + + def assert_equal_package_results(self, node, testmempoolaccept_result, submitpackage_result): + """Assert that a successful submitpackage result is consistent with testmempoolaccept + results and getmempoolentry info. Note that the result structs are different and, due to + policy differences between testmempoolaccept and submitpackage (i.e. package feerate), + some information may be different. + """ + for testres_tx in testmempoolaccept_result: + # Grab this result from the submitpackage_result + submitres_tx = submitpackage_result["tx-results"][testres_tx["txid"]] + assert_equal(submitres_tx["txid"], testres_tx["txid"]) + # No "allowed" if the tx was already in the mempool + if "allowed" in testres_tx and testres_tx["allowed"]: + assert_equal(submitres_tx["size"], testres_tx["vsize"]) + assert_equal(submitres_tx["fees"]["base"], testres_tx["fees"]["base"]) + entry_info = node.getmempoolentry(submitres_tx["txid"]) + assert_equal(submitres_tx["size"], entry_info["vsize"]) + assert_equal(submitres_tx["fees"]["base"], entry_info["fees"]["base"]) + + def test_submit_child_with_parents(self, num_parents, partial_submit): + node = self.nodes[0] + peer = node.add_p2p_connection(P2PTxInvStore()) + # Test a package with num_parents parents and 1 child transaction. + package_hex = [] + package_txns = [] + values = [] + scripts = [] + for _ in range(num_parents): + parent_coin = self.coins.pop() + value = parent_coin["amount"] + (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, parent_coin["txid"], value) + package_hex.append(txhex) + package_txns.append(tx) + values.append(value) + scripts.append(spk) + if partial_submit and random.choice([True, False]): + node.sendrawtransaction(txhex) + child_hex = create_child_with_parents(node, self.address, self.privkeys, package_txns, values, scripts) + package_hex.append(child_hex) + package_txns.append(tx_from_hex(child_hex)) + + testmempoolaccept_result = node.testmempoolaccept(rawtxs=package_hex) + submitpackage_result = node.submitpackage(package=package_hex) + + # Check that each result is present, with the correct size and fees + for i in range(num_parents + 1): + tx = package_txns[i] + wtxid = tx.hash + assert wtxid in submitpackage_result["tx-results"] + tx_result = submitpackage_result["tx-results"][wtxid] + assert_equal(tx_result, { + "txid": tx.rehash(), + "size": tx.get_vsize(), + "fees": { + "base": DEFAULT_FEE, + } + }) + + # submitpackage result should be consistent with testmempoolaccept and getmempoolentry + self.assert_equal_package_results(node, testmempoolaccept_result, submitpackage_result) + + # Package feerate is calculated for the remaining transactions after deduplication and + # individual submission. If only 0 or 1 transaction is left, e.g. because all transactions + # had high-feerates or were already in the mempool, no package feerate is provided. + # In this case, since all of the parents have high fees, each is accepted individually. + assert "package-feerate" not in submitpackage_result + + # The node should announce each transaction. No guarantees for propagation. + self.bump_mocktime(30) + peer.wait_for_broadcast([tx.hash for tx in package_txns]) + self.generate(node, 1) + + + def test_submit_cpfp(self): + node = self.nodes[0] + peer = node.add_p2p_connection(P2PTxInvStore()) + + # 2 parent 1 child CPFP. First parent pays high fees, second parent pays 0 fees and is + # fee-bumped by the child. + coin_rich = self.coins.pop() + coin_poor = self.coins.pop() + tx_rich, hex_rich, value_rich, spk_rich = make_chain(node, self.address, self.privkeys, coin_rich["txid"], coin_rich["amount"]) + tx_poor, hex_poor, value_poor, spk_poor = make_chain(node, self.address, self.privkeys, coin_poor["txid"], coin_poor["amount"], fee=0) + package_txns = [tx_rich, tx_poor] + hex_child = create_child_with_parents(node, self.address, self.privkeys, package_txns, [value_rich, value_poor], [spk_rich, spk_poor]) + tx_child = tx_from_hex(hex_child) + package_txns.append(tx_child) + + submitpackage_result = node.submitpackage([hex_rich, hex_poor, hex_child]) + + rich_parent_result = submitpackage_result["tx-results"][tx_rich.hash] + poor_parent_result = submitpackage_result["tx-results"][tx_poor.hash] + child_result = submitpackage_result["tx-results"][tx_child.hash] + assert_equal(rich_parent_result["fees"]["base"], DEFAULT_FEE) + assert_equal(poor_parent_result["fees"]["base"], 0) + assert_equal(child_result["fees"]["base"], DEFAULT_FEE) + # Package feerate is calculated for the remaining transactions after deduplication and + # individual submission. Since this package had a 0-fee parent, package feerate must have + # been used and returned. + assert "package-feerate" in submitpackage_result + assert_fee_amount(DEFAULT_FEE, rich_parent_result["size"] + child_result["size"], submitpackage_result["package-feerate"]) + + # The node will broadcast each transaction, still abiding by its peer's fee filter + self.bump_mocktime(30) + peer.wait_for_broadcast([tx.hash for tx in package_txns]) + self.generate(node, 1) + + + def test_submitpackage(self): + node = self.nodes[0] + + self.log.info("Submitpackage valid packages with 1 child and some number of parents") + for num_parents in [1, 2, 24]: + self.test_submit_child_with_parents(num_parents, False) + self.test_submit_child_with_parents(num_parents, True) + + self.log.info("Submitpackage valid packages with CPFP") + self.test_submit_cpfp() + + self.log.info("Submitpackage only allows packages of 1 child with its parents") + # Chain of 3 transactions has too many generations + chain_hex, _ = create_raw_chain(node, self.coins.pop(), self.address, self.privkeys, 3) + assert_raises_rpc_error(-25, "not-child-with-parents", node.submitpackage, chain_hex) + + if __name__ == "__main__": RPCPackagesTest().main() diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py index bb7fe5cf5a16..2e5715cbf3be 100755 --- a/test/functional/test_runner.py +++ b/test/functional/test_runner.py @@ -160,6 +160,7 @@ 'p2p_dns_seeds.py', 'wallet_abandonconflict.py --descriptors', 'feature_csv_activation.py', + 'rpc_packages.py', 'feature_reindex.py', 'feature_abortnode.py', # vv Tests less than 30s vv @@ -279,7 +280,6 @@ 'mempool_packages.py', 'mempool_package_onemore.py', 'rpc_createmultisig.py', - 'rpc_packages.py', 'mempool_package_limits.py', 'feature_versionbits_warning.py', 'rpc_preciousblock.py', From 5a1764e5c98e17c754d1d57582c9fb8a4a18820e Mon Sep 17 00:00:00 2001 From: MacroFake Date: Wed, 3 Aug 2022 11:11:44 +0200 Subject: [PATCH 11/16] Merge bitcoin/bitcoin#25379: test: use MiniWallet to simplify mempool_package_limits.py tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit f2f6068b69b1b532db92b276f024c89b56f38294 test: MiniWallet: add `send_self_transfer_chain` to create chain of txns (Andreas Kouloumos) 1d6b438ef0ccd05e1522ac38b44f847c1d93e72f test: use MiniWallet to simplify mempool_package_limits.py tests (Andreas Kouloumos) Pull request description: While `wallet.py` includes the MiniWallet class and some helper methods, it also includes some methods that have been moved there without having any direct relation with the MiniWallet class. Specifically `make_chain`, `create_child_with_parents` and `create_raw_chain` methods that were extracted from `rpc_packages.py` at f8253d69d6f02850995a11eeb71fedc22e6f6575 in order to be used on both `mempool_package_limits.py` and `rpc_packages.py`. Since that change, due to the introduction of additional methods in MiniWallet, the functionality of those methods can now be replicated with the existing MiniWallet methods and simultaneously simplify those tests by using the MiniWallet. This PR's goals are - to simplify the `mempool_package_limits.py` functional tests with usage of the MiniWallet. - to make progress towards the removal of the `make_chain`, `create_child_with_parents` and `create_raw_chain` methods of `wallet.py`. For the purpose of the aforementioned goals, a helper method `MiniWallet.send_self_transfer_chain` is introduced and method `bulk_transaction` has been integrated in `create_self_transfer*` methods using an optional `target_weight` option. ACKs for top commit: MarcoFalke: ACK f2f6068b69b1b532db92b276f024c89b56f38294 👜 Tree-SHA512: 3ddfa0046168cbf7904ec6b1ca233b3fdd4f30db6aefae108b6d7fb69f34ef6fb2cf4fa7cef9473ce1434a0cc8149d236441a685352fef35359a2b7ba0d951eb --- test/functional/mempool_package_limits.py | 308 +++++----------------- test/functional/test_framework/wallet.py | 47 ++-- 2 files changed, 95 insertions(+), 260 deletions(-) diff --git a/test/functional/mempool_package_limits.py b/test/functional/mempool_package_limits.py index f4ee1e49eac5..b7db228f5571 100755 --- a/test/functional/mempool_package_limits.py +++ b/test/functional/mempool_package_limits.py @@ -6,26 +6,15 @@ from decimal import Decimal -from test_framework.address import ADDRESS_BCRT1_P2SH_OP_TRUE +from test_framework.blocktools import COINBASE_MATURITY from test_framework.test_framework import BitcoinTestFramework from test_framework.messages import ( COIN, - CTransaction, - tx_from_hex, -) -from test_framework.script import ( - CScript, - OP_TRUE, ) from test_framework.util import ( assert_equal, ) -from test_framework.wallet import ( - bulk_transaction, - create_child_with_parents, - make_chain, - DEFAULT_FEE, -) +from test_framework.wallet import MiniWallet class MempoolPackageLimitsTest(BitcoinTestFramework): def set_test_params(self): @@ -33,19 +22,10 @@ def set_test_params(self): self.setup_clean_chain = True def run_test(self): - self.log.info("Generate blocks to create UTXOs") - node = self.nodes[0] - self.privkeys = [node.get_deterministic_priv_key().key] - self.address = node.get_deterministic_priv_key().address - self.coins = [] - # The last 100 coinbase transactions are premature - for b in self.generatetoaddress(node, 200, self.address)[:100]: - coinbase = node.getblock(blockhash=b, verbosity=2)["tx"][0] - self.coins.append({ - "txid": coinbase["txid"], - "amount": coinbase["vout"][0]["value"], - "scriptPubKey": coinbase["vout"][0]["scriptPubKey"], - }) + self.wallet = MiniWallet(self.nodes[0]) + # Add enough mature utxos to the wallet so that all txs spend confirmed coins. + self.generate(self.wallet, 35) + self.generate(self.nodes[0], COINBASE_MATURITY) self.test_chain_limits() self.test_desc_count_limits() @@ -62,22 +42,14 @@ def run_test(self): def test_chain_limits_helper(self, mempool_count, package_count): node = self.nodes[0] assert_equal(0, node.getmempoolinfo()["size"]) - first_coin = self.coins.pop() - spk = None - txid = first_coin["txid"] chain_hex = [] - chain_txns = [] - value = first_coin["amount"] - - for i in range(mempool_count + package_count): - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk) - txid = tx.rehash() - if i < mempool_count: - node.sendrawtransaction(txhex) - assert_equal(node.getmempoolentry(txid)["ancestorcount"], i + 1) - else: - chain_hex.append(txhex) - chain_txns.append(tx) + + chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=mempool_count) + # in-package transactions + for _ in range(package_count): + tx = self.wallet.create_self_transfer(utxo_to_spend=chaintip_utxo) + chaintip_utxo = tx["new_utxo"] + chain_hex.append(tx["hex"]) testres_too_long = node.testmempoolaccept(rawtxs=chain_hex) for txres in testres_too_long: assert_equal(txres["package-error"], "package-mempool-limits") @@ -123,48 +95,20 @@ def test_desc_count_limits(self): assert_equal(0, node.getmempoolinfo()["size"]) self.log.info("Check that in-mempool and in-package descendants are calculated properly in packages") # Top parent in mempool, M1 - first_coin = self.coins.pop() - parent_value = (first_coin["amount"] - Decimal("0.0002")) / 2 # Deduct reasonable fee and make 2 outputs - inputs = [{"txid": first_coin["txid"], "vout": 0}] - outputs = [{self.address : parent_value}, {ADDRESS_BCRT1_P2SH_OP_TRUE : parent_value}] - rawtx = node.createrawtransaction(inputs, outputs) - - parent_signed = node.signrawtransactionwithkey(hexstring=rawtx, privkeys=self.privkeys) - assert parent_signed["complete"] - parent_tx = tx_from_hex(parent_signed["hex"]) - parent_txid = parent_tx.rehash() - node.sendrawtransaction(parent_signed["hex"]) + m1_utxos = self.wallet.send_self_transfer_multi(from_node=node, num_outputs=2)['new_utxos'] package_hex = [] - - # Chain A - spk = parent_tx.vout[0].scriptPubKey.hex() - value = parent_value - txid = parent_txid - for i in range(12): - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk) - txid = tx.rehash() - if i < 11: # M2a... M12a - node.sendrawtransaction(txhex) - else: # Pa - package_hex.append(txhex) - - # Chain B - value = parent_value - Decimal("0.0001") - rawtx_b = node.createrawtransaction([{"txid": parent_txid, "vout": 1}], {self.address : value}) - tx_child_b = tx_from_hex(rawtx_b) # M2b - tx_child_b.vin[0].scriptSig = CScript([CScript([OP_TRUE])]) - tx_child_b_hex = tx_child_b.serialize().hex() - node.sendrawtransaction(tx_child_b_hex) - spk = tx_child_b.vout[0].scriptPubKey.hex() - txid = tx_child_b.rehash() - for i in range(12): - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk) - txid = tx.rehash() - if i < 11: # M3b... M13b - node.sendrawtransaction(txhex) - else: # Pb - package_hex.append(txhex) + # Chain A (M2a... M12a) + chain_a_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=11, utxo_to_spend=m1_utxos[0]) + # Pa + pa_hex = self.wallet.create_self_transfer(utxo_to_spend=chain_a_tip_utxo)["hex"] + package_hex.append(pa_hex) + + # Chain B (M2b... M13b) + chain_b_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12, utxo_to_spend=m1_utxos[1]) + # Pb + pb_hex = self.wallet.create_self_transfer(utxo_to_spend=chain_b_tip_utxo)["hex"] + package_hex.append(pb_hex) assert_equal(24, node.getmempoolinfo()["size"]) assert_equal(2, len(package_hex)) @@ -197,40 +141,18 @@ def test_desc_count_limits_2(self): node = self.nodes[0] package_hex = [] # M1 - first_coin_a = self.coins.pop() - parent_value = (first_coin_a["amount"] - DEFAULT_FEE) / 2 # Deduct reasonable fee and make 2 outputs - inputs = [{"txid": first_coin_a["txid"], "vout": 0}] - outputs = [{self.address : parent_value}, {ADDRESS_BCRT1_P2SH_OP_TRUE : parent_value}] - rawtx = node.createrawtransaction(inputs, outputs) - - parent_signed = node.signrawtransactionwithkey(hexstring=rawtx, privkeys=self.privkeys) - assert parent_signed["complete"] - parent_tx = tx_from_hex(parent_signed["hex"]) - parent_txid = parent_tx.rehash() - node.sendrawtransaction(parent_signed["hex"]) + m1_utxos = self.wallet.send_self_transfer_multi(from_node=node, num_outputs=2)['new_utxos'] # Chain M2...M24 - spk = parent_tx.vout[0].scriptPubKey.hex() - value = parent_value - txid = parent_txid - for i in range(23): # M2...M24 - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk) - txid = tx.rehash() - node.sendrawtransaction(txhex) + self.wallet.send_self_transfer_chain(from_node=node, chain_length=23, utxo_to_spend=m1_utxos[0]) # P1 - value_p1 = (parent_value - DEFAULT_FEE) - rawtx_p1 = node.createrawtransaction([{"txid": parent_txid, "vout": 1}], [{self.address : value_p1}]) - tx_child_p1 = tx_from_hex(rawtx_p1) - tx_child_p1.vin[0].scriptSig = CScript([CScript([OP_TRUE])]) - tx_child_p1_hex = tx_child_p1.serialize().hex() - txid_child_p1 = tx_child_p1.rehash() - package_hex.append(tx_child_p1_hex) - tx_child_p1_spk = tx_child_p1.vout[0].scriptPubKey.hex() + p1_tx = self.wallet.create_self_transfer(utxo_to_spend=m1_utxos[1]) + package_hex.append(p1_tx["hex"]) # P2 - (_, tx_child_p2_hex, _, _) = make_chain(node, self.address, self.privkeys, txid_child_p1, value_p1, 0, tx_child_p1_spk) - package_hex.append(tx_child_p2_hex) + p2_tx = self.wallet.create_self_transfer(utxo_to_spend=p1_tx["new_utxo"]) + package_hex.append(p2_tx["hex"]) assert_equal(24, node.getmempoolinfo()["size"]) assert_equal(2, len(package_hex)) @@ -262,32 +184,21 @@ def test_anc_count_limits(self): node = self.nodes[0] assert_equal(0, node.getmempoolinfo()["size"]) package_hex = [] - parents_tx = [] - values = [] - scripts = [] + pc_parent_utxos = [] self.log.info("Check that in-mempool and in-package ancestors are calculated properly in packages") # Two chains of 13 transactions each for _ in range(2): - spk = None - top_coin = self.coins.pop() - txid = top_coin["txid"] - value = top_coin["amount"] - for i in range(13): - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk) - txid = tx.rehash() - if i < 12: - node.sendrawtransaction(txhex) - else: # Save the 13th transaction for the package - package_hex.append(txhex) - parents_tx.append(tx) - scripts.append(spk) - values.append(value) + chain_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12) + # Save the 13th transaction for the package + tx = self.wallet.create_self_transfer(utxo_to_spend=chain_tip_utxo) + package_hex.append(tx["hex"]) + pc_parent_utxos.append(tx["new_utxo"]) # Child Pc - child_hex = create_child_with_parents(node, self.address, self.privkeys, parents_tx, values, scripts) - package_hex.append(child_hex) + pc_hex = self.wallet.create_self_transfer_multi(utxos_to_spend=pc_parent_utxos)["hex"] + package_hex.append(pc_hex) assert_equal(24, node.getmempoolinfo()["size"]) assert_equal(3, len(package_hex)) @@ -317,45 +228,29 @@ def test_anc_count_limits_2(self): """ node = self.nodes[0] assert_equal(0, node.getmempoolinfo()["size"]) - parents_tx = [] - values = [] - scripts = [] + pc_parent_utxos = [] self.log.info("Check that in-mempool and in-package ancestors are calculated properly in packages") # Two chains of 12 transactions each for _ in range(2): - spk = None - top_coin = self.coins.pop() - txid = top_coin["txid"] - value = top_coin["amount"] - for i in range(12): - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk) - txid = tx.rehash() - value -= Decimal("0.0001") - node.sendrawtransaction(txhex) - if i == 11: - # last 2 transactions will be the parents of Pc - parents_tx.append(tx) - values.append(value) - scripts.append(spk) + chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12) + # last 2 transactions will be the parents of Pc + pc_parent_utxos.append(chaintip_utxo) # Child Pc - pc_hex = create_child_with_parents(node, self.address, self.privkeys, parents_tx, values, scripts) - pc_tx = tx_from_hex(pc_hex) - pc_value = sum(values) - Decimal("0.0002") - pc_spk = pc_tx.vout[0].scriptPubKey.hex() + pc_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=pc_parent_utxos) # Child Pd - (_, pd_hex, _, _) = make_chain(node, self.address, self.privkeys, pc_tx.rehash(), pc_value, 0, pc_spk) + pd_tx = self.wallet.create_self_transfer(utxo_to_spend=pc_tx["new_utxos"][0]) assert_equal(24, node.getmempoolinfo()["size"]) - testres_too_long = node.testmempoolaccept(rawtxs=[pc_hex, pd_hex]) + testres_too_long = node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]]) for txres in testres_too_long: assert_equal(txres["package-error"], "package-mempool-limits") # Clear mempool and check that the package passes now self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=[pc_hex, pd_hex])]) + assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]])]) def test_anc_count_limits_bushy(self): """Create a tree with 20 transactions in the mempool and 6 in the package: @@ -371,31 +266,18 @@ def test_anc_count_limits_bushy(self): node = self.nodes[0] assert_equal(0, node.getmempoolinfo()["size"]) package_hex = [] - parent_txns = [] - parent_values = [] - scripts = [] + pc_parent_utxos = [] for _ in range(5): # Make package transactions P0 ... P4 - gp_tx = [] - gp_values = [] - gp_scripts = [] + pc_grandparent_utxos = [] for _ in range(4): # Make mempool transactions M(4i+1)...M(4i+4) - parent_coin = self.coins.pop() - value = parent_coin["amount"] - txid = parent_coin["txid"] - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, txid, value) - gp_tx.append(tx) - gp_values.append(value) - gp_scripts.append(spk) - node.sendrawtransaction(txhex) + pc_grandparent_utxos.append(self.wallet.send_self_transfer(from_node=node)["new_utxo"]) # Package transaction Pi - pi_hex = create_child_with_parents(node, self.address, self.privkeys, gp_tx, gp_values, gp_scripts) - package_hex.append(pi_hex) - pi_tx = tx_from_hex(pi_hex) - parent_txns.append(pi_tx) - parent_values.append(Decimal(pi_tx.vout[0].nValue) / COIN) - scripts.append(pi_tx.vout[0].scriptPubKey.hex()) + pi_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=pc_grandparent_utxos) + package_hex.append(pi_tx["hex"]) + pc_parent_utxos.append(pi_tx["new_utxos"][0]) # Package transaction PC - package_hex.append(create_child_with_parents(node, self.address, self.privkeys, parent_txns, parent_values, scripts)) + pc_hex = self.wallet.create_self_transfer_multi(utxos_to_spend=pc_parent_utxos)["hex"] + package_hex.append(pc_hex) assert_equal(20, node.getmempoolinfo()["size"]) assert_equal(6, len(package_hex)) @@ -419,52 +301,32 @@ def test_anc_size_limits(self): and in-package ancestors are all considered together. """ node = self.nodes[0] + parent_utxos = [] assert_equal(0, node.getmempoolinfo()["size"]) parents_tx = [] - values = [] - scripts = [] target_weight = 1000 * 30 # 30KvB high_fee = Decimal("0.003") # 10 sats/vB self.log.info("Check that in-mempool and in-package ancestor size limits are calculated properly in packages") # Mempool transactions A and B for _ in range(2): - spk = None - top_coin = self.coins.pop() - txid = top_coin["txid"] - value = top_coin["amount"] - (tx, _, _, _) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk, high_fee) - bulked_tx = bulk_transaction(tx, node, target_weight, self.privkeys) - node.sendrawtransaction(bulked_tx.serialize().hex()) - parents_tx.append(bulked_tx) - values.append(Decimal(bulked_tx.vout[0].nValue) / COIN) - scripts.append(bulked_tx.vout[0].scriptPubKey.hex()) + bulked_tx = self.wallet.create_self_transfer(target_weight=target_weight) + self.wallet.sendrawtransaction(from_node=node, tx_hex=bulked_tx["hex"]) + parent_utxos.append(bulked_tx["new_utxo"]) # Package transaction C - small_pc_hex = create_child_with_parents(node, self.address, self.privkeys, parents_tx, values, scripts, high_fee) - pc_tx = bulk_transaction(tx_from_hex(small_pc_hex), node, target_weight, self.privkeys) - pc_value = Decimal(pc_tx.vout[0].nValue) / COIN - pc_spk = pc_tx.vout[0].scriptPubKey.hex() - pc_hex = pc_tx.serialize().hex() + pc_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=parent_utxos, fee_per_output=int(high_fee * COIN), target_weight=target_weight) # Package transaction D - (small_pd, _, val, spk) = make_chain(node, self.address, self.privkeys, pc_tx.rehash(), pc_value, 0, pc_spk, high_fee) - prevtxs = [{ - "txid": pc_tx.rehash(), - "vout": 0, - "scriptPubKey": spk, - "amount": val, - }] - pd_tx = bulk_transaction(small_pd, node, target_weight, self.privkeys, prevtxs) - pd_hex = pd_tx.serialize().hex() + pd_tx = self.wallet.create_self_transfer(utxo_to_spend=pc_tx["new_utxos"][0], target_weight=target_weight) assert_equal(2, node.getmempoolinfo()["size"]) - testres_too_heavy = node.testmempoolaccept(rawtxs=[pc_hex, pd_hex]) + testres_too_heavy = node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]]) for txres in testres_too_heavy: assert_equal(txres["package-error"], "package-mempool-limits") # Clear mempool and check that the package passes now self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=[pc_hex, pd_hex])]) + assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]])]) def test_desc_size_limits(self): """Create 3 mempool transactions and 2 package transactions (25KvB each): @@ -482,50 +344,18 @@ def test_desc_size_limits(self): high_fee = Decimal("0.0021") # 10 sats/vB self.log.info("Check that in-mempool and in-package descendant sizes are calculated properly in packages") # Top parent in mempool, Ma - first_coin = self.coins.pop() - parent_value = (first_coin["amount"] - high_fee) / 2 # Deduct fee and make 2 outputs - inputs = [{"txid": first_coin["txid"], "vout": 0}] - outputs = [{self.address : parent_value}, {ADDRESS_BCRT1_P2SH_OP_TRUE: parent_value}] - rawtx = node.createrawtransaction(inputs, outputs) - parent_tx = bulk_transaction(tx_from_hex(rawtx), node, target_weight, self.privkeys) - node.sendrawtransaction(parent_tx.serialize().hex()) + ma_tx = self.wallet.create_self_transfer_multi(num_outputs=2, fee_per_output=int(high_fee / 2 * COIN), target_weight=target_weight) + self.wallet.sendrawtransaction(from_node=node, tx_hex=ma_tx["hex"]) package_hex = [] for j in range(2): # Two legs (left and right) # Mempool transaction (Mb and Mc) - mempool_tx = CTransaction() - spk = parent_tx.vout[j].scriptPubKey.hex() - value = Decimal(parent_tx.vout[j].nValue) / COIN - txid = parent_tx.rehash() - prevtxs = [{ - "txid": txid, - "vout": j, - "scriptPubKey": spk, - "amount": value, - }] - if j == 0: # normal key - (tx_small, _, _, _) = make_chain(node, self.address, self.privkeys, txid, value, j, spk, high_fee) - mempool_tx = bulk_transaction(tx_small, node, target_weight, self.privkeys, prevtxs) - else: # OP_TRUE - inputs = [{"txid": txid, "vout": 1}] - outputs = {self.address: value - high_fee} - small_tx = tx_from_hex(node.createrawtransaction(inputs, outputs)) - mempool_tx = bulk_transaction(small_tx, node, target_weight, None, prevtxs) - node.sendrawtransaction(mempool_tx.serialize().hex()) + mempool_tx = self.wallet.create_self_transfer(utxo_to_spend=ma_tx["new_utxos"][j], target_weight=target_weight) + self.wallet.sendrawtransaction(from_node=node, tx_hex=mempool_tx["hex"]) # Package transaction (Pd and Pe) - spk = mempool_tx.vout[0].scriptPubKey.hex() - value = Decimal(mempool_tx.vout[0].nValue) / COIN - txid = mempool_tx.rehash() - (tx_small, _, _, _) = make_chain(node, self.address, self.privkeys, txid, value, 0, spk, high_fee) - prevtxs = [{ - "txid": txid, - "vout": 0, - "scriptPubKey": spk, - "amount": value, - }] - package_tx = bulk_transaction(tx_small, node, target_weight, self.privkeys, prevtxs) - package_hex.append(package_tx.serialize().hex()) + package_tx = self.wallet.create_self_transfer(utxo_to_spend=mempool_tx["new_utxo"], target_weight=target_weight) + package_hex.append(package_tx["hex"]) assert_equal(3, node.getmempoolinfo()["size"]) assert_equal(2, len(package_hex)) diff --git a/test/functional/test_framework/wallet.py b/test/functional/test_framework/wallet.py index e350614b20cb..737b94f52413 100644 --- a/test/functional/test_framework/wallet.py +++ b/test/functional/test_framework/wallet.py @@ -7,7 +7,6 @@ from copy import deepcopy from decimal import Decimal from enum import Enum -from random import choice from typing import ( Any, List, @@ -94,6 +93,16 @@ def __init__(self, test_node, *, mode=MiniWalletMode.ADDRESS_OP_TRUE): def _create_utxo(self, *, txid, vout, value, height, coinbase, confirmations): return {"txid": txid, "vout": vout, "value": value, "height": height, "coinbase": coinbase, "confirmations": confirmations} + def _bulk_tx(self, tx, target_weight): + """Pad a transaction with extra outputs until it reaches a target weight (or higher). + returns the tx + """ + assert_greater_than_or_equal(target_weight, tx.get_weight()) + while tx.get_weight() < target_weight: + script_pubkey = ( b"6a4d0200" # OP_RETURN OP_PUSH2 512 bytes + + b"01" * 512 ) + tx.vout.append(CTxOut(0, script_pubkey)) + def get_balance(self): return sum(u['value'] for u in self._utxos) @@ -255,6 +264,7 @@ def create_self_transfer_multi( locktime=0, sequence=0, fee_per_output=1000, + target_weight=0 ): """ Create and return a transaction that spends the given UTXOs and creates a @@ -278,6 +288,9 @@ def create_self_transfer_multi( self.sign_tx(tx) + if target_weight: + self._bulk_tx(tx, target_weight) + txid = tx.rehash() return { "new_utxos": [self._create_utxo( @@ -293,7 +306,7 @@ def create_self_transfer_multi( "tx": tx, } - def create_self_transfer(self, *, fee_rate=Decimal("0.003"), fee=Decimal("0"), utxo_to_spend=None, locktime=0, sequence=0): + def create_self_transfer(self, *, fee_rate=Decimal("0.003"), fee=Decimal("0"), utxo_to_spend=None, locktime=0, sequence=0, target_weight=0): """Create and return a tx with the specified fee. If fee is 0, use fee_rate, where the resulting fee may be exact or at most one satoshi higher than needed.""" utxo_to_spend = utxo_to_spend or self.get_utxo() assert fee_rate >= 0 @@ -319,6 +332,17 @@ def sendrawtransaction(self, *, from_node, tx_hex, maxfeerate=0, **kwargs): self.scan_tx(from_node.decoderawtransaction(tx_hex)) return txid + def send_self_transfer_chain(self, *, from_node, chain_length, utxo_to_spend=None): + """Create and send a "chain" of chain_length transactions. The nth transaction in + the chain is a child of the n-1th transaction and parent of the n+1th transaction. + + Returns the chaintip (nth) utxo + """ + chaintip_utxo = utxo_to_spend or self.get_utxo() + for _ in range(chain_length): + chaintip_utxo = self.send_self_transfer(utxo_to_spend=chaintip_utxo, from_node=from_node)["new_utxo"] + return chaintip_utxo + def getnewdestination(address_type='legacy'): """Generate a random destination of the specified type and return the @@ -400,22 +424,3 @@ def create_raw_chain(node, first_coin, address, privkeys, chain_length=25): chain_txns.append(tx) return (chain_hex, chain_txns) - -def bulk_transaction(tx, node, target_weight, privkeys, prevtxs=None): - """Pad a transaction with extra outputs until it reaches a target weight (or higher). - returns CTransaction object - """ - tx_heavy = deepcopy(tx) - assert_greater_than_or_equal(target_weight, tx_heavy.get_weight()) - while tx_heavy.get_weight() < target_weight: - random_spk = "6a4d0200" # OP_RETURN OP_PUSH2 512 bytes - for _ in range(512*2): - random_spk += choice("0123456789ABCDEF") - tx_heavy.vout.append(CTxOut(0, bytes.fromhex(random_spk))) - # Re-sign the transaction - if privkeys: - signed = node.signrawtransactionwithkey(tx_heavy.serialize().hex(), privkeys, prevtxs) - return tx_from_hex(signed["hex"]) - # OP_TRUE - tx_heavy.vin[0].scriptSig = CScript([CScript([OP_TRUE])]) - return tx_heavy From ce306af8c551ae9311df7fc29c55c0278509625c Mon Sep 17 00:00:00 2001 From: Konstantin Akimov Date: Mon, 5 Jan 2026 02:06:55 +0700 Subject: [PATCH 12/16] fix: add missing changes and adjust fee rate to fix bitcoin/bitcoin#25379 Otherwise this error appears: test_framework.authproxy.JSONRPCException: min relay fee not met, 25500 < 30332 (-26) --- test/functional/mempool_package_limits.py | 5 +++-- test/functional/test_framework/wallet.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/test/functional/mempool_package_limits.py b/test/functional/mempool_package_limits.py index b7db228f5571..7b0a75f6db5e 100755 --- a/test/functional/mempool_package_limits.py +++ b/test/functional/mempool_package_limits.py @@ -306,10 +306,11 @@ def test_anc_size_limits(self): parents_tx = [] target_weight = 1000 * 30 # 30KvB high_fee = Decimal("0.003") # 10 sats/vB + fee_rate=Decimal("0.01") self.log.info("Check that in-mempool and in-package ancestor size limits are calculated properly in packages") # Mempool transactions A and B for _ in range(2): - bulked_tx = self.wallet.create_self_transfer(target_weight=target_weight) + bulked_tx = self.wallet.create_self_transfer(target_weight=target_weight, fee_rate=fee_rate) self.wallet.sendrawtransaction(from_node=node, tx_hex=bulked_tx["hex"]) parent_utxos.append(bulked_tx["new_utxo"]) @@ -317,7 +318,7 @@ def test_anc_size_limits(self): pc_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=parent_utxos, fee_per_output=int(high_fee * COIN), target_weight=target_weight) # Package transaction D - pd_tx = self.wallet.create_self_transfer(utxo_to_spend=pc_tx["new_utxos"][0], target_weight=target_weight) + pd_tx = self.wallet.create_self_transfer(utxo_to_spend=pc_tx["new_utxos"][0], target_weight=target_weight, fee_rate=fee_rate) assert_equal(2, node.getmempoolinfo()["size"]) testres_too_heavy = node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]]) diff --git a/test/functional/test_framework/wallet.py b/test/functional/test_framework/wallet.py index 737b94f52413..d16f2c710fd5 100644 --- a/test/functional/test_framework/wallet.py +++ b/test/functional/test_framework/wallet.py @@ -322,8 +322,9 @@ def create_self_transfer(self, *, fee_rate=Decimal("0.003"), fee=Decimal("0"), u assert send_value > 0 # create tx - tx = self.create_self_transfer_multi(utxos_to_spend=[utxo_to_spend], locktime=locktime, sequence=sequence, amount_per_output=int(COIN * send_value)) - assert_equal(tx["tx"].get_vsize(), vsize) + tx = self.create_self_transfer_multi(utxos_to_spend=[utxo_to_spend], locktime=locktime, sequence=sequence, amount_per_output=int(COIN * send_value), target_weight=target_weight) + if not target_weight: + assert_equal(tx["tx"].get_vsize(), vsize) return {"txid": tx["txid"], "hex": tx["hex"], "tx": tx["tx"], "new_utxo": tx["new_utxos"][0]} From ed3dbc006380becd16fc576f04420b2b26639260 Mon Sep 17 00:00:00 2001 From: MacroFake Date: Thu, 4 Aug 2022 19:21:39 +0200 Subject: [PATCH 13/16] Merge bitcoin/bitcoin#25773: test: Target exact weight in MiniWallet _bulk_tx fa2537cf0a7629d81df1bc5b4ae6a22dc572647b test: Target exact weight in MiniWallet _bulk_tx (MacroFake) Pull request description: Seems better to target the exact weight than a weight that is up to more than 2000 WU larger. Also, replace a broad `-acceptnonstdtxn=1` with `-datacarriersize=100000` to document the test assumptions better. ACKs for top commit: theStack: Code-review ACK fa2537cf0a7629d81df1bc5b4ae6a22dc572647b Tree-SHA512: cf02c3082a13195b8aa730866aeaf2575ce01974ae2b0244739d8cfc12e60c66312729ed703bb3214651744166a3b560bfaa8dc302ef46ed79fc4d1fe7fcc214 --- test/functional/mempool_package_limits.py | 4 ++-- test/functional/test_framework/wallet.py | 13 ++++++++----- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/test/functional/mempool_package_limits.py b/test/functional/mempool_package_limits.py index 7b0a75f6db5e..9b1ffa9feaba 100755 --- a/test/functional/mempool_package_limits.py +++ b/test/functional/mempool_package_limits.py @@ -34,8 +34,8 @@ def run_test(self): self.test_anc_count_limits_2() self.test_anc_count_limits_bushy() - # The node will accept our (nonstandard) extra large OP_RETURN outputs - self.restart_node(0, extra_args=["-acceptnonstdtxn=1"]) + # The node will accept (nonstandard) extra large OP_RETURN outputs + self.restart_node(0, extra_args=["-datacarriersize=100000"]) self.test_anc_size_limits() self.test_desc_size_limits() diff --git a/test/functional/test_framework/wallet.py b/test/functional/test_framework/wallet.py index d16f2c710fd5..b8838aa861ae 100644 --- a/test/functional/test_framework/wallet.py +++ b/test/functional/test_framework/wallet.py @@ -30,6 +30,7 @@ from test_framework.script import ( CScript, SignatureHash, + OP_RETURN, OP_TRUE, OP_NOP, SIGHASH_ALL, @@ -97,11 +98,13 @@ def _bulk_tx(self, tx, target_weight): """Pad a transaction with extra outputs until it reaches a target weight (or higher). returns the tx """ - assert_greater_than_or_equal(target_weight, tx.get_weight()) - while tx.get_weight() < target_weight: - script_pubkey = ( b"6a4d0200" # OP_RETURN OP_PUSH2 512 bytes - + b"01" * 512 ) - tx.vout.append(CTxOut(0, script_pubkey)) + tx.vout.append(CTxOut(nValue=0, scriptPubKey=CScript([OP_RETURN, b'a']))) + dummy_vbytes = (target_weight - tx.get_weight()) + tx.vout[-1].scriptPubKey = CScript([OP_RETURN, b'a' * dummy_vbytes]) + # Lower bound should always be off by at most 3 + assert_greater_than_or_equal(tx.get_weight(), target_weight) + # Higher bound should always be off by at most 3 + 12 weight (for encoding the length) + assert_greater_than_or_equal(target_weight + 15, tx.get_weight()) def get_balance(self): return sum(u['value'] for u in self._utxos) From d0140f3b5faef59f26134d1f50e5682b9aedac67 Mon Sep 17 00:00:00 2001 From: glozow Date: Mon, 28 Nov 2022 11:50:57 +0000 Subject: [PATCH 14/16] Merge bitcoin/bitcoin#25986: test: refactor `RPCPackagesTest` to use `MiniWallet` 17cad448516a6906ff637593ab57df332fade5d2 test: refactor `RPCPackagesTest` to use `MiniWallet` (w0xlt) Pull request description: This PR refactors `RPCPackagesTest` to use `MiniWallet` and removes `create_child_with_parents`, `make_chain`, and `create_raw_chain` from `test_framework/wallet`, as requested in https://github.com/bitcoin/bitcoin/issues/25965. Close https://github.com/bitcoin/bitcoin/issues/25965. ACKs for top commit: glozow: ACK 17cad448516a6906ff637593ab57df332fade5d2 pablomartin4btc: tested ACK 17cad44; went thru all changes and recommendations from @kouloumos & @glozow; also went up to #20833 to get a bit of background of the origin and purpose of these tests. kouloumos: ACK 17cad448516a6906ff637593ab57df332fade5d2 Tree-SHA512: 9228c532afaecedd577019dbc56f8749046d66f904dd69eb23e7ca3d7806e2132d90af29be276c7635fefb37ef348ae781eb3b225cd6741b20300e6f381041c3 --- test/functional/rpc_packages.py | 230 +++++++++-------------- test/functional/test_framework/wallet.py | 76 +++----- 2 files changed, 111 insertions(+), 195 deletions(-) diff --git a/test/functional/rpc_packages.py b/test/functional/rpc_packages.py index d0e070000a1a..5e13633f52bd 100755 --- a/test/functional/rpc_packages.py +++ b/test/functional/rpc_packages.py @@ -7,28 +7,23 @@ from decimal import Decimal import random -from test_framework.address import ADDRESS_BCRT1_P2SH_OP_TRUE -from test_framework.test_framework import BitcoinTestFramework +from test_framework.blocktools import COINBASE_MATURITY from test_framework.messages import ( tx_from_hex, ) from test_framework.p2p import P2PTxInvStore -from test_framework.script import ( - CScript, - OP_TRUE, -) +from test_framework.test_framework import BitcoinTestFramework from test_framework.util import ( assert_equal, assert_fee_amount, assert_raises_rpc_error, ) from test_framework.wallet import ( - create_child_with_parents, - create_raw_chain, DEFAULT_FEE, - make_chain, + MiniWallet, ) + class RPCPackagesTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 @@ -46,52 +41,53 @@ def assert_testres_equal(self, package_hex, testres_expected): assert_equal(shuffled_testres, self.nodes[0].testmempoolaccept(shuffled_package)) def run_test(self): - self.log.info("Generate blocks to create UTXOs") node = self.nodes[0] - self.privkeys = [node.get_deterministic_priv_key().key] - self.address = node.get_deterministic_priv_key().address - self.coins = [] - # The last 100 coinbase transactions are premature - for b in self.generatetoaddress(node, 220, self.address)[:-100]: - coinbase = node.getblock(blockhash=b, verbosity=2)["tx"][0] - self.coins.append({ + + # get an UTXO that requires signature to be spent + deterministic_address = node.get_deterministic_priv_key().address + blockhash = self.generatetoaddress(node, 1, deterministic_address)[0] + coinbase = node.getblock(blockhash=blockhash, verbosity=2)["tx"][0] + coin = { "txid": coinbase["txid"], "amount": coinbase["vout"][0]["value"], "scriptPubKey": coinbase["vout"][0]["scriptPubKey"], - }) + "vout": 0, + "height": 0 + } + + self.wallet = MiniWallet(self.nodes[0]) + self.generate(self.wallet, COINBASE_MATURITY + 120) # blocks generated for inputs + self.log.info("Create some transactions") # Create some transactions that can be reused throughout the test. Never submit these to mempool. self.independent_txns_hex = [] self.independent_txns_testres = [] for _ in range(3): - coin = self.coins.pop() - rawtx = node.createrawtransaction([{"txid": coin["txid"], "vout": 0}], - {self.address : coin["amount"] - Decimal("0.0001")}) - signedtx = node.signrawtransactionwithkey(hexstring=rawtx, privkeys=self.privkeys) - assert signedtx["complete"] - testres = node.testmempoolaccept([signedtx["hex"]]) + tx_hex = self.wallet.create_self_transfer(fee_rate=Decimal("0.0001"))["hex"] + testres = self.nodes[0].testmempoolaccept([tx_hex]) assert testres[0]["allowed"] - self.independent_txns_hex.append(signedtx["hex"]) + self.independent_txns_hex.append(tx_hex) # testmempoolaccept returns a list of length one, avoid creating a 2D list self.independent_txns_testres.append(testres[0]) self.independent_txns_testres_blank = [{ "txid": res["txid"]} for res in self.independent_txns_testres] - self.test_independent() + self.test_independent(coin) self.test_chain() self.test_multiple_children() self.test_multiple_parents() self.test_conflicting() self.test_submitpackage() - def test_independent(self): + def test_independent(self, coin): self.log.info("Test multiple independent transactions in a package") node = self.nodes[0] # For independent transactions, order doesn't matter. self.assert_testres_equal(self.independent_txns_hex, self.independent_txns_testres) self.log.info("Test an otherwise valid package with an extra garbage tx appended") - garbage_tx = node.createrawtransaction([{"txid": "00" * 32, "vout": 5}], {self.address: 1}) + address = node.get_deterministic_priv_key().address + garbage_tx = node.createrawtransaction([{"txid": "00" * 32, "vout": 5}], {address: 1}) tx = tx_from_hex(garbage_tx) # Only the txid is returned because validation is incomplete for the independent txns. # Package validation is atomic: if the node cannot find a UTXO for any single tx in the package, @@ -101,9 +97,8 @@ def test_independent(self): self.assert_testres_equal(package_bad, testres_bad) self.log.info("Check testmempoolaccept tells us when some transactions completed validation successfully") - coin = self.coins.pop() tx_bad_sig_hex = node.createrawtransaction([{"txid": coin["txid"], "vout": 0}], - {self.address : coin["amount"] - Decimal("0.0001")}) + {address : coin["amount"] - Decimal("0.0001")}) tx_bad_sig = tx_from_hex(tx_bad_sig_hex) tx_bad_sig_hex = tx_bad_sig.serialize().hex() testres_bad_sig = node.testmempoolaccept(self.independent_txns_hex + [tx_bad_sig_hex]) @@ -117,23 +112,22 @@ def test_independent(self): }]) self.log.info("Check testmempoolaccept reports txns in packages that exceed max feerate") - coin = self.coins.pop() - tx_high_fee_raw = node.createrawtransaction([{"txid": coin["txid"], "vout": 0}], - {self.address : coin["amount"] - Decimal("0.999")}) - tx_high_fee_signed = node.signrawtransactionwithkey(hexstring=tx_high_fee_raw, privkeys=self.privkeys) - assert tx_high_fee_signed["complete"] - tx_high_fee = tx_from_hex(tx_high_fee_signed["hex"]) - testres_high_fee = node.testmempoolaccept([tx_high_fee_signed["hex"]]) + tx_high_fee = self.wallet.create_self_transfer(fee=Decimal("0.999")) + testres_high_fee = node.testmempoolaccept([tx_high_fee["hex"]]) assert_equal(testres_high_fee, [ - {"txid": tx_high_fee.rehash(), "allowed": False, "reject-reason": "max-fee-exceeded"} + {"txid": tx_high_fee["txid"], "allowed": False, "reject-reason": "max-fee-exceeded"} ]) - package_high_fee = [tx_high_fee_signed["hex"]] + self.independent_txns_hex + package_high_fee = [tx_high_fee["hex"]] + self.independent_txns_hex testres_package_high_fee = node.testmempoolaccept(package_high_fee) assert_equal(testres_package_high_fee, testres_high_fee + self.independent_txns_testres_blank) def test_chain(self): node = self.nodes[0] - (chain_hex, chain_txns) = create_raw_chain(node, self.coins.pop(), self.address, self.privkeys) + + chain = self.wallet.create_self_transfer_chain(chain_length=25) + chain_hex = chain["chain_hex"] + chain_txns = chain["chain_txns"] + self.log.info("Check that testmempoolaccept requires packages to be sorted by dependency") assert_equal(node.testmempoolaccept(rawtxs=chain_hex[::-1]), [{"txid": tx.rehash(), "package-error": "package-not-sorted"} for tx in chain_txns[::-1]]) @@ -155,77 +149,57 @@ def test_chain(self): def test_multiple_children(self): node = self.nodes[0] - self.log.info("Testmempoolaccept a package in which a transaction has two children within the package") - first_coin = self.coins.pop() - value = (first_coin["amount"] - Decimal("0.0002")) / 2 # Deduct reasonable fee and make 2 outputs - inputs = [{"txid": first_coin["txid"], "vout": 0}] - outputs = [{self.address : value}, {ADDRESS_BCRT1_P2SH_OP_TRUE : value}] - rawtx = node.createrawtransaction(inputs, outputs) - parent_signed = node.signrawtransactionwithkey(hexstring=rawtx, privkeys=self.privkeys) - parent_tx = tx_from_hex(parent_signed["hex"]) - assert parent_signed["complete"] - parent_txid = parent_tx.rehash() - assert node.testmempoolaccept([parent_signed["hex"]])[0]["allowed"] - - parent_locking_script_a = parent_tx.vout[0].scriptPubKey.hex() - child_value = value - Decimal("0.0001") + parent_tx = self.wallet.create_self_transfer_multi(num_outputs=2) + assert node.testmempoolaccept([parent_tx["hex"]])[0]["allowed"] # Child A - (_, tx_child_a_hex, _, _) = make_chain(node, self.address, self.privkeys, parent_txid, child_value, 0, parent_locking_script_a) - assert not node.testmempoolaccept([tx_child_a_hex])[0]["allowed"] + child_a_tx = self.wallet.create_self_transfer(utxo_to_spend=parent_tx["new_utxos"][0]) + assert not node.testmempoolaccept([child_a_tx["hex"]])[0]["allowed"] # Child B - rawtx_b = node.createrawtransaction([{"txid": parent_txid, "vout": 1}], {self.address : child_value}) - tx_child_b = tx_from_hex(rawtx_b) - tx_child_b.vin[0].scriptSig = CScript([CScript([OP_TRUE])]) - tx_child_b_hex = tx_child_b.serialize().hex() - assert not node.testmempoolaccept([tx_child_b_hex])[0]["allowed"] + child_b_tx = self.wallet.create_self_transfer(utxo_to_spend=parent_tx["new_utxos"][1]) + assert not node.testmempoolaccept([child_b_tx["hex"]])[0]["allowed"] self.log.info("Testmempoolaccept with entire package, should work with children in either order") - testres_multiple_ab = node.testmempoolaccept(rawtxs=[parent_signed["hex"], tx_child_a_hex, tx_child_b_hex]) - testres_multiple_ba = node.testmempoolaccept(rawtxs=[parent_signed["hex"], tx_child_b_hex, tx_child_a_hex]) + testres_multiple_ab = node.testmempoolaccept(rawtxs=[parent_tx["hex"], child_a_tx["hex"], child_b_tx["hex"]]) + testres_multiple_ba = node.testmempoolaccept(rawtxs=[parent_tx["hex"], child_b_tx["hex"], child_a_tx["hex"]]) assert all([testres["allowed"] for testres in testres_multiple_ab + testres_multiple_ba]) testres_single = [] # Test accept and then submit each one individually, which should be identical to package testaccept - for rawtx in [parent_signed["hex"], tx_child_a_hex, tx_child_b_hex]: + for rawtx in [parent_tx["hex"], child_a_tx["hex"], child_b_tx["hex"]]: testres = node.testmempoolaccept([rawtx]) testres_single.append(testres[0]) # Submit the transaction now so its child should have no problem validating node.sendrawtransaction(rawtx) assert_equal(testres_single, testres_multiple_ab) - def test_multiple_parents(self): node = self.nodes[0] - self.log.info("Testmempoolaccept a package in which a transaction has multiple parents within the package") + for num_parents in [2, 10, 24]: # Test a package with num_parents parents and 1 child transaction. + parent_coins = [] package_hex = [] - parents_tx = [] - values = [] - parent_locking_scripts = [] + for _ in range(num_parents): - parent_coin = self.coins.pop() - value = parent_coin["amount"] - (tx, txhex, value, parent_locking_script) = make_chain(node, self.address, self.privkeys, parent_coin["txid"], value) - package_hex.append(txhex) - parents_tx.append(tx) - values.append(value) - parent_locking_scripts.append(parent_locking_script) - child_hex = create_child_with_parents(node, self.address, self.privkeys, parents_tx, values, parent_locking_scripts) - # Package accept should work with the parents in any order (as long as parents come before child) + # Package accept should work with the parents in any order (as long as parents come before child) + parent_tx = self.wallet.create_self_transfer() + parent_coins.append(parent_tx["new_utxo"]) + package_hex.append(parent_tx["hex"]) + + child_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=parent_coins, fee_per_output=2000) for _ in range(10): random.shuffle(package_hex) - testres_multiple = node.testmempoolaccept(rawtxs=package_hex + [child_hex]) + testres_multiple = node.testmempoolaccept(rawtxs=package_hex + [child_tx['hex']]) assert all([testres["allowed"] for testres in testres_multiple]) testres_single = [] # Test accept and then submit each one individually, which should be identical to package testaccept - for rawtx in package_hex + [child_hex]: + for rawtx in package_hex + [child_tx["hex"]]: testres_single.append(node.testmempoolaccept([rawtx])[0]) # Submit the transaction now so its child should have no problem validating node.sendrawtransaction(rawtx) @@ -233,40 +207,30 @@ def test_multiple_parents(self): def test_conflicting(self): node = self.nodes[0] - prevtx = self.coins.pop() - inputs = [{"txid": prevtx["txid"], "vout": 0}] - output1 = {node.get_deterministic_priv_key().address: 500 - 0.00125} - output2 = {ADDRESS_BCRT1_P2SH_OP_TRUE: 500 - 0.00125} + coin = self.wallet.get_utxo() # tx1 and tx2 share the same inputs - rawtx1 = node.createrawtransaction(inputs, output1) - rawtx2 = node.createrawtransaction(inputs, output2) - signedtx1 = node.signrawtransactionwithkey(hexstring=rawtx1, privkeys=self.privkeys) - signedtx2 = node.signrawtransactionwithkey(hexstring=rawtx2, privkeys=self.privkeys) - tx1 = tx_from_hex(signedtx1["hex"]) - tx2 = tx_from_hex(signedtx2["hex"]) - assert signedtx1["complete"] - assert signedtx2["complete"] + tx1 = self.wallet.create_self_transfer(utxo_to_spend=coin) + tx2 = self.wallet.create_self_transfer(utxo_to_spend=coin) # Ensure tx1 and tx2 are valid by themselves - assert node.testmempoolaccept([signedtx1["hex"]])[0]["allowed"] - assert node.testmempoolaccept([signedtx2["hex"]])[0]["allowed"] + assert node.testmempoolaccept([tx1["hex"]])[0]["allowed"] + assert node.testmempoolaccept([tx2["hex"]])[0]["allowed"] self.log.info("Test duplicate transactions in the same package") - testres = node.testmempoolaccept([signedtx1["hex"], signedtx1["hex"]]) + testres = node.testmempoolaccept([tx1["hex"], tx1["hex"]]) assert_equal(testres, [ - {"txid": tx1.rehash(), "package-error": "conflict-in-package"}, - {"txid": tx1.rehash(), "package-error": "conflict-in-package"} + {"txid": tx1["txid"], "package-error": "conflict-in-package"}, + {"txid": tx1["txid"], "package-error": "conflict-in-package"} ]) self.log.info("Test conflicting transactions in the same package") - testres = node.testmempoolaccept([signedtx1["hex"], signedtx2["hex"]]) + testres = node.testmempoolaccept([tx1["hex"], tx2["hex"]]) assert_equal(testres, [ - {"txid": tx1.rehash(), "package-error": "conflict-in-package"}, - {"txid": tx2.rehash(), "package-error": "conflict-in-package"} + {"txid": tx1["txid"], "package-error": "conflict-in-package"}, + {"txid": tx2["txid"], "package-error": "conflict-in-package"} ]) - def assert_equal_package_results(self, node, testmempoolaccept_result, submitpackage_result): """Assert that a successful submitpackage result is consistent with testmempoolaccept results and getmempoolentry info. Note that the result structs are different and, due to @@ -288,36 +252,26 @@ def assert_equal_package_results(self, node, testmempoolaccept_result, submitpac def test_submit_child_with_parents(self, num_parents, partial_submit): node = self.nodes[0] peer = node.add_p2p_connection(P2PTxInvStore()) - # Test a package with num_parents parents and 1 child transaction. - package_hex = [] + package_txns = [] - values = [] - scripts = [] for _ in range(num_parents): - parent_coin = self.coins.pop() - value = parent_coin["amount"] - (tx, txhex, value, spk) = make_chain(node, self.address, self.privkeys, parent_coin["txid"], value) - package_hex.append(txhex) - package_txns.append(tx) - values.append(value) - scripts.append(spk) + parent_tx = self.wallet.create_self_transfer(fee=DEFAULT_FEE) + package_txns.append(parent_tx) if partial_submit and random.choice([True, False]): - node.sendrawtransaction(txhex) - child_hex = create_child_with_parents(node, self.address, self.privkeys, package_txns, values, scripts) - package_hex.append(child_hex) - package_txns.append(tx_from_hex(child_hex)) + node.sendrawtransaction(parent_tx["hex"]) + child_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=[tx["new_utxo"] for tx in package_txns], fee_per_output=10000) #DEFAULT_FEE + package_txns.append(child_tx) - testmempoolaccept_result = node.testmempoolaccept(rawtxs=package_hex) - submitpackage_result = node.submitpackage(package=package_hex) + testmempoolaccept_result = node.testmempoolaccept(rawtxs=[tx["hex"] for tx in package_txns]) + submitpackage_result = node.submitpackage(package=[tx["hex"] for tx in package_txns]) # Check that each result is present, with the correct size and fees - for i in range(num_parents + 1): - tx = package_txns[i] - wtxid = tx.hash - assert wtxid in submitpackage_result["tx-results"] - tx_result = submitpackage_result["tx-results"][wtxid] + for package_txn in package_txns: + tx = package_txn["tx"] + assert tx.hash in submitpackage_result["tx-results"] + tx_result = submitpackage_result["tx-results"][tx.hash] assert_equal(tx_result, { - "txid": tx.rehash(), + "txid": package_txn["txid"], "size": tx.get_vsize(), "fees": { "base": DEFAULT_FEE, @@ -335,30 +289,25 @@ def test_submit_child_with_parents(self, num_parents, partial_submit): # The node should announce each transaction. No guarantees for propagation. self.bump_mocktime(30) - peer.wait_for_broadcast([tx.hash for tx in package_txns]) + peer.wait_for_broadcast([tx["tx"].hash for tx in package_txns]) self.generate(node, 1) - def test_submit_cpfp(self): node = self.nodes[0] peer = node.add_p2p_connection(P2PTxInvStore()) - # 2 parent 1 child CPFP. First parent pays high fees, second parent pays 0 fees and is - # fee-bumped by the child. - coin_rich = self.coins.pop() - coin_poor = self.coins.pop() - tx_rich, hex_rich, value_rich, spk_rich = make_chain(node, self.address, self.privkeys, coin_rich["txid"], coin_rich["amount"]) - tx_poor, hex_poor, value_poor, spk_poor = make_chain(node, self.address, self.privkeys, coin_poor["txid"], coin_poor["amount"], fee=0) + tx_poor = self.wallet.create_self_transfer(fee=0, fee_rate=0) + tx_rich = self.wallet.create_self_transfer(fee=DEFAULT_FEE) package_txns = [tx_rich, tx_poor] - hex_child = create_child_with_parents(node, self.address, self.privkeys, package_txns, [value_rich, value_poor], [spk_rich, spk_poor]) - tx_child = tx_from_hex(hex_child) + coins = [tx["new_utxo"] for tx in package_txns] + tx_child = self.wallet.create_self_transfer_multi(utxos_to_spend=coins, fee_per_output=10000) #DEFAULT_FEE package_txns.append(tx_child) - submitpackage_result = node.submitpackage([hex_rich, hex_poor, hex_child]) + submitpackage_result = node.submitpackage([tx["hex"] for tx in package_txns]) - rich_parent_result = submitpackage_result["tx-results"][tx_rich.hash] - poor_parent_result = submitpackage_result["tx-results"][tx_poor.hash] - child_result = submitpackage_result["tx-results"][tx_child.hash] + rich_parent_result = submitpackage_result["tx-results"][tx_rich["txid"]] + poor_parent_result = submitpackage_result["tx-results"][tx_poor["txid"]] + child_result = submitpackage_result["tx-results"][tx_child["tx"].hash] assert_equal(rich_parent_result["fees"]["base"], DEFAULT_FEE) assert_equal(poor_parent_result["fees"]["base"], 0) assert_equal(child_result["fees"]["base"], DEFAULT_FEE) @@ -370,10 +319,9 @@ def test_submit_cpfp(self): # The node will broadcast each transaction, still abiding by its peer's fee filter self.bump_mocktime(30) - peer.wait_for_broadcast([tx.hash for tx in package_txns]) + peer.wait_for_broadcast([tx["tx"].hash for tx in package_txns]) self.generate(node, 1) - def test_submitpackage(self): node = self.nodes[0] @@ -387,7 +335,7 @@ def test_submitpackage(self): self.log.info("Submitpackage only allows packages of 1 child with its parents") # Chain of 3 transactions has too many generations - chain_hex, _ = create_raw_chain(node, self.coins.pop(), self.address, self.privkeys, 3) + chain_hex = self.wallet.create_self_transfer_chain(chain_length=25)["chain_hex"] assert_raises_rpc_error(-25, "not-child-with-parents", node.submitpackage, chain_hex) diff --git a/test/functional/test_framework/wallet.py b/test/functional/test_framework/wallet.py index b8838aa861ae..fff2c5180422 100644 --- a/test/functional/test_framework/wallet.py +++ b/test/functional/test_framework/wallet.py @@ -25,7 +25,6 @@ CTransaction, CTxIn, CTxOut, - tx_from_hex, ) from test_framework.script import ( CScript, @@ -336,6 +335,28 @@ def sendrawtransaction(self, *, from_node, tx_hex, maxfeerate=0, **kwargs): self.scan_tx(from_node.decoderawtransaction(tx_hex)) return txid + def create_self_transfer_chain(self, *, chain_length): + """ + Create a "chain" of chain_length transactions. The nth transaction in + the chain is a child of the n-1th transaction and parent of the n+1th transaction. + + Returns a dic {"chain_hex": chain_hex, "chain_txns" : chain_txns} + + "chain_hex" is a list representing the chain's transactions in hexadecimal. + "chain_txns" is a list representing the chain's transactions in the CTransaction object. + """ + chaintip_utxo = self.get_utxo() + chain_hex = [] + chain_txns = [] + + for _ in range(chain_length): + tx = self.create_self_transfer(utxo_to_spend=chaintip_utxo) + chaintip_utxo = tx["new_utxo"] + chain_hex.append(tx["hex"]) + chain_txns.append(tx["tx"]) + + return {"chain_hex": chain_hex, "chain_txns" : chain_txns} + def send_self_transfer_chain(self, *, from_node, chain_length, utxo_to_spend=None): """Create and send a "chain" of chain_length transactions. The nth transaction in the chain is a child of the n-1th transaction and parent of the n+1th transaction. @@ -375,56 +396,3 @@ def address_to_scriptpubkey(address): # TODO: also support other address formats else: assert False - - -def make_chain(node, address, privkeys, parent_txid, parent_value, n=0, parent_locking_script=None, fee=DEFAULT_FEE): - """Build a transaction that spends parent_txid.vout[n] and produces one output with - amount = parent_value with a fee deducted. - Return tuple (CTransaction object, raw hex, nValue, scriptPubKey of the output created). - """ - inputs = [{"txid": parent_txid, "vout": n}] - my_value = parent_value - fee - outputs = {address : my_value} - rawtx = node.createrawtransaction(inputs, outputs) - prevtxs = [{ - "txid": parent_txid, - "vout": n, - "scriptPubKey": parent_locking_script, - "amount": parent_value, - }] if parent_locking_script else None - signedtx = node.signrawtransactionwithkey(hexstring=rawtx, privkeys=privkeys, prevtxs=prevtxs) - assert signedtx["complete"] - tx = tx_from_hex(signedtx["hex"]) - return (tx, signedtx["hex"], my_value, tx.vout[0].scriptPubKey.hex()) - -def create_child_with_parents(node, address, privkeys, parents_tx, values, locking_scripts, fee=DEFAULT_FEE): - """Creates a transaction that spends the first output of each parent in parents_tx.""" - num_parents = len(parents_tx) - total_value = sum(values) - inputs = [{"txid": tx.rehash(), "vout": 0} for tx in parents_tx] - outputs = {address : total_value - fee} - rawtx_child = node.createrawtransaction(inputs, outputs) - prevtxs = [] - for i in range(num_parents): - prevtxs.append({"txid": parents_tx[i].rehash(), "vout": 0, "scriptPubKey": locking_scripts[i], "amount": values[i]}) - signedtx_child = node.signrawtransactionwithkey(hexstring=rawtx_child, privkeys=privkeys, prevtxs=prevtxs) - assert signedtx_child["complete"] - return signedtx_child["hex"] - -def create_raw_chain(node, first_coin, address, privkeys, chain_length=25): - """Helper function: create a "chain" of chain_length transactions. The nth transaction in the - chain is a child of the n-1th transaction and parent of the n+1th transaction. - """ - parent_locking_script = None - txid = first_coin["txid"] - chain_hex = [] - chain_txns = [] - value = first_coin["amount"] - - for _ in range(chain_length): - (tx, txhex, value, parent_locking_script) = make_chain(node, address, privkeys, txid, value, 0, parent_locking_script) - txid = tx.rehash() - chain_hex.append(txhex) - chain_txns.append(tx) - - return (chain_hex, chain_txns) From bffd274f782464993f3fead3b87e014be28bff9b Mon Sep 17 00:00:00 2001 From: fanquake Date: Tue, 17 Jan 2023 16:28:36 +0000 Subject: [PATCH 15/16] Merge bitcoin/bitcoin#26625: test: Run mempool_packages.py with MiniWallet fa6b4021142154f52fc35360e409907360808801 test: Run mempool_packages.py with MiniWallet (MarcoFalke) fa448c27d2cdd70698a4188c7478996ff8804556 test: Return fee from MiniWallet (MarcoFalke) faec09f240213e8540e8559a3b4396ee93950cb4 test: Return chain of MiniWallet txs from MiniWallet chain method (MarcoFalke) faa12d4ccd2ed14a4b892816b4771d7ff8d0d8a0 test: Refactor MiniWallet sign_tx (MarcoFalke) fa2d82103fcd3b9084edd2268443f61ca5af969c test: Return wtxid from create_self_transfer_multi (MarcoFalke) Pull request description: This allows to run the test even when no wallet is compiled in. Also, it is a lot nicer to read now. ACKs for top commit: glozow: reACK fa6b402 Tree-SHA512: de0338068fd51db01d64ce270f94fd2982a63a6de597325cd1e1f11127e9075bd4aeacace0ed76d09a2db8b962b27237cf11edb4c1fe1a01134d397f8a11bd05 --- test/functional/mempool_package_limits.py | 12 +-- test/functional/mempool_packages.py | 116 ++++++++-------------- test/functional/rpc_packages.py | 6 +- test/functional/test_framework/util.py | 22 ---- test/functional/test_framework/wallet.py | 47 ++++----- 5 files changed, 72 insertions(+), 131 deletions(-) diff --git a/test/functional/mempool_package_limits.py b/test/functional/mempool_package_limits.py index 9b1ffa9feaba..480eae0bfdf6 100755 --- a/test/functional/mempool_package_limits.py +++ b/test/functional/mempool_package_limits.py @@ -44,7 +44,7 @@ def test_chain_limits_helper(self, mempool_count, package_count): assert_equal(0, node.getmempoolinfo()["size"]) chain_hex = [] - chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=mempool_count) + chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=mempool_count)[-1]["new_utxo"] # in-package transactions for _ in range(package_count): tx = self.wallet.create_self_transfer(utxo_to_spend=chaintip_utxo) @@ -99,13 +99,13 @@ def test_desc_count_limits(self): package_hex = [] # Chain A (M2a... M12a) - chain_a_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=11, utxo_to_spend=m1_utxos[0]) + chain_a_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=11, utxo_to_spend=m1_utxos[0])[-1]["new_utxo"] # Pa pa_hex = self.wallet.create_self_transfer(utxo_to_spend=chain_a_tip_utxo)["hex"] package_hex.append(pa_hex) # Chain B (M2b... M13b) - chain_b_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12, utxo_to_spend=m1_utxos[1]) + chain_b_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12, utxo_to_spend=m1_utxos[1])[-1]["new_utxo"] # Pb pb_hex = self.wallet.create_self_transfer(utxo_to_spend=chain_b_tip_utxo)["hex"] package_hex.append(pb_hex) @@ -144,7 +144,7 @@ def test_desc_count_limits_2(self): m1_utxos = self.wallet.send_self_transfer_multi(from_node=node, num_outputs=2)['new_utxos'] # Chain M2...M24 - self.wallet.send_self_transfer_chain(from_node=node, chain_length=23, utxo_to_spend=m1_utxos[0]) + self.wallet.send_self_transfer_chain(from_node=node, chain_length=23, utxo_to_spend=m1_utxos[0])[-1]["new_utxo"] # P1 p1_tx = self.wallet.create_self_transfer(utxo_to_spend=m1_utxos[1]) @@ -190,7 +190,7 @@ def test_anc_count_limits(self): # Two chains of 13 transactions each for _ in range(2): - chain_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12) + chain_tip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12)[-1]["new_utxo"] # Save the 13th transaction for the package tx = self.wallet.create_self_transfer(utxo_to_spend=chain_tip_utxo) package_hex.append(tx["hex"]) @@ -233,7 +233,7 @@ def test_anc_count_limits_2(self): self.log.info("Check that in-mempool and in-package ancestors are calculated properly in packages") # Two chains of 12 transactions each for _ in range(2): - chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12) + chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=12)[-1]["new_utxo"] # last 2 transactions will be the parents of Pc pc_parent_utxos.append(chaintip_utxo) diff --git a/test/functional/mempool_packages.py b/test/functional/mempool_packages.py index b922a7678bf4..f54ed9eaadcb 100755 --- a/test/functional/mempool_packages.py +++ b/test/functional/mempool_packages.py @@ -6,7 +6,6 @@ from decimal import Decimal -from test_framework.blocktools import COINBASE_MATURITY from test_framework.messages import ( COIN, DEFAULT_ANCESTOR_LIMIT, @@ -17,9 +16,8 @@ from test_framework.util import ( assert_equal, assert_raises_rpc_error, - chain_transaction, ) - +from test_framework.wallet import MiniWallet # custom limits for node1 CUSTOM_ANCESTOR_LIMIT = 5 @@ -42,43 +40,37 @@ def set_test_params(self): ], ] - def skip_test_if_missing_module(self): - self.skip_if_no_wallet() - def run_test(self): - # Mine some blocks and have them mature. + self.wallet = MiniWallet(self.nodes[0]) + self.wallet.rescan_utxos() + + if self.is_specified_wallet_compiled(): + self.nodes[0].createwallet("watch_wallet", disable_private_keys=True) + self.nodes[0].importaddress(self.wallet.get_address()) + peer_inv_store = self.nodes[0].add_p2p_connection(P2PTxInvStore()) # keep track of invs - self.generate(self.nodes[0], COINBASE_MATURITY + 1) - utxo = self.nodes[0].listunspent(10) - txid = utxo[0]['txid'] - vout = utxo[0]['vout'] - value = utxo[0]['amount'] - assert 'ancestorcount' not in utxo[0] - assert 'ancestorsize' not in utxo[0] - assert 'ancestorfees' not in utxo[0] - - fee = Decimal("0.0001") + # DEFAULT_ANCESTOR_LIMIT transactions off a confirmed tx should be fine - chain = [] + chain = self.wallet.create_self_transfer_chain(chain_length=DEFAULT_ANCESTOR_LIMIT) + witness_chain = [t["txid"] for t in chain] ancestor_vsize = 0 ancestor_fees = Decimal(0) - for i in range(DEFAULT_ANCESTOR_LIMIT): - (txid, sent_value) = chain_transaction(self.nodes[0], [txid], [0], value, fee, 1) - value = sent_value - chain.append(txid) + for i, t in enumerate(chain): + ancestor_vsize += t["tx"].get_vsize() + ancestor_fees += t["fee"] + self.wallet.sendrawtransaction(from_node=self.nodes[0], tx_hex=t["hex"]) # Check that listunspent ancestor{count, size, fees} yield the correct results - wallet_unspent = self.nodes[0].listunspent(minconf=0) - this_unspent = next(utxo_info for utxo_info in wallet_unspent if utxo_info['txid'] == txid) - assert_equal(this_unspent['ancestorcount'], i + 1) - ancestor_vsize += self.nodes[0].getrawtransaction(txid=txid, verbose=True)['size'] - assert_equal(this_unspent['ancestorsize'], ancestor_vsize) - ancestor_fees -= self.nodes[0].gettransaction(txid=txid)['fee'] - assert_equal(this_unspent['ancestorfees'], ancestor_fees * COIN) + if self.is_specified_wallet_compiled(): + wallet_unspent = self.nodes[0].listunspent(minconf=0) + this_unspent = next(utxo_info for utxo_info in wallet_unspent if utxo_info["txid"] == t["txid"]) + assert_equal(this_unspent['ancestorcount'], i + 1) + assert_equal(this_unspent['ancestorsize'], ancestor_vsize) + assert_equal(this_unspent['ancestorfees'], ancestor_fees * COIN) # Wait until mempool transactions have passed initial broadcast (sent inv and received getdata) # Otherwise, getrawmempool may be inconsistent with getmempoolentry if unbroadcast changes in between - peer_inv_store.wait_for_broadcast(chain) + peer_inv_store.wait_for_broadcast(witness_chain) # Check mempool has DEFAULT_ANCESTOR_LIMIT transactions in it, and descendant and ancestor # count and fees should look correct @@ -92,15 +84,20 @@ def run_test(self): ancestor_count = DEFAULT_ANCESTOR_LIMIT assert_equal(ancestor_fees, sum([mempool[tx]['fees']['base'] for tx in mempool])) + # Adding one more transaction on to the chain should fail. + next_hop = self.wallet.create_self_transfer(utxo_to_spend=chain[-1]["new_utxo"])["hex"] + assert_raises_rpc_error(-26, "too-long-mempool-chain", lambda: self.nodes[0].sendrawtransaction(next_hop)) + descendants = [] - ancestors = list(chain) + ancestors = [t["txid"] for t in chain] + chain = [t["txid"] for t in chain] for x in reversed(chain): # Check that getmempoolentry is consistent with getrawmempool entry = self.nodes[0].getmempoolentry(x) assert_equal(entry, mempool[x]) # Check that gettxspendingprevout is consistent with getrawmempool - witnesstx = self.nodes[0].gettransaction(txid=x, verbose=True)['decoded'] + witnesstx = self.nodes[0].getrawtransaction(txid=x, verbose=True) for tx_in in witnesstx["vin"]: spending_result = self.nodes[0].gettxspendingprevout([ {'txid' : tx_in["txid"], 'vout' : tx_in["vout"]} ]) assert_equal(spending_result, [ {'txid' : tx_in["txid"], 'vout' : tx_in["vout"], 'spendingtxid' : x} ]) @@ -186,9 +183,6 @@ def run_test(self): descendant_fees += entry['fees']['base'] assert_equal(entry['fees']['descendant'], descendant_fees + Decimal('0.00001')) - # Adding one more transaction on to the chain should fail. - assert_raises_rpc_error(-26, "too-long-mempool-chain", chain_transaction, self.nodes[0], [txid], [vout], value, fee, 1) - # Check that prioritising a tx before it's added to the mempool works # First clear the mempool by mining a block. self.generate(self.nodes[0], 1) @@ -225,28 +219,23 @@ def run_test(self): assert_equal(entry1['depends'], entry0['depends']) # Now test descendant chain limits - txid = utxo[1]['txid'] - value = utxo[1]['amount'] - vout = utxo[1]['vout'] - transaction_package = [] tx_children = [] # First create one parent tx with 10 children - (txid, sent_value) = chain_transaction(self.nodes[0], [txid], [vout], value, fee, 10) - parent_transaction = txid - for i in range(10): - transaction_package.append({'txid': txid, 'vout': i, 'amount': sent_value}) + tx_with_children = self.wallet.send_self_transfer_multi(from_node=self.nodes[0], num_outputs=10) + parent_transaction = tx_with_children["txid"] + transaction_package = tx_with_children["new_utxos"] # Sign and send up to MAX_DESCENDANT transactions chained off the parent tx chain = [] # save sent txs for the purpose of checking node1's mempool later (see below) for _ in range(DEFAULT_DESCENDANT_LIMIT - 1): utxo = transaction_package.pop(0) - (txid, sent_value) = chain_transaction(self.nodes[0], [utxo['txid']], [utxo['vout']], utxo['amount'], fee, 10) + new_tx = self.wallet.send_self_transfer_multi(from_node=self.nodes[0], num_outputs=10, utxos_to_spend=[utxo]) + txid = new_tx["txid"] chain.append(txid) if utxo['txid'] is parent_transaction: tx_children.append(txid) - for j in range(10): - transaction_package.append({'txid': txid, 'vout': j, 'amount': sent_value}) + transaction_package.extend(new_tx["new_utxos"]) mempool = self.nodes[0].getrawmempool(True) assert_equal(mempool[parent_transaction]['descendantcount'], DEFAULT_DESCENDANT_LIMIT) @@ -256,8 +245,8 @@ def run_test(self): assert_equal(mempool[child]['depends'], [parent_transaction]) # Sending one more chained transaction will fail - utxo = transaction_package.pop(0) - assert_raises_rpc_error(-26, "too-long-mempool-chain", chain_transaction, self.nodes[0], [utxo['txid']], [utxo['vout']], utxo['amount'], fee, 10) + next_hop = self.wallet.create_self_transfer(utxo_to_spend=transaction_package.pop(0))["hex"] + assert_raises_rpc_error(-26, "too-long-mempool-chain", lambda: self.nodes[0].sendrawtransaction(next_hop)) # Check that node1's mempool is as expected, containing: # - txs from previous ancestor test (-> custom ancestor limit) @@ -301,42 +290,19 @@ def run_test(self): # last block. # Create tx0 with 2 outputs - utxo = self.nodes[0].listunspent() - txid = utxo[0]['txid'] - value = utxo[0]['amount'] - vout = utxo[0]['vout'] - - send_value = (value - fee) / 2 - inputs = [ {'txid' : txid, 'vout' : vout} ] - outputs = {} - for _ in range(2): - outputs[self.nodes[0].getnewaddress()] = send_value - rawtx = self.nodes[0].createrawtransaction(inputs, outputs) - signedtx = self.nodes[0].signrawtransactionwithwallet(rawtx) - txid = self.nodes[0].sendrawtransaction(signedtx['hex']) - tx0_id = txid - value = send_value + tx0 = self.wallet.send_self_transfer_multi(from_node=self.nodes[0], num_outputs=2) # Create tx1 - tx1_id, _ = chain_transaction(self.nodes[0], [tx0_id], [0], value, fee, 1) + tx1 = self.wallet.send_self_transfer(from_node=self.nodes[0], utxo_to_spend=tx0["new_utxos"][0]) # Create tx2-7 - vout = 1 - txid = tx0_id - for _ in range(6): - (txid, sent_value) = chain_transaction(self.nodes[0], [txid], [vout], value, fee, 1) - vout = 0 - value = sent_value + tx7 = self.wallet.send_self_transfer_chain(from_node=self.nodes[0], utxo_to_spend=tx0["new_utxos"][1], chain_length=6)[-1] # Mine these in a block self.generate(self.nodes[0], 1) # Now generate tx8, with a big fee - inputs = [ {'txid' : tx1_id, 'vout': 0}, {'txid' : txid, 'vout': 0} ] - outputs = { self.nodes[0].getnewaddress() : send_value + value - 4*fee } - rawtx = self.nodes[0].createrawtransaction(inputs, outputs) - signedtx = self.nodes[0].signrawtransactionwithwallet(rawtx) - txid = self.nodes[0].sendrawtransaction(signedtx['hex']) + self.wallet.send_self_transfer_multi(from_node=self.nodes[0], utxos_to_spend=[tx1["new_utxo"], tx7["new_utxo"]], fee_per_output=40000) self.sync_mempools() # Now try to disconnect the tip on each node... diff --git a/test/functional/rpc_packages.py b/test/functional/rpc_packages.py index 5e13633f52bd..4d38a88e3774 100755 --- a/test/functional/rpc_packages.py +++ b/test/functional/rpc_packages.py @@ -125,8 +125,8 @@ def test_chain(self): node = self.nodes[0] chain = self.wallet.create_self_transfer_chain(chain_length=25) - chain_hex = chain["chain_hex"] - chain_txns = chain["chain_txns"] + chain_hex = [t["hex"] for t in chain] + chain_txns = [t["tx"] for t in chain] self.log.info("Check that testmempoolaccept requires packages to be sorted by dependency") assert_equal(node.testmempoolaccept(rawtxs=chain_hex[::-1]), @@ -335,7 +335,7 @@ def test_submitpackage(self): self.log.info("Submitpackage only allows packages of 1 child with its parents") # Chain of 3 transactions has too many generations - chain_hex = self.wallet.create_self_transfer_chain(chain_length=25)["chain_hex"] + chain_hex = [t["hex"] for t in self.wallet.create_self_transfer_chain(chain_length=25)] assert_raises_rpc_error(-25, "not-child-with-parents", node.submitpackage, chain_hex) diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py index 71c0b875a0d7..35c3ef38ffbf 100644 --- a/test/functional/test_framework/util.py +++ b/test/functional/test_framework/util.py @@ -591,28 +591,6 @@ def find_output(node, txid, amount, *, blockhash=None): raise RuntimeError("find_output txid %s : %s not found" % (txid, str(amount))) -def chain_transaction(node, parent_txids, vouts, value, fee, num_outputs): - """Build and send a transaction that spends the given inputs (specified - by lists of parent_txid:vout each), with the desired total value and fee, - equally divided up to the desired number of outputs. - - Returns a tuple with the txid and the amount sent per output. - """ - send_value = satoshi_round((value - fee)/num_outputs) - inputs = [] - for (txid, vout) in zip(parent_txids, vouts): - inputs.append({'txid' : txid, 'vout' : vout}) - outputs = {} - for _ in range(num_outputs): - outputs[node.getnewaddress()] = send_value - rawtx = node.createrawtransaction(inputs, outputs) - signedtx = node.signrawtransactionwithwallet(rawtx) - txid = node.sendrawtransaction(signedtx['hex']) - fulltx = node.getrawtransaction(txid, 1) - assert len(fulltx['vout']) == num_outputs # make sure we didn't generate a change output - return (txid, send_value) - - # Create large OP_RETURN txouts that can be appended to a transaction # to make it large (helper for constructing large transactions). The # total serialized size of the txouts is about 66k vbytes. diff --git a/test/functional/test_framework/wallet.py b/test/functional/test_framework/wallet.py index fff2c5180422..b129d188fc86 100644 --- a/test/functional/test_framework/wallet.py +++ b/test/functional/test_framework/wallet.py @@ -158,11 +158,11 @@ def sign_tx(self, tx, fixed_length=True): tx.vin[0].scriptSig = CScript([der_sig + bytes(bytearray([SIGHASH_ALL]))]) tx.rehash() elif self._mode == MiniWalletMode.RAW_OP_TRUE: - for i in range(len(tx.vin)): - tx.vin[i].scriptSig = CScript([OP_NOP] * 24) # pad to identical size + for i in tx.vin: + i.scriptSig = CScript([OP_NOP] * 24) # pad to identical size elif self._mode == MiniWalletMode.ADDRESS_OP_TRUE: - for i in range(len(tx.vin)): - tx.vin[i].scriptSig = CScript([CScript([OP_TRUE])]) + for i in tx.vin: + i.scriptSig = CScript([CScript([OP_TRUE])]) else: assert False @@ -281,10 +281,13 @@ def create_self_transfer_multi( inputs_value_total = sum([int(COIN * utxo['value']) for utxo in utxos_to_spend]) outputs_value_total = inputs_value_total - fee_per_output * num_outputs amount_per_output = amount_per_output or (outputs_value_total // num_outputs) + assert amount_per_output > 0 + outputs_value_total = amount_per_output * num_outputs + fee = Decimal(inputs_value_total - outputs_value_total) / COIN # create tx tx = CTransaction() - tx.vin = [CTxIn(COutPoint(int(utxo_to_spend['txid'], 16), utxo_to_spend['vout']), nSequence=seq) for utxo_to_spend,seq in zip(utxos_to_spend, sequence)] + tx.vin = [CTxIn(COutPoint(int(utxo_to_spend['txid'], 16), utxo_to_spend['vout']), nSequence=seq) for utxo_to_spend, seq in zip(utxos_to_spend, sequence)] tx.vout = [CTxOut(amount_per_output, bytearray(self._scriptPubKey)) for _ in range(num_outputs)] tx.nLockTime = locktime @@ -303,6 +306,7 @@ def create_self_transfer_multi( coinbase=False, confirmations=0, ) for i in range(len(tx.vout))], + "fee": fee, "txid": txid, "hex": tx.serialize().hex(), "tx": tx, @@ -321,52 +325,45 @@ def create_self_transfer(self, *, fee_rate=Decimal("0.003"), fee=Decimal("0"), u else: assert False send_value = utxo_to_spend["value"] - (fee or (fee_rate * vsize / 1000)) - assert send_value > 0 # create tx tx = self.create_self_transfer_multi(utxos_to_spend=[utxo_to_spend], locktime=locktime, sequence=sequence, amount_per_output=int(COIN * send_value), target_weight=target_weight) if not target_weight: assert_equal(tx["tx"].get_vsize(), vsize) + tx["new_utxo"] = tx.pop("new_utxos")[0] - return {"txid": tx["txid"], "hex": tx["hex"], "tx": tx["tx"], "new_utxo": tx["new_utxos"][0]} + return tx def sendrawtransaction(self, *, from_node, tx_hex, maxfeerate=0, **kwargs): txid = from_node.sendrawtransaction(hexstring=tx_hex, maxfeerate=maxfeerate, **kwargs) self.scan_tx(from_node.decoderawtransaction(tx_hex)) return txid - def create_self_transfer_chain(self, *, chain_length): + def create_self_transfer_chain(self, *, chain_length, utxo_to_spend=None): """ Create a "chain" of chain_length transactions. The nth transaction in the chain is a child of the n-1th transaction and parent of the n+1th transaction. - - Returns a dic {"chain_hex": chain_hex, "chain_txns" : chain_txns} - - "chain_hex" is a list representing the chain's transactions in hexadecimal. - "chain_txns" is a list representing the chain's transactions in the CTransaction object. """ - chaintip_utxo = self.get_utxo() - chain_hex = [] - chain_txns = [] + chaintip_utxo = utxo_to_spend or self.get_utxo() + chain = [] for _ in range(chain_length): tx = self.create_self_transfer(utxo_to_spend=chaintip_utxo) chaintip_utxo = tx["new_utxo"] - chain_hex.append(tx["hex"]) - chain_txns.append(tx["tx"]) + chain.append(tx) - return {"chain_hex": chain_hex, "chain_txns" : chain_txns} + return chain - def send_self_transfer_chain(self, *, from_node, chain_length, utxo_to_spend=None): + def send_self_transfer_chain(self, *, from_node, **kwargs): """Create and send a "chain" of chain_length transactions. The nth transaction in the chain is a child of the n-1th transaction and parent of the n+1th transaction. - Returns the chaintip (nth) utxo + Returns a list of objects for each tx (see create_self_transfer_multi). """ - chaintip_utxo = utxo_to_spend or self.get_utxo() - for _ in range(chain_length): - chaintip_utxo = self.send_self_transfer(utxo_to_spend=chaintip_utxo, from_node=from_node)["new_utxo"] - return chaintip_utxo + chain = self.create_self_transfer_chain(**kwargs) + for t in chain: + self.sendrawtransaction(from_node=from_node, tx_hex=t["hex"]) + return chain def getnewdestination(address_type='legacy'): From 50c42a2812a23710e48cdad6ae9367067f9b16ea Mon Sep 17 00:00:00 2001 From: glozow Date: Thu, 30 Mar 2023 18:46:38 +0100 Subject: [PATCH 16/16] Merge bitcoin/bitcoin#27350: test: refactor: dedup mempool_package_limits.py subtests via decorator e669833943bda13b2840a174dc8e45194187fc8e test: dedup package limit checks via decorator in mempool_package_limits.py (Sebastian Falbesoner) 72f25e238c1f791f9fd3018152d76f9127b745e2 test: refactor: use Satoshis for fees in mempool_package_limits.py (Sebastian Falbesoner) Pull request description: The subtests in the functional test mempool_package_limits.py all follow the same pattern: 1. first, check that the mempool is currently empty 2. create and submit certain single txs to the mempool, prepare list of hex transactions 3. check that `testmempoolaccept` on the package hex fails with a "package-mempool-limits" error on each tx result 4. after mining a block, check that submitting the package succeeds Note that steps 1,3,4 are identical for each of the subtests and only step 2 varies, so this might be a nice opportunity to deduplicate code by using a newly introduced decorator which executes the necessary before and after the essential part of the subtest. This also makes it easier to add new subtests without having to copy-paste those parts once again. In addition, the first commit switches the fee unit from BTC to Satoshis, which allows to get rid of some imports (`COIN` and `Decimal`) and a comment for the `test_desc_size_limits` subtest is fixed (s/25KvB/21KvB/). ACKs for top commit: ismaelsadeeq: ACK e669833943bda13b2840a174dc8e45194187fc8e glozow: utACK e669833943bda13b2840a174dc8e45194187fc8e Tree-SHA512: 84a85e739de7387391c13bd46aeb015a74302ea7c6f0ca3d4e2b1b487d38df390dc118eb5b1c11d3e4206bff316a4dab60ef6b25d8feced672345d4e36ffd205 --- test/functional/mempool_package_limits.py | 124 +++++++++------------- 1 file changed, 50 insertions(+), 74 deletions(-) diff --git a/test/functional/mempool_package_limits.py b/test/functional/mempool_package_limits.py index 480eae0bfdf6..9df8899895b8 100755 --- a/test/functional/mempool_package_limits.py +++ b/test/functional/mempool_package_limits.py @@ -8,14 +8,36 @@ from test_framework.blocktools import COINBASE_MATURITY from test_framework.test_framework import BitcoinTestFramework -from test_framework.messages import ( - COIN, -) from test_framework.util import ( assert_equal, ) from test_framework.wallet import MiniWallet +# Decorator to +# 1) check that mempool is empty at the start of a subtest +# 2) run the subtest, which may submit some transaction(s) to the mempool and +# create a list of hex transactions +# 3) testmempoolaccept the package hex and check that it fails with the error +# "package-mempool-limits" for each tx +# 4) after mining a block, clearing the pre-submitted transactions from mempool, +# check that submitting the created package succeeds +def check_package_limits(func): + def func_wrapper(self, *args, **kwargs): + node = self.nodes[0] + assert_equal(0, node.getmempoolinfo()["size"]) + package_hex = func(self, *args, **kwargs) + testres_error_expected = node.testmempoolaccept(rawtxs=package_hex) + assert_equal(len(testres_error_expected), len(package_hex)) + for txres in testres_error_expected: + assert_equal(txres["package-error"], "package-mempool-limits") + + # Clear mempool and check that the package passes now + self.generate(node, 1) + assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=package_hex)]) + + return func_wrapper + + class MempoolPackageLimitsTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 @@ -39,9 +61,9 @@ def run_test(self): self.test_anc_size_limits() self.test_desc_size_limits() + @check_package_limits def test_chain_limits_helper(self, mempool_count, package_count): node = self.nodes[0] - assert_equal(0, node.getmempoolinfo()["size"]) chain_hex = [] chaintip_utxo = self.wallet.send_self_transfer_chain(from_node=node, chain_length=mempool_count)[-1]["new_utxo"] @@ -50,13 +72,7 @@ def test_chain_limits_helper(self, mempool_count, package_count): tx = self.wallet.create_self_transfer(utxo_to_spend=chaintip_utxo) chaintip_utxo = tx["new_utxo"] chain_hex.append(tx["hex"]) - testres_too_long = node.testmempoolaccept(rawtxs=chain_hex) - for txres in testres_too_long: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=chain_hex)]) + return chain_hex def test_chain_limits(self): """Create chains from mempool and package transactions that are longer than 25, @@ -75,6 +91,7 @@ def test_chain_limits(self): # 13 transactions in the mempool and 13 in the package. self.test_chain_limits_helper(13, 13) + @check_package_limits def test_desc_count_limits(self): """Create an 'A' shaped package with 24 transactions in the mempool and 2 in the package: M1 @@ -92,7 +109,6 @@ def test_desc_count_limits(self): package transactions). """ node = self.nodes[0] - assert_equal(0, node.getmempoolinfo()["size"]) self.log.info("Check that in-mempool and in-package descendants are calculated properly in packages") # Top parent in mempool, M1 m1_utxos = self.wallet.send_self_transfer_multi(from_node=node, num_outputs=2)['new_utxos'] @@ -112,14 +128,9 @@ def test_desc_count_limits(self): assert_equal(24, node.getmempoolinfo()["size"]) assert_equal(2, len(package_hex)) - testres_too_long = node.testmempoolaccept(rawtxs=package_hex) - for txres in testres_too_long: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=package_hex)]) + return package_hex + @check_package_limits def test_desc_count_limits_2(self): """Create a Package with 24 transaction in mempool and 2 transaction in package: M1 @@ -156,15 +167,9 @@ def test_desc_count_limits_2(self): assert_equal(24, node.getmempoolinfo()["size"]) assert_equal(2, len(package_hex)) - testres = node.testmempoolaccept(rawtxs=package_hex) - assert_equal(len(testres), len(package_hex)) - for txres in testres: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=package_hex)]) + return package_hex + @check_package_limits def test_anc_count_limits(self): """Create a 'V' shaped chain with 24 transactions in the mempool and 3 in the package: M1a M1b @@ -182,7 +187,6 @@ def test_anc_count_limits(self): and in-package ancestors are all considered together. """ node = self.nodes[0] - assert_equal(0, node.getmempoolinfo()["size"]) package_hex = [] pc_parent_utxos = [] @@ -202,14 +206,9 @@ def test_anc_count_limits(self): assert_equal(24, node.getmempoolinfo()["size"]) assert_equal(3, len(package_hex)) - testres_too_long = node.testmempoolaccept(rawtxs=package_hex) - for txres in testres_too_long: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=package_hex)]) + return package_hex + @check_package_limits def test_anc_count_limits_2(self): """Create a 'Y' shaped chain with 24 transactions in the mempool and 2 in the package: M1a M1b @@ -227,7 +226,6 @@ def test_anc_count_limits_2(self): and in-package ancestors are all considered together. """ node = self.nodes[0] - assert_equal(0, node.getmempoolinfo()["size"]) pc_parent_utxos = [] self.log.info("Check that in-mempool and in-package ancestors are calculated properly in packages") @@ -244,14 +242,9 @@ def test_anc_count_limits_2(self): pd_tx = self.wallet.create_self_transfer(utxo_to_spend=pc_tx["new_utxos"][0]) assert_equal(24, node.getmempoolinfo()["size"]) - testres_too_long = node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]]) - for txres in testres_too_long: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]])]) + return [pc_tx["hex"], pd_tx["hex"]] + @check_package_limits def test_anc_count_limits_bushy(self): """Create a tree with 20 transactions in the mempool and 6 in the package: M1...M4 M5...M8 M9...M12 M13...M16 M17...M20 @@ -264,7 +257,6 @@ def test_anc_count_limits_bushy(self): combined, PC has 25 in-mempool and in-package parents. """ node = self.nodes[0] - assert_equal(0, node.getmempoolinfo()["size"]) package_hex = [] pc_parent_utxos = [] for _ in range(5): # Make package transactions P0 ... P4 @@ -281,14 +273,9 @@ def test_anc_count_limits_bushy(self): assert_equal(20, node.getmempoolinfo()["size"]) assert_equal(6, len(package_hex)) - testres = node.testmempoolaccept(rawtxs=package_hex) - for txres in testres: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=package_hex)]) + return package_hex + @check_package_limits def test_anc_size_limits(self): """Test Case with 2 independent transactions in the mempool and a parent + child in the package, where the package parent is the child of both mempool transactions (30KvB each): @@ -302,10 +289,9 @@ def test_anc_size_limits(self): """ node = self.nodes[0] parent_utxos = [] - assert_equal(0, node.getmempoolinfo()["size"]) - parents_tx = [] - target_weight = 1000 * 30 # 30KvB - high_fee = Decimal("0.003") # 10 sats/vB + target_vsize = 30_000 + high_fee = 10 * target_vsize # 10 sats/vB + target_weight = target_vsize fee_rate=Decimal("0.01") self.log.info("Check that in-mempool and in-package ancestor size limits are calculated properly in packages") # Mempool transactions A and B @@ -315,22 +301,17 @@ def test_anc_size_limits(self): parent_utxos.append(bulked_tx["new_utxo"]) # Package transaction C - pc_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=parent_utxos, fee_per_output=int(high_fee * COIN), target_weight=target_weight) + pc_tx = self.wallet.create_self_transfer_multi(utxos_to_spend=parent_utxos, fee_per_output=high_fee, target_weight=target_weight) # Package transaction D pd_tx = self.wallet.create_self_transfer(utxo_to_spend=pc_tx["new_utxos"][0], target_weight=target_weight, fee_rate=fee_rate) assert_equal(2, node.getmempoolinfo()["size"]) - testres_too_heavy = node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]]) - for txres in testres_too_heavy: - assert_equal(txres["package-error"], "package-mempool-limits") - - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=[pc_tx["hex"], pd_tx["hex"]])]) + return [pc_tx["hex"], pd_tx["hex"]] + @check_package_limits def test_desc_size_limits(self): - """Create 3 mempool transactions and 2 package transactions (25KvB each): + """Create 3 mempool transactions and 2 package transactions (21KvB each): Ma ^ ^ Mb Mc @@ -340,12 +321,12 @@ def test_desc_size_limits(self): and in-package descendants are all considered together. """ node = self.nodes[0] - assert_equal(0, node.getmempoolinfo()["size"]) - target_weight = 21 * 1000 - high_fee = Decimal("0.0021") # 10 sats/vB + target_vsize = 21_000 + high_fee = 10 * target_vsize # 10 sats/vB + target_weight = target_vsize self.log.info("Check that in-mempool and in-package descendant sizes are calculated properly in packages") # Top parent in mempool, Ma - ma_tx = self.wallet.create_self_transfer_multi(num_outputs=2, fee_per_output=int(high_fee / 2 * COIN), target_weight=target_weight) + ma_tx = self.wallet.create_self_transfer_multi(num_outputs=2, fee_per_output=high_fee // 2, target_weight=target_weight) self.wallet.sendrawtransaction(from_node=node, tx_hex=ma_tx["hex"]) package_hex = [] @@ -360,13 +341,8 @@ def test_desc_size_limits(self): assert_equal(3, node.getmempoolinfo()["size"]) assert_equal(2, len(package_hex)) - testres_too_heavy = node.testmempoolaccept(rawtxs=package_hex) - for txres in testres_too_heavy: - assert_equal(txres["package-error"], "package-mempool-limits") + return package_hex - # Clear mempool and check that the package passes now - self.generate(node, 1) - assert all([res["allowed"] for res in node.testmempoolaccept(rawtxs=package_hex)]) if __name__ == "__main__": MempoolPackageLimitsTest().main()