Release 2.0.11.2.0

.github/actions/bitcoin-int-tests/Dockerfile.bitcoin-tests

@@ -21,6 +21,7 @@ RUN cargo test -- --test-threads 1 --ignored tests::neon_integrations::bitcoind_
 RUN cargo test -- --test-threads 1 --ignored tests::neon_integrations::liquid_ustx_integration
 RUN cargo test -- --test-threads 1 --ignored tests::neon_integrations::stx_transfer_btc_integration_test
 RUN cargo test -- --test-threads 1 --ignored tests::neon_integrations::bitcoind_forking_test
+RUN cargo test -- --test-threads 1 --ignored tests::neon_integrations::should_fix_2771
 RUN cargo test -- --test-threads 1 --ignored tests::neon_integrations::pox_integration_test
 RUN cargo test -- --test-threads 1 --ignored tests::bitcoin_regtest::bitcoind_integration_test
 RUN cargo test -- --test-threads 1 --ignored tests::should_succeed_handling_malformed_and_valid_txs

CHANGELOG.md

@@ -5,6 +5,48 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to the versioning scheme outlined in the [README.md](README.md).
 
+## [2.0.11.2.0]
+
+NOTE: This change resets the `testnet`. Users running a testnet node will need
+to reset their chain states.
+
+### Added
+
+- `clarity-cli` will now also print a serialized version of the resulting
+  output from `eval` and `execute` commands. This serialization is in
+  hexademical string format and supports integration with other tools. (#2684)
+- The creation of a Bitcoin wallet with BTC version `> 0.19` is now supported
+  on a private testnet. (#2647)
+- `lcov`-compatible coverage reporting has been added to `clarity-cli` for
+  Clarity contract testing. (#2592)
+- The `README.md` file has new documentation about the release process. (#2726)
+
+### Changed
+
+- This change resets the testnet. (#2742)
+- Caching has been added to speed up `/v2/info` responses. (#2746)
+
+### Fixed
+
+- PoX syncing will only look back to the reward cycle prior to divergence,
+  instead of looking back over all history. This will speed up running a
+  follower node. (#2746)
+- The UTXO staleness check is re-ordered so that it occurs before the RBF-limit
+  check. This way, if stale UTXOs reached the "RBF limit" a miner will recover
+  by resetting the UTXO cache. (#2694)
+- Microblock events were being sent to the event observer when microblock data
+  was received by a peer, but were not emitted if the node mined the
+  microblocks itself. This made something like the private-testnet setup
+  incapable of emitting microblock events. Microblock events are now sent
+  even when self-mined. (#2653)
+- A bug is fixed in the mocknet/helium miner that would lead to a panic if a
+  burn block occurred without a sortition in it. (#2711)
+- A bug that caused problems syncing during a bitcoin reorg has been fixed
+  (#2771).
+- Syncing with the burnchain has been improved (#2780).
+- Documentation is fixed in cases where string and buffer types are allowed
+  but not covered in the documentation.  (#2676)
+
 ## [2.0.11.1.0]
 
 This software update is our monthly release. It introduces fixes and features for both developers and miners. 

Cargo.toml

@@ -43,7 +43,6 @@ name = "block_limits"
 harness = false
 
 [dependencies]
-tini = "0.2"
 rand = "=0.7.2"
 rand_chacha = "=0.2.2"
 serde = "1"

README.md

@@ -171,7 +171,7 @@ cargo run --bin blockstack-cli publish b8d99fd45da58038d630d9855d3ca2466e8e0f89d
 You can observe the state machine in action locally by running:
 
 ```bash
-cargo testnet start --config=./testnet/stacks-node/conf/testnet-follower-conf.toml
+cargo stacks-node start --config=./testnet/stacks-node/conf/testnet-follower-conf.toml
 ```
 
 `testnet-follower-conf.toml` is a configuration file that you can use for setting genesis balances or configuring Event observers. You can grant an address an initial account balance by adding the following entries:
@@ -317,6 +317,91 @@ You can automatically reformat your commit via:
 cargo fmt --all
 ```
 
+## Non-Consensus Breaking Release Process
+
+For non-consensus breaking releases, this project uses the following release process:
+
+1. The release must be timed so that it does not interfere with a *prepare
+phase*.  The timing of the next Stacking cycle can be found
+[here](https://stacking.club/cycles/next). A release to `mainnet` should happen
+at least 24 hours before the start of a new cycle, to avoid interfering
+with the prepare phase. So, start by being aware of when the release can
+happen.
+
+1. Before creating the release, the release manager must determine the *version
+number* for this release.  The factors that determine the version number are
+discussed in [Versioning](#versioning).  We assume, in this section,
+that the change is not consensus-breaking.  So, the release manager must first
+determine whether there are any "non-consensus-breaking changes that require a
+fresh chainstate". This means, in other words, that the database schema has
+changed. Then, the release manager should determine whether this is a feature
+release, as opposed to a hot fix or a patch. Given the answers to these
+questions, the version number can be computed.
+
+1. The release manager enumerates the PRs or issues that would _block_
+   the release. A label should be applied to each such issue/PR as
+   `2.0.x.y.z-blocker`. The release manager should ping these
+   issue/PR owners for updates on whether or not those issues/PRs have
+   any blockers or are waiting on feedback.
+
+1. The release manager should open a `develop -> master` PR. This can be done before
+   all the blocker PRs have merged, as it is helpful for the manager and others
+   to see the staged changes.
+
+1. The release manager must update the `CHANGELOG.md` file with summaries what
+was `Added`, `Changed`, and `Fixed`. The pull requests merged into `develop`
+can be found
+[here](https://github.com/blockstack/stacks-blockchain/pulls?q=is%3Apr+is%3Aclosed+base%3Adevelop+sort%3Aupdated-desc). Note, however, that GitHub apparently does not allow sorting by
+*merge time*, so, when sorting by some proxy criterion, some care should
+be used to understand which PR's were *merged* after the last `develop ->
+master` release PR.  This `CHANGELOG.md` should also be used as the description
+of the `develop -> master` so that it acts as *release notes* when the branch
+is tagged.
+
+1. Once the blocker PRs have merged, the release manager will create a new tag
+   by manually triggering the [`stacks-blockchain` Github Actions workflow](https://github.com/blockstack/stacks-blockchain/actions/workflows/stacks-blockchain.yml)
+   against the `develop` branch, inputting the release candidate tag, `2.0.x.y.z-rc0`,
+   in the Action's input textbox.
+
+1. Once the release candidate has been built, and docker images, etc. are available,
+   the release manager will notify various ecosystem participants to test the release
+   candidate on various staging infrastructure:
+
+   1. Stacks Foundation staging environments.
+   1. Hiro PBC regtest network.
+   1. Hiro PBC testnet network.
+   1. Hiro PBC mainnet mock miner.
+
+   The release candidate should be announced in the `#stacks-core-devs` channel in the
+   Stacks Discord. For coordinating rollouts on specific infrastructure, the release
+   manager should contact the above participants directly either through e-mail or
+   Discord DM. The release manager should also confirm that the built release on the
+   [Github releases](https://github.com/blockstack/stacks-blockchain/releases/)
+   page is marked as `Pre-Release`.
+
+1. The release manager will test that the release candidate successfully syncs with
+   the current chain from genesis both in testnet and mainnet. This requires starting
+   the release candidate with an empty chainstate and confirming that it synchronizes
+   with the current chain tip.
+
+1. If bugs or issues emerge from the rollout on staging infrastructure, the release
+   will be delayed until those regressions are resolved. As regressions are resolved,
+   additional release candidates should be tagged. The release manager is responsible
+   for updating the `develop -> master` PR with information about the discovered issues,
+   even if other community members and developers may be addressing the discovered
+   issues.
+
+1. Once the final release candidate has rolled out successfully without issue on the
+   above staging infrastructure, the release manager tags 2 additional `stacks-blockchain`
+   team members to review the `develop -> master` PR.
+
+1. Once reviewed and approved, the release manager merges the PR, and tags the release
+   via the [`stacks-blockchain` Github action]((https://github.com/blockstack/stacks-blockchain/actions/workflows/stacks-blockchain.yml))
+   by clicking "Run workflow" and providing the release version as the tag (e.g.,
+   `2.0.11.1.0`) This creates a release and release images. Once the release has been
+   created, the release manager should update the Github release text with the
+   `CHANGELOG.md` "top-matter" for the release.
+
 ## Copyright and License
 
 The code and documentation copyright are attributed to blockstack.org for the year of 2020.

deployment/helm/stacks-blockchain/values.yaml

@@ -231,13 +231,13 @@ config:
     rpc_port: 18443
     peer_port: 18444
   ustx_balance:
-    - address: STB44HYPYAT2BB2QE513NSP81HTMYWBJP02HPGK6
+    - address: ST2QKZ4FKHAH1NQKYKYAYZPY440FEPK7GZ1R5HBP2
       amount: "10000000000000000"
-    - address: ST11NJTTKGVT6D1HY4NJRVQWMQM7TVAR091EJ8P2Y
+    - address: ST319CF5WV77KYR1H3GT0GZ7B8Q4AQPY42ETP1VPF
       amount: "10000000000000000"
-    - address: ST1HB1T8WRNBYB0Y3T7WXZS38NKKPTBR3EG9EPJKR
+    - address: ST221Z6TDTC5E0BYR2V624Q2ST6R0Q71T78WTAX6H
       amount: "10000000000000000"
-    - address: STRYYQQ9M8KAF4NS7WNZQYY59X93XEKR31JP64CP
+    - address: ST2TFVBMRPS5SSNP98DQKQ5JNB2B6NZM91C4K3P7B
       amount: "10000000000000000"
 
   ## Uncommenting this block will give you greater control over the settings in the configmap
@@ -262,16 +262,16 @@ config:
   #   peer_port = 18444
 
   #   [[ustx_balance]]
-  #   address = "STB44HYPYAT2BB2QE513NSP81HTMYWBJP02HPGK6"
+  #   address = "ST2QKZ4FKHAH1NQKYKYAYZPY440FEPK7GZ1R5HBP2"
   #   amount = 10000000000000000
   #   [[ustx_balance]]
-  #   address = "ST11NJTTKGVT6D1HY4NJRVQWMQM7TVAR091EJ8P2Y"
+  #   address = "ST319CF5WV77KYR1H3GT0GZ7B8Q4AQPY42ETP1VPF"
   #   amount = 10000000000000000
   #   [[ustx_balance]]
-  #   address = "ST1HB1T8WRNBYB0Y3T7WXZS38NKKPTBR3EG9EPJKR"
+  #   address = "ST221Z6TDTC5E0BYR2V624Q2ST6R0Q71T78WTAX6H"
   #   amount = 10000000000000000
   #   [[ustx_balance]]
-  #   address = "STRYYQQ9M8KAF4NS7WNZQYY59X93XEKR31JP64CP"
+  #   address = "ST2TFVBMRPS5SSNP98DQKQ5JNB2B6NZM91C4K3P7B"
   #   amount = 10000000000000000
 
   ## Annotations to be added to the Configmap

docs/event-dispatcher.md

@@ -27,6 +27,9 @@ These events are sent to the configured endpoint at two URLs:
 This payload includes data related to a newly processed block,
 and any events emitted from Stacks transactions during the block.
 
+If the transaction originally comes from the parent microblock stream 
+preceding this block, the microblock related fields will be filled in.
+
 Example:
 
 ```json
@@ -58,15 +61,21 @@ Example:
       "raw_tx": "0x808000000004008bc5147525b8f477f0bc4522a88c8339b2494db50000000000000002000000000000000001015814daf929d8700af344987681f44e913890a12e38550abe8e40f149ef5269f40f4008083a0f2e0ddf65dcd05ecfc151c7ff8a5308ad04c77c0e87b5aeadad31010200000000040000000000000000000000000000000000000000000000000000000000000000",
       "status": "success",
       "tx_index": 0,
-      "txid": "0x3e04ada5426332bfef446ba0a06d124aace4ade5c11840f541bf88e2e919faf6"
+      "txid": "0x3e04ada5426332bfef446ba0a06d124aace4ade5c11840f541bf88e2e919faf6",
+      "microblock_sequence": "None",
+      "microblock_hash": "None",
+      "microblock_parent_hash": "None"
     },
     {
       "contract_abi": null,
       "raw_result": "0x03",
       "raw_tx": "0x80800000000400f942874ce525e87f21bbe8c121b12fac831d02f4000000000000000000000000000003e800006ae29867aec4b0e4f776bebdcea7f6d9a24eeff370c8c739defadfcbb52659b30736ad4af021e8fb741520a6c65da419fdec01989fdf0032fc1838f427a9a36102010000000000051ac2d519faccba2e435f3272ff042b89435fd160ff00000000000003e800000000000000000000000000000000000000000000000000000000000000000000",
       "status": "success",
       "tx_index": 1,
-      "txid": "0x738e4d44636023efa08374033428e44eca490582bd39a6e61f3b6cf749b4214c"
+      "txid": "0x738e4d44636023efa08374033428e44eca490582bd39a6e61f3b6cf749b4214c",
+      "microblock_sequence": "3",
+      "microblock_hash": "0x9304fcbcc6daf5ac3f264522e0df50eddb5be85df6ee8a9fc2384c54274daaac",
+      "microblock_parent_hash": "0x4893ab44636023efa08374033428e44eca490582bd39a6e61f3b6cf749b474bd"
     }
    ],
    "matured_miner_rewards": [
@@ -116,6 +125,63 @@ Example:
   PoX commitments during this block. These addresses may not actually receive rewards during
   this block if the block is faster than miners have an opportunity to commit.
 
+### `POST /new_microblocks`
+
+This payload includes data related to one or more microblocks that are either emmitted by the 
+node itself, or received through the network. 
+
+Example:
+
+```json
+{
+  "parent_index_block_hash": "0x999b38d44d6af72703a476dde4cea683ec965346d9e9a7ded2d773fb4f257a3b",
+  "events": [
+    {
+      "event_index": 1,
+      "committed": true,
+      "stx_transfer_event": {
+        "amount": "1000",
+        "recipient": "ST31DA6FTSJX2WGTZ69SFY11BH51NZMB0ZZ239N96",
+        "sender": "ST3WM51TCWMJYGZS1QFMC28DH5YP86782YGR113C1"
+      },
+      "txid": "0x738e4d44636023efa08374033428e44eca490582bd39a6e61f3b6cf749b4214c",
+      "type": "stx_transfer_event"
+    }
+  ],
+  "transactions": [
+    {
+      "contract_abi": null,
+      "raw_result": "0x03",
+      "raw_tx": "0x808000000004008bc5147525b8f477f0bc4522a88c8339b2494db50000000000000002000000000000000001015814daf929d8700af344987681f44e913890a12e38550abe8e40f149ef5269f40f4008083a0f2e0ddf65dcd05ecfc151c7ff8a5308ad04c77c0e87b5aeadad31010200000000040000000000000000000000000000000000000000000000000000000000000000",
+      "status": "success",
+      "tx_index": 0,
+      "txid": "0x3e04ada5426332bfef446ba0a06d124aace4ade5c11840f541bf88e2e919faf6",
+      "microblock_sequence": "3",
+      "microblock_hash": "0x9304fcbcc6daf5ac3f264522e0df50eddb5be85df6ee8a9fc2384c54274daaac",
+      "microblock_parent_hash": "0x4893ab44636023efa08374033428e44eca490582bd39a6e61f3b6cf749b474bd"
+    },
+    {
+      "contract_abi": null,
+      "raw_result": "0x03",
+      "raw_tx": "0x80800000000400f942874ce525e87f21bbe8c121b12fac831d02f4000000000000000000000000000003e800006ae29867aec4b0e4f776bebdcea7f6d9a24eeff370c8c739defadfcbb52659b30736ad4af021e8fb741520a6c65da419fdec01989fdf0032fc1838f427a9a36102010000000000051ac2d519faccba2e435f3272ff042b89435fd160ff00000000000003e800000000000000000000000000000000000000000000000000000000000000000000",
+      "status": "success",
+      "tx_index": 1,
+      "txid": "0x738e4d44636023efa08374033428e44eca490582bd39a6e61f3b6cf749b4214c",
+      "microblock_sequence": "4",
+      "microblock_hash": "0xfcd4fc34c6daf5ac3f264522e0df50eddb5be85df6ee8a9fc2384c5427459e43",
+      "microblock_parent_hash": "0x9304fcbcc6daf5ac3f264522e0df50eddb5be85df6ee8a9fc2384c54274daaac"
+    }
+  ],
+  "burn_block_hash": "0x4eaabcd105865e471f697eff5dd5bd85d47ecb5a26a3379d74fae0ae87c40904",
+  "burn_block_height": 331,
+  "burn_block_timestamp": 1651301734
+}
+```
+
+* `burn_block_{}` are the stats related to the burn block that is associated with the stacks 
+  block that precedes this microblock stream.
+* Each transaction json object includes information about the microblock the transaction was packaged into. 
+
 ### `POST /new_mempool_tx`
 
 This payload includes raw transactions newly received in the