[WIP] der: remove lifetime from BitStringRef [BREAKING]#2300
Closed
tarcieri wants to merge 2 commits into
Closed
Conversation
tarcieri
force-pushed
the
der/remove-bitstring-lifetime
branch
2 times, most recently
from
0977fc1 to
5819bf0
Compare
tarcieri
commented
Apr 23, 2026
| // was constructed from, and `inner` contains the original pointer. | ||
| #[allow(unsafe_code)] | ||
| unsafe { | ||
| slice::from_raw_parts(self.inner.as_ptr().cast(), byte_length(self.inner.len())) |
Member
Author
There was a problem hiding this comment.
This one, it seems, fails under stacked borrows:
https://github.com/RustCrypto/formats/actions/runs/24847701760/job/72739437052#step:5:78
error: Undefined Behavior: trying to retag from <177556> for SharedReadOnly permission at alloc64685[0x0], but that tag does not exist in the borrow stack for this location
--> der/src/asn1/bit_string2.rs:88:13
|
88 | slice::from_raw_parts(self.inner.as_ptr().cast(), byte_length(self.inner.len()))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this error occurs as part of retag at alloc64685[0x0..0x3]
|
= help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
= help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <177556> would have been created here, but this is a zero-size retag ([0x0..0x0]) so the tag in question does not exist anywhere
@RalfJung I don't suppose you know a way to pull off this particular hack that would actually be compatible with stacked borrows? (trying to make a bitstring-like reference type that can be used with Borrow and ToOwned patterns so it works with Cow) Apparently stacked borrows doesn't like me using a ZST slice like [UnsafeCell<()>] so I can store the bit length in a fat pointer.
Member
Author
There was a problem hiding this comment.
This issue and comment seem relevant: rust-lang/unsafe-code-guidelines#256 (comment)
There was a problem hiding this comment.
I assume the thread you started on Zulip is the same question? Thanks for moving it there, that means other people can help answer when I am swamped (which is my default state these days ;).
tarcieri
force-pushed
the
der/remove-bitstring-lifetime
branch
3 times, most recently
from
d9463a2 to
6bf8653
Compare
Following the pattern of #1921 and what #1998 did for `OctetStringRef`, this removes the lifetime from the struct, instead changing `BitStringRef` to a proper reference type to be used as `&BitStringRef`, which can implement the `Borrow` and `ToOwned` patterns needed to work with `Cow`. To make this work, this uses a hack described in #2298 where we abuse a fat pointer, namely a ZST slice `[UnsafeCell<()>]`, to carry the pointer to the `*const u8` buffer that backs the `BitStringRef`, and carries the bit length along as the length of the ZST slice. To get the original byte length back we can `div_ceil(8)` and then reconstruct the original `[u8]`.
tarcieri
force-pushed
the
der/remove-bitstring-lifetime
branch
from
6bf8653 to
be413f1
Compare
Miri passes with this
Member
Author
|
This fails under stacked borrows, but passes with 72211d8 which enables Given this is for a data type used to represent untrusted data that lives at the network edge, I'm disinclined to merge something with soundness that appears somewhat unclear:
It seems stacked borrows and tree borrows disagree about how to handle this scenario and it's unclear to me which one is "right": is stacked borrows being too strict, or is this an edge case in tree borrows that isn't being handled correctly? That's further compounded by it being a breaking change. So, I'm going to go ahead and close this, but perhaps we can reconsider something like this in the next breaking release of |
This is a grey area of the language where no decision has been made yet. If you want to write code that you can be sure will work with all future versions of Rust, then you can't do tricks like this. That's why Stacked Borrows rejects the code. If you want to write code that should work with today's Rust, but that is not covered by Rust's stability guarantees, then you can do tricks like this. That's why Tree Borrows accepts the code. But if you do then be aware that while the code should not be miscompiled by today's rustc, it might well be miscompiled in the future when we narrow down the spec. There's a reason Tree Borrows is not the default yet. We will likely want to allow the |
Member
Author
|
I think grey areas are something we probably want to avoid. Even if it so happens to work today I can imagine we would still have to deal with people asking us why it fails under Miri. Hopefully by the time we're ready to ship the next breaking release of this crate in a year or two there will have been some developments in Rust the language like proper custom DSTs or even just some decision being made about how this is specified so we know there won't be future miscompilation. |
|
Yeah, hopefully by then someone will step up to drive such developments forwards. |
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
tarcieri
added a commit
to RustCrypto/utils
that referenced
this pull request
May 15, 2026
Adds a crate providing a `&BitSlice`/`&mut BitSlice` type which is constructable from `&[u8]` but provides slicing at the granularity of individual bits. The name of the crate is a play on `bitvec`, which provides a similar type. However, the implementation in this crate is significantly simpler with a much smaller code surface and minimal use of `unsafe` code. The implementation is a generalization of RustCrypto/formats#2300 which sought to implement a similar data structure as a reference type for representing ASN.1 BIT STRINGs. However, using this approach was deferred because the implementation relies on a conversion which is sound under Tree Borrows (as verified by Miri) but unsound under Stacked Borrows as it loses provenance. See rust-lang/unsafe-code-guidelines#134 There are several places such a data structure is potentially useful for RustCrypto projects. Beyond the previously mentioned ASN.1 BIT STRING use case, being able to iterate over bits is useful in many numerical algorithms with applications in cryptography, notably in `crypto-bigint` and for elliptic curves. Elliptic curve scalar multiplication is generally implemented as a loop over the bits of a scalar. Having an iterator type for this purpose avoids problems relating to the endianness of how scalars are serialized when implementing generic scalar multiplication algorithms, e.g. wNAF (see RustCrypto/group#12). Given the current open soundness story, I'm not rushing to use this in `crypto-bigint` until that changes. Where we could use it today though is as an optional dependency to `der`, where it can act as an ASN.1 BIT STRING type, but implement `ToOwned` producing a `der::asn1::BitString` (which, to make `ToOwned` work, needs to impl `Borrow<BitSlice>`). This would make it optionally possible to use `Cow` for copy-on-write BIT STRINGs today with `BitSlice` as the borrowed form, but leaving the preferred default data structure for that purpose as `der::asn1::BitStringRef`, which is a lifetime-parameterized struct that avoids the open soundness questions around `BitSlice`. From there we can see what develops around the soundness story and SB/TB discrepancy, and beyond that new Rust features like custom DSTs which may make expressing structures like this less of a hack.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following the pattern of #1921 and what #1998 did for
OctetStringRef, this removes the lifetime from the struct, instead changingBitStringRefto a proper reference type to be used as&BitStringRef, which can implement theBorrowandToOwnedpatterns needed to work withCow.To make this work, this uses a hack described in #2298 where we abuse a fat pointer, namely a ZST slice
[UnsafeCell<()>], to carry the pointer to the*const u8buffer that backs theBitStringRef, and carries the bit length along as the length of the ZST slice.To get the original byte length back we can
div_ceil(8)and then reconstruct the original[u8].