Experimental: External Payload Storage

[jmaeagle99]

What was changed

Add external storage APIs to allow customers to specify how large payloads or payloads that match some specific customer-defined quality can be stored via customer-supplied external storage systems.

• Customers can choose their external storage endpoints by providing storage drivers. They can either use prebuilt ones or write their own via the StorageDriver type. This change does not contain any prebuilt drivers, just provides the primitives to make them.
• Customers can register several storage drivers. This allows workflows and clients to be able to retrieve payloads that have been stored by different driver types.
• Customers can dynamically choose which driver to use for storing each payload by specifying a driver selector. The selector can be a simple callable or could be a StorageDriverSelector to participate in serialization context transforms.
• External storage is situated outside of the workflow sandbox and runs after encoding for outbound requests and runs before decoding for inbound responses.
• External storage is off by default and must be opted into via specifying DataConverter.external_storage. There are no breaking changes regardless of opted into or not.

Why?

• Allow customers to mitigate situations where payload size may be too large to store into temporal history.
• Allow customers to externally store payloads based on other payload and contextual qualities such a serialization context and payload metadata.

Checklist

1. Closes External Payload Storage Foundation - Python #1342
2. How was this tested: Add additional unit and workflow tests.
3. Any docs updates needed? Yes, new docs for external storage should be created.

[drewhoskins-temporal]

Very excited! One important comment (which we already covered in our afternoon sync) and a bunch of polish suggestions.

[drewhoskins-temporal]

Is there enough information here for users to understand and debug what happened? I'm not seeing it here.

[drewhoskins-temporal]

Suggested change

	== "Driver 'bad-driver' returned 0 claims, expected 1"
	== "ExternalStorageDriver 'bad-driver' returned 0 claims, expected 1."

Here and on other diagnostics, please make sure to set context and to make it actionable.

• Context. What can't we do because this happened? "Cannot store a reference to uploaded payload." or something.
• Actionability. What should the user do to debug/fix? I'm guessing in this case it'd be "This is probably a bug in the ExternalStorageDriver."

[drewhoskins-temporal]

Can we assert the message here? And as above, make sure we make an actionable suggestion for the user.

[drewhoskins-temporal]

nit: I know this is just test code, but should echo the bad data (bucket ID) in the error message or its metadata.