Repository navigation
As a user I'd like to add Ingestion specs/Datasources programatically #168
Description
I'd like to have a CustomResource/ConfigMap that I can use to define all my Datasources/Ingestions in Druid.
We decided to just take the verbatim JSON as it is generated by the Druid Web UI and have this in a CustomResource.
I thought about having just a ConfigMap instead of a CR but that opens the question on how to reference the Druid cluster in question.
It could be the other way around: A "config map selector" in Druid that selects all ConfigMaps that should be added but that also doesn't seem very agile. Either way this needs to be authorized later.
- Please come up with a final architecture (CustomResource vs. other options)
- Bring up the architecture at the weekly architecture meeting for discussion
- Implement
- Add Integration Tests
- Add documentation
Metadata
Metadata
Assignees
Labels
No labels
Activity
razvan commentedon Mar 3, 2022
If each druid cluster runs in it's own namespace, there is no need to reference it. The ConfigMap is created in the appropriate namespace.
If multiple clusters can run in the same namespace, a label can be used to select the cluster.
siegfriedweber commentedon Mar 11, 2022
Ingestion Specs
Ingestion specs can be created with the wizard in the Druid web UI. If there is already an ingestion spec as JSON then it can be pasted into the web UI and submitted. Druid begins ingesting data once the ingestion spec is submitted. The web UI just calls a REST endpoint which also can be used directly:
The endpoint is not about declaring an ingestion spec, it is used for creating a task which ingests the data according to the spec. The ingestion is idempotent iff some conditions are met (see the druid documentation). For instance, if the user sets
appendToExistingtotruein the ingestion spec then it is not idempotent. In this case the chosen solution must not create a task more than once. So it is best to not rely on idempotency.Discovery
In the previous comments there was already a discussion how the ingestion specs should be discovered an assigned to Druid clusters. If the ingestion specs are part of the cluster specification then this discussion is unnecessary. If the ingestion specs are separate resources, like config maps or custom resources, then the according references can be made in the cluster specification or in the ingestion specs if the specs are custom resources. The references can be explicitly named or implicitly described by labels.
In my opinion, we should not inline the ingestion specs to keep the cluster specification small and to allow ingestion specs to be referenced by other clusters (if this is ever a real use case). I prefer config maps over a custom resource definition for the ingestion specs because I see no benefit of using custom resources. The config maps should be referenced in the cluster specification because that is what we do in all other cases, too. I have no opinion if they should be referenced by (config map) names or by labels.
Options
Custom resource managed by the operator
A little digression to Superset: The database used by Superset must be initialized. The data necessary for the initialization (a secret with the admin user data and a flag indicating if examples should be loaded) are provided in the specification of the Superset cluster. The Superset operator uses a custom resource of kind
SupersetDBto ensure that the database is initialized with the given data. The custom resource is created by the operator and the SupersetDB controller handles the reconciliation. TheSupersetDBresource then provides information about the status of the database initialization. The initialization is only performed once. (Not important here but for the sake of completeness: Another reason for this approach is that the Druid connection can only be set up after the database is initialized. The Druid connection is also modelled as a custom resource. The controller of the Druid connection watches the status of the database resource and creates the job after the database is ready.)The database initialization in Superset solves a similar problem as given in this ticket. The user creates config maps with the ingestion specs and references them in the Druid cluster specification. The Druid operator then creates a custom resource to handle the ingestion after the cluster is up-and-running. The status of the ingestion could also be reflected in the custom resource.
A job creates the ingestion tasks
The ingestion specs are stored in config maps and referenced by the Druid cluster. A job is created together with the cluster which waits until the cluster is up-and-running and then creates the ingestion tasks. This is a simple solution but I do not like the busy waiting.
No implementation
Druid already offers a very convenient way of adding ingestion specs. Users will probably use the option which is the most convenient one. Therefore, not implementing this functionality in the Stackable platform is also an option.
siegfriedweber commentedon Mar 11, 2022
As discussed in the daily, the implementation part will be outsourced to a separate ticket.
lfrancke commentedon Mar 21, 2022
Thank you, very thorough.
Do the ingestion specs have a "name" (I assume so because as far as I understood you they do create tables in the end) which means multiple specs could conflict with each other?
siegfriedweber commentedon Mar 23, 2022
The datasources have a name (e.g. "wikipedia"):
The ingestion spec contains the name of the datasource:
{ "spec": { "dataSchema": { "dataSource": "wikipedia",If a new task (with a different ingestion spec) is created for the same datasource then the datasource is changed accordingly. The tasks have unique identifiers:
Therefore the order in which ingestion specs for the same datasource are applied, matters.
However, I have no experience with Druid. So, @adwk67, please correct me if my explanation is inaccurate.
adwk67 commentedon Mar 23, 2022
In default mode an ingestion spec for an existing datasource will just overwrite it. There is an append option, so data can be extended, although deduplication is not attempted in any kind of thorough way (if the same data exists in different segments, then they exist as duplicates within that datasource. See here.
sbernauer commentedon Apr 4, 2022
Meeting with @lfrancke regarding deeper understanding outstanding and decision
lfrancke commentedon Apr 6, 2022
Thank you!
@sbernauer do you want to be part of that meeting?
sbernauer commentedon Apr 6, 2022
Not from my side, thanks. I just commented that in our daily as i was sharing my screen
soenkeliebau commentedon Apr 25, 2022
ping @lfrancke - do you remember why this is still open?
soenkeliebau commentedon Apr 29, 2022
Feedback from Lars: Waiting for a meeting on this, but holiday season interferred, will be picked up next week.
soenkeliebau commentedon May 3, 2022
@siegfriedweber @adwk67 @lfrancke just had a brief meeting on this and took the following decision:
Ingestion specs can be defined by the user via a customresource, which is watched by a controller in the Druid operator that then provisions these specs.
The crd will contain at least the following:
These objects will initially be considered read-only, so changes to them will not be propagated to Druid by the controller.
soenkeliebau commentedon May 3, 2022
implementation will be tracked in #226
lfrancke commentedon May 3, 2022
Thank you @soenkeliebau for the ticket and @adwk67 @siegfriedweber for the preparation and help.
I think this issue can be closed then in favor of #226.