Platform
Connectors
FAQs and Troubleshooting
Billing
This section gives you an overview of how you will get billed for Daton usage.
Basics
We have made a conscious attempt to keep Daton billing simple, and easy to track. The most granular element for billing in Daton is the number of rows replicated.
What is considered a replicated row?
- A newly created row in the destination that has been replicated through Daton.
- Each row in fully replicated tables
- Newly created rows in an incrementally loaded table
- An updated row in the destination
- A row that has been created through un-nesting nested columns in tables.
- Un-nested tables are created as sub-tables with the destination.
- Applicable to certain data warehouses.
- Rows processed as a result of reload and table setting operations.
Every Daton subscription grants users a limit of replicated rows per month. The number of rows replicated limit resets at the end of every billing cycle, i.e, in a given month, if the number of rows replicated is lower than the limit granted by the subscription, the remaining rows are lost and are not carried forward to the next billing cycle.
Overages
If the row limit for a certain month is breached, integrations continue to function, but an additional overage charge is applied at the end of the billing cycle. Some common occurrences for overages are
- When new integrations are setup which will perform full loads
- Fully replicated tables
- Replicating nested columns.
Strategies to Mitigate Overage Charges
A replication schedule is attached at the source integration level. Any table that is being replicated within that integration will follow the same schedule. There will be instances where certain tables (APIs) within a source only support full replication while some others have replication field based replication. To keep the system lean and billing in check, users can adopt some of the following strategies.
Full vs Incremental load separation
- Mixing full load only tables with incremental load tables will result in potentially unnecessary loads of full load tables.
- A recommended approach would be to configure two or more integrations for the same source to handle incremental and full load separately.
Example:
- Table 1:
- Can only be replicated fully due to a lack of source system support or lack of a replication field.
- New rows are added to the table once in a few days.
- Table 2
- Can be replicated based on a replication field.
- 10 rows added to the table every hour.
If both the tables in the example are added as a part of the same integration that is running at a replication frequency of 1 hour, then the number of rows processed at the end of 4 hours will be:
Table 1: 100 + 100 + 100 + 100 Table 2: 100 + 10 + 10 + 10 Total rows processed = 100 4 + 100 + 310 = 530
If both the tables in the example are added as a part of the separate integrations that is running at a replication frequency of 24 hours for Table 1 and 1 hour for table 2, then the number of rows processed at the end of 2 hours will be:
Table 1: 100 ( next load will run after 24 hours) Table 2: 100 + 10 + 10 + 10 Total rows processed = 100 + 100 + 3*10 = 230
As you can see, the number of rows utilized can add up quickly for fully replicated tables and therefore planning your integration setup is important.
SaaS applications
Daton supports a variety of SaaS applications. Most SaaS applications these days support extracting data through REST APIs. However, some SaaS applications like analytics applications, generate more data than other applications like the ones for customer support. Knowing what data to pull from these sources is important to keep replicated rows to an optimal level.
Full Replication:
Certain SaaS applications do not permit extracting incremental data based on a previous stored state. In such cases, only full loads are possible. Please follow the recommended approach above for these type of SaaS integrations as well.
Nested Columns: Quite a few SaaS applications these days use nested columns within tables to store data. For example, Shopify stores order line data in a column within the order header table. When a relational database like Oracle, Postgres or MySQl is used as the destination, Daton attempts to un-nest these nested columns, creates new tables for these columns.
In the shopify example highlighted above, users may choose to unselect the nested column while configuring header table and configure a separate table for order lines.
Documentation Please refer to our SaaS documentation before embarking on setting up an integration to understand how the SaaS application integration works and how different tables are replicated.
Pausing integrations and removing tables from integration
​Edit integration functionality allows users to unselect tables in the integration that are no longer being leveraged so that replication no longer happens for these tables. Please make the changes only after you have sufficiently determined that no one downstream is dependent on the table that you are unselecting from the integration.
​Pausing an integration will immediately signal Daton to no longer run any pipeline jobs pertaining to that integration. If you pause the integration while a job for that integration is currently running, that job will not be impact and those rows will be counted toward the monthly quota. It is not a common practice to pause any integration but is an option that can be considered if necessary.
Free vs Paid Integrations
In addition to the number of rows replicated per month, users could also be charged an additional fee over the base subscription price if they subscribe to certain paid integrations. At this point in time, all paid integrations are custom in nature, so please reach out to us if you have any specific needs,, and we would be happy to engage.
Visibility into processed rows
Daton provides three different way for users to track data being processed by the data pipeline. All three views offer different perspectives in the data processed for the account.
Dashboard
Our dashboard gives you access to the summary of data processed by Daton. More details on the charts and graphs in the dashboard can be found here​
Integrations
Integration details comprises of three sections. The section that pertains to billing is the section that highlights the number of rows processed by the most recent job for an integration. Clicking on the details icon include a small screenshot on the integration, takes the user to the details on an integration.
Integration Details
On this screen, users can view the total rows processed for a specific source integration, a list of all the jobs for that source, and other job related data.
Last modified 2yr ago