SSIS Looping Incremental ETL

Here we show how to build a looping on incremental ETL in SSIS:

Add a parameter “BatchSize” to your incremental parameterized query, having a limit on the batch size (“Top 1000” for SQL Server, or “Limit 1000″ for MySQL”):
image19

Add a BatchActualSize variable and let it get assigned the rowCount of the input:

image23

– Add a For Loop Container around the Get DW Key and ETL tasks:

image20

– Test it > It should be successful – repeating and resetting the following scene several times as the data allow:

image22

Test the DW population meanwhile. Then test and document the performance per total number of rows that it processed.

image21

If you need to log its performance, look at this article.

Leave a Reply

Your email address will not be published. Required fields are marked *