ASP.NET Core Playground - 4. Process data from SQL in batches

So I had to make a background worker that was supposed to process a lot of data coming directly from the database and run all that data through a microservice via an API. Waiting to download everything needed from the database seemed slow and a bad API experience, but also I didn’t want to flood the network by calling the API for every single data row.

So I came up with a nice and easy way of batching the data while streaming it.

We need two packages.

  • First one is System.Linq.Async. Installing it allows us to use normal and clean LINQ syntax with IAsyncEnumerable.

  • Second one is System.Reactive. This one allows us to use all the awesome features of reactive programming parading inside C#. If you know rxjs you will understand System.Reactive.

Now to the point. This is an example how to process data coming from the database in batches.

using System.Reactive.Linq;

return _dbContext.WeatherForecasts
    .Select(d => d.Count)

This we get an async enumerator that returns each iteration a list of 1000 rows. Best of both worlds!

This post is licensed under CC BY 4.0 by the author.