ASP.NET Core Playground - 4. Process data from SQL in batches

So I had to make a background worker that was supposed to process a lot of data coming directly from the database and run all that data through a microservice via an API. Waiting to download everything needed from the database seemed slow and a bad API experience, but also I didn’t want to flood the network by calling the API for every single data row.

So I came up with a nice and easy way of batching the data while streaming it.

We need two packages.

• First one is System.Linq.Async. Installing it allows us to use normal and clean LINQ syntax with IAsyncEnumerable.

• Second one is System.Reactive. This one allows us to use all the awesome features of reactive programming parading inside C#. If you know rxjs you will understand System.Reactive.

Now to the point. This is an example how to process data coming from the database in batches.

using System.Reactive.Linq;

return _dbContext.WeatherForecasts
    .AsAsyncEnumerable()
    .ToObservable()
    .Buffer(1000)
    .Select(d => d.Count)
    .ToAsyncEnumerable();

This we get an async enumerator that returns each iteration a list of 1000 rows. Best of both worlds!