Batch processing ActiveRecord in Yii 2

Optimise the memory and CPU usage

Most of the beginners might have seen the following kind of error while processing a large amount of data in PHP.


PHP Fatal error:  Out of memory (allocated 1707606016) (tried to allocate 426680697 bytes)

Even when you have ini_set("memory_limit", "-1"); in your script, you still face the same issue. This is because the amount of memory available for the execution of the script is no longer available.

While processing heavy records in Yii 2 or for that matter in any other framework the main concern is that of the memory and the CPU usage. In order to avoid the Out Of Memory Exception in PHP, we can go with the batch processing. Let's take a look at how we can implement it in the Yii 2 application.

As usual, we'll go with the example. Suppose you add a new field in the user table, username. Now you want to generate a username for each user and update the record in the database, assuming that there is some function, generateUsername(), which will generate the username.

Following code shows how it can be done without considering the 'Out Of Memory Exception'. This function fetches all the records from the database into the memory and then processes them one by one. Thus consuming heavy memory.


/* Unoptimised code */
public function actionAutoGenerateUsername()
{
    $users = User::find()->all();
    foreach ($users as $user) {
        
        $user->username = $user->generateUsername();
        if ($user->save()) {
            
            echo 'Username generated and saved';
            
        } else {
            
            echo 'Failed to generate and save username';
        }
    }
}

Now coming to the optimised code, look at the following code. Here, we first count the number of records we want to process, in this case, all the users. Then we create a batch of 100 records and fetch them into the memory and for these 100 records generate the username (you can have your own logic here). This is achieved using the limit and the offset parameters for getting the ActiveRecord.


/* Optimised code */
public function actionAutoGenerateUsername()
{
    $totalUsers = User::find()->count();

    $limit = 100;
    $batch = ceil($totalUsers / $limit);

    for ($i = 0; $i < $batch; ++$i) {

        $limitStart = $i * $limit;
        // $limitEnd = ($i + 1) * $limit;

        $users = User::find()
            ->limit($limit)
            ->offset($limitStart)
            ->all();

        foreach ($users as $user) {
            $user->username = $user->generateUsername();
            if ($user->save()) {

                echo 'Username generated and saved';

            } else {

                echo 'Failed to generate and save username';
            }
        }

    }
}

Yii 2 provides a better option for batch processing. It has built in method through which you can iterate over the objects of the Model Class in batches. Following is the sample code for our example.


public function actionAutoGenerateUsername()
{
    foreach (User::find()->each(100) as $user) {
        $user->username = $user->generateUsername();
        if ($user->save()) {

            echo 'Username generated and saved';

        } else {

            echo 'Failed to generate and save username';
        }
    }
}