magento2 – How to set an offset for category pages?

I wish the first page of a category product grid to display 18 products, while all other pages must display 20 products. I wrote a plugin for Magento Catalog Block Product ProductList ToolbarThe getLimit method that accomplishes this. However, page 1 shows products 1 to 18 and page 2, products 21 to 40 and it is impossible to see products # 19 and # 20.

Did I choose the wrong approach? Or is there a way to change the pagination offset?