In general, it's up to you, the creator of a package, to define the dependencies of your package. In the end, you tell future users of your package: you must want and be able to use these dependencies, otherwise you will not be able to use this package.
You can decide to be as minimalist as you want or use as many mysterious and hard-to-install libraries as only the most ardent and motivated soul will want to take care of your contribution (there is certainly more than some of these floating software!). Some libraries specializing in computer vision and neural networks could be an example – you would not want it if you did not really need it for your package.
The reasonable in-between says: "Do these dependencies put a heavy burden on people for whom I want to be able to use my software?" If the answer is "not as burdensome as I would be if I did not use those libraries myself," then that's it: include them, it's literally what they're the!
The need to use extremely common libraries is generally less of a burden, as you expect most users to have it installed or will do so anyway for their own use (and
magrittr is very popular in the R world, so I would be more concerned about the packages you have found hard to find and install successfully).
If this was part of a large project or if it had very specific hardware / software limitations, I would implore you to collect the requirements for the types of systems, operating systems, and versions you need to take in charge. Then you only need to compare your dependencies to the requirements you have collected. If you do not have these limitations, you can simply focus on your own productivity as a developer and use what helps you do your job.