mariadb – MySQL is claiming all of my memory when it’s only using a few gigs

So, I have been having an issue with memory being “buffered” or “active”, it seems my MariaDB installation is just marking all the memory on my system as it’s own? I don’t know for sure what is going on but it will show that I have almost no memory free but 24Gb available on free -wh ? I need to know if this is an issue and if yes please tell me what settings I need to change.

My system has 32GB of RAM and it’s constantly having to use SWAP because of this memory being buffered/active (swappiness is set to 1). Please check my server/mariadb settings below.

free -wh
mariadb.conf
smem -p
innodb engine status
cat /proc/meminfo
htop