Whenever mysqld is restarted the memory consumption slowly builds up to about 75 to 90% of my total memory (while only 8 out of 32GB is allocated to mysqld).
Sometimes this takes 5 hours, sometimes this takes 16 hours.
Usually it sticks around that percentage for a good amount of hours.
If that was just it it would be fine, but after some time it then starts using even more memory, Until it overflows, causing my 32GB swap drive to fill up to 100% sometimes.
Now things start to rapidly slow down to a crawl with mostly timeouts. It does recover after some time to the point it will no longer timeout for a while, but the memory never frees up so its constantly on the verge of dropping out again.
Anyone have an insight on how can this be?
Why is mysqld using so much memory and not returning any free mem?
And why is it not respecting at all the buffer limits that were set?
I have played around for some time with the mariadb.conf, but even reverting all back to default doesn’t seem to have much impact on the memory usage.
mysqld Ver 10.3.27-MariaDB-0+deb10u1
mysql 20 0 33.2g 25.1g 3760 S 521.6 80.1 1122:26 mysqld
Memory/buffer specs from mysql tuner
(--) Physical Memory : 31.3G
(--) Max MySQL memory : 8.0G
(--) Other process memory: 1.4G
(--) Total buffers: 7.5G global + 2.9M per thread (151 max threads)
(--) P_S Max memory usage: 104M
(--) Global Buffers
(--) +-- Key Buffer: 128.0M
(--) +-- Max Tmp Table: 256.0M
(--) Query Cache Buffers
(--) +-- Query Cache: OFF - DISABLED
(--) +-- Query Cache Size: 0B
(--) Per Thread Buffers
(--) +-- Read Buffer: 128.0K
(--) +-- Read RND Buffer: 256.0K
(--) +-- Sort Buffer: 2.0M
(--) +-- Thread stack: 292.0K
(--) +-- Join Buffer: 256.0K
I’m about to build PC from parts. I’m a software developer, so PC will be used for programming purposes mostly (PHP Storm, Visual Studio, Photoshop, Chrome 50+ tabs at time), I’m not a big gamer, but maybe I will try one or other game with a new PC.
I can spend up to $2300 (maximum $3500), I want to forget annoying lag spikes (if possible for this price).
I’m totally out of knowledge in nowadays software so I’ve decided to ask your help.
I found a this website where I can try to build my own PC from parts.
I’ve selected these parts for $3448.94, I would like to get your advices if I chosen it correctly for my needs. Maybe I can choose some parts cheaper?
AMD Threadripper 3960X 3.8 GHz 24-Core Processor
This processor seems powerful enough, isn’t it? Or maybe I should take a look to another? Maybe I have to search for CPU with integrated video card?
CPU Cooler $129.99
NZXT Kraken X53 73.11 CFM Liquid CPU Cooler
Totally I have no idea about coolers, chosen random from the list for an average price. Is it 1 cooler enough for PC at all?
I have no idea what parameters to look at when choosing motherboard. Is this motherboard good enough?
Crucial Ballistix MAX 16 GB (2 x 8 GB) DDR4-4400 CL19 Memory
For RAM I’ve chosen DDR4, total 16 GB 4400 speed, looks like should be fine. At this website I’ve found other RAM a bit cheaper but it provided me warning that too many voltage or etc, so have have chosen these.
Samsung 970 Evo 1 TB M.2-2280 NVME Solid State Drive
For Storage I’ve chosen 1 TB SSD, but there are other parameters too, so maybe you could give me advices if it’s good choice or I have to search for other. I’ve seen tat there is parameter Interface SATA 6 Gb/s or M.2 – what is difference?
Video Card $725.00
Asus GeForce GTX 1660 SUPER 6 GB TUF GAMING OC Video Card
I know that video card is playing big role in PC when playing games, but as I’m not a big gamer (at least never played before new, high quality graphic games) I have no idea what graphic card to choose, prices very different between video cards. Maybe I don’t need any at all and it’s enough integrated in CPU?
Lian Li PC-O11 Dynamic ATX Full Tower Case
Chosen randomly, I don’t know if I should take a look at anything when choosing Case.
Power Supply $134.99
Corsair RMx (2018) 750 W 80+ Gold Certified Fully Modular ATX Power Supply
Chosen randomly too. Any tips about it?
At the website that I provided before there are additional parts for PC like:
Case Accessories, Case Fans, Fan Controllers, Thermal Compound, UPS Systems Expansion Cards / Networking, Sound Cards, Wired Network Adapters, Wireless Network Adapters
Any of them are required?
I mean if I buying brand new motherboard isn’t it with integrated sound card. wired network adapter? So I have to buy them additionally? So maybe I have to buy something else? Maybe Case Fan also are required?
I started to write a function which had a pointer to a vector as a parameter so that it could modify that vector to output results (as the actual return value was an error code), when I started to think about the memory behind that.
For example, if I have the following code:
std::vector<int> *vect = new std::vector<int>();
for (uint32_t i = 0; i < 10; i++)
std::cout << "Ptr: " << vect << " Size " << vect->size() << " Max Size " << vect->capacity();
std::cout << " elements 0: " << (*vect)(0) << ", " << i << " :" << (*vect)(i) << std::endl;
And I run it, I get the following output:
Ptr: 0x557c393f9e70 Size 0 Max Size 0 elements 0: 0, 0 :0
Ptr: 0x557c393f9e70 Size 1 Max Size 1 elements 0: 0, 1 :1
Ptr: 0x557c393f9e70 Size 2 Max Size 2 elements 0: 0, 2 :2
Ptr: 0x557c393f9e70 Size 3 Max Size 4 elements 0: 0, 3 :3
Ptr: 0x557c393f9e70 Size 4 Max Size 4 elements 0: 0, 4 :4
Ptr: 0x557c393f9e70 Size 5 Max Size 8 elements 0: 0, 5 :5
Ptr: 0x557c393f9e70 Size 6 Max Size 8 elements 0: 0, 6 :6
Ptr: 0x557c393f9e70 Size 7 Max Size 8 elements 0: 0, 7 :7
Ptr: 0x557c393f9e70 Size 8 Max Size 8 elements 0: 0, 8 :8
Ptr: 0x557c393f9e70 Size 9 Max Size 16 elements 0: 0, 9 :9
It seems as though this could cause major memory issues – because if the vector needs to expand, it could be writing into space which is already being utilized, because it looks like that pointer does not change. Even running this over a much larger loop, this pointer looks like it is constant.
I’m still a (relatively) new programmer, and am not sure that I have the grasp on memory allocation that I would like to. Is my understanding correct – will this cause buffer errors and overwrite adjacent memory? Or is there some protection in std::vector that I am not considering?
I am creating a vectorized reinforcement learning environment. It handles multiple instances of a board game in synchron. Up until now, the state of the environment was a specific representation of the board. However, the need arose for running tests with different board representations and later it might expand to include data aside the representation (altough very unlikely).
Structure-wise I am utilizing Data Oriented Design and not Object Oriented Proremming. As result I have contiguous vectors for each environment attribute (board, turns, score) and not a vector of environments. This way of processing resulted with a 5-6x speed increase with otherwise same functions.
What options do I have in supporting multiple state types, especially if performance is a key aspect? (There is no need to support multiple types in the same instance of a vectorized environment.)
In my initial approach (after using a single std::vector with “fancy” indexing) I utilized a std::vector of std::arrays. This was sufficient for handling one board representation, but not for multiple. Therefore I switched to a custom templated class in my last approach with a std::array as its member. The size of the array was was set by an enum template argument. (I wanted my data to be in contiguous memory, therefore kept the vector of arrays design, and used tempalting to determine the size for holding all the features of the representation.) This method had advantages such as type safety and handling different scenarios by simple overloads. The disadvantages include the requirement of an extra tempalte argument (who would have though) and – the most importantly – I am not convined whether it could later be expanded into supporting a state that consists of something other than the board.
(To clarify the last part, if the state only requires the boards, then no issues there for handling multiple representations. However, if for a certain representation it needs an extra argument then the overloading only works if I add that extra argument as a dummy argument to all other functions.)
My ideas regarding the next possible approach include switching the type enum from a template argument to a member variable. The states could be handled with a single std::vector and fancy indexing or each representation would have their own storage allocated and only the selected one used. The proper function calls would be handled by branching (so multi argument state creation is no issue). This appraoch seems to be rather flexible, but maybe it would cost in performance.
What did I miss? How could my approaches be improved? What other solutions do you see?
I started a transaction with bitcoin-0.21.0-win64, a Windows client. It may be that the fee is set too low, which leads to the unsuccessful transaction. What should I do in the future? What do I need to do? How to implement the specific operation steps? Below is the transaction details, please help! Thank you very much！
Status: 0 / unacknowledged, not in memory pool
Date: 2021 / 3 / 8 12:31
To: coin an 1kgsxf7krv1ab99tex1wjm643tsilczbg6
Expenditure: – 0.05193365 BTC
Transaction fee: – 0.00120000 BTC
Net amount: – 0.05313365 BTC
Transaction ID: aceea84e8a72b24654cba7b1d042561d8a9d36f7828855b52012de8a5a0b9c12
Total transaction size: 11767 bytes
Transaction Virtual Size: 11767 bytes
Output index: 0
“The card is similarly unrecognisable on my computer CF card reader.”
Basically for software to work, the card has to be detected in Windows Disk Management with correct capacity.
Verify with a different reader, if card still isn’t recognized then in general this can’t be recovered using software. If I am asked to handle a case like that I:
Open the ‘case’ and inspect the PCB using multimeter and microscope. If I can spot defects I’ll try repairing them, and if successful I can now access the card. Some times it’s enough to reflow the solder under the NAND chips. If a working and matching donor board is available, often NAND chips can be transferred to donor PCB.
If unable to repair I unsolder NAND chips and ‘dump’ them using a reader. Using specialized software I then convert the dump to a logical file system using software that emulates the controller (or tries as good as it can) from which files can be recovered. Ideally result is a coherent file system, however sometimes raw recovery is highest achievable. It’s the logical image reconstruction part that is most complex and time consuming.
NAND readers are not overly expensive (but still too for a single case probably), it’s the software that is needed to convert the raw dump to a logical image that is. I use the Soft Center reader and software (called Flash Extractor).
With regards to logical photo recovery software, often mentioned names are PhotoRec and Recuva, also free and very good is R-Photo made by the company that produces R-Studio which is used in many labs for logical data recovery.
If card (whatever card, CF, SD etc.) is detected but physical capacity is incorrect:
In these cases dumping the NAND is almost always required. Typically physical capacity is decreased to MBs rather than GBs. It’s an indication controller is working but in let’s call it a safe mode. It is either unable to detect the NAND memory -or- NAND translation table is corrupt. See this translation table as a ‘FAT’ for the firmware, only this table is used by firmware to map LBA addresses to actual NAND pages.