mysql – Is using strings as keys for reference to other tables bad in terms of memory usage?

Yes MySql and all other rdms will store the complete email as varchar and reserve space for the number of bytes.

Integer with up to 8 bytes for a big integer will on löy use these bytes, and are so faster when referencing.

In terms of speed you use INTEGER, and consider other like varchar(36) for uuids when the need arises, for example different servers have to save data in the same table.

whith emails a unique and so be indexed for referencing, you should take the extra mile and use Integer if you expect you will have big tables.

mysql – Mysqld draining memory, using way more then assigned and not giving any back

This is a production server.

Whenever mysqld is restarted the memory consumption slowly builds up to about 75 to 90% of my total memory (while only 8 out of 32GB is allocated to mysqld).
Sometimes this takes 5 hours, sometimes this takes 16 hours.
Usually it sticks around that percentage for a good amount of hours.
If that was just it it would be fine, but after some time it then starts using even more memory, Until it overflows, causing my 32GB swap drive to fill up to 100% sometimes.
Now things start to rapidly slow down to a crawl with mostly timeouts. It does recover after some time to the point it will no longer timeout for a while, but the memory never frees up so its constantly on the verge of dropping out again.

Anyone have an insight on how can this be?
Why is mysqld using so much memory and not returning any free mem?
And why is it not respecting at all the buffer limits that were set?
I have played around for some time with the mariadb.conf, but even reverting all back to default doesn’t seem to have much impact on the memory usage.

Running

mysqld  Ver 10.3.27-MariaDB-0+deb10u1

top stats

mysql     20   0   33.2g  25.1g   3760 S 521.6  80.1   1122:26 mysqld

Memory/buffer specs from mysql tuner

(--) Physical Memory     : 31.3G
(--) Max MySQL memory    : 8.0G
(--) Other process memory: 1.4G                                                                                                                                                                                                               
(--) Total buffers: 7.5G global + 2.9M per thread (151 max threads)                                                                                                                                                                           
(--) P_S Max memory usage: 104M
(--) Global Buffers
(--)  +-- Key Buffer: 128.0M
(--)  +-- Max Tmp Table: 256.0M
(--) Query Cache Buffers
(--)  +-- Query Cache: OFF - DISABLED
(--)  +-- Query Cache Size: 0B
(--) Per Thread Buffers
(--)  +-- Read Buffer: 128.0K
(--)  +-- Read RND Buffer: 256.0K
(--)  +-- Sort Buffer: 2.0M
(--)  +-- Thread stack: 292.0K
(--)  +-- Join Buffer: 256.0K

Mariadb conf

skip-name-resolve
performance_schema      = ON
query_cache_type        = 0
query_cache_size        = 0   
tmp_table_size          = 256M
max_heap_table_size     = 256M
innodb_log_files_in_group       = 4
innodb_buffer_pool_size        = 7G 
innodb_status_file                      #extra reporting
innodb_file_per_table                   #enable always
innodb_flush_log_at_trx_commit  = 2     #2/0 = perf, 1 = ACID
innodb_table_locks              = 0  
innodb_lock_wait_timeout        = 60
innodb_thread_concurrency       = 24 
innodb_commit_concurrency       = 2 
innodb_log_file_size   = 384M
innodb_buffer_pool_instances = 7  

memory – Building PC from parts

I’m about to build PC from parts. I’m a software developer, so PC will be used for programming purposes mostly (PHP Storm, Visual Studio, Photoshop, Chrome 50+ tabs at time), I’m not a big gamer, but maybe I will try one or other game with a new PC.

I can spend up to $2300 (maximum $3500), I want to forget annoying lag spikes (if possible for this price).

I’m totally out of knowledge in nowadays software so I’ve decided to ask your help.

I found a this website where I can try to build my own PC from parts.

I’ve selected these parts for $3448.94, I would like to get your advices if I chosen it correctly for my needs. Maybe I can choose some parts cheaper?

CPU $1399.00

AMD Threadripper 3960X 3.8 GHz 24-Core Processor

This processor seems powerful enough, isn’t it? Or maybe I should take a look to another? Maybe I have to search for CPU with integrated video card?


CPU Cooler $129.99

NZXT Kraken X53 73.11 CFM Liquid CPU Cooler

Totally I have no idea about coolers, chosen random from the list for an average price. Is it 1 cooler enough for PC at all?


Motherboard $499.99

Gigabyte TRX40 AORUS MASTER EATX sTRX4 Motherboard

I have no idea what parameters to look at when choosing motherboard. Is this motherboard good enough?


Memory $219.99

Crucial Ballistix MAX 16 GB (2 x 8 GB) DDR4-4400 CL19 Memory

For RAM I’ve chosen DDR4, total 16 GB 4400 speed, looks like should be fine. At this website I’ve found other RAM a bit cheaper but it provided me warning that too many voltage or etc, so have have chosen these.


Storage $159.99

Samsung 970 Evo 1 TB M.2-2280 NVME Solid State Drive

For Storage I’ve chosen 1 TB SSD, but there are other parameters too, so maybe you could give me advices if it’s good choice or I have to search for other. I’ve seen tat there is parameter Interface SATA 6 Gb/s or M.2 – what is difference?


Video Card $725.00

Asus GeForce GTX 1660 SUPER 6 GB TUF GAMING OC Video Card

I know that video card is playing big role in PC when playing games, but as I’m not a big gamer (at least never played before new, high quality graphic games) I have no idea what graphic card to choose, prices very different between video cards. Maybe I don’t need any at all and it’s enough integrated in CPU?


Case $159.99

Lian Li PC-O11 Dynamic ATX Full Tower Case

Chosen randomly, I don’t know if I should take a look at anything when choosing Case.

__

Power Supply $134.99

Corsair RMx (2018) 750 W 80+ Gold Certified Fully Modular ATX Power Supply

Chosen randomly too. Any tips about it?


At the website that I provided before there are additional parts for PC like:

Case Accessories, Case Fans, Fan Controllers, Thermal Compound, UPS Systems Expansion Cards / Networking, Sound Cards, Wired Network Adapters, Wireless Network Adapters

Any of them are required?

I mean if I buying brand new motherboard isn’t it with integrated sound card. wired network adapter? So I have to buy them additionally? So maybe I have to buy something else? Maybe Case Fan also are required?

Thank you for any tips!

database design – Redis or MongoDB in Memory

I am working on processing of documents and storing their hash in Redis to identify duplicate records and some system files.

Recently we migrated to MongoDB from SQL Server as our primary database.

So now does it make sense to manage both Redis and MongoDB? as we can also run MongoDB in memory?

C++: Is a pointer to a vector going to cause memory issues?

I started to write a function which had a pointer to a vector as a parameter so that it could modify that vector to output results (as the actual return value was an error code), when I started to think about the memory behind that.

For example, if I have the following code:

std::vector<int> *vect = new std::vector<int>();

for (uint32_t i = 0; i < 10; i++)
{
    std::cout << "Ptr: " << vect << " Size " << vect->size() <<  " Max Size " << vect->capacity();
    vect->push_back(i);
    std::cout << " elements 0: " << (*vect)(0) << ", " << i << " :" << (*vect)(i) << std::endl; 
}

And I run it, I get the following output:

Ptr: 0x557c393f9e70 Size 0 Max Size 0 elements 0: 0, 0 :0
Ptr: 0x557c393f9e70 Size 1 Max Size 1 elements 0: 0, 1 :1
Ptr: 0x557c393f9e70 Size 2 Max Size 2 elements 0: 0, 2 :2
Ptr: 0x557c393f9e70 Size 3 Max Size 4 elements 0: 0, 3 :3
Ptr: 0x557c393f9e70 Size 4 Max Size 4 elements 0: 0, 4 :4
Ptr: 0x557c393f9e70 Size 5 Max Size 8 elements 0: 0, 5 :5
Ptr: 0x557c393f9e70 Size 6 Max Size 8 elements 0: 0, 6 :6
Ptr: 0x557c393f9e70 Size 7 Max Size 8 elements 0: 0, 7 :7
Ptr: 0x557c393f9e70 Size 8 Max Size 8 elements 0: 0, 8 :8
Ptr: 0x557c393f9e70 Size 9 Max Size 16 elements 0: 0, 9 :9

It seems as though this could cause major memory issues – because if the vector needs to expand, it could be writing into space which is already being utilized, because it looks like that pointer does not change. Even running this over a much larger loop, this pointer looks like it is constant.

I’m still a (relatively) new programmer, and am not sure that I have the grasp on memory allocation that I would like to. Is my understanding correct – will this cause buffer errors and overwrite adjacent memory? Or is there some protection in std::vector that I am not considering?

design – What approaches are there for handling elements in contiguous memory while being dependent on other user input

I am creating a vectorized reinforcement learning environment. It handles multiple instances of a board game in synchron. Up until now, the state of the environment was a specific representation of the board. However, the need arose for running tests with different board representations and later it might expand to include data aside the representation (altough very unlikely).

Structure-wise I am utilizing Data Oriented Design and not Object Oriented Proremming. As result I have contiguous vectors for each environment attribute (board, turns, score) and not a vector of environments. This way of processing resulted with a 5-6x speed increase with otherwise same functions.

What options do I have in supporting multiple state types, especially if performance is a key aspect? (There is no need to support multiple types in the same instance of a vectorized environment.)

In my initial approach (after using a single std::vector with “fancy” indexing) I utilized a std::vector of std::arrays. This was sufficient for handling one board representation, but not for multiple. Therefore I switched to a custom templated class in my last approach with a std::array as its member. The size of the array was was set by an enum template argument. (I wanted my data to be in contiguous memory, therefore kept the vector of arrays design, and used tempalting to determine the size for holding all the features of the representation.) This method had advantages such as type safety and handling different scenarios by simple overloads. The disadvantages include the requirement of an extra tempalte argument (who would have though) and – the most importantly – I am not convined whether it could later be expanded into supporting a state that consists of something other than the board.

(To clarify the last part, if the state only requires the boards, then no issues there for handling multiple representations. However, if for a certain representation it needs an extra argument then the overloading only works if I add that extra argument as a dummy argument to all other functions.)

My ideas regarding the next possible approach include switching the type enum from a template argument to a member variable. The states could be handled with a single std::vector and fancy indexing or each representation would have their own storage allocated and only the selected one used. The proper function calls would be handled by branching (so multi argument state creation is no issue). This appraoch seems to be rather flexible, but maybe it would cost in performance.

What did I miss? How could my approaches be improved? What other solutions do you see?

client – A wrong deal: the status is “0 / unacknowledged, not in memory pool “

I started a transaction with bitcoin-0.21.0-win64, a Windows client. It may be that the fee is set too low, which leads to the unsuccessful transaction. What should I do in the future? What do I need to do? How to implement the specific operation steps? Below is the transaction details, please help! Thank you very much!

Status: 0 / unacknowledged, not in memory pool
Date: 2021 / 3 / 8 12:31
To: coin an 1kgsxf7krv1ab99tex1wjm643tsilczbg6
Expenditure: – 0.05193365 BTC
Transaction fee: – 0.00120000 BTC
Net amount: – 0.05313365 BTC
Transaction ID: aceea84e8a72b24654cba7b1d042561d8a9d36f7828855b52012de8a5a0b9c12
Total transaction size: 11767 bytes
Transaction Virtual Size: 11767 bytes
Output index: 0

Find offset and page number given paged virtual memory address

I ran into this question in class:
Assume a machine that is 64-bit and has 8GB memory. They use a paged virtual memory where the page size is 4KB. You run the following program:

int main() {
   int *ptr = malloc(16);
   printf("%pn", ptr);
   return 0;
}

and get output:
0x555555756260

We’re to find the virtual page number of the address and the offset.

What I’m thinking is that I first convert that to binary, figure out how many bits and which bits represent the address/offset.

But I’m not sure how to figure out the latter. How can I determine which bits in the address correspond to which?

I’d really love some guidance

How can I recover photo files from a memory card with a corrupt filesystem?

“The card is similarly unrecognisable on my computer CF card reader.”

Basically for software to work, the card has to be detected in Windows Disk Management with correct capacity.

enter image description here

Verify with a different reader, if card still isn’t recognized then in general this can’t be recovered using software. If I am asked to handle a case like that I:

  • Open the ‘case’ and inspect the PCB using multimeter and microscope. If I can spot defects I’ll try repairing them, and if successful I can now access the card. Some times it’s enough to reflow the solder under the NAND chips. If a working and matching donor board is available, often NAND chips can be transferred to donor PCB.

  • If unable to repair I unsolder NAND chips and ‘dump’ them using a reader. Using specialized software I then convert the dump to a logical file system using software that emulates the controller (or tries as good as it can) from which files can be recovered. Ideally result is a coherent file system, however sometimes raw recovery is highest achievable. It’s the logical image reconstruction part that is most complex and time consuming.

NAND readers are not overly expensive (but still too for a single case probably), it’s the software that is needed to convert the raw dump to a logical image that is. I use the Soft Center reader and software (called Flash Extractor).

With regards to logical photo recovery software, often mentioned names are PhotoRec and Recuva, also free and very good is R-Photo made by the company that produces R-Studio which is used in many labs for logical data recovery.

If card (whatever card, CF, SD etc.) is detected but physical capacity is incorrect:

In these cases dumping the NAND is almost always required. Typically physical capacity is decreased to MBs rather than GBs. It’s an indication controller is working but in let’s call it a safe mode. It is either unable to detect the NAND memory -or- NAND translation table is corrupt. See this translation table as a ‘FAT’ for the firmware, only this table is used by firmware to map LBA addresses to actual NAND pages.

root access – Is there any way to force apps to use internal memory only? In Rooted Android device

I have oneplus 5t 8gb ram + 128 gb internal storage
100gb free space is available in phone.

I Rooted the device.
Now I am facing the problem regarding installing apps from Google play Store.

When I try to Install any random app.

Its showing

” App Requires External Storage”
“It cannot be downloaded.Insert an <a href=”https://support.google.com/android-one/answer/6088895?hl=en’>SD card or USB storage with sufficient storage, and try again.

And Also No Apps is Using the Internal Storage for storing the data like Camera , Whatsapp etc

I even not able to download and see the whatsapp images or any document shared and not able to download anything from chrome too.

If anyone can help me I will appreciate it and very thankful to him/her.