algorithms – Function replication – Stack Exchange software engineering

You are given an arbitrary function f. You can invoke it with arbitrary parameters and read the return values, as well as note the exceptions that it triggers, but you can not read its implementation.

How would you like to write a function g which reproduces fImplementation?

Note that by "function" I do not want to restrict f to be a mathematical function; it can work on any type of data, not just numbers. "Program" can be a better word for that. However, if this makes it easier to think about the problem, it may be helpful to start thinking only in terms of numbers.

Also note that I do not think certainty / proof of correction is possible here; only error correction is.

postgresql 10 – Why the pg_wal directory does not clean up during logical replication

My main question – Why pg_wal dir grows extremely?

I have two postgresql instances.

The first one was created as a master:

wal_level: "logical"
max_wal_senders: 5
max_replication_slots: 10

In addition, I enabled archiving for pg_wal:

archive_mode: on
archive_command: 'gzip < %p > /.../data/pg_wal_archive/%f'

After that, PUBLICATION was created for some of my tables (not ALL TABLES)

For replication psql instance – I use the default psql configuration, created only SUBSCRIPTION control.

And I have two questions:

  1. I run a script on the master, which pulls ~ 10 Gb of data to a table, which was not included in the PUBLICATION. And I see that my pg_wal the size of the dir increases anyway. Why?

  2. Why does not pg_wal dir clean up? Is this normal? Looks like I have data for the restore table that does not use replication, how to delete that data?

mongodb – Connectivity issues in the replication process

I have tried to add a new node to a replica set without success because of these errors:

2019-08-12T06: 48: 15.031 + 0200 I ASIO (NetworkInterfaceASIO-RS-0)
End of the connection to the host mongo03: 27017 due to poor connection status;
1 connections with this host remain open 2019-08-12T06: 48: 15.031 + 0200 I
REPL (replication-224) Reboot the oplog request after an error:
NetworkInterfaceExceededTimeLimit: Error in the recovery batch callback:
The operation has expired. Last recovered optime (with hash): {ts:
Timestamp (1565585283, 1317), t: 9} (1741433982986566690). restarts
remaining: 10 2019-08-12T06: 48: 15.032 + 0200 I REPL
(replication-224) New scheduled oplog request fetcher source:
mongo03: 27017 database: local query: {find: "oplog.rs", filter: {ts:
{$ gte: Timestamp (1565585283, 1317)}}, tailable: true, oplogReplay:
true, waitData: true, maxTimeMS: 2000, batchSize: 13981010, duration: 9,
readConcern: {afterOpTime: {ts: Timestamp (1565585283, 1317), t: 9}
}} Query metadata: {$ replData: 1, $ oplogQueryData: 1,
$ readPreference: {mode: "secondaryPreferred"}} active: 1
findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms closing
down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler
request: Remote control 2262327 – target: mongo03: 27017 db: local cmd: {
find: "oplog.rs", filter: {ts: {$ gte: Timestamp (1565585283, 1317)}
}, tailable: true, oplogReplay: true, waitData: true, maxTimeMS:
2000, batchSize: 13981010, term: 9, readConcern: {afterOpTime: {ts:
Timestamp (1565585283, 1317), t: 9}}} active: 1
callbackHandle.valid: 1 callbackHandle.cancelated: 0 attempts: 1
retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis: -1ms
2019-08-12T06: 48: 17.637 + 0200 I ASIO (NetworkInterfaceASIO-RS-0)
Connection to mongo03: 27017 2019-08-12T06: 48: 17.645 + 0200 I ASIO
(NetworkInterfaceASIO-RS-0) successfully connected to mongo03: 27017,
took 8ms (2 connections now open for mongo03: 27017)
2019-08-12T06: 49: 36.961 + 0200 I – (worker writer replica 13)
Progress of the clone of the collection DB.CollectionName: 221390504/643329790
34% (copied documents) 2019-08-12T06: 51: 52.859 + 0200 I ASIO
(NetworkInterfaceASIO-RS-0) End of the connection to the host mongo03: 27017
due to a bad connection status; 1 connections to this host remain open
2019-08-12T06: 51: 52.859 + 0200 I REPL (replication-229) Restart
Oplog request because of an error: NetworkInterfaceExceededTimeLimit: error in
Batch fetcher reminder: the operation has expired. Last recovered optime (with
hash): {ts: Timestamp (1565585502, 204), t: 9} (3285716995067894969).
Remaining Restart: 10 2019-08-12T06: 51: 52.859 + 0200 I ASIO
(NetworkInterfaceASIO-RS-0) Connection to mongo03: 27017
2019-08-12T06: 51: 52.859 + 0200 I REPL (replication-229) scheduled
new query oplog Fetcher source: mongo03: 27017 database: local query: {
find: "oplog.rs", filter: {ts: {$ gte: Timestamp (1565585502, 204)}
}, tailable: true, oplogReplay: true, waitData: true, maxTimeMS:
2000, batchSize: 13981010, term: 9, readConcern: {afterOpTime: {ts:
Timestamp (1565585502, 204), t: 9}}} Query metadata: {$ replData: 1,
$ oplogQueryData: 1, $ readPreference: {mode: "secondaryPreferred"}}
active: 1 findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms
stop ?: 0 first: 1 firstCommandScheduler:
Request RemoteCommandRetryScheduler: RemoteCommand 2272962 –
target: mongo03: 27017 db: local cmd: {find: "oplog.rs", filter: {ts: {
$ gte: Timestamp (1565585502, 204)}}, tailable: true, oplogReplay:
true, waitData: true, maxTimeMS: 2000, batchSize: 13981010, duration: 9,
readConcern: {afterOpTime: {ts: Timestamp (1565585502, 204), t: 9}}
} active: 1 callbackHandle.valid: 1 callbackHandle.cancelated: 0
attempt: 1 retryPolicy: RetryPolicyImpl maxAttempts: 1 maxTimeMillis:
-1ms

I am a little lost as to how to debug it.
This is a 250 GB database, the synchronization is done via a 100 MB private network. There is an iptables in all the servers but the private network traffic is allowed:

-A INPUT -i eth1 -p tcp -m tcp --dport 27017 -m state --state NEW,ESTABLISHED -j ACCEPT

At some point, the replication process fails:

2019-08-12T02: 22: 37.582 + 0200 I REPL (replication-143) Restart
Oplog request because of an error: ExceededTimeLimit: error in the recovery batch
callback: runtime error while searching for command :: caused by :: errmsg:
"Operation exceeded the time limit". Last recovered optime (with hash): {
ts: Timestamp (1565569328, 710), t: 9} (7908135344534364729). restarts
remaining: 1 2019-08-12T02: 22: 37.582 + 0200 I REPL (replication-143)
Planned source for the new oplog request Fetcher: mongo03: 27017
local query: {find: "oplog.rs", filter: {ts: {$ gte:
Timestamp (1565569328, 710)}}, tailable: true, oplogReplay: true,
waitData: true, maxTimeMS: 2000, batchSize: 13981010, duration: 9,
readConcern: {afterOpTime: {ts: Timestamp (1565569328, 710), t: 9}}
} query metadata: {$ replData: 1, $ oplogQ ueryData: 1,
$ readPreference: {mode: "secondaryPreferred"}} active: 1
findNetworkTimeout: 7000ms getMoreNetworkTimeout: 10000ms closing
down?: 0 first: 1 firstCommandScheduler: RemoteCommandRetryScheduler
request: Remote control 1381614 – target: mongo03: 27017 db: local cmd: {
find: "oplog.rs", filter: {ts: {$ gte: Timestamp (1565569328, 710)}
}, tailable: true, oplogReplay: true, waitData: true, maxTimeMS:
2000, baSchSize: 13981010, term: 9, readConcern: {afterOpTime: {ts:
Timestamp (1565569328, 710), t: 9}}} active: 1 callbackHandle.valid:
1 callbackHandle.cancelled: 0 retryPolicy: RetryPolicyImpl
maxAttem points: 1 maxTimeMillis: -1ms 2019-08-12T02: 22: 39.612 + 0200 I
REPL (replication-142) Error returned by an oplog request (no more)
Query restarts on the left): ExceededTimeLimit: Error in the recovery batch
callback: runtime error while searching for command :: caused by :: errmsg:
"Operation exceeded in the allotted time" 2019-08-12T02: 22: 39.615 + 0200 I REPL
(replication-142) The oplog extraction completed during the initial synchronization:
ExceededTimeLimit: Error in the batch reminder of the recovery tool: executor error
during the find :: cause command :: errms g: "operation exceeded
limit. "Last recovered optime and hash: {ts: Timestamp (1565569328,
710), t: 9} (7908135344534364729) 2019-08-12T02: 22: 43.034 + 0200 I REPL
(replication-142) CollectionCloner ns: DB.CollectionName completed
cloning with status: IllegalOperation: AsyncResultsMerger killed
2019-08-12T02: 22: 44.551 + 0200 W REPL Collection (replication-142)
The clone of & # 39; DB.CollectionName & # 39; failed because of IllegalOperation: While
cloning the collection & # 39; DB.CollectionName & # 39; there was a mistake
& # 39; AsyncResultsMerger killed & # 39; 2019-08-12T02: 22: 44.551 + 0200 I REPL
(replication-142) CollectionCloner :: called start, on ns: DB.CollectionName2
2019-08-12T02: 22: 44.551 + 0200 W REPL database (replication-142)
The clone 'dbName & # 39; (9 out of 9) failed due to ShutdownInProgress collection
cloner completed 2019-08-12T02: 22: 44.551 + 0200 I REPL
(replication-142) Completed clone data: ShutdownInProgress:
clone collection completed. Beginning of the oplog replay.
2019-08-12T02: 22: 44.551 + 0200 I REPL (replication-142) Initial sync
try to finish.

And that must start again.

I do not know if I could set a longer timeout to avoid these disconnections.

Thank you in advance.

mysql – Replication of the MariaDB slave has stopped after the slave upgrade to 10.1.41

After the recent update of MariaDB 10.1.41, which took place on 01 / Aug / 2019, few of our slaves stopped synchronizing the master's relay logs, which is on 10.1.40. Slaves automatically updated to 10.1.41 now have the following status. The slave I / O thread is in a state of readiness and the logs are not written. I've redone a slave and initiated a master change with the positions but still getting the same. The other slave servers contained in MariaDB 10.1.40, 10.1.33, etc. operate normally. Those who were upgraded to 10.1.41 have a problem?

Does anyone have an idea about this?

MariaDB ((none))> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: NULL
                  Master_Host: a.b.c.d
                  Master_User: xxxxxxxx
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: xxxxxxxxxxxxxx-bin.001135
          Read_Master_Log_Pos: 262316421
               Relay_Log_File: xxxxxxxxxxxxxxxxxxxxx-bin.003410
                Relay_Log_Pos: 4
        Relay_Master_Log_File: xxxxxxxxx-bin.001135
             Slave_IO_Running: Preparing
            Slave_SQL_Running: Yes
              Replicate_Do_DB: xxxxxxxxx
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 262316421
                        Relay_Log_Space: 498
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 111
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
                   Using_Gtid: No
                  Gtid_IO_Pos: 
      Replicate_Do_Domain_Ids: 
  Replicate_Ignore_Domain_Ids: 
                Parallel_Mode: conservative

mariadb – MySQL Replication Stuck During Preparation

My replication of MariaDB was blocked during the "preparation".
This occurs after an update and restart of the server yesterday. Since then, replication does not work.

Configuration: 2 MariaDB servers with master-master replication. Version: 10.4 (Updated a few weeks ago from 10.1 – replication worked after upgrade)

Since the output of both servers for SHOW SLAVE STATUS is similar, I publish only from my first server:

MariaDB ((none))> show slave statusG;
*************************** 1. row ***************************
                Slave_IO_State: NULL
                   Master_Host: data2
                   Master_User: replusr
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: mariadb-bin.018865
           Read_Master_Log_Pos: 14137515
                Relay_Log_File: relay-bin.017274
                 Relay_Log_Pos: 4
         Relay_Master_Log_File: mariadb-bin.018865
              Slave_IO_Running: Preparing
             Slave_SQL_Running: Yes
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table: 
                    Last_Errno: 0
                    Last_Error:
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 14137515
               Relay_Log_Space: 768
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: No
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master: NULL
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 0
                 Last_IO_Error:
                Last_SQL_Errno: 0
                Last_SQL_Error:
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 2
                Master_SSL_Crl:
            Master_SSL_Crlpath:
                    Using_Gtid: No
                   Gtid_IO_Pos:
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: conservative
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
              Slave_DDL_Groups: 0
Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 0
1 row in set (0.000 sec)

ERROR: No query specified

The logs do not display anything, and the binary files explode because they can not be rotated because the replication will not continue.

I have never seen anything like it. Ideas?

SQL Server – Transactional Replication: Problems with the Identity Column in the Subscriber

I have a transactional replication configured from Server A to Server B.
I have a table that has an identity column as the primary key.
In my editor, for the table tb1, the identity column,Id1 was defined as "Not for Replication" like yes.

I had the habit of receiving the message of error "Cannot insert explicit value for idntity column in table tb1 when IDENTITY_INSERT is set to OFF"
But when I checked my subscriber, I saw that the same column defined as "Not for Replication" as n °
I have therefore used the script below to modify the setting of my subscriber TB1.

alter table [dbo].[tb1] alter column [Id1] add not for replication; 

And the "Not for Replicaiton" is also set to "Yes" in my subscriber.

Thus, regardless of the Id1 value of the source, it must also be replicated in the subscriber table.

But now I'm getting the error message below
"Explicit value must be specified for identity column in table tb1 when IDENTOTY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identtity column"

It looks like the identity value explained in Id1 is not passed on by the publisher.

Any suggestions on this.

mysql – Replication of the MariaDB slave has stopped after the slave upgrade to 10.1.41

After the recent update of MariaDB 10.1.41, which took place on 01 / Aug / 2019, few of our slaves stopped synchronizing the master's relay logs, which is on 10.1.40. Slaves automatically updated to 10.1.41 now have the following status. The slave I / O thread is in a state of readiness and the logs are not written. I did a reset slave and initiated a master change with the positions again, but still get the same. The other slave servers contained in MariaDB 10.1.40, 10.1.33, etc. operate normally. Those who were upgraded to 10.1.41 have a problem?

Does anyone have an idea about this?

MariaDB ((none))> show slave statusG
*************************** 1. row ***************************
               Slave_IO_State: NULL
                  Master_Host: a.b.c.d
                  Master_User: xxxxxxxx
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: xxxxxxxxxxxxxx-bin.001135
          Read_Master_Log_Pos: 262316421
               Relay_Log_File: xxxxxxxxxxxxxxxxxxxxx-bin.003410
                Relay_Log_Pos: 4
        Relay_Master_Log_File: xxxxxxxxx-bin.001135
             Slave_IO_Running: Preparing
            Slave_SQL_Running: Yes
              Replicate_Do_DB: xxxxxxxxx
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 262316421
                        Relay_Log_Space: 498
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 111
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
                   Using_Gtid: No
                  Gtid_IO_Pos: 
      Replicate_Do_Domain_Ids: 
  Replicate_Ignore_Domain_Ids: 
                Parallel_Mode: conservative

Replication mysql – How to build a multi-region API server without database conflict

Currently, I'm building a web application that has a global audience, no specific region. To reduce latency, I've followed the usual steps of having a cache in Redis memory, local caching in the browser, but I remain concerned about overall latency.

I have an API server built in PHP that communicates with clients and with a MySQL database that is the current source of the truth.

I use my API server and my database on two DigitalOcean droplets in the SG region (where I am), but for a user located in the United States or the United Kingdom, this latency would have been higher. How can I distribute my servers so that there is more local access for other regions, while maintaining a database without any conflicts or duplication of data?

replication – Does a read replica help if the problem is for heavy queries?

The application that our team is responsible for has more and more problems with database performance. The application that queries it does not generate many simultaneous queries, the problem we encounter is rather heavy queries joining a few tables. These queries are executed every few minutes and their execution rarely overlaps. However, they sometimes cause the database to choke, and as the system contains more data, the problem continues to grow. Until now, the solution has been to optimize these queries, for example by trying to extract fewer fields and to scale the database.

I am relatively new to the team and today, I asked if a replica in reading had been considered. I do not know much about DBs, so it seemed natural to me, but a senior engineer said that it would barely help, because the replica in reading would pose the same problems as the master: it would still have to do the same writings as the masters, and the heavy demands would be just as heavy for him and the chances of his expiration, the same. The only gain would be the amount of them each instance should serve.

Said like that, it looks very reasonable. My questions are:

  1. Is a reading replica really no help in this situation?
  2. What are the alternatives?

PS: in case it matters, it's PostgreSQL.

Replication – The RDS MySQL replica database is no longer updated. Bonus: Tips for AWS beginners?

Hello everyone and thank you for your help, I'm in trouble.

I've been on a project to replace for someone who is overwhelmed by the weather, so I'm not very familiar with the setup of everything and I'm not yet an AWS expert.

We have a project on a restricted budget with a production instance used for collecting data from forms. We have a replica of this instance used exclusively for the subsequent execution of large queries on this data. After making a few changes to a view and changing the size of the root instance, the replicate database stopped synchronization / updating.

I have updated the replica database so that it has the same storage. I hope that this will solve the problem, but I could help you if you can not debug it. Does any one have a good crash course on how to … learn what someone did when he started a project like this with little d & # 39; integration?

If I were asked to set up a new project, I could do it with little reading, but it does not use Elastic Beanstalk, and I can not ask the person who set it up. So where is the right place to start learning what this thing is actually doing?