Fastest way to bulk merge mongodb collection

I have two MongoDB collections that I am merging.
Both of them have billions of records.

I want to nest collection within another collection.
I am going to traverse both Collections. Collection A has an array of objectId that reference Collection B.

Right now, I was looping it and updating data but it was taking weeks to update the whole dataset.
I am doing this for another dataset.

I am wondering if there is a faster way to do it.

Right now, I am updating data in batches of 5000 using BulkWrite API.

FFMPEG : Merge multiple image inputs over video input using ‘xfade’ filter

I’ve been trying to figure out the way to merge image inputs over one video.
Basically I have few images which I takes as input and then applies some filters and then merge it with video like below (multiple images with zoompan filter)

ffmpeg -i "inputs/image0.png" -i "inputs/image1.png" -i "inputs/image2.png" -i "inputs/image3.png" -i "inputs/background_video.mp4" -filter_complex "(4)split=2(color)(alpha);(color)crop=iw/2:ih:0:0(color);(alpha)crop=iw/2:ih:iw/2:0(alpha);(color)(alpha)alphamerge(ovrly);(0)scale=540:960,setsar=1(0_scaled);(1)scale=540:960,setsar=1(1_scaled);(2)scale=540:960,setsar=1(2_scaled);(3)scale=540:960,setsar=1(3_scaled);(0_scaled)scale=2700x4800,zoompan=z='min(zoom+0.0010,1.20)':x='iw/2-iw*(1/2-0/100)*on/201-iw/zoom/2':y='ih/2-ih*(1/2-0/100)*on/201-ih/zoom/2':d=25*8.04:s=540x960(v0);(1_scaled)scale=2700x4800,zoompan=z='min(zoom+0.0013,1.20)':x='iw/2-iw*(1/2-100/100)*on/151-iw/zoom/2':y='ih/2-ih*(1/2-0/100)*on/151-ih/zoom/2':d=25*6.04:s=540x960(v1);(2_scaled)scale=2700x4800,zoompan=z='min(zoom+0.0010,1.20)':x='iw/2-iw*(1/2-100/100)*on/201-iw/zoom/2':y='ih/2-ih*(1/2-100/100)*on/201-ih/zoom/2':d=25*8.04:s=540x960(v2);(3_scaled)scale=2700x4800,zoompan=z='min(zoom+0.0010,1.20)':x='iw/2-iw*(1/2-50/100)*on/201-iw/zoom/2':y='ih/2-ih*(1/2-50/100)*on/201-ih/zoom/2':d=25*8.04:s=540x960(v3);(v0)(v1)(v2)(v3)concat=n=4:v=1:a=0,format=yuv420p(concatenated_video);(concatenated_video)(ovrly)overlay=0:0" "outputs/zoomInTest.mp4"

I’m tried several similar ideas to get xfade working out but I’m getting issues mentioned below.

ffmpeg -loop 1 -t 8.04 -i "inputs/image0.png" -loop 1 -t 5.60 -i "inputs/image1.png" -loop 1 -t 8.36 -i "inputs/image2.png" -loop 1 -t 8.16 -i "inputs/image3.png" -i "inputs/background_video.mp4" -filter_complex "(4)split=2(color)(alpha);(color)crop=iw/2:ih:0:0(color);(alpha)crop=iw/2:ih:iw/2:0(alpha);(color)(alpha)alphamerge(ovrly);(0)(1)xfade=transition=circleopen:duration=1.00:offset=7.54(v0);(1)(2)xfade=transition=diagbl:duration=1.00:offset=13.14(v1);(2)(3)xfade=transition=slideright:duration=1.00:offset=21.00(v2);(3)fade=t=in:st=29.0:d=1.0(v3);(v0)(v1)(v2)(v3)concat=n=4:v=1:a=0,format=yuv420p(concatenated_video);(concatenated_video)(ovrly)overlay=0:0" "outputs/fadeTestNew.mp4" 

Below picture is reference of my process.
enter image description here

1.Output wasn’t being animated at all (solved)

I solved this issue by specifying -loop 1 -t 8.04 -i instead of only -i to all input images.

2.Filter wasn’t mapped because was unused etc. (solved)

I solved this issue by removing scale filter which was producing outputs like (0_scaled) which were unused.

3.xfade filter doesn’t work properly. (unsolved)

Watch above 2nd ffmpeg command which is the last update on my command. the thing is first two input image is being animated perfectly(like I want first input to be fade out by transition circle open at around 7.54s and it works well too but rest are just messed up and can’t describe what happens!

I found out that xfade filter requires 2x input so I could only use transition on first 3 input and for 4th image I used fade filter to fade it out on last second.

(TLDR): All I want is to use all this 4 image. transition them via xfade filter at different time like first transition at 8sec, 2nd at 14sec and 3rd at 22sec, and map them all on my background video and generate final output video!

Thanks!

go – Merge Intervals(Golang) – Code Review Stack Exchange

Solving the merge intervals problem in golang(link: https://leetcode.com/problems/merge-intervals/).

The Problem

Given an array of intervals where intervals(i) = (starti, endi), merge all overlapping intervals, and return an array of the non-overlapping intervals that cover all the intervals in the input.

 

Example 1:

Input: intervals = ((1,3),(2,6),(8,10),(15,18))
Output: ((1,6),(8,10),(15,18))
Explanation: Since intervals (1,3) and (2,6) overlaps, merge them into (1,6).
Example 2:

Input: intervals = ((1,4),(4,5))
Output: ((1,5))
Explanation: Intervals (1,4) and (4,5) are considered overlapping.

Code

package main

import (
    "fmt"
    "sort"
)

func main() {

    var intervals ()()int
    intervals = append(intervals, ()int{7, 10})
    intervals = append(intervals, ()int{3, 4})
    intervals = append(intervals, ()int{2, 5})

    mergeIntervals(intervals)

}

func mergeIntervals(intervals ()()int) ()()int {

    if len(intervals) < 2{
        return intervals
    }
    var mergedIntervals ()()int
    fmt.Println("intervals :", intervals)
    sort.Slice(intervals, func(i, j int) bool {
        return intervals(i)(0) < intervals(j)(0)
    })

    previousEndTime := intervals(0)(1)
    for index, _ := range intervals {

        if index >= len(intervals)-1 {

            if index == len(intervals)-1 {
                mergedIntervals = append(mergedIntervals, ()int{intervals(index)(0), intervals(index)(1)})
            }
            break
        }
        if index != 0 {
            //skip current interval if previous interval end time is higher
            if previousEndTime > intervals(index)(1) {
                continue
            }
        }
        fmt.Println("comparing intervals ", intervals(index), "and ", intervals(index+1))
        if intervals(index)(1) > intervals(index+1)(1) {
            //check if current end time greater than end time of next interval
            mergedIntervals = append(mergedIntervals, ()int{intervals(index)(0), intervals(index)(1)})
            previousEndTime = intervals(index)(1)
        } else if intervals(index)(1) >= intervals(index+1)(0) {
            //the actual merge for overlapping  values
            mergedIntervals = append(mergedIntervals, ()int{intervals(index)(0), intervals(index+1)(1)})
            previousEndTime = intervals(index+1)(1)
        } else {
            //just include current interval with no overlap
            mergedIntervals = append(mergedIntervals, ()int{intervals(index)(0), intervals(index)(1)})
            previousEndTime = intervals(index)(1)
        }
    }

    fmt.Println("merged intervals ", mergedIntervals)
    return mergedIntervals
}

Looking for

  • Code Bugs.
  • Feel this code is too verbose.Any suggestions on how to optimize
    the code further.

Merge Staff/Members Online Widgets

Admin submitted a new resource:

Merge Staff/Members Online Widgets – merges the staff/members online widgets!

I ran a modification that did this back on XF1 (pretty sure it was by @Russ) – but it never got ported.. until now! 🙂

This is original code and uses only the same idea.

Before:

(IMG…

Read more

.(tagsToTranslate)xenforo nulled(t)nulled(t)whmcs nulled(t)vbulletin nulled(t)whmcs themes nulled(t)whmcs modules nulled(t)whmcs template nulled(t)nulled forum(t)best nulled scripts site(t)whmcs templates nulled(t)phpfox nulled(t)nulled script(t)xenforo 2 nulled(t)best nulled sites(t)xenforo nulled themes(t)whmcs nulled(t)nulled scripts(t)vbulletin 5 nulled(t)whmcs addons nulled(t)arrowchat nulled(t)cs-cart nulled(t)xfilesharing pro nulled(t)nulled script forum(t)cs cart nulled(t)nulled sites(t)blesta nulled

Merge two python lists that have some overlapping region

I have several signals (the current one is delayed from the previous one). By properly compensating the delay, I can make an overlapping region between 2 successive signals. I want to create one signal by merging the two signals, so that overlapping region doesn’t count twice. And I want to do the same for others. It’s like creating a panorama with images.

First I’m searching how to merge two signals, so that I can iterate through a loop. This is the two signals with an overlapping region. Any idea of how to achieve this is highly appreciated.

I’m taking each signal from a large matrix of signals

image1_data = horzmat.iloc(100:101,:)
image2_data = horzmat.iloc(101:102,:)
image3_data = horzmat.iloc(102:103,:)
These are the two signals I'm looking to stitch

How to merge an image taken from four cameras placed in four corners of a room?

For my application, I need to

  1. place four cameras on the corners of a room
  2. Capture image from all 4 cameras
  3. Merge all four of them
  4. Finally if there are someone in room, i want to detect number of people accurately irrespective of how close they are and all.

Is it possible to do this using Jetson Multimedia board?

database – Merge migrations from two different servers

I’m working on a project that has two separate versions of the API, v1, and v2 because they wanted to migrate from Javascript to Typescript. Both versions use the same database.

Recently I’m having some trouble as one part of the migrations are in the v1, and the most recent updates are on version v2. I would like to have all the migration in the same place, how can this be done?

One of the problems I’m facing, for example, is that I would like to set up an in-memory DB for tests, but as I don’t have all the migrations in v2 I can’t do this.

Tech stack used in v1: javascript, node, sequelize.

Tech stack used in v2: typescript, node, typeorm.

sql server – How do I merge three update query into one single update query

I am trying to update a promotion table with a single update statement. Although the three update statements below can do it, I want to merge the three statements into one. How can I do that?

UPDATE promos SET next_promotion = (last_promotion + 4) WHERE grade_level IN (14,15,16,17,18)

UPDATE promos SET next_promotion = (last_promotion + 3) WHERE grade_level IN (7,8,9,10,11,12,13)

UPDATE promos SET next_promotion = (last_promotion + 2) WHERE grade_level IN (1,2,3,4,5,6)

c# – Linq query to merge overlapping elements

I have this class representing a period of time:

public class Period
{
    public Period(DateTime dateFrom)
    {
        DateFrom = dateFrom;
    }

    public Period(DateTime dateFrom, DateTime? dateTo)
    {
        DateFrom = dateFrom;
        DateTo = dateTo;
    }

    public Period() { }
    public DateTime DateFrom { get; set; }
    public DateTime? DateTo { get; set; }


    public bool IsOverlapping(Period other)
    {
        if (!DateTo.HasValue)
        {
            return DateFrom.Value <= other.DateTo.Value;
        }
        
        if (!other.DateTo.HasValue)
        {
            return other.DateFrom.Value <= DateTo.Value;
        }

        return DateFrom.Value <= other.DateTo.Value && other.DateFrom.Value <= DateTo.Value;
    }

    public bool IsFinite => DateTo.HasValue;

    public bool IsInfinite => !IsFinite;

    // Equals/GetHashCode have been overrided
}

Now a have a list of periods and for each of them I have to perform an network call so to minimize them I decide to merge all overlapping periods.

So a list like that :

  • 2020-01-01 -> 2020-01-10
  • 2020-02-05 -> 2020-02-10
  • 2020-02-07 -> 2020-02-15
  • 2020-02-13 -> 2020-02-20
  • 2020-03-01 -> 2020-03-10
  • 2020-03-25 -> 2020-03-31
  • 2020-03-30 ->

Should become :

  • 2020-01-01 -> 2020-01-10
  • 2020-02-05 -> 2020-02-20
  • 2020-03-01 -> 2020-03-10
  • 2020-03-25 ->

I tried this code

    periods.OrderBy(p => p.DateFrom.Value)
            .Aggregate(new List<Period>(), (ps, p) =>
            {
                if (!ps.Any())
                {
                    ps.Add(p);
                    return ps;
                }

                var last = ps.Last();
                if (last.IsOverlapping(p))
                {
                    if (last.IsInfinite || p.IsInfinite)
                    {
                        ps(ps.Count() - 1) = new Period(DateTimeHelpers.Min(last.DateFrom.Value, p.DateFrom.Value), null);
                    }
                    else
                    {
                        ps(ps.Count() - 1) = new Period(DateTimeHelpers.Min(last.DateFrom.Value, p.DateFrom.Value), DateTimeHelpers.Max(last.DateTo.Value, p.DateTo.Value));
                    }

                    return ps;
                }

                ps.Add(p);
                return ps;

            });

It’s working properly but I’m not satisfied with it so I wonder if there is a more performant/elegant/readable way to do it ?

I’m not talking about just refactoring to extract some methods but a fundamentally different solution, it’s possible I missed a useful LINQ operator.

Here is my test if you want to reproduce it (MsTest + FluentAssertions). The period are not in the right order to ensure it will be take care of by the method itself:

        // Arrange
        var periods = new List<Period>()
        {
            new Period(new DateTime(2020, 2, 13), new DateTime(2020, 2, 20)),
            new Period(new DateTime(2020, 3, 1), new DateTime(2020, 3, 10)),
            new Period(new DateTime(2020, 3, 25), new DateTime(2020, 3, 31)),
            new Period(new DateTime(2020, 3, 30)),
            new Period(new DateTime(2020, 1, 1), new DateTime(2020, 1, 10)),
            new Period(new DateTime(2020, 2, 5), new DateTime(2020, 2, 10)),
            new Period(new DateTime(2020, 2, 7), new DateTime(2020, 2, 15))
        };

        // Act
        var mergedPeriods = Implementation(periods);

        // Assert
        mergedPeriods.Should().HaveCount(4);
        mergedPeriods(0).Should().Be(new Period(new DateTime(2020, 1, 1), new DateTime(2020, 1, 10)));
        mergedPeriods(1).Should().Be(new Period(new DateTime(2020, 2, 5), new DateTime(2020, 2, 20)));
        mergedPeriods(2).Should().Be(new Period(new DateTime(2020, 3, 1), new DateTime(2020, 3, 10)));
        mergedPeriods(3).Should().Be(new Period(new DateTime(2020, 3, 25)));