numpy – Find pixel indices in a shape: Opencv and Python

Suppose I have a hollow, curved (and not necessarily convex) mask that I received from my pretreatment steps:

Hollow circle mask

I now want to try to select all the pixels that occur inside this shape and add them to the mask, as follows:

Filled circle mask

How can I do this in Python?


Code to generate the examples:

import cv2
import numpy as np
import matplotlib.pyplot as plt

# Parameters for creating the circle
COLOR_BLUE = (255, 0, 0)
IMAGE_SHAPE = (256, 256, 3)
CIRCLE_CENTER = tuple(np.array(IMAGE_SHAPE) // 2)(:-1)
CIRCLE_RADIUS = 30
LINE_THICKNESS = 5 # Change to -1 for example of filled circle

# Draw on a circle
img = np.zeros(IMAGE_SHAPE, dtype=np.uint8)
img_circle = cv2.circle(img, CIRCLE_CENTER, CIRCLE_RADIUS, COLOR_BLUE, LINE_THICKNESS)
circle_mask = img_circle(:, :, 0)

# Show the image
plt.axis("off")
plt.imshow(circle_mask)
plt.show()

Fill in the values ​​in the numpy array that are between a certain value

Let's say I have a table that looks like this:

a = np.array((0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0))

I want to fill in the values ​​between 1 and 1.
This would therefore be the desired output:

a = np.array((0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0))

I took a look at this answer, which gives the following:

array((0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
   1, 1)) 

I'm sure this answer is really close to the exit I want. However, although I have tried countless times, I cannot change this code to make it work as I want, as I am not very proficient with numpy arrays.
Any help is greatly appreciated!

numpy – Get rid of loops for slow in Python

I have created a small python script that will generate test sets for my project.

The script generates 2 datasets with the same dimensions n*m. One contains 0.1 binary values ​​and the other contains floats.

# Probabilities must sum to 1
AMOUNT1 = {0.6 : get_10_20,
           0.4 : get_20_30}

AMOUNT2 = {0.4 : get_10_20,
           0.6 : get_20_30}

OUTCOMES = (AMOUNT1, AMOUNT2)

def pick_random(prob_dict):
    '''
    Given a probability dictionary, with the first argument being the probability,
    Returns a random number given the probability dictionary
    '''
    r, s = random.random(), 0
    for num in prob_dict:
        s += num
        if s >= r:
            return prob_dict(num)()


def compute_trade_amount(action):
    '''
    Select with a probability, depending on the action.
    '''
    return pick_random(OUTCOMES(action))


ACTIONS = pd.DataFrame(np.random.randint(2, size=(n, m)))
AMOUNTS = CLIENT_ACTIONS.applymap(compute_trade_amount)

The script runs correctly and generates the output that I need, but if I want to scale to for many dimensions the for in loop pick_random() slows down my computation time.
How can I get rid of it? Maybe with some understanding of the table using numpy?

What throws my reasoning is the if-stmt. Because sampling has to happen with probability.

numpy – MNIST python machine learning dataset

I am a noob with machine learning and I have been struggling with this for a few days now and I don't understand why my neural network is having trouble classifying the mnist dataset. I checked my calculations and used the gradient check, but I can't seem to find the problem.

import pickle as pc
import numpy as np
import matplotlib.pyplot as mb
class MNIST:
#fix: gradient checking maybe not working, maybe backprop not working, symetrical updating, check if copying correctly


    def processImg(self):

        '''
        #slower than pickle file
        inTrset = np.loadtxt("mnist_train.csv", delimiter = ",");
        inTestSet = np.loadtxt("mnist_test.csv", delimiter = ",");
        fullX = np.asfarray(inTrset(:,1:))
        fullY = np.asfarray(inTrset(:, :1))
        '''

        with open("binaryMNIST.pkl", "br") as fh:
            data = pc.load(fh)


        img_dim = 28;
        features = 784;
        m = 60000
        test_m = 10000;

        fullX = (np.asfarray(data(0)))
        bias = np.ones((60000, 1))
        fullX = np.hstack((bias, fullX))

        fullY = np.asfarray(data(1))

        testX = (np.asfarray(data(2)))
        bias2 = np.ones((10000, 1))
        testX = np.hstack((bias2, testX))

        testY = np.asfarray(data(3))

        fullY = fullY.astype(int)
        testY = testY.astype(int)

        iden = np.identity(10, dtype = np.int)
        oneHot = np.zeros((m, 10), dtype = np.int)
        oneHot_T = np.zeros((test_m, 10), dtype = np.int)

        #creates m number of one, zeros vector indicating the class
        for i in range(test_m):
            oneHot_T(i) = iden(testY(i), :)

        for i in range(m):
            oneHot(i) = iden(fullY(i), :)

        trainX = fullX(:40000, :)
        trainY = oneHot(:40000, :)

        valX = np.asfarray(fullX(40000:, :))
        valY = np.asfarray(oneHot(40000:, :))


        self.trainX = trainX
        self.trainY = trainY
        self.valX = valX
        self.valY = valY
        self.testX = testX
        self.oneHot_T = oneHot_T


    def setThetas(self):
        #784 features
        #5 nodes per layer (not including bias)
        #(nodes in previous layer, nodes in next layer)
        #theta1(785, 5) theta2(6, 5) theta3(6, 10)

        #after finishing, do big 3d matrix of theta and vectorize backprop

        params = np.random.rand(4015)
        self.params = params



    def fbProp(self, theta1, theta2, theta3):

        #after calculating a w/sig(), add bias
        m = np.shape(self.trainY)(0)
        z1 = np.array(np.dot(self.trainX, theta1), dtype = np.float64)

        a1 = self.sig(z1)
        bias = np.ones((40000, 1))
        a1 = np.hstack((bias, a1))
        z2 = np.dot(a1, theta2)
        a2 = self.sig(z2)
        a2 = np.hstack((bias, a2))
        z3 = np.dot(a2, theta3)
        hyp = self.sig(z3)

        g3 = 0
        g2 = 0
        g1 = 0

        for i in range(m):
            dOut = hyp(i, :) - self.trainY(i, :)
            d2 = np.dot(np.transpose(dOut), np.transpose(theta3))
            d2 = d2(1:) * self.sigG(z2(i, :))
            d1 = np.dot(d2, np.transpose(theta2))
            d1 = d1(1:) * self.sigG(z1(i, :))

            g3 = g3 + np.dot(np.transpose(np.array(a2(i, :), ndmin = 2)), np.array(dOut, ndmin = 2))
            g2 = g2 + np.dot(np.transpose(np.array(a1(i, :), ndmin = 2)), np.array(d1, ndmin = 2))
            g1 = g1 + np.dot(np.transpose(np.array(self.trainX(i, :), ndmin = 2)), np.array(d1, ndmin = 2))

        self.theta1G = (1/m) * g1
        self.theta2G = (1/m) * g2
        self.theta3G = (1/m) * g3


    def gradDescent(self):

        params = np.array(self.params)
        theta1 = params(0:3925)
        theta1 = np.resize(theta1, (785, 5))
        theta2 = params(3925:3955)
        theta2 = np.resize(theta2, (6, 5))
        theta3 = params(3955:4015)
        theta3 = np.resize(theta3, (6, 10))

        for i in range(self.steps):
            J = self.error(theta1, theta2, theta3, self.trainX, self.trainY)
            print("Iteration: ", i+1, " | error: ", J)
            self.fbProp(theta1, theta2, theta3)
            theta1 = theta1 - (self.alpha * self.theta1G)
            theta2 = theta2 - (self.alpha * self.theta2G)
            theta3 = theta3 - (self.alpha * self.theta3G)



        #On test set
        correct = self.test(theta1, theta2, theta3)
        print(correct/100, "%")


    def error(self, params, X, y):
        theta1 = params(0:3925)
        theta1 = np.resize(theta1, (785, 5))
        theta2 = params(3925:3955)
        theta2 = np.resize(theta2, (6, 5))
        theta3 = params(3955:4015)
        theta3 = np.resize(theta3, (6, 10))


        bias = np.ones((np.shape(y)(0), 1))
        a1 = self.sig(np.dot(X, theta1))
        a1 = np.hstack((bias, a1))
        a2 = self.sig(np.dot(a1, theta2))
        a2 = np.hstack((bias, a2))
        hyp = self.sig(np.dot(a2, theta3))

        #10 classes
        pt1 = ((-np.log(hyp) * y) - (np.log(1-hyp) * (1-y))).sum()
        J = 1/(40000) * pt1.sum()

        return J


    def error(self, theta1, theta2, theta3, X, y):
        bias = np.ones((np.shape(y)(0), 1))
        a1 = self.sig(np.dot(X, theta1))
        a1 = np.hstack((bias, a1))
        a2 = self.sig(np.dot(a1, theta2))
        a2 = np.hstack((bias, a2))
        hyp = self.sig(np.dot(a2, theta3))
        print(hyp(0, :))

        #10 classes
        pt1 = ((np.log(hyp) * y) + (np.log(1-hyp) * (1-y))).sum()
        J = - (1/(40000)) * pt1.sum()

        return J



    #def validate(self):

    def test(self, theta1, theta2, theta3):
        X = self.testX
        y = self.oneHot_T
        bias = np.ones((np.shape(y)(0), 1))
        a1 = self.sig(np.dot(X, (theta1)))
        a1 = np.hstack((bias, a1))
        a2 = self.sig(np.dot(a1, (theta2)))
        a2 = np.hstack((bias, a2))
        hyp = self.sig(np.dot(a2, (theta3)))

        correct = 0
        ans = np.array((0, 1, 2, 3, 4, 5, 6, 7, 8, 9))

        for i in range(np.shape(y)(0)):
            #fix backprop and forward prop then this
            guess = np.argmax(hyp(i, :))
            match = np.argmax(y(i, :))
            print("guess: ", guess, "| ans: ", match)
            if guess == match:
                correct = correct + 1;

        return correct



    def gradientCheck(self):
        params = np.array(self.params)
        theta1 = params(0:3925)
        theta1 = np.resize(theta1, (785, 5))
        theta2 = params(3925:3955)
        theta2 = np.resize(theta2, (6, 5))
        theta3 = params(3955:4015)
        theta3 = np.resize(theta3, (6, 10))
        self.fbProp(theta1, theta2, theta3)

        grad = self.theta1G.ravel()
        grad = np.append(grad, self.theta2G.ravel())
        grad = np.append(grad, self.theta3G.ravel())


        print("got grads")
        epsilon = 0.00001

        params2 = np.array(self.params)
        check = np.zeros(np.shape(params))
        for i in range(3965, np.size(params)):
            temp = params(i)
            params(i) = params(i) + epsilon
            params2(i) = params2(i) - epsilon
            check(i) = (self.error(params, self.trainX, self.trainY) - self.error(params2, self.trainX, self.trainY)) / (2 * epsilon)
            params(i) = temp
            params2(i) = temp
            print(grad(i), " ", check(i))



    def sigG(self, z):
        return (self.sig(z) * (1-self.sig(z)))


    def sig(self, z):
        return 1/(1+(np.exp(-z)))


    def printPictures(self):
        #number of training examples to iterate over

        for i in range(3):
            img = self.trainX(i, 1:).reshape((28,28))
            mb.title('Digit = {}'.format(np.argmax(self.trainY(i,:))))
            mb.imshow(img, cmap = 'gray_r')
            mb.show()



    def __init__(self, steps, alpha, nodes, h_layers):
        self.steps = steps
        self.alpha = alpha
        self.nodes = nodes
        self.h_layers = h_layers



obj = MNIST(100, 0.1, 5, 1);
obj.processImg();
obj.setThetas();
obj.gradDescent()

#obj.gradientCheck()
#obj.printPictures()

python – improved speed of this diffraction calculator based on numpy

I will simulate diffraction patterns of a normal incident Gaussian profile beam from a 2D network of point diffusers with a height distribution.

The 2D table of diffuser positions X, Y and Z each has a size N x N and these are summarized in each call to E_ab(a, b, positions, w_beam). It's done M x M times to build the diffraction model.

If I estimate ten floating point operations per dispersion site per pixel and one nanosecond per flop (which my laptop does for numpy small matrices), I would expect time to be 10 M^2 N^2 1E-09 seconds. For the small N, this runs a factor of 50 or 100 slower than that, and for the large N (bigger than say 2000), it slows down even more. I guess it has something to do with the pagination of the large paintings in memory.

What can i do to increase the speed of fat N?

Note: Right now, the height variation Z is random, in the future I plan to also include an additional systematic pitch variation term, so even if the purely Gaussian variation might have an analytical solution, I have to do it numerically.


Since I randomly distribute the height of pZ here, the plots will be a little different each time. My output (run on a laptop) is this, and I can't even begin to understand why it takes longer (~ 16 seconds) when w_beam is small only when it is large (~ 6 seconds).

My estimator 10 M^2 N^2 1E-09 suggests 0.25 seconds, these are about 50 times slower, so there can be substantial room for improvement.

1 16.460583925247192
2 14.861294031143188
4 8.405776023864746
8 6.4988932609558105

Python script:

import numpy as np
import matplotlib.pyplot as plt
import time

def E_ab(a, b, positions, w_beam):
    X, Y, Z = positions
    Rsq = X**2 + Y**2
    phases = k0 * (a*X + b*Y + (1 + np.sqrt(1 - a**2 - b**2))*Z)
    E = np.exp(-Rsq/w_beam**2)  * np.exp(-j*phases)
    return E.sum() / w_beam**2 # rough normalization

twopi, j = 2*np.pi, np.complex(0, 1)

wavelength = 0.08
k0  = twopi/wavelength

z_noise = 0.05 * wavelength

N, M = 100, 50
x = np.arange(-N, N+1)
X, Y = np.meshgrid(x, x)
Z = z_noise * np.random.normal(size=X.shape) # use random Z noise for now
positions = (X, Y, Z)

A = np.linspace(0, 0.2, M)

answers = ()
for w_beam in (1, 2, 4, 8):
    E = ()
    tstart = time.time()
    for i, a in enumerate(A):
        EE = ()
        for b in A:
            e = E_ab(a, b, positions, w_beam)
            EE.append(e)
        E.append(EE)
    print(w_beam, time.time() - tstart)
    answers.append(np.array(E))

if True:
    plt.figure()
    for i, E in enumerate(answers):
        plt.subplot(2, 2, i+1)
        plt.imshow(np.log10(np.abs(E)), vmin=0.0001)
        plt.colorbar()
    plt.show()

enter description of image here

algorithm – Efficiently move items to the back of a python / numpy array

I'm looking for a way to make this code more efficient, using any way possible.

In (1): def move_to_back(l, value):
   ...:     total_count = l.count(value)
   ...: 
   ...:     i = 0
   ...:     while i != total_count:
   ...: 
   ...:         if l(i) == value:
   ...:             l.pop(i)
   ...:             l.append(value)
   ...: 
   ...:         i += 1
   ...:     return l
   ...:

In (2): l = (24,24,24,1,2,5,2)

In (3): move_to_back(l,24)
Out(3): (1, 2, 5, 2, 24, 24, 24)

In (4): %timeit move_to_back(l, 24)
427 ns ± 5.32 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

If anyone knows how this could be improved in any way (using external libraries is 100% right for me), this would be greatly appreciated.

python – Numpy – how to assign a value to indices of different dimensions?

Suppose I have a matrix and some clues

a = np.array(((1, 2, 3), (4, 5, 6)))
a_indices = np.array(((0,2), (1,2)))

Is there an effective way to perform the following operation?

for i in range(2):
    a(i, a_indices(i)) = 100

# a: np.array(((100, 2, 100), (4, 100, 100)))

python – replaces a custom value when a null value is encountered in the Numpy array

I need to replace a custom value when a null value is encountered in the Numpy table
I found something like this, but it gives incorrect results.

def func(string, replace_with):
string(np.where(string == "")) = replace_with
return string

for

func(np.array(("1","2" ,"", "12", "")),"100")

his gift

('1' '2' '10' '12' '10') 

instead of

('1' '2' '100' '12' '100')

Understand the python and numpy syntax to hide and filter the array

Please understand how the 2 lines below work. Although I understand the code, I can not write it intuitively.
1.How does a pair of parentheses create an array, then the individual elements go through the logical state check and create a new array.
2. How to package the filter elements in square brackets to create another sub-array

If it is difficult to explain it completely, do not hesitate to indicate additional reading material.

import numpy as np
my_vector = np.array ((- 17, 4, 0, 2, 21, 37, 105))
mon_vecteur

array((-17,   4,   0,   2,  21,  37, 105))

zero_mod_7_mask = 0 == (my_vector% 7) # First question refers to this line

zero_mod_7_mask

array((False, False,  True, False,  True, False,  True))

my_subarray = my_vector (zero_mod_7_mask) # Second question refers to this line
my_subarray

array((  0,  21, 105))

Python: Lack of Memory When You Try to Convert Images to Numpy Arrays

I have a list of about 15,000 images of 120 x 90 pixels high and wide. I'm trying to convert them to a Numpy table, but when I try to convert them, my computer runs out of memory (8 GB of swap + 12 GB). Once done, I will save to a file for future machine learning training.

dataSet = genDataSet()
for image in dataSet:
    pixelImages.append((imageToRGB(image(0), True),image(1)))

def imageToRGB(inputFile, normalise = False):
    os.chdir("/home/spchee/CodeProjects/School Project/images")
    img = Image.open(inputFile) #Opens File
    pixels = np.asarray(img) #Converts it to a numpy array
    pixels = np.rint(pixels)
    if normalise: #This normalises it between the values of 0 and 1
        pixels= pixels/255
    img.close()
    return pixels

The function genDataSet () returns a list in the form of ((filepath1, genre), (filepath2, genre) …).

When you run this code, it runs out of memory, my computer crashes almost completely and I am forced to forcibly stop it.