python – optimize binary classification method for speed

I have the following code for determining TP, TN, FP and FN values for binary classification given two sparse vectors as input (using the sparse library):

def confused(sys1, ann1):
    # True Positive (TP): we predict a label of 1 (positive), and the true label is 1.
    TP = np.sum(np.logical_and(ann1 == 1, sys1 == 1))

    # True Negative (TN): we predict a label of 0 (negative), and the true label is 0.
    TN = np.sum(np.logical_and(ann1 == 0, sys1 == 0))

    # False Positive (FP): we predict a label of 1 (positive), but the true label is 0.
    FP = np.sum(np.logical_and(ann1 == 0, sys1 == 1))

    # False Negative (FN): we predict a label of 0 (negative), but the true label is 1.
    FN = np.sum(np.logical_and(ann1 == 1, sys1 == 0))
    
    return TP, TN, FP, FN

I’m trying to find a way to optimize this for speed. This is based on how-to-compute-truefalse-positives-and-truefalse-negatives-in-python-for-binary-classification-problems where my addition was to add the sparse arrays to optimize for memory usage, since the input vectors for the current problem I am trying to solve have over 7.9 M elements, and the positive cases (i.e., 1), are few and far between wrt the negative cases (i.e., 0).

I’ve done profiling of my code and about half the time is spent in this method.

topology – Classification of the Summit supercomputer

Can we classify a supercomputer in more than one group? For example, Flynn’s classification, classification according to topology, classification according to memory access.
For the Summit supercomputer, I have found no sources on these topics, except for Flynn’s classification. How can I classify it in other categories, for example by topology?

machine learning – Tweet Classification into topics- What to do with data

Good evening,
First of all, I want to apologize if the title is misleading.
I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into topics. I am working on topic modelling with LDA getting the right number of topics (I guess) thanks to this R package, which calculates the value of three metrics(“CaoJuan2009”, “Arun2010”, “Deveaud2014”). Since I am very new to this, I just thought about a few questions that might be obvious for some of you, but I can’t find online.

  1. I have removed, before cleaning the data (removing mentions, stopwords, weird characters, numbers etc), all duplicate instances (having all three columns in common), in order to avoid them influencing the results of topic modelling. Is this right?

  2. Should I, for the same reason mentioned before, remove also all retweets?

  3. Until now, I thought about classifing using the “per-document-per-topic” probability. If I get rid of so many instances, do I have to classify them based on the “per-word-per-topic” probability?

  4. Do I have to divide the dataset into testing and training? I thought that is a thing only in supervised training, since I cannot really use the testing dataset to measure quality of classification.

  5. Antoher goal would be to classify twitterers based the topic they most are passionate about. Do you have any idea about how to implement this?

Thank you all very much in advance.

automobiles – Roads classification in Spain

While I am looking for some nice roads in Spain for my next roadtrip, I noticed that the roads there have different numbera A-, B-, N- etc, some of them have even a letter at the end (C-1412a)

What is the meaning of all those letters and numbers? Does a bigger number mean a smaller road?

map of random region in Spain

python – Transfer learning CNNs for image classification in TensorFlow

This code works, and I’m pretty sure it’s mathematically/algorithmically correct:

model = tf.keras.applications.MobileNet(include_top=False,input_shape=INPUT_SHAPE)
model.trainable = False
model = tf.keras.Sequential((
    model,
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(
        1, "sigmoid"
    ),
))

try:
    model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
                  optimizer=tf.keras.optimizers.RMSprop(),
                  metrics=('accuracy', tf.keras.metrics.AUC(curve="PR")))

    model.fit(train_dataset,
              epochs=2,
              validation_data=validation_dataset)
finally:
    for directory in tempdirs:
        rmtree(directory)

For completion, here is my dataset setup and imports, I’m being very verbose and not using tf.data.Dataset.from_tensor_slices and/or builtin random dataset utility functions, to keep things like my real workflow:

from tempfile import mkdtemp
from os import path, mkdir
from shutil import rmtree
import tensorflow as tf
import numpy as np
from PIL import Image

NUM_CLASSES = 2
INPUT_SHAPE = 256, 256, 3


def create_random_ds(tempdir):
    for class_num in range(NUM_CLASSES):
        directory = path.join(tempdir, 'class{}'.format(class_num))
        mkdir(directory)
        for i in range(20):
            Image 
              .fromarray((np.random.rand(*INPUT_SHAPE) * 255).astype('uint8')) 
              .convert('RGBA') 
              .save(path.join(directory, 'image{}.png'.format(i)))

    return tf.keras.preprocessing.image_dataset_from_directory(
        tempdir, labels='inferred', label_mode='int',
    )

tempdirs = mkdtemp(), mkdtemp()

train_dataset = create_random_ds(tempdirs(0))
validation_dataset = create_random_ds(tempdirs(1))

What do you think, is it correct, and how can it be improved?

c++ – Difference between shader input element classification between D3D12 and Vulkan

I’m confused about the difference between the shader input element classifications in D3D12 and Vulkan. For example, in Vulkan I could have the following declarations:

struct vertex
{
    glm::vec3 pos;
    glm::vec3 col;
};

VkVertexInputBindingDescription input_binding_description
{
    .binding = 0,
    .stride = sizeof(vertex),
    .inputRate = VK_VERTEX_INPUT_RATE_VERTEX
};

std::array<VkVertexInputAttributeDescription, 2> input_attribute_descriptions {
    VkVertexInputAttributeDescription{
        .location = 0,
        .format = VK_FORMAT_R32G32B32_SFLOAT,
        .offset = offsetof(vertex, pos)
    },
        VkVertexInputAttributeDescription{
        .location = 1,
        .format = VK_FORMAT_R32G32B32_SFLOAT,
        .offset = offsetof(vertex, col)
    }
};

Here the input rate is specified per vertex and not per attribute. On the other hand, in D3D12, we would have

struct Vertex
{
    XMFLOAT3 pos;
    XMFLOAT3 col;
};

std::array<D3D12_INPUT_ELEMENT_DESC, 2> input_element_descs =
{
    { "POSITION", 0, DXGI_FORMAT_R32G32B32_FLOAT, 0, 0,
    D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0 },
    { "COLOR", 0, DXGI_FORMAT_R32G32B32_FLOAT, 0, 12,
    D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0 }
};

And as you can see, the input rate is specified per attribute. Why is that? Is there a difference between the meaning of the input classification in D3d12 and Vulkan that I’m missing? I’m not familiar with D3D12, but at first glance, it doesn’t make sense to me to have different input classifications for the attributes of a vertex.

computability – Classification of recursively enumerable sets

It’s quite easy to show that there are only three recursive sets up to m-equivalence, namely $emptyset,{1},mathbb N.$ Can we state something “similarly short” for recursively enumerable sets, like there are only 4 recursively enumerable sets up to m-equivalence – $emptyset,{1},K,mathbb N$ (where $K={imidvarphi_i(i)text{ is def.}}$)? More specifically, is every non-recursive recursively enumerable set m-equivalent to $K$?

best idea for vlan classification

I have a Server Room with about 40 servers. some servers are DB. Some of them is for web .
some of them need internet and some no.
what is the best idea for vlan classification them?
putting on DB sever and WEB sever in one Vlan is Dangerous؟؟
what is the best policy for vlan classification them?
thanks.

python – How to Use Class Weights with Focal Loss in PyTorch for Imbalanced dataset for MultiClass Classification

I am working on Multiclass Classification (4 classes) for Language Task and I am using the BERT model for classification task. I am following this blog as reference. My BERT Fine Tuned model returns nn.LogSoftmax(dim=1).

My data is pretty imbalanced so I used sklearn.utils.class_weight.compute_class_weight to compute weights of the classes and used the weights inside the Loss.

class_weights = compute_class_weight('balanced', np.unique(train_labels), train_labels)
weights= torch.tensor(class_weights,dtype=torch.float)
cross_entropy  = nn.NLLLoss(weight=weights) 

My results were not so good so I thought of Experementing with Focal Loss and have a code for Focal Loss.

class FocalLoss(nn.Module):
  def __init__(self, alpha=1, gamma=2, logits=False, reduce=True):
    super(FocalLoss, self).__init__()
    self.alpha = alpha
    self.gamma = gamma
    self.logits = logits
    self.reduce = reduce

  def forward(self, inputs, targets):
    BCE_loss = nn.CrossEntropyLoss()(inputs, targets)

    pt = torch.exp(-BCE_loss)
    F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss

    if self.reduce:
      return torch.mean(F_loss)
    else:
      return F_loss

I have 3 questions now. First and the Most important is

  1. Should I use Class Weight with Focal Loss?
  2. If I have to Implement weights inside this Focal Loss, can I use weights parameters inside nn.CrossEntropyLoss()
  3. If this implement is incorrect, what should be the proper code for this one including the weights (if possible)

algorithms – Classification and complexity of generating all possible combinations: P, NP, NP-Complete or NP-Hard

The algorithm needs to generate all possible combinations from a given list (empty set excluded).

list          =>         (1, 2, 3)
combinations  =>         ({1}, {2}, {3}, {1,2}, {1,3}, {2,3}, {1,2,3})

This algorithm would take O(2n) time complexity to generate all combinations. However, I’m not sure if an improved algorithm can bring this time complexity down. If an improved algorithm exists, please do share your knowledge!

In the case that it takes O(2n) which is exponential, I would like some insight regarding which class this algorithm belongs to P, NP, NP-Complete, NP-Hard, or whether it does not belong to any of them. Thanks in advance 🙂