code review – Robust PCA (better Mathematica style)

I’ve parsed (almost verbatim) Python RPCA implementation to WM.
Can it be rewritten using a better WM style? In particular, I’m not happy with Clip(..., {0, Infinity}) and While() loop (can be replaced with Do()).

ClearAll(shrink) ;
shrink(matrix_, tau_) := Sign(matrix)*Clip(Abs(matrix) - tau, {0, Infinity})

ClearAll(threshold) ;
threshold(matrix_, tau_) := Block(
    {u, s, v},
    {u, s, v} = SingularValueDecomposition(matrix) ;
    Dot(u, Dot(shrink(s, tau), Transpose(v)))
) ;

ClearAll(rpca) ;
rpca(matrix_?MatrixQ, mu_Real, lambda_Real, tolerance_Real, limit_Integer) := Block(
    {inverse, count, error, sk, yk, lk},
    inverse = 1.0/mu ; 
    count = 0 ;
    sk = yk = lk = ConstantArray(0.0, Dimensions(matrix)) ;
    While(
        count < limit,
        lk = threshold(matrix - sk + inverse*yk, inverse) ;
        sk = shrink(matrix - lk + inverse*yk, inverse*lambda) ;
        error = matrix - lk - sk ;
        yk = yk + mu*error ;
        error = Norm(error, "Frobenius") ;
        count++ ;
        If(error < tolerance, Break()) ;
    ) ;
    {lk, sk, {count, error}}
) ;

Example:

(* https://github.com/dganguli/robust-pca *)
(* "12.1.1 for Linux x86 (64-bit) (June 19, 2020)" *)

(* generate test matrix *)

n = 100 ;
num$groups = 3 ;
num$values = 40 ;
matrix = N(ConstantArray(Flatten(Transpose(ConstantArray(10*Range(num$groups), num$values))), n)) ;
{n, m} = Dimensions(matrix)

(* set selected elements to zero *)

SeedRandom(1) ;
ln = RandomInteger({1, n}, 20) ;
lm = RandomInteger({1, m}, 20) ;
Table(matrix((ln((i)), lm((i)))) = 0.0, {i, 1, 20}) ;
matrix = Developer`ToPackedArray(matrix) ;

(* -- python zeros
ln = (81, 15, 1, 68, 4, 66, 24, 98, 69, 75, 16, 25, 5, 91, 84, 71, 2, 31, 49, 26)
lm = (45, 74, 107, 70, 57, 48, 29, 69, 27, 69, 11, 87, 77, 44, 34, 45, 87, 94, 19, 39)
for x, y in zip(ln, lm):
    D(x, y) = 0
*)

(* set parameters *)
mu = 1/4*1/Norm(matrix, 1)*Apply(Times, Dimensions(matrix)) ;
lambda = 1/Sqrt(N(Max(Dimensions(matrix)))) ;
tolerance = 10.0^-7*Norm(matrix, "Frobenius") ;
limit = 1000 ;

(* rpca *)
result = rpca(matrix, mu, lambda, tolerance, limit) ;

(* # of iterations and error *)
Last(result)

(* low rank *)
+Table(result((1))((ln((i)), lm((i)))), {i, 1, 20})

(* sparse *)
-Table(result((2))((ln((i)), lm((i)))), {i, 1, 20})

(* {100, 120} *)
(* {39, 0.000167548} *)
(* {20., 20., 30., 20., 20., 20., 10., 20., 10., 20., 10., 23.0449, 20., 20., 10., 20., 23.0449, 30., 10., 10.} *)
(* {20., 20., 30., 20., 20., 20., 10., 20., 10., 20., 10., 23.0449, 20., 20., 10., 20., 23.0449, 30., 10., 10.} *)

linear algebra – Robust estimation of $Ax=b$

Problem setting
$min_{x} |Ax-b|$, where $A in mathcal{R}^{m times n}, m>>n $, full rank.

L1 loss is used for robust estimation using IRLS. The corresponding equation to solve turns out to be $ A^{T}WAx=A^{T}Wb$, where $W=diag(d_i), d_i=1/|e_{i}|$, $e_{i}=a_{i}^{T}x-b_{i}$, $a_{i}$ is the ith row of $A$, $b_{i}$ is the ith element of $b$. For $e_{i}$ close to $0$, the value of $d_{i}$ is very large. For my specific case, the range of $d_i$ is from $10^{-3}$ to $10^5$.

To avoid high values of $d_i$, it is taken as $d_i=1/(|e_i|+delta)$ where $delta>0$ is a small number near $0$. Let $delta=10^{-3}$. This brings the range of $d_i$ as $10^{-3}$ to $10^3$. The range of values of $d_i$ is still high to bring numerical instability. It makes
$ A^{T}WA$ near singular matrix.

Please suggest a way to avoid numerical instability.

Thanks in advance!

matrix – Doubt about notation in Robust Optimization

I’m studying the pricing model under Robust Programming here described from page 41 to page 45. I don’t understand what’s the meaning of subscripts $i$ and “second” $t$ referred to dual variables of model. As a comparison, take the model at page 38: dual variable $p_{1,t}$ (included in the second constraint of case I) refers to the set $begin{Bmatrix}
p_{c,t}
end{Bmatrix}_{c=1}^4,_{t=1}^T$
, where $c$ indicates the constraint subject to conversion as part of reference set of four constraints and $t$ the subperiod to which is associated the single realization of returns. Matrix disequality of page 37 clears it up. Instead, in the model that I’m studying now, the reference set becomes $begin{Bmatrix}
p_{c,t,i}
end{Bmatrix}_{c=1}^4,_{t=1}^T,_{i=1}^T$
.

So:

  1. What does $i$ say?

  2. Why, for example in case t.I third constraint, do we have $p_{t,1,t}$? What does the “second” $t$ mean?

Thanks in advance for any help.

making use of our robust cryptocurrency wallet and payment solutions – Cryptocurrencies Corner

Transfer digital currencies effortlessly by utilizing our high-quality cryptocurrency wallet and payment solutions. Our features include automatic session logout, access to detailed transaction history, two-factor authentication, an exclusive QR code scanner, and provision of real-time push notifications. Contact our impeccable developer team and achieve a plethora of benefits soon. 

hashcode – Iter 2: Reusable, robust c++ std::hash for GMP’s big integer type

This is the 2nd iteration of a code review. The 1st iteration (completed) is at
Iter 1: Reusable, robust c++ std::hash<mpz_class> for GMP’s big integer type

1. Goal

My intention is to provide a fast hashing algorithm to hash GMP’s big integer type mpz_class and mpz_t so I can use these types as keys for an unordered_map. The code shall be reusable for others.

2. Current Approach

string_view is used to wrap the data of the big integer’s absolute value. Since C++17, the standard library provides the specialization hash<string_view> which is used to produce the initial hash value. If the big integer is negative, the initial hash value is scambled once to produce different hash values for positive and for negative big integers of the same magnitude.

3. Code

File hash_mpz.h:

#ifndef HASH_MPZ_H_
#define HASH_MPZ_H_

#include <gmpxx.h>

namespace std {

template<> struct hash<mpz_srcptr> {
    size_t operator()(const mpz_srcptr x) const;
};

template<> struct hash<mpz_t> {
    size_t operator()(const mpz_t x) const;
};

template<> struct hash<mpz_class> {
    size_t operator()(const mpz_class &x) const;
};

}

#endif /* HASH_MPZ_H_ */

File hash_mpz.cpp:

#include "hash_mpz.h"
#include <cstddef>
#include <string_view>

constexpr size_t pi_size_t() {
    if (sizeof(size_t) == 4) {
        return 0xc90fdaa2; // floor(pi/4 * 2^32)
    } else if (sizeof(size_t) == 8) {
        return 0xc90fdaa22168c234; // floor(pi/4 * 2^64)
    } else {
        throw std::logic_error(
                "current sizeof(size_t) not supported. only 32 or 64 bits are supported.");
    }
}

inline size_t scramble(size_t v) {
    return v ^ (pi_size_t() + (v << 6) + (v >> 2));
}

namespace std {

size_t std::hash<mpz_srcptr>::operator()(const mpz_srcptr x) const {
    string_view view { reinterpret_cast<char*>(x->_mp_d), abs(x->_mp_size)
            * sizeof(mp_limb_t) };
    size_t result = hash<string_view> { }(view);

    // produce different hashes for negative x
    if (x->_mp_size < 0) {
        result = scramble(result);
    }

    return result;
}

size_t hash<mpz_t>::operator()(const mpz_t x) const {
    return hash<mpz_srcptr> { }(static_cast<mpz_srcptr>(x));
}

size_t hash<mpz_class>::operator()(const mpz_class &x) const {
    return hash<mpz_srcptr> { }(x.get_mpz_t());
}

}

File main.cpp:

#include <iostream>
#include <gmpxx.h>
#include <unordered_map>

#include "hash_mpz.h"

using namespace std;

int main() {
    mpz_class a;

    mpz_ui_pow_ui(a.get_mpz_t(), 168, 16);

    cout << "a      : " << a << endl;
    cout << "hash( a): " << (hash<mpz_class> { }(a)) << endl;
    cout << "hash(-a): " << (hash<mpz_class> { }(-a)) << endl;

    unordered_map<mpz_class, int> map;
    map(a) = 2;
    cout << "map(a) : " << map(a) << endl;

    return 0;
}

4. Question

Is there anything which can benefit from further improvement?

Essentials for a robust leverage crypto exchange!   – Cryptocurrencies Corner

 

         Crypto exchange platforms with leverage are currently the most trending platforms and are becoming more popular with each day. They are considered to be one of the highly profitable trading techniques. This is because, unlike any other exchange existing in the market, they allow investors to borrow leverage LARGER than the initial sum they deposit in the exchange. This is highly beneficial for investors to expand their market position exponentially and pocket huge profits. 

 

To state an honest fact, leverage trading cryptocurrency exchange is considered to be risky and as we all know, crypto markets are more volatile than anything else. But beyond the fact, we can still see potential investors interested to invest in this exchange and make use of this advanced strategy. If you are one of them, all you have to do is choose the right platform with right features such as, 

 

  • Perpetual contracts

  • Social/Copy Trading

  • Partial Close Trades

  • Deleveraging option

  • GDPR compliance

  • Risk management

  • Trading bonus

 

With a team of qualified, seasoned experts, Blockchain App Factory will help you with next-gen features and trading strategies for your leverage exchange software development, which will help you boost profits for your business without being prone to any risks. To know more, reach out to us. 

.

architecture – How does a robust http server handle requests?

Maximize throughput requires the use of asynchronous techniques, of which C is very bad. The problem with a while loop that accepts and handles new connections is that we can only process one request at a time, and that processing of that request is usually tied to I / O: the speed depends on the connection to the client, and even the client itself. If we don't receive the next TCP packet, the entire loop is blocked until the connection times out!

So, what servers typically do in C is fork () immediately after accepting a request, or passing the accepted file descriptor to a pre-forked worker pool (see: Apache), or using threads instead of process. Now, the number of requests that can be in flight is limited by the number of threads / work processes. There are considerable overhead costs with this approach, limiting the extent to which it can evolve. There is a context switching overhead of juggling all of these processes, and there is a memory overhead of at least a few KB for each worker. Most workers will still be inactive because they are waiting for the end of an I / O operation (read or write).

A scalable server should therefore use non-blocking techniques driven by events. To a small extent, this can be handled with select () or poll () in POSIX. Non-portable APIs like epoll () on Linux or kqueue () on FreeBSD (which all predate IOCP on Windows NT) are more practical. You wouldn't use them directly, but use a library of event loops like libuv or libevent.

Nginx precisely uses these event I / O facilities to efficiently multiplex connections. Node.JS combines the V8 JavaScript engine with libuv and is probably one of the most accessible platforms for writing asynchronous servers. However, we now also see systems programming languages ​​get good abstractions for asynchronous operations. The Async / Wait syntax was launched in .NET languages ​​and is also available in Rust and C ++ 20. This significantly reduces overhead costs per in-flight request and relatively easily integrates integration. an event loop, without having to manually code a state machine.

File System – Is It A Fast And Robust Way To Get The Length Of A Saying Name In C?

With dirent.h, if I readdir(), I can assign the return value to a struct dirent * I noticed that there is in them a property called d_namlen. They are basically the length of the d_name. But as pointed out here, there is no reliability. So, is the following code efficient, but robust, since it only measures length manually if I need it? I use the C language, on all platforms.

unsigned short NameLen;
if (!(NameLen = Entry->d_namlen)) { //No entry should have a zero length string, so testing if d_namelen is zero might do the trick?
    NameLen = strlen(Entry->d_name);
}

Is making this code a good idea?

legal – Do we need to specifically support older browsers to conform to WCAG or do they follow the robust principle enough?

I understand that WCAG is technology independent, so it does not specify on which browsers, devices and operating systems a website should be accessible.

However, our site is not responding on some platforms, such as Opera or IE6 / Windows XP. This affects accessibility, because suddenly the content will be broken and inaccessible. For example, in IE6, black text on a white background ends up hovering on a black background, which makes it unreadable.

But it is impossible to take into account every device, browser and operating system. So I feel like my only option is to be compatible with the main browsers / devices, and then to follow the Robust principle. So even if the contrast of the text breaks on IE6 / Windows XP, I will not be prosecuted for non-compliance.

Our objective here is not to be prosecuted for non-compliance with WCAG. We do not know to what extent we will be prosecuted for not supporting Windows XP, but the owner of the site is very concerned about the legal aspects.

Learn more about understanding compliance

Legal – Is the Robust principle sufficient to comply with WCAG?

I understand that WCAG is technology independent, so it does not specify on which browsers, devices and operating systems a website should be accessible.

However, our site is not responding on some platforms, such as Opera or IE6 / Windows XP. This affects accessibility, because suddenly the content will be broken and inaccessible. For example, in IE6, black text on a white background ends up hovering on a black background, which makes it unreadable.

But it is impossible to take into account every device, browser and operating system. So I feel like my only option is to be compatible with the main browsers / devices, and then to follow the Robust principle. So even if the contrast of the text breaks on IE6 / Windows XP, I will not be prosecuted for non-compliance.

Our objective here is not to be prosecuted for non-compliance with WCAG. We do not know to what extent we will be prosecuted for not supporting Windows XP, but the owner of the site is very concerned about the legal aspects.

Learn more about understanding compliance