mysql – Where can I get the appropriate dataset to perform normalization techniques?

I am new to databases and have just started learning different techniques of Data Warehousing. I have been assigned the task of performing normalization techniques on a data set of at least 1000 rows with a minimum of three tables.

I am not able to find such data for the same.

I have tried, kaggle and uci but all to no avail. Can I also perform normalization techniques using MySQL Workbench? Please help me .. Thanks in advance.

database theory – Space-efficient data structure for the analytical interrogation of multiple branching evolutions of a dataset

description of the problem

i have a data state space: a set of data sets, each of which can be modeled as a collection of arbitrary key-value pairs. These datasets are each a branch of the evolution of a parent dataset, forming a tree (not a network; branch of datasets but do not merge.) The root of the story tree is an empty dataset.

I'm not looking at / not exploring the entirety of this data state space. Rather, I have a list of all the leaf node datasets that exist in the real world. I only care about these leaf node data sets and (sometimes) their ancestors.

I want to find (or create!) A persistent data structure (on disk, distributed / fragmented if necessary) to store and query these data sets. I would also accept a complete database management system built around such a structure.


This data structure would require the following operations:

  • to define a new set of data in the store, in terms of parent dataset identifier / identifier; and one set of "written" key-values which would create this dataset if it were applied to the referenced parent;

  • open memory relative to a particular dataset identifier, returning a dataset descriptor;

  • request the store handle + data set for a value of a particular key;

  • request the storage handle + data set for the key-value pairs in a data range of keys;

  • request the store handle + data set for a spill of all the key-value pairs in the dataset.

  • (possibly) get a the cursor compared to a store + data set handle + initial key, and use it to repeat forward / backward through key-value pairs, each iteration returning a key-value pair.

Constraints imposed by the data:

  • Many leaf node datasets in the state space will contain Billions key-value pairs.

  • Many leaf node datasets are more than 10 million levels "deep" or "high" away from the root.

  • Many datasets are trivial changes from their parent, consisting of a update or even zero updates (but will keep a separate identity in the case of the zero update.)

  • Many datasets are not trivial, hundreds of thousands updates from their parent.

  • Each branch node in the tree has, on average, 1.5 children (most have only one; some have two; very few have more than two.) The tree consists mainly of long linear segments of nodes, with branch nodes getting branching off between a new "main branch" on one side; and a short and terminal "side branch" on the other.

Operational constraints:

  • the Lily operations must be fast (i.e. soft real-time / limited latency), because the purpose of this datastore is to serve analytical requests with intensive reading.

  • Insertion new datasets in the store have to ladder in the millions without going exponential; but otherwise, the inserts can be quite slow, taking the order of a few seconds to validate a new data set definition in the store.

  • Opening a data set in the store can be time consuming (again ~ seconds), but again, this overhead must increase slowly enough to open a buried data set "deep" in the store. This "free" time can be used to decompress the data set of any delta compression / encoding format, to cache the required data or intermediate data structures from disk to memory, etc.

  • The store must also be space-saving on disk as possible (allowing the overhead of disk space needed to satisfy the other constraints.) I don't have petabytes to expand this store!


Consideration of the use of disk space is where things get interesting, IMHO.

Without a requirement for disk space efficiency, the naïve solution is to simply have full copies of each separate data set stored separately in their own read-indexed storage files, and then distribute sets of Separate data on their own network fragments in order to read requests can be routed to them separately.

But this naïve solution would throw the storage requirements for this state space (reminder, ~ billion KV pairs each, ~ million knots) into a territory of several petabytes, and I don't have that kind of Space.

Intuitively, thanks to my experience in tuning analytical database systems, I well disk data structure for OLAP backup storage (including all required indexes) to introduce at most 10x overhead in addition to a modified data capture representation of the source data. The modified data capture representation of all these data sets (i.e. the representations that to define above) currently stands at ~ 50 GB; So I would intuitively expect this data structure to require no more than ~ 500 GB of disk. Am I crazy to expect this?

I know I can get some easy gains in terms of storage overhead for general "deduplication" of datasets, just by relying on a file system with a copy-on-write block level, where each dataset in the state space becomes its copy-on-write snapshot and the snapshots form a tree structure. But it looks like would not evolve operationally, because either I would use a sort of sorted flat file dataset format (in which case the insertions "in the middle" of the data cause an explosion of storage in descending snapshots), or I & # 39; Would use something like a disc LSM tree (eg LevelDB) or B + tree (eg LMDB) (in which case each snapshot would add another "level" to the tree, causing either an explosion of the inodes of file in the case of LevelDB, i.e. a fragmentation of each file into tiny scopes per layer in the case of LMDB), which ultimately means that reading in a data set of one million branches in one such a store would have a lot of overhead in file system accounting.

I guess one well the data structure for this would involve, at some level:

  • tests (HAMT?)
  • a notion of "keyframes" versus "interstitial frames", to represent sets of trivial changes
  • a notion of "highly connected roads" through the tree, where the "major branches" are repackaged (defragmented?) and the "minor branches" are re-stored in terms of difference from a "main branch"

I am aware of Datomic, which seems to have an architecture and a set of operations similar to what I am looking for; but which only supports a linear chronology, rather than a tree of chronological branches. I do not know if its architectural design could be extended to support connection time without fundamental changes.

I also know what blockchain systems (eg Ethereum) do with merkle patricia trie– storage room. I evaluated exactly that, but – at least when the blockchains themselves implement this approach – the read performance born scale for analytical workloads. (This works for these systems, since their evolutionary stages are OLTP workloads, almost always calculating against a "main branch" state – the result of the most recent previous calculation – which is therefore set almost completely hides in memory. Most recent "main branch" state can be considered "cold". None of this is true for an OLAP use case; OLAP queries examine arbitrary branches of a state space at arbitrary times, with neither branch nor node being "hot".) I would expect, however, that some modification (relaxation?) of this approach is appropriate.

How to subtract anomalies received from the dataset

We have a dataset

data = ResourceData["Sample Data: Boston Homes"]

Find abnormal examples in the dataset.

anomalies = FindAnomalies[data]

Now i want to delete the anomalous rows received from the dataset. How can it be done?

import – How to recover only the dataset required for a query in ImportJSON?

I am trying to retrieve the latest BTC / INR price from an HTTP JSON URL from an exchange.

I use the following formula in Google Sheet to retrieve the latest price from the URL:


But it shows all the prices. I just wanted to get the latest price.
Can you help me filter the / price request to retrieve only the latest price.

I am using this ImportJSON script:

Thank you

tracing – Visualization of a 5-D dataset with a 3-D surface, a color and a manipulation?

I am relatively new to Mathematica. Hoping for help to get started, as the graphics capabilities of Mathematica look promising.

I have a 5-dimensional list of numerical data generated by simulation. So i have five variables (a B C D E) and their associated numeric values.

I would like to visualize a 3D surface interpolating the data points with a, b, c like the variables to visualize, re as a color setting, and e as input for Manipulate.

performance – Efficient uniqueness check on a large dataset

I'm refactoring a health surveillance system that requires certain attributes of an entity to be unique through the system.
The attributes of an entity are configurable by the end user and the user can choose one or more attributes to be unique (either "universally" unique or unique in a geographic area).

Currently, the solution works very poorly when looking for these unique values ​​(we use Postgres). The use of Postgres partial indexes alleviates the performance problem, but, on large datasets (500 million rows, which is not unusual), performance is not acceptable.

One solution I am considering is to hash the attribute + value using a trigger before INSERT and UPDATE. The trigger would check this unique-index "hash" before authorizing the insert. If the hash is missing, it inserts it otherwise it blocks the operation.

Is there a better solution to this problem, given the size of the dataset?

python – How to normalize an image dataset

How to normalize an image dataset? the only formatting that I know of and that I've seen some use is to divide the matrix by 255, but I don't understand why, since that only changes the scale of values.
There are several measures of the tendency for centering and dispersion, but I don't know how to apply them to images.
Note: I am facing a classification problem.

How to sort a dataset where each row includes a sequence of strings and integers

I got this type of data:

initDataS = Dataset@<|"Count" -> <|ID01 -> 41667 "Train",
ID02 -> 23288 "Tail" + 18379 "Train",
ID03 -> 30907 "Tail" + 10760 "Train",
ID04 -> 34058 "Tail" + 7100 "Train" + 509 "Loop",
ID05 -> 36256 "Tail" + 5411 "Train",
ID06 -> 37548 "Tail" + 3700 "Train" + 419 "Loop"|>|>

InitDataS formatted

I would like to sort it effectively this way:

FinalDataS = Dataset@ <|"ID01" -> <|"Train" -> 41667, "Tail" -> 0, "Loop" -> 0|>,
"ID02" -> <|"Train" -> 18379, "Tail" -> 23288, "Loop" -> 0|>,
"ID03" -> <|"Train" -> 10760, "Tail" -> 30907, "Loop" -> 0|>,
"ID04" -> <|"Train" -> 7100, "Tail" -> 34058, "Loop" -> 509|>,
"ID05" -> <|"Train" -> 5411, "Tail" -> 36256, "Loop" -> 0|>,
"ID06" -> <|"Train" -> 3700, "Tail" -> 37548, "Loop" -> 419|> |>

Formatted FinalDataS

How could I do this?
Thanks in advance for your help!

plotting – curve fitting to a dataset

I am trying to adapt a polynomial to the data listed as {{0,6.67}, {6,17.33}, {10,42.67}, {13,37.33}, {17,30.1}, {20, 29.31}, {28,28.74}}. When I use LinearModelFit [data, {x, x ^ 2, x ^ 3, x ^ 4, x ^ 5, x ^ 6}, x] I get FittedModel [6.67-42.6435x + 16.1427x ^ 2- <<19>> x ^ 3 + <<20>> x ^ 4-0.00367168x ^ 5 + 0.0000409458x ^ 6. I don't understand why I don't get coefficients for the terms x ^ 3 and x ^ 4 which make sense and I would like to 39; help to solve this problem.

machine learning – Sckit regression on the power dataset

How to perform a linear regression on each subset of data frames in a loop with the linear regression of scikit-learn?

    def sub_lists(list1):
  sublist = (()) 
  for i in range(len(list1) + 1): 
        for j in range(i + 1, len(list1) + 1): 
            sub = list1(i:j) 
            return sublist

X = sub_lists(df5);y = df4;

I have regressed on this, but it continues to generate errors, it is a .dta (stata) file.