Tag: data

conventions – When exporting data, is it alright if we don’t show the data?

my question may be rather confusing without the context, please allow me to explain and I will really appreciate it if you will give me your suggestions and insights 🙂

I’m currently designing a part of our in-house portal and there’s this page where I show a table of data such as the statuses of our devices. This data can be changed according to the user’s needs, such as if they want to see the data for the month of June, they can select ‘June’ and the data will change accordingly. Then, there is the export button that will export the data using the information on the table, into another format via csv. Ideally, I should show the data for easy reference and from what I researched, it is the convention.

However, I was questioned (most likely due to efficient use of resources) if the presentation of the data is even needed. Because the user simply needs the exported csv only. Which is also true because assuming that our portal does everything for her accurately, she doesn’t need to check and refer to the table. And if she wants to check, she can just look at her csv and edit from there.

Hence, I’m stuck. Because sometimes conventions aren’t the best but at the same time conventions are what people are used to.

noSQL database structure for dynamic data

I am currently prototyping with Flutter and want to use Google Firestore to store my data.

Part of my data is structured like a Google form. It could contain a title, text passages, new lines, formatting (bold, italic, …) and also images, input fields, radio buttons, labels, etc. on different positions inside the text.

My goal is to map the complete structure of a form in the database.

Now I have the following two questions:

  • Is it really a good idea to use Firestore mainly because of the good integration? Later on there will be added functionalities for which Firestore is predestined. My intial decision for Firestore was because the Firebase universe is fully integrated in Flutter, I use Firebase Auth for authentication, Firebase makes it easy for me to sync data and improves my developing speed for prototyping. Does it make sense to rely on 2 different databases in this case?
  • How to store such data containing information about different data types in a structured way? Both in a noSQL and in a relational database? I was thinking of using custom formatting e.g. xml. Then I can parse the text in my app and display it formatted. I could also use this to identify text fields etc and insert them in the right place. I’m pretty sure there are some best practices for that?

google apps script – Duplicate and rename a sheet (only it`s data) at fixed hr and day

every monday I’ve to make a copy of a sheet which is called week_nn, where nn is a progressive number of the current week for that sheet from its beginning. At some point of the day, we have to fix those data to make a history, and so I duplicate the sheet which take the standard name of “copy_of_week_nn_WIP” to “copy_of_week_nn_FRZ”. If possible, I’d like to automathize this with a triggered script (time/day, in this case, every monday), because sometimes it happened if forgot the task and also, because it`s a stupid job…how I could possible do it?

R how to turn all rows data in a dataframe to a single string?

I have a tibble df, and if I do the below, I get all rows data in below format.

df = tibble(A1=c("A03816","A03817"))
> df(("A1"))
(1) "A03816" "A03817"

But I want those 2 words in the same string like below, how can I do it?


(1) "A03816 A03817"

plotting – How to extract data from the solution of NDsolve with specific parameter range

I’ll use the Lotka-Volterra model as an example since I can’t copy your code.

First way:
Use Show to get the forward t>0 and backward t<0 solutions to be different colors.

{xsol, ysol} = NDSolveValue({
   x'(t) == x(t) - 2 x(t) y(t),
   y'(t) == x(t) y(t) - y(t),
   x(0) == y(0) == 1},
  {x, y}, {t, -20, 20})


Show({
  ParametricPlot({xsol(t), ysol(t)}, {t, -4, 0}, PlotStyle -> Red, 
   PlotRange -> {{0, 2.5}, {0, 1.5}},
   Prolog -> {PointSize(Large), Blue, Point({1, 1})}),
  ParametricPlot({xsol(t), ysol(t)}, {t, 0, 4}, PlotStyle -> Green)})

Using Show

Second way:
Have a forward and a backward NDSolve.

Forward solution

{xsolf, ysolf} = NDSolveValue({
   x'(t) == x(t) - 2 x(t) y(t),
   y'(t) == x(t) y(t) - y(t),
   x(0) == y(0) == 1},
  {x, y}, {t, 0, 20})

Backward solution

{xsolb, ysolb} = NDSolveValue({
   x'(t) == -(x(t) - 2 x(t) y(t)),
   y'(t) == -(x(t) y(t) - y(t)),
   x(0) == y(0) == 1},
  {x, y}, {t, 0, 20})

Notice that the backward solve has the same initial conditions and time range as the forward solve, but that the equations are negated.

This lets you use only a single ParametricPlot for both forward and backward solutions.

ParametricPlot({{xsolf(t), ysolf(t)}, {xsolb(t), ysolb(t)}}, {t, 0, 4}, 
  PlotStyle -> {Red, Green}, 
  Prolog -> {PointSize(Large), Blue, Point({1, 1})})

enter image description here

I’ve also gone with NDSolveValue instead of the regular NDSolve since you asked to store the data in a table which you can do pretty easily with

xfData = Table(xsolf(t), {t, 0, 5, .1})

If you use the second way, remember that for the backward solve time is “negative time” so you might need to do something like

xbData = Table(xsolb(t), {t, 10, 0, -.1})

calculus and analysis – Fitting data to a model

I am using the following code with the following data (which I called fn):

data (fn) here: https://pastebin.com/WwANBjNx

fn = {datneg30, datneg10, dat5, dat25, dat40, dat45, dat50, dat55, 
   dat60, dat65, dat70, dat75, dat80, dat85, dat87, dat90, dat92};
Ta = {-30, -10, 5, 25, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 87, 90,
    92};

Tre = 75 + 273.2;
unagednematic = 0.35;


avnem(t_, T_, h22_, k2ref_, C1_, C2_, n2_) :=
  (h22 - unagednematic) (1 - 
     Exp(-( k2ref Exp(
          C1 (T + 273.2 - Tre)/2.3/Abs((C2 + T + 273.2 - Tre))) t^
          n2)))(*Nematic*);

datLC2 = Flatten(
  Table({fn((i))((j, 1)), Ta((i)), fn((i))((j, 3))}, {i, 13, 15}, {j, 
    1, Length(fn((i)))}), 
  1);(*Creates a list of {ta,Ta,LC} from Ta=80C to Ta=87C*)

nlmLC2 = NonlinearModelFit(
  datLC2, {avnem(t, T, h22, k2ref, C1, C2, n2), 0 < h22 < 1.3, 
   0.1 < k2ref < 10, 0 < C1 < C2, 80 > C2 > 0, 
   0.6 < n2 < 1}, {{k2ref, 0.28}, {h22, 1.38}, {C1, 17}, {C2, 
    40}, {n2, 0.75}}, {t, T});

nlmLC2("RSquared")
nlmLC2("BestFitParameters")


fpLC2b = Table(
  nlmLC2(t, Ta((i))), {i, 5, 
   15});(*Reemplace the fitting of nlmLC2 using Ta=40 to Ta=87. Get 
the equation as a function of t only. The 2 at the beginning is the 
index.*)

Show(LogLinearPlot({fpLC2b}, {t, 0.001, 10000}, 
  PlotStyle -> {Directive(Purple), Directive(Darker(Yellow)), 
    Directive(Orange)}), 
 ListLogLinearPlot(Table(fn((i))((All, {1, 5})), {i, 5, 15}), 
  PlotMarkers -> {Automatic, 13}, AspectRatio -> 1 , 
  PlotLegends -> 
   Placed({40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 87}, {0.14, 
     0.66})), Frame -> True, PlotRange -> {All, {0, 1.4}}, 
 Axes -> False, AspectRatio -> 1, FrameStyle -> Directive(Black, 13), 
 FrameLabel -> {Style("!(*SubscriptBox((t), (a))) / s", 16), 
   Style("(CapitalDelta)H Liquid crystal 
(!(*SuperscriptBox((Jg), (-1))))", 16)})

How can I fit the data to the model of the code (e.g, h22 - unagednematic) (1 - Exp(-( k2ref Exp( C1 (T + 273.2 - Tre)/2.3/Abs((C2 + T + 273.2 - Tre))) t^n2))))?

I am getting a very bad fitting as the figure show and I know the model should be able to describe the data:

enter image description here

windows 10 – Does data recovery of file depends on its extension?

A file is basically data on a physical file system and a reference on logical file system. The file name including its extension is part of the logical layer. Once a file is removed from this layer, the physical blocks (now without a reference from the logical layer) are marked as available. In this stage, the contents of the file are still readable until it gets overwritten with another file, but the name and extension are gone.

Therefore, data recovery of a removed file depends more on the data format than the extension. Some formats are, unless the file was fragmented, easier to recognize by clear and strictly defined beginnings and endings. Some examples include:

  • JPEG images (and derived formats) are sequences of segments, each beginning and most ending with a marker, like 0xFFD8 Start Of Image and 0xFFD9 End of Image.
  • A ZIP archive consist of file entries and ends with a central directory referencing these files. It is like a small file system within a single file.
  • TAR archives have a 512 byte header for each file followed by the file rounded up to multiple of 512 bytes and padded (typically with zeroes).
  • From text files, e.g., XML is easy to recognize, as it may start with <?xml and consists of root element and child elements, each having a defined ending.
  • Modern Microsoft Office files a.k.a. Office Open XML files are ZIP packed XML-based documents.

As these files are that recognizable, they are faster to recover. The time depends on how many formats there has to be defined, recognized and analyzed. E.g., if you know you are only after JPEG images you can narrow this process down to JPEG markers.

design – What is the most suitable data structure for IPv4 addresses with intersecting ranges?

Mostly in all routers and operating systems the Longest Prefix Match algorithm is used for searching the trie of IPv4 addresses.
Packet filters like Iptables don’t need some special data structure for finding IPv4 addresses.

I need to implement some data structure, where I could efficiently (log(N)) find IPv4 addresses, which are specified in firewall rules. But there could also be IPv4 address ranges. These ranges could be intersected and have exclusions (exclude one range or IP from another one). But I thing exclusions could be resolved in time of constructing (deploying) data structure.
So the Longest Prefix Match doesn’t fit.
I was considering the Interval Tree but I’m not sure that this is the most effective way.

P.S. inserting and deleting is not a big problem but it would be good to be also log(N)

integrity – E2E-Encrypted Storage: How to detect that data has been rolled back?

As far as I understand this would still decrypt without any issues because the old auth tag matches with the old encrypted data?

Right

Your approach would generally work. However, I find a bit odd your description of adding “the timestamp from the latest changes to the HMAC of the root directory”. I’m not sure if you mean into, or just the . If the root directory contain the hash/HMAC of the file, then you can detect the rolling back of that file. No need to check the file timestamps at all (you could store this last update metadata either with the file itself, or in the directory entry). It is enough for you to store the last change timestamp to the “cloud filesystem” in that root directory, so that you can show to the user the version they are browsing (and detect rollbacks).

I would recommend to store a sequential id as well. While the actual timestamp would work reasonably well for people, the application checking it is not being given a rollbacked view would work best with monotonically increasing ids (e.g. suppose that there are two updates in the same second).

Also, you need to be careful to ensure your update to the root directory (or other intermediate directories) cannot be mixed with two clients changing files at the same time and overwriting an older version (you could use atomic updates of the blob, checking the prior contents are those it knew about).

PS: I disagree with @mti2935 ‘blockchain solution’. Any secure communication channel would be enough to detect the rollback. Sure, you could do this in a cryptocurrency blockchain, but that’s completely overkill. And certainly not “easier” than doing so in the client app, which would be free. Simply a server in a different provider, albeit with less space could serve as notary of the cloud provider. Or even a service like twitter.

plotting – How can I label a table of histograms with the names of the data symbols, not the data themselves?

How can I label a table of histograms with the names of the data symbols, not the data themselves?

If I want to make a table of plots with each plot labeled with the name of the function it plots, this is simple:

f[x_] := x; g[x_] := x^2;
Table[Plot[p[x], {x, -1, 1}, PlotLabel -> p], {p, {f, g}}]

However, if I want to do the same for Histogram, I have not figured out how. I have tried various combinations using ToString, Hold, HoldForm, and Unevaluated, as well as ChartLabels. For example,

l = RandomReal[{0, 1}, 20]; m = RandomReal[{0, 1}, 20];
Table[Histogram[p, PlotLabel -> p], {p, {l, m}}]

does not work, as the labels are the values of l and m, not the strings l and m.

The same question applies to the format analogous to

Plot[#[x], {x, -1, 1}, PlotLabel -> #] & /@ {f, g}

which works for Plot but not for Histogram.