big data – How do megasites like Youtube perform backups?

How do megasites like Youtube perform backups? According to https://www.quora.com/Where-does-YouTube-store-so-many-videos, 2014 they stored 76 PB every year, a number that most certainly has increased a lot since then. Is it even possible to backup this or do they solve the problems backups usually are used for in some other way (such as so much redundancy that it simple can’t fail)?

Bash script for user backups with BorgBackup

I use BorgBackup to handle backups for my personal computer.
The following is a bash script that creates backups of my system to various different targets.
I have little experience with bash and hope to get some pointers regarding best practices and possible security issues/unanticipated edge cases.

One specific question is about the need to unset environment variables (in particular BORG_PASSPHRASE) like I have seen people doing (for example here)
From my understanding this should not be necessary because the environment is only local.

Ideally I would also like to automatically ensure the integrity of the Borg repositories. I know there is borg check which I could run from time to time, but I am not sure if this is even necessary when using create which supposedly already makes sure the repository is in a healthy state?

Some notes regarding the code below:

  • noti is a very simple script for notifications with i3status but could be replaced with anything else
  • Some paths and names are replaces with dummies
  • I can not exit properly on errors of borg create because I backup /etc where some files have wrong permissions and BorgBackup will throw errors
  • The TODO comments in the code are things I may want to look into at some time, but the code works for now

Bash script:

#!/bin/bash

# User data backup script using BorgBackup with options for different target repositories
# usage: backup.sh <targetname>
# Each target must have a configuration file backup-<targetname>.conf that provides:
# - pre- and posthook functions
# - $repository - path to a valid borg repository or were one should be created
# - $backup - paths or exclusion paths
# - $pruning - borg pruning scheme
# Additional borg environment variables may be provided and will not be overwritten.
# Output is logged to LOGFILE="$HOME/.local/log/backup/<date>"

# INSTALLATION
# Place script and all target configs in $HOME/.local/scripts
# $HOME/.config/systemd/user/borg-backup.service
# ```
# (Unit)
# Description=Borg User Backup
# (Service)
# Environment=SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
# ExecStart=%h/.local/scripts/backup.sh target1
# Nice=19
# IOSchedulingClass=2
# IOSchedulingPriority=7
# ```
#
# $HOME/.config/systemd/user/borg-backup.timer
# ```
# (Unit)
# Description=Borg User Backup Timer
# (Timer)
# OnCalendar=*-*-* 8:00:00
# Persistent=true
# RandomizedDelaySec=10min
# WakeSystem=false
# (Install)
# WantedBy=timers.target
# ```
# $ systemctl --user import-environment PATH
# reload the daemon
# $ systemctl --user daemon-reload
# start the timer with
# $ systemctl --user start borg-backup.timer
# and confirm that it is running
# $ systemctl --user list-timer
# you can also run the service manually with
# $ systemctl --user start borg-backup


function error () {
    RED='33(0;91m'
    NC='33(0m'
    printf "${RED}%s${NC}n" "${1}"
    notify-send -u critical "Borg" "Backup failed: ${1}"
    noti rm "BACKUP"
    noti add "BACKUP FAILED"
    exit 1
}


## Targets

if ( $# -lt 1 ); then
    echo "$0: Missing arguments"
    echo "usage: $0 targetname"
    exit 1
fi

case "$1" in
    "target1"|"target2"|"target3")
        target="$1"
        ;;
    *)
        error "Unknown target"
        ;;
esac

# TODO abort if specified target is already running
# exit if borg is already running, maybe previous run didn't finish
#if pidof -x borg >/dev/null; then
#    error "Backup already running."
#fi


## Logging and notification

# notify about running backup
noti add "BACKUP"

# write output to logfile
log="$HOME/.local/log/backup/backup-$(date +%Y-%m-%d-%H%M%S).log"
exec > >(tee -i "$log")
exec 2>&1

echo "$target"


## Global Prehook

# create list of installed software
pacman -Qeq > "$HOME/.local/log/package_list.txt"

# create list of non backed up resources
ls -R "$HOME/misc/" > "$HOME/.local/log/resources_list.txt"

# create list of music titles
ls -R "$HOME/music/" > "$HOME/music/music_list.txt"


## Global Config

# set repository passphrase
export BORG_PASSCOMMAND="cat $HOME/passwd.txt"

compression="lz4"


## Target specific Prehook and Config

CONFIGDIR="$HOME/.local/scripts"
source "$CONFIGDIR"/backup-"$target".conf

# TODO make non mandatory and only run if it is defined
prehook || error "prehook failed"


## Borg

# TODO use env variables in configs instead?
# export BORG_REPO=$1
# export BORG_REMOTE_PATH=borg1
# borg create ::'{hostname}-{utcnow:%Y-%m-%dT%H:%M:%S}' $HOME

SECONDS=0
echo "Begin of backup $(date)."

borg create                    
    --verbose                  
    --stats                    
    --progress                 
    --compression $compression 
    "$repository"::"{hostname}-{utcnow:%Y-%m-%d-%H%M%S}" 
    $backup
#  || error "borg failed"

# use prune subcommand to maintain archives of this machine
borg prune                 
    --verbose              
    --list                 
    --progress             
    "$repository"          
    --prefix "{hostname}-" 
    $pruning               
  || error "prune failed"

echo "End of backup $(date). Duration: $SECONDS Seconds"


## Cleanup

posthook
noti rm "BACKUP"
echo "Finished"

exit 0

Example configuration file:

backup="$HOME
--exclude $HOME/movie
--exclude $HOME/.cache
--exclude $HOME/.local/lib
--exclude $HOME/.thumbnails
--exclude $HOME/.Xauthority
"

pruning="--keep-daily=6 --keep-weekly=6 --keep-monthly=6"

repository="/run/media/username/DRIVE"

prehook() { :; }  # e.g. mount drives/network storage

posthook() { :; }  # unmount ...

Super Fast SSD CLOUD VPS Servers| Cheap Pricing + Backups from HostingSource! | NewProxyLists

HostingSource, Inc. is a leading hosting provider, offering reliable, scalable solutions for customers of all sizes and services. We supply all of the servers, software, bandwidth and management tools needed to run almost any web hosted application – from small to enterprise server solutions.
Our staff has over 20 years of experience in the IT field.

Our team of dedicated professionals, specializing in managed hosting solutions, software development, website design, website marketing, database integration & programming, & e-commerce solutions, can offer you the service and support that you need to make your online presence a reality.

HostingSource Advantages:
– High Performance SSD
– Redundant Hardware
– 24/7/365 Dedicated Support
– 100% Uptime Guarantee
– Windows Hyper-V
– Linux KVM Hypervisor
– Fast VPS Deployment
– No Contract, Cancel Anytime

Save Big on Hosting – 50% off 1st month on all Servers/VPS plans.

CHECK OUT HostingSource HIGH PERFORMANCE SSD CLOUD VPS PLans:

VPS – I
1 CPU Core
20GB – SSD DISK
5TB – DATA TRANSFER
1GB – RAM
Price – $8/mo
==>>> ORDER

VPS – II
2 CPU Core
40GB – SSD DISK
6TB – DATA TRANSFER
2GB – RAM
Price – $15/mo
==>>> ORDER

VPS – III
3 CPU Core
60GB – SSD DISK
7TB – DATA TRANSFER
4GB – RAM
Price – $22/mo
==>>> ORDER

CLOUD-I
4 CPU Core
100GB – SSD DISK
10TB – DATA TRANSFER
8GB – RAM
Price – $29/mo
==>>> ORDER

CLOUD-II
8 CPU Core
150GB – SSD DISK
15TB – DATA TRANSFER
12GB – RAM
Price – $39/mo
==>>> ORDER

CLOUD-III
12 CPU Core
200GB – SSD DISK
20TB – DATA TRANSFER
16GB – RAM
Price – $59/mo
==>>> ORDER

FOR MORE DETAILS CHECK: http://www.hostingsource.com/vps-hosting/

HostingSource, Inc., is committed to providing our customers the highest level of quality service and support. We are also committed to serving future customers by rapidly responding to sales or informational inquiries.
Please contact us at your convenience via the methods listed below:
– If you have any questions, feel free to call us toll-free: 1-888-586-1101
– Sales Department – For all sales inquiries or non technical questions.
– Technical Support Department – Technical Support available 24 hours a day, 7 days a week.
– FAQ Support – Use our in-depth Knowledgebase to get additional tips. It’s a great way to learn how to fine tune your website, enhance your service and more.

OVH US Destroyed My TOTAL Business | HELP LESS | DON'T BUY WITH OVH US | NO BACKUPS GIVEN

I had 2 Servers with OVH US (BV1 & BV2) With them,In that one of the server got 2 Spamhaus issue and which we had removed that account and i… | Read the rest of https://www.webhostingtalk.com/showthread.php?t=1827959&goto=newpost

50% OFF – Reseller Hosting – cPanel & WHM | SSD | CloudLinux | Backups | 99.9% Uptime – VernalWeb | NewProxyLists

VernalWeb is offering 50% discount on the first invoice (Up to 12 months) for all SSD Reseller Hosting plans. For a limited time only, new customers can receive 50% DISCOUNT on any of our reseller web hosting plans.

Coupon Code: SmartWay

Code will be applied on regular price & Discount Price will be shown at Checkout

Coupon rules:
• Valid for all customers.
• This coupon can’t be used with any other special offer.
• This coupon is only available on 1, 6 or 12-month billing terms.
——————————————————————

Lots Of Free Gifts
• Free Domain (Yearly Term)
• Free CloudFlare CDN
• Free SSL
• Free Migration
• Free Daily Backups(No Extra Charge)
• Free DDoS Protection
• Free SEO Tools

Included with all shared hosting accounts:
* WHM Control Panel
* cPanel Control Panel
* Softaculous Script Auto-installer (Hundreds of one-click install scripts!)
* SNI Supported
* High-Redundancy RAID 10 Arrays
* WordPress Optimized
* Developer Friendly Atmosphere
* E-commerce Ready!
* Secure IMAP & POP3
* Apache Handlers
* FREE Website Migrations (We will migrate your current website, free of charge!)

Blazing Fast Server Hardware, Premium Bandwidth:
We SPARE NO EXPENSE for our offer! Our newest shared servers are purpose-built for performance using enterprise components, including RAID-10 SSDs for all storage.

SSD Reseller Hosting Plans

StartMe Plan
25GB Pure SSD Storage
250GB Data Transfer
25 cPanel Accounts
Free Domain Registration
Unlimited Subdomains
Unlimited E-mail Addresses
Unlimited MySQL Databases
Unlimited FTP Accounts
1-Click Software Installations
Instant Activation

Order Now!

GoodFace Plan
50GB Pure SSD Storage
500GB Data Transfer
50 cPanel Accounts
Free Domain Registration
Unlimited Subdomains
Unlimited E-mail Addresses
Unlimited MySQL Databases
Unlimited FTP Accounts
1-Click Software Installations
Instant Activation

Order Now!

FineGrow Plan
100GB Pure SSD Storage
1000GB Data Transfer
100 cPanel Accounts
Free Domain Registration
Unlimited Subdomains
Unlimited E-mail Addresses
Unlimited MySQL Databases
Unlimited FTP Accounts
1-Click Software Installations
Instant Activation

Order Now!

If you need any further information or help please feel free to contact us at Help Desk

Why VernalWeb?
• Instant Activation
• Multiple Locations
• Solid State Drive makes your website 20X faster
• 24X7 In-house Support via Phone or Email
• Lowest Pricing in the industry
• Enterprise Grade Server with reliability and Stability
• CloudLinux
• Multiple PHP versions (PHP4.4, PHP5.1, PHP5.2, PHP5.3, PHP5.4, PHP5.5, PHP5.6, PHP7.0, PHP7.1, PHP7.2, PHP7.3)
• 99.9% Uptime Guarantee
• 30 Days Money Back Guarantee
• 1 Click Software Installer
• No overloaded Servers
• Easy Control Panel(WHM)

Get in touch with us:
Facebook | Twitter | Pinterest | LinkedIn

Super Fast Ssd Cloud Vps Servers| Cheap Pricing + Backups From Hostingsource!

HostingSource, Inc. is a leading hosting provider, offering reliable, scalable solutions for customers of all sizes and services. We supply all of the servers, software, bandwidth and management tools needed to run almost any web hosted application – from small to enterprise server solutions.
Our staff has over 20 years of experience in the IT field.

Our team of dedicated professionals, specializing in managed hosting solutions, software development, website design, website marketing, database integration & programming, & e-commerce solutions, can offer you the service and support that you need to make your online presence a reality.

HostingSource Advantages:
– High Performance SSD
– Redundant Hardware
– 24/7/365 Dedicated Support
– 100% Uptime Guarantee
– Windows Hyper-V
– Linux KVM Hypervisor
– Fast VPS Deployment
– No Contract, Cancel Anytime

Save Big on Hosting – 50% off 1st month on all Servers/VPS plans.

CHECK OUT HostingSource HIGH PERFORMANCE SSD CLOUD VPS PLans:

VPS – I
1 CPU Core
20GB – SSD DISK
5TB – DATA TRANSFER
1GB – RAM
Price – $8/mo
==>>> ORDER

VPS – II
2 CPU Core
40GB – SSD DISK
6TB – DATA TRANSFER
2GB – RAM
Price – $15/mo
==>>> ORDER

VPS – III
3 CPU Core
60GB – SSD DISK
7TB – DATA TRANSFER
4GB – RAM
Price – $22/mo
==>>> ORDER

CLOUD-I
4 CPU Core
100GB – SSD DISK
10TB – DATA TRANSFER
8GB – RAM
Price – $29/mo
==>>> ORDER

CLOUD-II
8 CPU Core
150GB – SSD DISK
15TB – DATA TRANSFER
12GB – RAM
Price – $39/mo
==>>> ORDER

CLOUD-III
12 CPU Core
200GB – SSD DISK
20TB – DATA TRANSFER
16GB – RAM
Price – $59/mo
==>>> ORDER

FOR MORE DETAILS CHECK: http://www.hostingsource.com/vps-hosting/

HostingSource, Inc., is committed to providing our customers the highest level of quality service and support. We are also committed to serving future customers by rapidly responding to sales or informational inquiries.
Please contact us at your convenience via the methods listed below:
– If you have any questions, feel free to call us toll-free: 1-888-586-1101
– Sales Department – For all sales inquiries or non technical questions.
– Technical Support Department – Technical Support available 24 hours a day, 7 days a week.
– FAQ Support – Use our in-depth Knowledgebase to get additional tips. It’s a great way to learn how to fine tune your website, enhance your service and more.

[WTS] Super Fast SSD CLOUD VPS Servers| Cheap Pricing + Backups from HostingSource!

HostingSource, Inc. is a leading hosting provider, offering reliable, scalable solutions for customers of all sizes and services. We supply all of the servers, software, bandwidth and management tools needed to run almost any web hosted application – from small to enterprise server solutions.
Our staff has over 20 years of experience in the IT field.

Our team of dedicated professionals, specializing in managed hosting solutions, software development, website design, website marketing, database integration & programming, & e-commerce solutions, can offer you the service and support that you need to make your online presence a reality.

HostingSource Advantages:
– High Performance SSD
– Redundant Hardware
– 24/7/365 Dedicated Support
– 100% Uptime Guarantee
– Windows Hyper-V
– Linux KVM Hypervisor
– Fast VPS Deployment
– No Contract, Cancel Anytime

Save Big on Hosting – 50% off 1st month on all Servers/VPS plans.

CHECK OUT HostingSource HIGH PERFORMANCE SSD CLOUD VPS PLans:

VPS – I
1 CPU Core
20GB – SSD DISK
5TB – DATA TRANSFER
1GB – RAM
Price – $8/mo
==>>> ORDER

VPS – II
2 CPU Core
40GB – SSD DISK
6TB – DATA TRANSFER
2GB – RAM
Price – $15/mo
==>>> ORDER

VPS – III
3 CPU Core
60GB – SSD DISK
7TB – DATA TRANSFER
4GB – RAM
Price – $22/mo
==>>> ORDER

CLOUD-I
4 CPU Core
100GB – SSD DISK
10TB – DATA TRANSFER
8GB – RAM
Price – $29/mo
==>>> ORDER

CLOUD-II
8 CPU Core
150GB – SSD DISK
15TB – DATA TRANSFER
12GB – RAM
Price – $39/mo
==>>> ORDER

CLOUD-III
12 CPU Core
200GB – SSD DISK
20TB – DATA TRANSFER
16GB – RAM
Price – $59/mo
==>>> ORDER

FOR MORE DETAILS CHECK: http://www.hostingsource.com/vps-hosting/

HostingSource, Inc., is committed to providing our customers the highest level of quality service and support. We are also committed to serving future customers by rapidly responding to sales or informational inquiries.
Please contact us at your convenience via the methods listed below:
– If you have any questions, feel free to call us toll-free: 1-888-586-1101
– Sales Department – For all sales inquiries or non technical questions.
– Technical Support Department – Technical Support available 24 hours a day, 7 days a week.
– FAQ Support – Use our in-depth Knowledgebase to get additional tips. It’s a great way to learn how to fine tune your website, enhance your service and more.

.

iPadOS 14.1 backups files from “On My iPad” to iCloud without option to exclude them

After upgrade to iPadOS 14.1 my iPad wants to backup every file from “On my iPad” folder, which includes around 50 GB, so the backup is always larger than my iCloud quota. I cannot find an option to exclude these files and I couclude that it must be these files because if I delete some files from the “On My iPad” folder the size of the backup decreases correspondingly. I tried to log out of iCloud and then log on again as somebody suggested somewhere, but the behaviour is still the same. I have not observerd such a problem with iPhone after upgrade to iOS 14.1.

Any idea how to exclude the files from “On My iPad” from iCloud backups ? Is there an setting I have not found yet ?

Storing Backups on Google Cloud (free/lowcost how ?)

I have been using a different 3rd party server provider for years, my backups are done nightly, sites are zipped and posted to AWS bucket.

I recently moved to Google Cloud and I see my egree traffic is shooting through the roof each night and costing a fortune.

So now I need a new solution, How do folks manage to backup servers on Google Compute that doest cost a fortune.

50% OFF – Reseller Hosting – cPanel & WHM | SSD | CloudLinux | Backups | 99.9% Uptime – VernalWeb | NewProxyLists

VernalWeb is offering 50% discount on the first invoice (Up to 12 months) for all SSD Reseller Hosting plans. For a limited time only, new customers can receive 50% DISCOUNT on any of our reseller web hosting plans.

Coupon Code: SmartWay

Code will be applied on regular price & Discount Price will be shown at Checkout

Coupon rules:
• Valid for all customers.
• This coupon can’t be used with any other special offer.
• This coupon is only available on 1, 6 or 12-month billing terms.
——————————————————————

Lots Of Free Gifts
• Free Domain (Yearly Term)
• Free CloudFlare CDN
• Free SSL
• Free Migration
• Free Daily Backups(No Extra Charge)
• Free DDoS Protection
• Free SEO Tools

Included with all shared hosting accounts:
* WHM Control Panel
* cPanel Control Panel
* Softaculous Script Auto-installer (Hundreds of one-click install scripts!)
* SNI Supported
* High-Redundancy RAID 10 Arrays
* WordPress Optimized
* Developer Friendly Atmosphere
* E-commerce Ready!
* Secure IMAP & POP3
* Apache Handlers
* FREE Website Migrations (We will migrate your current website, free of charge!)

Blazing Fast Server Hardware, Premium Bandwidth:
We SPARE NO EXPENSE for our offer! Our newest shared servers are purpose-built for performance using enterprise components, including RAID-10 SSDs for all storage.

SSD Reseller Hosting Plans

StartMe Plan
25GB Pure SSD Storage
250GB Data Transfer
25 cPanel Accounts
Free Domain Registration
Unlimited Subdomains
Unlimited E-mail Addresses
Unlimited MySQL Databases
Unlimited FTP Accounts
1-Click Software Installations
Instant Activation

Order Now!

GoodFace Plan
50GB Pure SSD Storage
500GB Data Transfer
50 cPanel Accounts
Free Domain Registration
Unlimited Subdomains
Unlimited E-mail Addresses
Unlimited MySQL Databases
Unlimited FTP Accounts
1-Click Software Installations
Instant Activation

Order Now!

FineGrow Plan
100GB Pure SSD Storage
1000GB Data Transfer
100 cPanel Accounts
Free Domain Registration
Unlimited Subdomains
Unlimited E-mail Addresses
Unlimited MySQL Databases
Unlimited FTP Accounts
1-Click Software Installations
Instant Activation

Order Now!

If you need any further information or help please feel free to contact us at Help Desk

Why VernalWeb?
• Instant Activation
• Multiple Locations
• Solid State Drive makes your website 20X faster
• 24X7 In-house Support via Phone or Email
• Lowest Pricing in the industry
• Enterprise Grade Server with reliability and Stability
• CloudLinux
• Multiple PHP versions (PHP4.4, PHP5.1, PHP5.2, PHP5.3, PHP5.4, PHP5.5, PHP5.6, PHP7.0, PHP7.1, PHP7.2, PHP7.3)
• 99.9% Uptime Guarantee
• 30 Days Money Back Guarantee
• 1 Click Software Installer
• No overloaded Servers
• Easy Control Panel(WHM)

Get in touch with us:
Facebook | Twitter | Pinterest | LinkedIn