Web scraping code using selenium and Python

I have a code to delete data from a web page using selenium and python.
The code selects a URL from the database and searches for it using the selenium chrome driver and retrieves the result and inserts it into a postgreSQL database.

Can anyone revise this code and let me know what I need to modify to make it better and organized.

The code below searches for a URL in google, for example the example "https://www.google.com/search?&q=samung+galaxy+note+10", so results will be displayed in the bar Google suggestions (adbar) from that to fetch the Product name, company name, location, product URL and price.

Get this in 5 lists and merge these results using the ZIP function and the final result will be a TUPLE named "final_results", the content of which must be inserted into the database.

Note: the code works fine now, but it is not optimized and I would like to get recommendations on how to optimize this.
In addition, after some research, Google blocks the display of the search to select the question "Whether it is a robot" and there after the script fails. Another way to recover?

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import psycopg2
import os
import glob
import datetime

final_results=()
positions=()

option = webdriver.ChromeOptions()
option.add_argument("—-incognito")
browser = webdriver.Chrome(executable_path='/users/user_123/downloads/chrome_driver/chromedriver', chrome_options=option)


#def db_connect():
try:
    #Database connection string
     DSN = "dbname='postgres' user='postgres' host='localhost' password='postgres' port='5432'"
     #DWH table to which data is ported
     TABLE_NAME = 'staging.search_url'
     #Connecting DB..
     conn = psycopg2.connect(DSN)
     print("Database connected...")
     #conn.set_client_encoding('utf-8')
     cur = conn.cursor()
     cur.execute("SET datestyle='German'")
except (Exception, psycopg2.Error) as error:
     print('database connection failed')
     quit()

def get_products(url):
    browser.get(url)
    names = browser.find_elements_by_xpath("//span(@class='pymv4e')")
    # simplified the filtering
    # note that python objects have a "truth" value,
    # and that empty strings and empty lists have a False value
    #product = (x.text for x in names if x.text.strip())

    upd_product_name_list=list(filter(None, names))
    product_name = (x.text for x in upd_product_name_list)
    product = (x for x in product_name if len(x.strip()) > 2)

    upd_product_name_list.clear()
    product_name.clear()
    return product
    #print(product)      


##################################
search_url_fetch="""select url_to_be_searched from staging.search_url where id in(65,66,67,68)"""
psql_cursor = conn.cursor()
psql_cursor.execute(search_url_fetch)
serach_url_list = psql_cursor.fetchall()
print('Fetched DB values')
##################################

for row in serach_url_list:
    passed_url=''
    passed_url=str(row)
    passed_url=passed_url.replace(',)','')
    passed_url=passed_url.replace('(','')
    new_url=''
    new_url=passed_url(1:len(passed_url)-1)
    print('Passed URL :'+new_url)
    print("n")

    filtered=()
    filtered.clear()
    filtered = get_products(new_url)
    if not filtered:
        new_url=new_url+'+kaufen'
        get_products(new_url)
        print('Modified URL :'+new_url)

    if filtered:
         print(filtered)
         positions.clear()
         for x in range(1, len(filtered)+1):
           positions.append(str(x))
         gobal_position=0
         gobal_position=len(positions)
         print('global postion first: '+str(gobal_position))
         print("n")

         company_name_list = browser.find_elements_by_xpath("//div(@class='LbUacb')")
         # use list comprehension to get the actual repo titles and not the selenium objects.
         company = ()
         company.clear()
         company = (x.text for x in company_name_list)
         # print out all the titles.
         print('Company Name:')
         print(company, 'n')


         price_list = browser.find_elements_by_xpath("//div(@class='e10twf T4OwTb')")
         # use list comprehension to get the actual repo titles and not the selenium objects.
         price = ()
         price.clear()
         price = (x.text for x in price_list)
         print('Price:')
         print(price)
         print("n")

         urls=()
         urls.clear()
         find_href = browser.find_elements_by_xpath("//a(@class='plantl pla-unit-single-clickable-target clickable-card')")
         for my_href in find_href:
             url_list=my_href.get_attribute("href")
             urls.append(url_list)
             #print(my_href.get_attribute("href"))
         print('URLS:')
         print(urls)
         print("n")

         print('Final Result: ')
         result = zip(positions,filtered, urls, company,price)
         final_results.clear()
         final_results.append(tuple(result))
         print(final_results)
         print("n")


         print('global postion end :'+str(gobal_position))
         i=0
         #try:
         for d in final_results:
                print( d(i))
                while i <= gobal_position:
                  cur.execute("""INSERT into staging.pla_crawler_results(position, product_name, url,company,price) VALUES (%s, %s, %s,%s, %s)""", d(i))
                  print('Inserted succesfully')
                  conn.commit()
                  i=i+1
         #except (Exception, psycopg2.Error) as error:
             #pass

Why HTTP status code has the following

Hi. I hope I post this in the right section.

The HTTP status code checker for my website (hosted by my fairly new host) has the following result:

<HTTP / 1.1 200 OK
<Connection: Keep-Alive
<Cache-Control: public, max-age = 2592000
<Expires: Fri 21 Feb 2020 17:44:30 GMT
<Content type: text / html
<Last update: Fri, May 10 2019 19:22:26 GMT
<Accept-Ranges: bytes
<Content Length: 37358
<Date: Wed 22 Jan 2020 17:44:30 GMT
<Server: LiteSpeed
<Alt-Svc: quic = ": 443"; ma = 2592000; v = "39.43.46.50", h3-Q039 = ": 443"; ma = 2592000, h3-Q043 = ": 443"; ma = 2592000, h3-Q046 = ": 443"; ma = 2592000, h3-Q050 = ": 443"; ma = 2592000, h3-23 = ": 443"; ma = 2592000, h3-24 = ": 443"; ma = 2592000

What is the "Alt-Svc:"?

I searched for information on this on developer.mozilla. website org and it indicates that this may not be compatible with these certain browsers – Edge, IE, Safari, Safari on IOS. Should this be used when many of my clients are using IOS to access my site?

When checking header status codes, I don't remember receiving this result when I was with previous hosts.

Any help / advice on this, could this be a problem?

Thanks in advance for your help.

java – How do I configure the Calculator web service code example in netbeans to use x509 certificates for authentication?

I have successfully followed the steps described here: https://netbeans.org/kb/docs/websvc/jax-ws.html

Everything is working properly (i.e. I can run a client jar file from the terminal and have it send two numbers to add; the calculator web service receives them and returns the correct sum in the response SOAP).

I now want to add authentication using x509 certificates, but I can't find any specific documentation on how to do this. The closest link I found is a secure calculator here:

https://netbeans.org/kb/docs/websvc/wsit.html#Exercise_2_2

But that seems to use "Username authentication with symmetric keys", which is not what I'm looking for.

I'm looking for the calculator client to send their x509 certificate to the calculator web service. The calculator's web service authenticates the x509 certificate it just received from the client. If the authentication succeeds, it will proceed to add the two numbers sent by the client. Otherwise, it returns "invalid certificate".

Sounds like a simple thing to do, but I can't find any documentation or an example of a netbeans project that does it.

This website looks promising: https://docs.oracle.com/cd/E17802_01/webservices/webservices/reference/tutorials/wsit/doc/WSIT_Security9.html#wp162511

More specifically the example: Mutual Certificates Security (MCS). However, when I get to the "Securing the Sample Web Service Client Application (MCS) section that says:

Select the WSIT Configuration tab from the CalculatorWSService dialog.

Netbeans does not have a WSIT configuration tab. This therefore seems obsolete since I started using Netbeans 8.2.

I would appreciate any help from the community.

wp query – What code to use in an array to call the current subcategory?

This function lists only the products of the subcategory called – bags:

$args = array( 'post_type' => 'product', 'product_cat' => 'bags')
$loop = new WP_Query( $args ); while ( $loop->have_posts() ) : $loop->the_post();

If I run this code on a category page, how can I change the hard coded "bags" to dynamically produce the products in the active subcategory of the category page?

How can I install Angular Generator in VS Code?

In the first angular guide https://angular.io/start, they show the following image:

enter description of image here

How can I install this extension in VS Code?

Autocorrelation code and power spectrum of a Lorentz ODE 3D system

I need codes for:

1) Autocorrelation

2) Power spectrum (db v / s Hz)

For a Lorentz ODE 3-D system. A MATLAB / Mathematica code will be preferred.

Thanks in advance.

web server – Does a library have to be present on the computer / server to run the code based on that?

Suppose that a programmer in a technology company codes software in a particular programming language and uses a library of it. So, if this code is also deployed on a server, should this server have this library in its system?

I guess the answer is yes. If that is the answer, then the web hosting company (the one that hosts the programmer's company) wouldn't it be in an awkward position because it hosts many sites and even with one company, many libraries used?

So, would they download (maybe even manually) each library / module that their client uses?

How could they handle this?

javascript – Toy Robot Code – Improvements?

I know this code worked as I tested it with the case reviews discussed below in the code, and I even added some of mine, but the developers who tested me came back with the following reviews:

  1. Everything in one file, no attempt to modularize or separate concerns
  2. No tests at all. You could possibly forgive an entry level
    0% test submission, but seniors should demonstrate their
    value
  3. Everything is declared as a let variable which suggests
    can be canceled. I guess most of them should be consts.
    Again, the senior should demonstrate this
  4. The submission can be difficult to read with the excessive comments and all that together

I have included their original tests, as well as three of mine, commented in the code below … I don't know if they were ignored or rejected … Someone could- enlighten us on how this code could have been improved? If I receive another coding test, I would like to have an overview of what is expected in terms of code formatting compared to this kind of comments …

The description

  • The application is a simulation of a toy robot moving on a square table, dimensions 5 units x 5 units.
  • There are no other obstacles on the table surface.
  • The robot is free to move around the table surface, but must be prevented from falling to destruction. Any movement that
    prevent the robot from falling off the table,
    however, other valid motion commands must still be authorized.

Create an application that can read commands like this:

PLACE X,Y,F
MOVE
LEFT
RIGHT
REPORT
  • PLACE will place the toy robot on the table in position X, Y and oriented NORTH, SOUTH, EAST or WEST.
  • Origin (0,0) can be considered as the most SOUTHWEST corner.
  • The first valid command for the robot is a PLACE command, after which any sequence of commands can be issued, in any order, including
    another PLACE command. The app should delete all orders
    in order until a valid PLACE command has been executed.
  • MOVE moves the toy robot one unit forward in the direction it is currently facing.
  • LEFT and RIGHT will rotate the robot 90 degrees in the specified direction without changing the robot's position.
  • REPORT will announce the X, Y and F of the robot. It can be in any form, but standard output is sufficient.

  • A robot that is not on the table can choose to ignore the MOVE, LEFT, RIGHT and REPORT commands.

  • The input can come from a file or from standard input, depending on the choice of the developer.
  • Provide test data to exercise the application.

constraints

  • The toy robot must not fall off the table during movement. This also includes the initial placement of the toy robot.
  • Any movement that would cause the robot to fall should be ignored.

Example of entry and exit

Example a

PLACE 0,0,NORTH
MOVE
REPORT

Expected production:

0,1,NORTH

Example b

PLACE 0,0,NORTH
LEFT
REPORT

Expected production:

0,0,WEST

Example c

PLACE 1,2,EAST
MOVE
MOVE
LEFT
MOVE
REPORT

Expected production

3,3,NORTH

deliverables

Ruby source files, test data, and any test code.

There is no need to provide graphical output showing the
toy robot movement.

// Toy Robot!

/* A few caveats for this solution
    1) Input is said to be of a text format, we will make this an array with each element containing a command
        e.g. ("PLACE 1,2,NORTH", "MOVE", "LEFT", "MOVE", "REPORT")
    2) I'm doing this in NodeJS (JavaScript)
    3) To run this one could run this directly in the Node terminal, or copy and paste it into JSBin.com and execute it there; 
        or reference it in an HTML file; so in saying that, the executed input will be located at the bottom of the functional definition.

*/

/* CODE STARTS */

"use strict"
let Robot = function(aCommands) {

    let iTableX = 5; // Width of Table
    let iTableY = 5; // Height of Table
    let aDir = ("NORTH","EAST","SOUTH","WEST"); // Directions
    let sRobotDir = 0; // Sets the default based off the above array
    let aCommandSplit = ();

    // Sets default setting for robot's coordinates X, Y
    let iRobotX = 0;
    let iRobotY = 0;
    let bRobotPlaced = false; // Determines if the robot has been placed on the board...
    let aTokens = (); // Sets up a basic array to hold our tokens

    // Can I tell you I'm already enjoying programming this? :D

    /* Let's set up some basic rules here
    1) We split each element into a command and optional options (options only available with PLACE command)
    2) We iterate through each array element, anything but a starting PLACE command is ignored.
    3) As we proceed through each element, a switch command will assess the value of the command, and then takes an appropriate action.
    */

    // Tokenizing engine
    for(let iLoop = 0, iLen = aCommands.length; iLoop < iLen; iLoop++) {
        aCommandSplit = aCommands(iLoop).toUpperCase().split(" "); // Initiates primary split, ensuring all commands are upper case
        if(aCommandSplit.length == 2) {
            aCommandSplit(1) = aCommandSplit(1).split(","); // split parameters
        } else {
            aCommandSplit(1) = (); // defafult
        }
        aTokens.push(aCommandSplit);            
    }

    // Invalid Command
    let invalidCommand = function(item) {
        console.log(">> INVALID COMMAND: '"+item.join(" ")+"' IGNORED");
    };

    // Checks if value is below zero
    let isBelowZero = (val) => (val < 0);

    // Checks if value is equal to or above a certain limit
    let isBeyondTableLimit = (val,limit) => (val >= limit);

    // Checks if string represents a positive value
    let isPosInteger = function (str) {
        var n = Math.floor(Number(str));
        return n !== Infinity && String(n) === str && n >= 0;
    };

    // Checks to see if the string is a direction in the direction array
    let isDirection = (str) => !!~aDir.indexOf(str);

    // Returns the index of the direction string
    let whichDirection = (str) => aDir.indexOf(str);

    // Validates the parameters for placement
    let checkPlaceParams = (arr) => (
            (arr != ()) &&                              // array isn't empty
            (arr.length == 3) &&                        // array contains three elements
            isPosInteger(arr(0)) &&                     // element 0 is a positive integer or zero
            isPosInteger(arr(1)) &&                     // element 1 is a positive integer or zero
            (typeof(arr(2)) == "string") &&             // element 2 is a string
            isDirection(arr(2)) &&                      // element 2 is a direction
            !isBelowZero(+arr(0)) &&                    // element 0 is equal to zero or above
            !isBelowZero(+arr(1)) &&                    // element 1 is equal to zero or above
            !isBeyondTableLimit(+arr(0), iTableX) &&    // element 0 is not bigger than the table size
            !isBeyondTableLimit(+arr(1), iTableY)       // element 1 is not bigger than the table size
        );

    // Sets the placement
    let place = function(item) {
        if(checkPlaceParams(item(1))) {
            iRobotX = +item(1)(0);
            iRobotY = +item(1)(1);
            sRobotDir = whichDirection(item(1)(2));
            bRobotPlaced = true;
        }
    };

    // Rotates the robot left or right.
    let rotate = function(turn) {
        let newDir = (sRobotDir + ((turn=="LEFT")?3:1))%4;
        sRobotDir = newDir;
    };
    // Checks to see if the robot can move in a certain direction
    let canMove = function() {
        switch(sRobotDir) {
            case 0: // North
                return (!isBeyondTableLimit(iRobotY+1, iTableY));
                break;
            case 1: // East
                return (!isBeyondTableLimit(iRobotX+1, iTableX));
                break;
            case 2: // South
                return (!isBelowZero(iRobotY-1));
                break;
            case 3: // West
                return (!isBelowZero(iRobotX-1));
                break;
        }
    }
    // Moves the robot
    let move = function() {
        if(sRobotDir % 2 == 1) { // If east or west
            if(sRobotDir == 1) { // if east
                iRobotX++;
            } else {
                iRobotX--;
            }
        } else {
            if(sRobotDir == 0) { // if north
                iRobotY++;
            } else {
                iRobotY--;
            }
        }
    };

    // Reports position and direction facing
    let report = function() {
        console.log((iRobotX,iRobotY,aDir(sRobotDir)).join(","));
    };

    // Step through the tokens
    aTokens.forEach(function(item, index) {
        switch (item(0)) {
            case "PLACE":
                place(item);
                break;
            case "LEFT":
            case "RIGHT":
                bRobotPlaced ? rotate(item(0)) : invalidCommand(item);
                break;
            case "MOVE":
                (bRobotPlaced && canMove()) ? move() : invalidCommand(item);
                break;
            case "REPORT":
                bRobotPlaced ? report() : invalidCommand(item);
                break;
            default:
                invalidCommand(item);
                break;
        }
    });
};  

/* CODE ENDS */

/* EXECUTE BLOCK STARTS */

    // Original Test Cases
    // Robot(("PLACE 0,0,NORTH","MOVE","REPORT"));  // 0,1,NORTH
    // Robot(("PLACE 0,0,NORTH","LEFT","REPORT")); // 0,0,WEST
    // Robot(("PLACE 1,2,EAST","MOVE","MOVE","LEFT","MOVE","REPORT"));  // 3,3,NORTH

    // My Test Cases
    // Robot(("PLACE 1,2,NORTH", "MOVE", "LEFT", "MOVE", "REPORT"));      // Expected result: 0,3,WEST
    // Robot(("MOVE","PLACE 0,3,WEST","MOVE","RIGHT","MOVE","MOVE","REPORT")); // Expected result: 0,4,NORTH
    /* NOTE: Robot should ignore first MOVE command because it hasn't been PLACE-d yet, and then ignore the second MOVE command because it can't go any further west, be able to move on the third MOVE command, but then ignore the last MOVE command because it can't go any further north. */
    //Robot(("PLACE 2,2,NORTH","KEFT","NOVE","MOVE","REPORT"));
    /* Should ignore the "KEFT" command as it can't rotate "KEFT", then ignore the "NOVE" command, but obey the "MOVE" command. 
        Final result: 2,3,NORTH */

/* EXECUTE BLOCK ENDS */

physics – How to solve the problem of "non-numeric values" for the following code?

In the following code, I added? NumericQ whenever necessary but I get the mentioned error:

rp(t_) := {f(t), g(t), h(t)}; r(x_, y_, z_) := {x, y, z}

intB(x_, y_, z_, t_) := Cross(rp(t), 
 r(x, y, z) - rp(t))/((r(x, y, z) - rp(t)).(r(x, y, z) - rp(t)))^3/ 2

ex = {1, 0, 0}; ey = {0, 1, 0}; ez = {0, 0, 1};

intBx(x_, y_, z_, t_) := intB(x, y, t).ex;
intBy(x_, y_, z_, t_) := intB(x, y, t).ey;
intBz(x_, y_, z_, t_) := intB(x, y, t).ez;

Bx(x_?NumericQ, y_?NumericQ, z_?NumericQ, a_?NumericQ, b_?NumericQ, 
 t_?NumericQ) := NIntegrate(intBx(x, y, z, t), {t, a, b});
By(x_, y_, z_, a_, b_) := NIntegrate(intBy(x, y, z, t), {t, a, b});
Bz(x_, y_, z_, a_, b_) := NIntegrate(intBz(x, y, z, t), {t, a, b});
f(t_?NumericQ) := Cos(t);
g(t_?NumericQ) := Sin(t); 
h(t_?NumericQ) := 0.05 t;a = -5; b = 5;

Plot(Bx(x, 0, 1, a, b), {x, 0, 5}, PlotRange -> All)

8 – How not to cache part of the TWIG code?

The following code is used to display whether users are online, absent, or offline.

My problem is that the current date "timestamp" is cached, so there is no change in user status unless I manually clear the cache with "drush cr" .

How can I fix it?

user – full.html.twig:

  {% if (date().timestamp - user.access.value) < 900 %}
    
Online
{% elseif (date().timestamp - user.access.value) < 1800 %}
Absent
{% else %}
Offline
{% endif %}