Google Indexing cURL 28 error

I have a website in WordPress and I am showing a sitemap to google with about 6,800 urls.

Some days ago there were 6,750 urls indexed. Today there are about 500.

Webmaster tools reports:
5,000 urls sent and indexed (this has been constant for a while now)
1,800 discovered and not indexed: Excluded

The excluded urls are not blocked in robots.txt or have a noindex follow tag. I tested some of them in google page speed insights and it reads the urls.

SEO Yoast health tool sais there is a CURL 28 ERROR that reads the next:

“Ryte offers a free indexability check for Yoast SEO users. The request to Ryte to check whether your site can be found by search engines failed due to an error.

Error details: cURL error 28: Operation timed out after 5001 milliseconds with 0 bytes received (http_request_failed)
SEMrush

If this is a live site, it is recommended that you figure out why the check failed.

So I investigated online and found this 5 points to check with the server:

  • Make sure your server is running a recent version of PHP and the cURL library.
  • Try to increase your Server Memory Limit settings.
  • The cURL error can be a dns related issue. Your hosting company might need to switch dns configuration to OpenDNS
  • Ask your host if there is some limitation with wp-cron, or if loopback is disabled.
  • Ask your host if there a firewall or security modules (e.g. mod_security ) that could block the outgoing cURL requests.

I checked them all and they say everything is ok. They just increased the memory limit.

I am not sure if this cURL 28 error could be causing an indexing problem with my site. What do you think? Any Ideas?

Thanks.

 

How to reference different wallets with curl JSONRPC requests Bitcoin core

I have a headless Bitcoin core 0.17.1 running on Debian 10, and have created a second wallet “johns-wallet”.

I’d like to get the walletinfo (or any wallet request) of that specific wallet via curl, not the cli.

The Bitcoin core API reference doesn’t offer an example of how to target different wallets when not using the CLI (and neither did my Google search).

I’ve tried:

curl --user johnsmith --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getwalletinfo", "params": () }' -H 'content-type: text/plain;' bitcoind:18332/wallets/johns-wallet/

and also

curl --user johnsmith --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getwalletinfo", "params": ("/wallets/johns-wallet/") }' -H 'content-type: text/plain;' bitcoind:18332

And also variations on the path itself to the wallet – none of them return a result, let alone an error.

Running without trying to specify a wallet returns:

{"result":null,"error":{"code":-19,"message":"Wallet file not specified (must request wallet RPC through /wallet/<filename> uri-path)."},"id":"curltest"}

(In case it matters, the Bitcoind is running in a docker container, with a named volume)

curl and wget fail to verify wildcard Let’s Encrypt certificate, but browsers work fine

We’ve obtained a wildcard certificate from Let’s Encrypt for civility.social and *.civility.social, using certbot. This works fine on all browsers and with curl and wget when verifying https://civility.social, or https://graphql.civility.social. The A records for those (sub)domains point to the same server, from which the certbot challenge was completed.

Later we’ve added a subdomain hosted on a different server, meet.. To use the same certificate on a different server, we copied the files making up the wildcard certificate from the original server to meet.. Both servers use NGINX. The problem is that wget and curl fail to fetch anything from meet.civility.social, even though browsers don’t complain. wget fails even with --no-check-certificate.

$ $ wget -v --debug --no-check-certificate https://meet.civility.social
Setting --check-certificate (checkcertificate) to 0
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.20.3 on linux-gnu.

Reading HSTS entries from ~/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)
--2020-06-26 21:39:15--  https://meet.civility.social/
Resolving meet.civility.social (meet.civility.social)... 157.245.170.94
Caching meet.civility.social => 157.245.170.94
Connecting to meet.civility.social (meet.civility.social)|157.245.170.94|:443... connected.
Created socket 3.
Releasing 0x000055ae59be63e0 (new refcount 1).
Initiating SSL handshake.
SSL handshake failed.
Closed fd 3
Unable to establish SSL connection.


$ curl https://meet.civility.social/
curl: (60) SSL certificate problem: unable to get local issuer certificate

What might be going on here?

Use curl to upload files to SharePoint Online – 403 FORBIDDEN

I have to upload files from linux box to SharePoint Online. I have used following command:

curl -k -ntlm --user myusername:password --upload-file test.txt "https://url/test.txt" -v

But I get error:

403 FORBIDDEN.

However, wget command works perfectly to download files.

wget --cookies=on --load-cookies cookies.txt --keep-session-cookies --user myusername --password mypassword "https://url/test.txt"

Is there a way to make it work?

magento2 – Curl magento 2 store on docker

I’m trying to curl a docker dev install from itself

using MagentoFrameworkHTTPClientCurl

I get a failure and the result SSL certificate problem: self signed certificate

Got the options set

$this->curl->setOption(CURLOPT_SSL_VERIFYHOST, 0);
$this->curl->setOption(CURLOPT_SSL_VERIFYPEER, 0);

Because it’s local/docker I can’t resolve the host

Therefore using CURLOPT_PROXY. Syntax looks like this: https://web.magento2.docker:443

Also tried this option: CURLOPT_HTTPPROXYTUNNEL

If I tail nginx logs they look like this

2020/06/21 23:41:48 (info) 16#16: *75 SSL_do_handshake() failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 172.27.0.3, server: 0.0.0.0:443

Has anyone experienced similar issue and got workaround?

On command line I can use this curl https://web.magento2.docker --insecure

curl – a fail-safe way to upload via HTTP

I use curl to upload zip files to Nexus. Recently it failed, but my ant build continued because the exit code was 0. So I added the –fail flag. However,

This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).

So how can I get a fail-safe method to do this, with curl or anything else?

php – cURL RETURNTRANSFER function in wp_remote_post

i have this cURL code below which is running on PHP yet when i did it with wordpress it didn’t work, i tried using WP_REMOTE_POST i couldn’t do the return transfer function of Curl, what can be a replacement for that or how can i use Curl in my wordpress website. (PS cURL is enabled and work fine for get method but have errors in post method.

my cURL code

try{
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
            curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: application/xml', 'Content-length: '. strlen($xml)) );
            curl_setopt($ch, CURLOPT_POST, true); 
            curl_setopt($ch, CURLOPT_HEADER, false);
            curl_setopt($ch, CURLOPT_URL,$url);  
            curl_setopt($ch, CURLOPT_POSTFIELDS, $input_xml);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
            $data = curl_exec($ch);
            curl_close($ch);
            }
            catch (Exception $e) { echo 'Caught exception: ', $e->getMessage(), "n";
            }

my wp_remote_post

$response = wp_remote_post( 
        $url, 
        array(
            'method' => 'POST',
            'timeout' => 45,
            'redirection' => 5,
            'httpversion' => '1.0',
            'headers' => array(
                'Content-Type' => 'application/xml'
            ),
            'body' => $input_xml,
            'sslverify' => false
        )
    );

multithreading – Use cURL and threads to make API calls

Requirement

We use a tool where we have Jython as one of the scripting languages available for making custom scripts. The requirement is that we need to make multiple API calls (in parallel) and aggregate those results in a single JSON using a script.

Previous attempts

My go-to instinct was to use requests but it didn’t work out. Please refer to the following StackOverflow question.

Jython: Getting SSLError when doing requests get

In my local environment, the simple Java way of doing it worked but when trying to do the same thing in the tool, it gave me an error.

Working implementation

So, the only other option that we could think of was using cURL. Following is a similar implementation that we have.

Note: I used the jsonplaceholder APIs for doing this test.

import json
import subprocess
from threading import Thread
import pprint


def execute_get_api(url, key, results):
    p1 = subprocess.Popen((
        'curl', '--location', '--request', 'GET', url
    ), stdout=subprocess.PIPE, shell=False)
    out, err = p1.communicate()
    results(key) = json.loads(out)


def get_post(post_id):
    post_url = 'https://jsonplaceholder.typicode.com/posts/%s' % post_id
    comments_url = 'https://jsonplaceholder.typicode.com/posts/%s/comments' % post_id

    # Temporary dictionary to store responses from the API calls
    results = {
        "post": "",
        "comments": ""
    }

    # Making the API calls
    threads = ()
    get_posts = Thread(target=execute_get_api,
                       args=(post_url, 'post', results))
    get_comments = Thread(target=execute_get_api,
                          args=(comments_url, 'comments', results))

    get_posts.start()
    get_comments.start()

    threads.append(get_posts)
    threads.append(get_comments)

    for th in threads:
        th.join()

    return results


def driver():
    pp = pprint.PrettyPrinter(indent=4)
    pp.pprint(get_post(1))


if __name__ == '__main__':
    driver()

Output

{   'comments': (   {   u'body': u'laudantium enim quasi est quidem magnam voluptate ipsam eosntempora quo necessitatibusndolor quam autem quasinreiciendis et nam sapiente accusantium',
                        u'email': u'Eliseo@gardner.biz',
                        u'id': 1,
                        u'name': u'id labore ex et quam laborum',
                        u'postId': 1},
                    {   u'body': u'est natus enim nihil est dolore omnis voluptatem numquamnet omnis occaecati quod ullam atnvoluptatem error expedita pariaturnnihil sint nostrum voluptatem reiciendis et',
                        u'email': u'Jayne_Kuhic@sydney.com',
                        u'id': 2,
                        u'name': u'quo vero reiciendis velit similique earum',
                        u'postId': 1},
                    {   u'body': u'quia molestiae reprehenderit quasi aspernaturnaut expedita occaecati aliquam eveniet laudantiumnomnis quibusdam delectus saepe quia accusamus maiores nam estncum et ducimus et vero voluptates excepturi deleniti ratione',
                        u'email': u'Nikita@garfield.biz',
                        u'id': 3,
                        u'name': u'odio adipisci rerum aut animi',
                        u'postId': 1},
                    {   u'body': u'non et atquenoccaecati deserunt quas accusantium unde odit nobis qui voluptatemnquia voluptas consequuntur itaque dolornet qui rerum deleniti ut occaecati',
                        u'email': u'Lew@alysha.tv',
                        u'id': 4,
                        u'name': u'alias odio sit',
                        u'postId': 1},
                    {   u'body': u'harum non quasi et rationentempore iure ex voluptates in rationenharum architecto fugit inventore cupiditatenvoluptates magni quo et',
                        u'email': u'Hayden@althea.biz',
                        u'id': 5,
                        u'name': u'vero eaque aliquid doloribus et culpa',
                        u'postId': 1}),
    'post': {   u'body': u'quia et suscipitnsuscipit recusandae consequuntur expedita et cumnreprehenderit molestiae ut ut quas totamnnostrum rerum est autem sunt rem eveniet architecto',
                u'id': 1,
                u'title': u'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
                u'userId': 1}}

I’m not sure if there is any other way of doing this. Also, do let me know if this approach can be improved.

Any help will be appreciated 🙂

brute force – curl to wfuzz translation

I am trying to run wfuzz to match the curl command which works, I know valid credentials but it doesn’t seem to pass it properly.

wfuzz -c -w user -w pass -b "session=cookie" --digest FUZZ:FUZ2Z "http://192.119.2.3/"

(user and pass files contain user and pass accordingly)

curl -c cookie --digest -u user:pass http://192.119.2.3

The target is running Gunicorn web server