Executing multiple cURL requests

Asked
Active3 hr before
Viewed126 times

8 Answers

executingmultiple
90%

Tells curl to use a separate operation for the following URL and associated options. This allows you to send several URL requests, each with their own specific options, for example, such as different user names or custom requests for each.,This is very useful to save network bandwidth, client and server resources, and overall the need of using multiple curl commands, as curl by default closes the connection when end of command is reached.,I have read that curl accepts multiple URLs in one command, but I'm unsure if this would process the URLs sequentially or in "parallel".,Starting curl version 7.36.0, the --next or -: command-line option allows to chain multiple requests, and usable both in command-line and scripting.

put it in a file like thus:

curl http: //example.com/?update_=1
   curl http: //example.com/?update_=3
   curl http: //example.com/?update_=234
   curl http: //example.com/?update_=65
load more v
88%

By default, you can run only 1 cURL command at a time. We will use xargs command to make multiple cURL requests in parallel.,Here is how to execute cURL command in parallel using xargs.,If you want to make requests to 3 separate domains in parallel, then here is the command to do it.,In the above commands, -P 5 option tells xargs to run 5 commands in parallel. If you do not include it, then xargs will run cURL requests sequentially.

xargs command accepts one or more strings as argument, from standard input, and generates separate command for each string. Here is an example.

echo "test1 test2 test3" | xargs mkdir
load more v
72%

Allows the processing of multiple cURL handles asynchronously. , This example will create two cURL handles, add them to a multi handle, and process them asynchronously. ,curl_multi_close() - Close a set of cURL handles, Returns a cURL multi handle on success, false on failure.

load more v
65%

Assuming I'm a big unix rookie, - I'm running a curl request through cron every 15 minutes. - Curl basically is used to load a web page (php) that given some arguments, acts as a script like:,Write a script with two curl requests in desired order and run it by cron, like,I have read that curl accepts multiple urls in one command but i'm unsure if this would process the ulrs sequentially or in "parallel". ,I think this uses more native capabilities

1) make a file called curlrequests.sh 2) put it in a file like thus:

curl http: //mysite.com/?update_=1
   curl http: //mysite.com/?update_=3
   curl http: //mysite.com/?update_=234
   curl http: //mysite.com/?update_=65

3) save the file and make it executable with chmod:

chmod + x curlrequests.sh

4) run your file:

. / curlrequests.sh

or

/path/to / file / curlrequests.sh

As a side note, you can chain requests with &&, like this:

curl http: //mysite.com/?update_=1 && curl http://mysite.com/?update_=2 && curl http://mysite.com/?update_=3

And execute in parallel using &:

curl http: //mysite.com/?update_=1 & curl http://mysite.com/?update_=2 & curl http://mysite.com/?update_=3
load more v
75%

I'm Carlos Roso. I'm a Software Engineer at Amazon. Former digital nomad at Toptal and Crossover. In love with open source and design.,I write about remote work, software and personal growth. Subscribe to enjoy my articles every once in a while.,Note: The code snippet above uses jq to parse the JSON response.,curl is not an async operation (unless appending a &)

show_top_hn() {

      # 1 st curl: fetch the top stories.Get the first element with jq.Store the result in 'post_id'.
      post_id = $(curl - s https: //hacker-news.firebaseio.com/v0/topstories.json?print=pretty | jq '.[0]')

         # Create a placeholder
         for the string replacement.We 'll replace '
         item_id ' with the real item ID later on.
         item_request = https: //hacker-news.firebaseio.com/v0/item/item_id.json?print=pretty

         # 2n d curl: fetch the top story 's info. Use string replacement to plug the id into the '
         item_request ' string.
         item_data = $(curl - s "${item_request/item_id/$post_id}")

         # Grab the title from the returned JSON title = $(echo $item_data | jq - r '.title')

         # Grab the URL from the returned JSON url = $(echo $item_data | jq - r '.url')

         # Compose the final string echo "$title"
         "->"
         "$url"

      }
40%

Use '&' after a command to background a process, and 'wait' to wait for them to finish. Use '()' around the commands if you need to create a sub-shell.,Connect and share knowledge within a single location that is structured and easy to search., How can a single creature safely flee from a combat? , Are there countries that ban public sector unions, but allow private sector ones?

Use '&' after a command to background a process, and 'wait' to wait for them to finish. Use '()' around the commands if you need to create a sub-shell.

#!/bin/bash

curl - s - o foo http: //example.com/file1 && echo "done1" &
   curl - s - o bar http: //example.com/file2 && echo "done2" & 
   curl - s - o baz http: //example.com/file3 && echo "done3" &

   wait
load more v
22%

Why we are discussing this(multiple cURL requests/curl_multi_exec).,Let’s take an example where we need to fetch records from the server based on ids assuming that server will return data for single id in a single request.,But if use multiple cURL requests then response time reduces significantly.,Result for each cURL request is in $result array some what like this:

Now, if we use cURL then we need to iterate loop for each id and request server therefore if there are 10 ids then on an average response time will be 10X of a single request.

/**
 * Webkul Software.
 *
 * @category Webkul
 * @author Webkul
 * @copyright Copyright (c) 2010-2016 Webkul Software Private Limited (https://webkul.com)
 * @license https://store.webkul.com/license.html
 */

$result = array();
foreach($ids as $id) {
   // URL from which data will be fetched
   $fetchURL = 'https://webkul.com&customerId='.$id;
   $ch = curl_init();
   curl_setopt($ch, CURLOPT_URL, $fetchURL);
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
   $result[] = curl_exec($ch);
   curl_close($ch);
}
load more v
60%

   curl http: //example.com/?update_=1 & curl http://example.com/?update_=2 & curl http://example.com/?update_=3

Other "executing-multiple" queries related to "Executing multiple cURL requests"