1 |
curl [options] [URL...] |
When you specify no additional options for curl, it will simply fetch the data using the default communication protocol HTTP. The below command will retrieve the source code for the given site.
1 |
$ curl www.example.com |
As you can guess, simply downloading the source of a website is rarely useful to normal users. Thankfully, you can easily specify what file to download. Take a look at the below example to see this in action.
1 |
$ curl -o pic-of-the-day.jpg https: //apod .nasa.gov /apod/image/1912/Orava_Duskova_WinterHexagon .jpg |
We can also download files and save them using their default names. Youll need to utilize the -O option for this purpose. The below command demonstrates this using the same image file in the above example.
1 2 |
$ curl -O https: //apod .nasa.gov /apod/image/1912/Orava_Duskova_WinterHexagon .jpg $ curl --remote-name https: //apod .nasa.gov /apod/image/1912/Orava_Duskova_WinterHexagon .jpg |
cURL also allows us to download multiple files at the same time. Its pretty straightforward, and all you need to do is supply their URLs one after another after the -O option.
1 2 3 |
$ curl -O https: //apod .nasa.gov /apod/image/1912/Orava_Duskova_WinterHexagon .jpg \ -O https: //persiadigest .com /uploads/gallery/1547451742 .jpg \ -O https: //persiadigest .com /uploads/gallery/1547452172 .jpg |
Sometimes your downloads may get interrupted for some reason. Thankfully, cURL allows us to resume paused downloads using the -C option. This option requires an offset to determine from where to continue the download. This offset is the exact number of bytes that have been downloaded already, and you want to skip.
1 2 |
$ curl -C- -O https: //persiadigest .com /uploads/gallery/1547452172 .jpg $ curl -- continue -at - -O https: //persiadigest .com /uploads/gallery/1547452172 .jpg |
The cURL command in Linux allows us to select parts of URLs by specifying them as a set inside square braces. Take a quick look at the below illustrations to see them in action.
1 |
$ curl -O https: //persiadigest .com /uploads/gallery/1547452 [200-300:25].jpg |
Its possible to download remote data using URLs from a file. Youll need to leverage xargs for this purpose. First, create a test file containing the URLs for your files. Now, take a look at the next example to understand how this works.
1 |
$ xargs -n 1 curl -O < urls.txt |
HTTP headers may contain several name-value pairs which enable the client or server to transmit additional information regarding a web request/response. They usually contain information like content type, user agent, and encoding. The -I option tells cURL to fetch this data from remote servers.
1 2 |
$ curl -I https: //google .com/ $ curl -- head https: //google .com/ |
The HTTP/2 is a significant revamp of HTTP and will replace it as the standard web protocol in the future. Many sites have started to adopt it, and you can easily check whether a site supports HTTP/2 or not using the below cURL command in Linux.
1 |
$ curl -I --http2 https: //google .com/ | grep HTTP |
Sometimes you may query for a file thats been moved to a different server. By default, curl returns the new location of this file but doesnt follow the new location. You can tell curl to follow the redirects in such situations by using the -L or location option, as demonstrated below.
1 2 3 |
$ curl google.com $ curl -L google.com $ curl --location google.com |
The cURL command in Linux allows us to limit the transfer rate using the limit-rate parameter. This is a useful feature for Linux shell scripts or automated jobs like CRON.
1 |
$ curl --limit-rate 1m -O https: //github .com /JetBrains/kotlin/releases/download/v1 .3.61 /kotlin-native-linux-1 .3.61. tar .gz |
You can tell cURL to only fetch a file if it has been modified after/before a pre-defined date. The -z option is used for specifying the date parameters, as shown by the below examples.
1 2 |
$ curl -z 01-Dec-19 -O https: //persiadigest .com /files/en/news_albums/7596/83/thumbnails/thm_4188_934 .jpg $ curl -- time -cond 01-Dec-19 -O https: //persiadigest .com /files/en/news_albums/7596/83/thumbnails/thm_4188_934 .jpg |
1 |
$ curl -z -01-Dec-19 https: //persiadigest .com /files/en/news_albums/7596/83/thumbnails/thm_4188_934 .jpg |
The curl command allows users to pass authentication information for remote servers. This comes in handy when fetching documents from sites protected via credentials. Were demonstrating a simple mock example to illustrate this issue for our audience below.
1 2 |
$ curl -u USER: PASSWORD www.example.com /some/file $ curl --user USER: PASSWORD www.example.com /some/file |
We can very easily leverage cURL to send or receive files from FTP servers. The below commands will illustrate how to establish an FTP connection using the cURL command in Linux. Youll also learn how to download documents from remote FTP servers.
1 |
$ curl -u FTP_USER:FTP_PASSWORD ftp : //ftp .example.com /public/dir/ |
1 |
$ curl -u FTP_USER:FTP_PASSWORD -O ftp : //ftp .example.com /public/dir/image .jpg |
Uploading documents to FTP servers is also pretty straightforward. All you need to do is pass the -T option, followed by the name of your file or files. The following examples demonstrate how to upload a single file, multiple files, and a range of files over FTP.
1 2 3 |
$ curl -T image1.jpg -u FTP_USER:FTP_PASSWORD ftp : //ftp .example.com /public/dir/ $ curl -T image1.jpg image2.jpg image3.jpg -u FTP_USER:FTP_PASSWORD ftp : //ftp .example.com /public/dir/ $ curl -T image[4-10].jpg -u FTP_USER:FTP_PASSWORD ftp : //ftp .example.com /public/dir/ |
The cURL utility makes it effortless to transfer form data using the HTTP POST request. You can upload not only text but also binary data. So, its possible to send both files and text input. Additionally, curl allows you to specify the content type.
1 2 3 |
$ curl -F 'username=user' 'password=userpassword' www.example.com /login $ curl -F 'username=user' 'password=@/etc/passwd' www.example.com /login $ curl -F 'username=user' 'password=<passfile' www.example.com /login |
When you send an HTTP request, it contains the information about your client using the user agent field. By default, cURL command in Linux will send curl/<version> as the user agent. Many admins block such requests to prevent potential scraping attempts. So, its a good practice to change the user agent into something more common.
1 2 3 |
$ curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" https: //getfedora .org/ $ curl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" https: //getfedora .org/" \ www.example.com |
The HTTP protocol allows multiple request methods for communicating with remote servers. Each method has its own purpose, such as POST for posting data, PUT for updating data, and DELETE for deleting. By default, cURL performs an HTTP GET request that simply fetches the specified data.
1 2 |
$ curl -d "username=user&password=passphrase" -X POST www.example.com /login $ curl --data "username=user&password=passphrase" -H "Content-Type: application/x-www-form-urlencoded" -X POST www.example.com /login |
HTTP PUT requests allow users to update the value of a field over remote connections. It works almost similar to the above curl command, as shown in the below example.
1 2 |
$ curl -d "{\"parameter\":\"value\"}" -H "Content-Type: application/json" -X PUT www.example.com /api $ curl --data "{\"parameter\":\"value\"}" -H "Content-Type: application/json" -X PUT www.example.com /api |
The HTTP DELETE verb enables us to remove a value from a remote field. Its often used in Linux bash scripts and offers great flexibility.
1 2 |
$ curl -d "{\"parameter\":\"value\"}" -X "DELETE" www.example.com /api $ curl --data "{\"parameter\":\"value\"}" -X "DELETE" www.example.com /api |
Cookies are small pieces of data sent to your system by remote servers when you visit a page there. Theyre usually used for increasing the user experience. The curl command allows us to save cookies from a website for later inspection. This is a straightforward task and requires a simple option called cookie-jar.
1 2 3 |
$ curl --cookie-jar up-cookies.txt https: //google .com/ $ curl --cookie-jar persia-cookies.txt https: //persiadigest .com /uploads/gallery/1547451742 .jpg -O $ curl --cookie-jar - https: //persiadigest .com /uploads/gallery/1547451742 .jpg |
The curl command also enables us to send data as cookies to remote HTTP servers. You will need to utilize the -b or cookie option for this purpose, as demonstrated by the below examples. You need to pass the data in name-value pairs separated by equal signs.
1 2 |
$ curl --cookie up-cookies.txt https: //google .com/ $ curl -b "USER_TOKEN=Yes" https: //google .com/ |
Linux curl command allows users to use HTTP proxies. If not specified, the default port is assumed to be 1080. Since all communications made over an HTTP proxy is converted to HTTP, some protocol-dependent functionalities might become unavailable.
1 2 |
$ curl -x 192.168.99.1:8888 https: //example .com/ $ curl -U user:passphrase -x 192.168.99.1:8888 https: //example .com/ |
Often site admins want to test their websites locally before making them available to the rest of the word. The following cURL command in Linux will illustrate how to resolve domain names for this purpose.
1 |
$ curl --resolve www.example.com:80:localhost https: //www .example.com/ |
We can leverage the curl command for sending emails from automated CRON jobs or scripts. The cURL utility uses the SMTP(Simple Mail Transfer Protocol) for this job. Youre required to provide information such as address details and mail server alongside the message.
1 |
$ curl --mail-from admin@ test .com --mail-rcpt user@ test .com smtp: //testmailserver .com |
Although many curl commands have a default timeout, its not the case with all options. The curl utility allows us to specify the time limit for operations like downloads or uploads. The -m or max-time options can be used to enable this, as illustrated in the below examples.
1 2 |
$ curl -m 120 -O www.example.com /pics/image [1-10].jpg $ curl --max- time 120 -O www.example.com /pics/image [1-10].jpg |
Contrary to operation timeouts, connection timeouts only specify the maximum amount of time curl spends behind to establish a remote connection. The curl command exposes the connect-timeout option to do this. Its also provided in seconds. Take a quick glance at the next example to see this in action.
1 |
$ curl --connect-timeout 60 -O www.example.com /pics/image [1-10].jpg |
Sometimes you may want to download a large file in chunks of various sizes. Its a useful feature for automated scripts or when you need to cap the download amount. The range option provides a dynamic way to handle this using curl, as demonstrated below.
1 2 3 |
$ curl --range 0-99999999 -O https: //osdn .net /dl/manjaro/manjaro-xfce-18 .1.4-191210-linux54.iso $ curl --range 100000000-199999999 -O https: //osdn .net /dl/manjaro/manjaro-xfce-18 .1.4-191210-linux54.iso $ curl --range 200000000- -O https: //osdn .net /dl/manjaro/manjaro-xfce-18 .1.4-191210-linux54.iso |
The curl command allows you to enable silent mode. In this mode, curl will refrain from displaying the progress bar and error messages, if any. However, youll still get the data you request in your terminal. To redirect it to a file, you can use either the -O or -o option.
1 2 |
$ curl -s -O https: //www .example.com /picture .jpg $ curl --silent -O https: //www .example.com /picture .jpg |
The fail mode allows curl to suppress any output on server errors. Normally in the case of silent mode, curl will download an HTTP document that states the state of the requested resource. But, enabling fail mode makes curl stop whenever it encounters an error on the server-side.
1 2 |
$ curl -f -O https: //www .example.com /picture .jpg $ curl --fail -O https: //www .example.com /picture .jpg |
Earlier, we saw how to send emails from the terminal using curl. You can also use this tool to read your incoming email messages. Take a look at the following example to understand this in detail.
1 |
$ curl -u username:passphrase imap: //mail .example.com |
1 |
$ curl -u username:password imap: //mail .example.com -X 'UID FETCH 123' |
Many modern-day servers implement authentication based on certificates. cURL allows you to specify the certificate file for communicating with such secure servers. You will need to utilize the -E or cert option to do this, as demonstrated via the below examples.
1 2 |
$ curl -E path /to/certificate .crt:passphrase ftp : //example .com $ curl --cert path /to/certificate .crt:passphrase ftp : //example .com |
You may want to verify the certificates of remote servers for many reasons, one being security. The curl utility provides the cacert option to do this. It takes the name of the certificate as its argument. Youll understand it more clearly by inspecting the below example.
1 |
$ curl --cacert path /to/certificate .crt:passphrase ftp : //example .com |
Some sites on the internet are hosted using self-signed SSL certificates. When querying these resources, curl will print an SSL certificate warning message. We can utilize the -k option to ignore these warnings and carry on with our connection.
1 2 |
$ curl -k ftp : //example .com /images/ [1-10].jpg $ curl --insecure ftp : //example .com /images/ [1-10].jpg |
The curl command allows us to query for the status of a remote site quite effortlessly. Take a quick look at the below example to see this in action.
1 |
$ curl -Is https: //www .facebook.com -L | head -n 1 |
You can also leverage curl to find out the geo-location of a remote resource. The following command will print out all information related to your physical location.
1 |
$ curl ipinfo.io |
1 |
$ curl ipinfo.io /197 .143.162.90 |
The default progress meter shows various information such as transfer amount, percentages, and so on. The curl command also allows you to use a simple progress bar instead of the default meter. You can activate this using the -# option, as shown in the below examples.
1 |
You can use curl to effortlessly upload image files from your system to a remote server. The following example demonstrates a simple but effective example highlighting this technique.
1 2 |
$ curl -F "img_file=@~/Pictures/Personal/image.jpg" https: //example .com /upload .php $ curl --form "img_file=@~/Pictures/Personal/image.jpg" https: //example .com /upload .php |
Short links are a common way of sharing URLs among friends and colleagues. However, its hard to say what exactly a short link points to since the shortening process makes the URL total cryptic. Thankfully, we can use curl to expand these short links and determine whether they are safe for our system or not.
1 |
$ curl -sIL https: //bit .ly /2MFIhgv | grep -i ^location; |
Another funny curl command I often use to grab a random number is listed below. You can leverage this command when looking for random integer values inside bash scripts.
Copy this command and run it multiple times. Youll get a new number each time you run it in your Linux terminal.
You can easily utilize curl to create automated or scheduled tweets. Youll need to provide your twitter credentials using the username-passphrase combo seen earlier. The below command shows a quick demonstration.
1 |
$ curl -u user: pass -d status= "Handling my tweets from cURL" https: //twitter .com /statuses/update .xml |
The cURL command also allows users to convert documents into one format from another. The following command illustrates how to convert an HTML document to a PDF file using Linux cURL.
1 2 3 4 5 |
$ curl \ https: //c .docverter.com /convert \ -F from=html \ -F to=pdf \ -F input_files[]=@<( echo hello) > FILENAME.PDF |
1 |
$ file FILENAME.pdf |
Transfer.sh is a simple but effective file sharing facility for the terminal. It allows users to upload files up-to 10G and encrypt them if required. The uploaded files will be available for a period of 10 days. First, create a simple text file called test.txt and then run the below command.
1 |
$ cat test .txt|gpg -ac -o-|curl -X PUT --upload- file "-" https: //transfer .sh /test .txt |
1 |
$ curl https: //transfer .sh /1h4tfs/test .txt|gpg -o- > test .txt |
You can have a real-time look at what the workings of cURL by adding the verbosity flag -v. This will print out the details of the ongoing communication. Try out the following commands to see how this works in cURL.
1 2 |
$ curl - v https: //www .google.com /search ?q=ubuntu $ curl --verbose https: //www .google.com /search ?q=ubuntu |
The trace option enables us to view how incoming and outgoing data are being sent and received. They are often very useful in debugging and provides essential information on remote data transfers.
1 2 |
$ curl --trace dumpfile https: //www .google.com /search ?q=ubuntu $ curl --trace - https: //www .google.com /search ?q=ubuntu |
The cUrl command in Linux has seen numerous revisions and version changes. Knowing your curl version is crucial since some options discussed in this guide wont work in some older versions of curl. Take a look at the following curl commands.
1 2 |
$ curl -V $ curl --version |
The help page of curl, like other widely used terminal commands, prints out a concise summary of all available options. Its a great place to start whenever youre stuck at some point.
1 |
$ curl --help |
The curl command comes with excellent, in-depth documentation which makes it easy to learn the different usage of its vast array of options. No matter whether youre a beginner or a seasoned user, you will always learn something new from the manual page of curl.
1 |
$ man curl |