While Linux offers several options for moving directories and files, many users prefer rsync because of its efficiency, flexibility, and versatility.
One of the reasons behind the popularity of the rsync utility is the option to exclude specific files and directories from the synchronization process. This capability is incredibly useful when dealing with large data stores or when certain files or directories are not required at the destination.
In this tutorial, we’ll delve into the practicality of using rsync‘s exclusion options. We’ll start with a short overview of the rsync command and then discuss 6 scenarios where you can apply the rsync’s exclusion capability to execute real-world tasks.
Table Of Contents
- Understanding the rsync Exclude Option
- Use rsync to Exclude Directories and Files
- Prerequisites
- Scenario #1: Exclude a Specific File
- Scenario #2: Exclude a Specific Directory
- Scenario #3: Exclude Multiple Files or Directories Based on a Pattern
- Scenario #4: Exclude Multiple Files and Directories
- Scenario #5: Exclude Files by Size
- Scenario #6: Exclude Files and Directories from a List
- Conclusion
- FAQs
Understanding the rsync Exclude Option
The basic usage of the rsync utility is to synchronize everything from a specified location. In many real-world cases, you might wish to exclude directories and files that already exist at the target location. This capability can shorten the time for backing up files because the process now moves the selected files (for instance, only the new or updated files).
The rsync command’s –exclude flag utilizes relative paths from the source directory. To exclude specific files or directories, simply add –exclude to the rsync command, followed by the relative path of the item you wish to omit.
Here’s a basic format of how to use the rsync exclude option:
# rsync [OPTIONS] --exclude 'file_or_directory' source/ destination/
In this syntax, replace source/ with the source directory for data transfer. The destination/ will be the target directory name where rsync will transfer the data. If this directory doesn’t exist, rsync will create it and then proceed with the file transfer.
The presence or absence of the trailing slash (/) at the end of the source directory affects rsync‘s behavior. Omitting the slash results in rsync copying the source directory itself into the destination, followed by the directory’s contents. Including the slash means rsync only copies the source directory content without adding an extra directory level.
To understand different use cases of the rsync command with examples, please refer to our guide on rsync command usage.
Use rsync to Exclude Directories and Files
Let’s now see how you can use rsync to exclude specific directories and files from the synchronization process.
Prerequisites
Before we dive into the specifics of excluding files and directories, let’s ensure you have the necessary prerequisites in place:
- Root access on the system
- Terminal or command line access
- Ensure rsync is installed on your system. If not, you can typically install it using your package manager.
Scenario #1: Exclude a Specific File
Sometimes, you need to exclude a single file from the data transfer process.
You must mention the file or its directory path to exclude it from the rsync process.
Consider this example usage:
# rsync -av --exclude 'example.txt' myfolder/ backupfolder/
This command replicates all the files from myfolder to backupfolder, except for the example.txt file’.
Let’s look at the contents of the source
and destination folders.
In our examples, we’ll use rsync with the -a (archive) and -v (verbose) options combined as -av. The -a option ensures a recursive directory sync while maintaining permissions, symbolic links, and ownership and group settings. The -v option, though not mandatory, provides a detailed view of the rsync command’s progress.
Scenario #2: Exclude a Specific Directory
You can scale the process you used in the previous scenario (rsync –exclude file) to exclude a directory from the synchronization process.
Here’s the rsync command syntax to exclude an entire directory from the process:
# rsync -av --exclude='directory_name/' source_directory/ destination_directory/
This syntax ensures that directory_name and its contents are excluded from the data transfer process.
Scenario #3: Exclude Multiple Files or Directories Based on a Pattern
You can use wildcards to define a pattern that the rsync command uses to exclude multiple files or directories:
For instance, run the following command to exclude all files that start with sample:
# rsync -av --exclude 'sample*' sourcedir/ destinationdir/
Any file or directory corresponding to this pattern (sample*) will be omitted from the transfer.
You can also use a similar wildcard strategy to exclude all directories based on a specific pattern.
For instance, execute the following command to exclude all directories ending with the text reports:
# rsync -av --exclude '*reports' sourcedir/ destinationdir/
Employing an asterisk before and after a pattern provides an additional refinement to the –exclude criterion.
The rsync utility enables the exclusion of specific file types during data synchronization. Utilize an asterisk (*) followed by the file type extension you wish to exclude.
For instance, if you intend to back up a directory containing numerous .log or .tmp files that you don’t wish to include in the backup, execute the following command to exclude those specific file type:
# rsync -av --exclude='*.log' --exclude='*.tmp' source_directory/ destination_directory/
This command excludes all files with the .log and .tmp extensions from the transfer.
Scenario #4: Exclude Multiple Files and Directories
You can use the –exclude option multiple times in your rsync command syntax to exclude multiple files and directories. The good thing is that you can combine rsync –exclude folder and rsync –exclude directory commands to enable the selective transfer of essential data.
# rsync -av --exclude='*.txt' --exclude='dir1/' --exclude='file2.log' source_directory/ destination_directory/
This example excludes all .txt files, file2.log, and dir1/ from the data transfer.
You can string up as many –exclude statements as required. We recommend using curly braces within the –exclude option to specify files and directories for a neat command structure. Additionally, you can separate the patterns with commas for more clarity.
For instance, the above command can be streamlined as follows:
# rsync -av --exclude={'*.txt','file2.log','dir1'} sourcedir/ destinationdir/
As you can see, only dir2 and dir3 were copied to the destination directory. The command excluded the rest of the files and directories.
Scenario #5: Exclude Files by Size
When you use rsync for file transfers, you have the option to specify the minimum or maximum size of files for exclusion.
This option is useful if your source directory includes numerous large files that you wish to omit from the backup. For this, you can employ the –max-size option and specify file size in megabytes or gigabytes.
For instance, the following command excludes files exceeding 100 kilobytes:
# rsync -av --max-size='100K' source_directory/ destination_directory/
Alternatively, if you aim to exclude files smaller than a particular size, utilize the –min-size option.
For instance, consider the scenario where you need to transfer a directory containing images. However, in addition to the main image files, the folder also contains numerous thumbnail files. Now, considering that the image files are usually larger than 1MB, you can exclude the thumbnail files smaller than that size by executing the following command:
# rsync -av --min-size=1m sourcedir/ destinationdir/
Tip: Avoid using the –exclude option when specifying the maximum or minimum size.
Scenario #6: Exclude Files and Directories from a List
In some cases, you may have a list of files and directories to exclude in the form of a text file.
You can use the –exclude-from option in this case. The command syntax will be:
# rsync -av --exclude-from='exclude_list.txt' source_directory/ destination_directory/
Note that this method bypasses all files and directories listed in the file. You can incorporate any pattern explained in this guide.
Conclusion
Excluding directories with rsync elevates the data synchronization precision. Whether streamlining backups or orchestrating server transfers, rsync’s flexibility empowers users. When it comes to reliable hosting for seamless data management, consider the robust solutions offered by RedSwitches, a leading bare metal hosting provider. Their performance-driven infrastructure ensures unparalleled security and scalability.
As you embark on refined data synchronization with rsync, trust RedSwitches for a hosting environment that meets the demands of today’s dynamic digital landscape. Efficiency, security, and performance – RedSwitches delivers excellence in bare metal hosting.
At RedSwitches, we offer the best dedicated server pricing and deliver instant dedicated servers, usually on the same day the order gets approved. Whether you need a dedicated server, a traffic-friendly 10Gbps dedicated server, or a powerful bare metal server, we are your trusted hosting partner.
FAQs
Q. Can rsync be used for real-time synchronization?
While rsync itself is not designed for real-time synchronization, it can be scheduled periodically using cron jobs or other scheduling mechanisms to achieve near-real-time synchronization.
Q. How can I exclude hidden files during data transfer with the rsync?
Use the –exclude=’.*’ option to exclude hidden files and directories.
Q. Is rsync compatible with Windows systems?
Certainly, rsync can be employed on Windows systems using tools such as Cygwin or through native Windows implementations.
Q. What is rsync and how does it work?
rsync is a powerful command-line tool for synchronizing and transferring files and directories between systems. It efficiently copies and transfers only the differences between the source and destination, minimizing data transfer and speeding up the process.
Q. How do I exclude files or directories with the rsync?
You can exclude specific files or directories from being transferred using the –exclude option with rsync command. This allows you to specify the files and directories you want to exclude based on their name, size, or other criteria.
Q. Can I exclude multiple files and directories when using rsync?
Yes, you can exclude multiple files and directories by listing them in a text file and then using the –exclude-from option with rsync command. This method allows you to specify multiple files and directories to be excluded in a single operation.
Q. How can I exclude a particular directory from rsync transfer?
To exclude a specific directory from rsync transfer, you can use the –exclude option followed by the relative path to the directory you want to exclude. This ensures that the specified directory and its contents are not included in the data transfer.
Q. Is it possible to exclude large files from rsync transfer?
Yes, you can exclude large files from rsync transfer by using the –max-size option followed by the size limit of the files you want to exclude. This allows you to exclude files larger than the specified size from being transferred.
Q. Can I exclude files and directories based on their name during the rsync transfer?
Yes, you can specify the files and directories you want to exclude based on their name using the –exclude option followed by the specific file or directory name pattern. rsync allows you to exclude files and directories based on their names or patterns.
Q. Can I exclude one or more files or directories using a text file with rsync?
Yes, you can create a text file listing the files and directories you want to exclude and then use the –exclude-from option with rsync to exclude those specific files and directories from the data transfer process. This provides a convenient way to manage exclusions.
Q. How do I use rsync to exclude files and directories based on their type or extension?
You can use the –exclude option with a file extension or type pattern to exclude specific files based on their type or extension during rsync transfer. For example, to exclude all .jpg files, you can use –exclude=’*.jpg` in the rsync command.
Q. Is it possible to exclude files and directories using rsync based on their size or timestamp?
Yes, you can specify the files and directories you want to exclude based on their size or timestamp using the –exclude option with appropriate size or time-based criteria. This allows for a flexible approach to excluding files and directories during rsync transfers.
