This is the old page. The new page is here SummaryMcTool is a program for Moving and Copying files. Since XCopy wouldn't do what I wanted, and I had been having troubles with TotalCopy running out of resources when trying to copy large numbers (millions) of files, like any good developer, instead of resorting to an existing tool (like RoboCopy), I simply re-invented the copy process. It was developed in C# and requires the Microsoft .NET 4.0 (or higher) framework.It's designed to copy multiple (relatively small) files between high-performance networked storage devices and may not work well for other purposes. In this situation, McTool usually gets 5-10 times the performance of other copy tools (Windows Explorer, batch file copy, Xcopy, RoboCopy/RichCopy, Xxcopy, Total Copy, etc.) depending on the hardware and files being processed. Of course, it also provides features that the other tools don't (GUI interface, e-mail reports, real-time statistics, etc.). Things McTool is not good at (or even capable of):
Change HistoryDownload
InstallationExtract the single file from the ZIP archive and copy it somewhere locally (it won't run from a network drive without modifying the default .NET permissions) and run it.
Command Line Options
To make the most use out of the command line capabilities,
create a settings file with the UI in McTool manually the first time
and then edit/modify/use it as a template to create new files as you
actually need them for your process.You may set up desktop shortcuts with settings files to perform repetitive tasks easily. You can also create batch (or script) files to automate multiple copy processes.
Filename Pattern Matching with Regular ExpressionsI haven't put together any super-interesting patterns yet to show as examples, but I'm sure someone will come up with some. Here are a couple of samples:
More information about Regular Expression and pattern matching can be found here: http://www.regular-expressions.info/tutorial.html Tips & TricksA few things that may come in handy someday. If you are MOVING files, prescanning must complete before any copying begins (otherwise things could get moved out from under the scan process). If you are copying, the prescanning occurs alongside the copying, but it's ONLY for your information - you don't NEED to prescan the files. In either case (move or copy), if speed is the most important thing, simply turn it off - you will still get a report at the end of what it actually did.I have seen throughput speeds of up to 95 MB/sec over a Gbit network using a single instance of the tool. With certain types of files (lots of very small ones - say, 100K or less), running multiple instances on multiple machines could potentially speed things up even more because of the way SMB works (it's very chatty). SMB2 could help with this, but I don't have any networks using it to test with. Generally, getting the application to run directly on either the source or the destination machine is better than using a "man in the middle" machine to do the work. ThreadsIf you are using fast networked storage systems, you should be able to use a good number of threads. If the drives you are using aren't set up as arrays, or if they have a relatively small number of drives in the array (8 is a relatively small number of drives), sometimes using less threads can help. I have only seen less threads be helpful in a couple of cases though, usually, more works better.McTool uses a single thread to process an entire directory. Subdirectories are handled as separate directories with their own threads. This means that if all the files are in a single directory (yikes!), it will only ever use one thread. Keeping huge numbers of files (100,000 is a huge number of files to me) in a single location causes file handling issues (slowness, memory problems, etc.) in many Windows applications so I haven't seen very many instances where this was a severe limitation. If you are using a single external USB drive, 23 threads probably won't work well (try 5 in this case). The bottom line is that you'll want to experiment and see what works best for your situation. One thing to know when changing the number of threads while the program is running is that since each thread works on a single directory, once a thread starts running, it must finish processing that directory before it will end (watch the "number of threads running" stat counter). Some threads are used by the program, so going too low may not allow copying to occur. If you expand the program's window size horizontally, you can see
the threads that are running. If there are significantly fewer "scan"
threads than "normal" threads running, it usually means that the source
system is having a hard time keeping up. The tool allocates the same
number number of prescan threads as copy threads, so during this
process, it can use up to twice as many threads as you have set.
Copying the directory structure, but not the filesSet a bogus file pattern (something that will never match anything) and it won't copy any files, but it will create the directory structure in the destination location. Check source size and then waitIf you're not sure that the files from the source will fit on the destination, you can set the number of threads to zero (0). It will scan the source directory(ies) and then send you an e-mail telling you how big it was, and how much space is free on the destination. You can then either increase the number of threads to let it copy, or cancel if there's not enough room. Database TrackingMcTool can log its operations to a SQL Server database if a valid connection string is present in File | Preferences | Reporting. Structure for the History table CREATE TABLE [dbo].[History]( Structure of the PathsCopied table CREATE TABLE [dbo].[PathsCopied]( Querying the databaseHere are a few queries I have found to be useful: Running jobs (condensed) select SourcePath, DestPath, replace(replace(status,'Time Remaining (@ last/total rate): ',''),'complete ','') as Status, Current copy speed of ALL running jobs in MBPS select sum(cast(left(right(status, len(status) - charindex('[', status)), Totals from the History log select replace(convert (varchar, convert(money, sum(FilesCopied)), 1), '.00', '') as FilesCopied, Misc select * from History where destpath like '%gbrmil1914r4pt2%'
EnhancementsIf you think of other things that could be included
in the program (or might be worth considering), or if it doesn't work
for what you need to do, let me know. On the list or done already (
|
 | |
|
|