Sunday, January 31, 2010

Splitting Still Image Sequences for use with QuickTime Movies

This is another piece of the automation for my current timelapse video project.
Update: I've rewritten all my scripts in Python and will post those soon.

In "Automating Conversion of Still Image Sequences to QuickTime Movies" I showed how to use a script (qtMovieFromStills.js) with QuickTime's COM interface to generate a movie from a sequence of still images.  I pointed out that QuickTime has an upper limit on the size of the movie that can be generated this way and mentioned that I split up my image sequences to work around that problem.  In this article I post a script splitFolders.js that manages the splitting automatically.

You can run this script to split your image sequences, then run qtMovieFromStills.js on each of the parts.

The last script was in JScript and ran inside MicroSoft's Windows Script Host (WSH) environment, largely because that was the example I started from.  I use JavaScript a lot, but not previously for host scripts.  But it was a reasonably good experience so for consistency I decided to stick with it -- I might choose to put them together into a single, larger program in the future.

splitFolders.js takes three arguments:
  1. workFolder: Full path to the folder containing your image sequence.
  2. sequenceName: Image sequences are expected to have names and sequence numbers, like "Foo-001", "Foo-002", ...  In this example the sequenceName is "Foo" or "Foo-".  My script infers "-" or "_" after the sequenceName, so you can enter just "Foo".
  3. maxSize: The maximum number of still images that you want in a single QuickTime movie.  So if you have 12,345 images and enter 5000 for maxSize, they will be split into three folders.
The split folders keep the name of your workFolder, with a number added like this:  if your workFolder is called "Further", the newly created split folders are named "Further-1", "Further-2" and so forth.  They are created with the same parent folder as your workFolder.

Here's an example.  You have the following folder containing 12,345 images:
L:\Images\TL\Further\
The images have names like "FX-0001.JPG", "FX-0002.JPG" and so forth.  The folder also contains some files that aren't part of the image sequence: resume.doc, shopping.txt.

You enter the following command in the Windows shell:
cscript splitFolders.js L:\Images\TL\Further FX 5000
It crunches along for a while, and when it finishes, here's what you'll find:

L:\Images\TL\Further
Now contains only the files that were not part of the image sequence: resume.doc and shopping.txt.
L:\Images\TL\Further-1
Contains FX-0001.JPG through FX-5000.JPG
L:\Images\TL\Further-2
Contains FX-5001.JPG through FX-10000.JPG
L:\Images\TL\Further-3
Contains FX-10001.JPG through FX-12345.JPG
Here's the script.  Copy and paste it into an editor, then save it as splitFolders.js on your computer.  Do be cautious: it's moving your precious images around and I always like to have a backup copy before doing this kind of thing.  If you're really worried, edit the script to use "CopyFile" instead of "MoveFile".  Or comment out the "MoveFile" and just run it to see what it will do.
// WSH JScript
// Author: Chip Chapin <cchapin@gmail.com>
// Last Updated: 2010-01-30

// Split an image sequence into smaller chunks to faciliate processing
// by QuickTime. See http://cchapin.blogspot.com/

function usage(s) {
  WScript.Echo(s);
  WScript.Echo("usage: splitFolders.js workFolder sequenceName maxSize");
  WScript.Echo(
"  workFolder -- Full path to folder containing the image sequence.\n" +
"  sequenceName -- Sequences have file names like Foo-001, Foo-002 ...\n" +
"      In this case the sequenceName would be 'Foo' or 'Foo-'.\n" +
"      (Dash and underscore are ignored at the end of the sequenceName)\n" +
"  maxSize -- Maximum number of sequence files to put in a folder.");
  WScript.Quit();
}

// Get script arguments
if (WScript.Arguments.Length != 3) {
  usage("Expected three arguments");
}
var workFolderName = WScript.Arguments(0);
var sequenceName = WScript.Arguments(1);
var maxSize = WScript.Arguments(2);

var fso =  WScript.CreateObject("Scripting.FileSystemObject");

if (!fso.FolderExists(workFolderName)) {
  usage("Invalid folder path '" + workFolderName + "'");
}
var workFolder = fso.GetFolder(workFolderName);

// Make a list of files that match the sequenceName pattern.
var sequenceFiles = [];
var sequencePattern = RegExp().compile(sequenceName + "[-_]?[0-9]+\.", "i");
var files = new Enumerator(workFolder.Files);
while (!files.atEnd()) {
  var f = files.item();
  if (f.Name.match(sequencePattern)) {
    sequenceFiles.push(f);
  }
  files.moveNext();
}
WScript.Echo("Folder '" + workFolder.Path +
             "' contains " + sequenceFiles.length +
             " images in sequence '" + sequenceName + "'.");
if (sequenceFiles.length == 0) {
  WScript.Quit();
}
if (sequenceFiles.length <= maxSize) {
  WScript.Echo("No need to split.");
  WScript.Quit();
}

// Iterate over the sequence files in the base folder, moving them
// to successive split folders.
var numSplits = Math.ceil(sequenceFiles.length / maxSize);
WScript.Echo("Splitting image sequence '" + sequenceName +
             "' into " + numSplits + " parts.");
var baseFolderName = fso.GetBaseName(workFolderName);
var nextFileInSplit = 1;
var currentSplit = 0;
var files = new Enumerator(workFolder.Files);
for (var fileCount = 0; fileCount < sequenceFiles.length; fileCount++) {
  if (fileCount == 0 || nextFileInSplit > maxSize) {
    currentSplit++;
    nextFileInSplit = 1;
    var splitFolder = workFolder.ParentFolder.Path + "\\" +
                      baseFolderName + "-" + currentSplit;
    WScript.Echo("New split folder '" + splitFolder + "' (" + fileCount + ")");
    fso.CreateFolder(splitFolder);
  }
  var f = sequenceFiles[fileCount];
  // Move a file from workFolder to splitFolder
  WScript.Echo(f.Name + " to " + splitFolder);
  fso.MoveFile(f.Path, splitFolder + "\\" + f.Name);
  nextFileInSplit++;
}

WScript.Echo("Split " + fileCount +
             " files in '" + workFolder +
             "' into " + currentSplit + " folders.");

// End.

References

Saturday, January 30, 2010

Automating Conversion of Still Image Sequences to QuickTime Movies

For my current timelapse video project, I collect over 12,000 images almost every day, and need to automate the processing as much as possible.  In my notes on the first video, Mountain View Rainstorm, I explained why I chose to create a QuickTime movie from the still image sequence before video editing, and in this article I explore how to automate the QuickTime movie production from the stills.

I thought this article would be short and simple, but it raised many questions ... "You take the red pill - you stay in Wonderland and I show you how deep the rabbit-hole goes." -- The Matrix 1999.

Try Built-In Automation First

QuickTime Pro already provides a very convenient way to make a movie from a sequence of stills.  If you're only doing this operation occasionally, you don't need to automate it further.  Simply open QuickTime and use the "Open Image Sequence" command.  When it asks you to open a file: give it the first file in your sequence.  For example, if my files are named "MTV_0001" to "MTV_12345", I specify "MTV_0001".

The "Open A File" dialog is also where you specify the framerate of your movie.

Yes, your files must be numbered.  If you're trying to make a movie out of "red.jpg", "orange.jpg", "yellow.jpg" etc. it won't work.  Fix the names.

For most of us that's all we need.  Thanks for stopping by!

Scripting QuickTime Operations

A Google search for "quicktime automate" leads very quickly Luc-Eric Rousseau's 2006 XSIBlog post Automating QuickTime at the Command Line on Windows in which he provides a script to do almost exactly what I want.  Hooray!  But there were a few tricks involved in making it work the way I wanted -- truthfully it took me several days -- and in the process I learned more about QuickTime, its COM interface, and the "Windows Script Host" (WSH) facility.

Using qtMovieFromStills.js

My adaptation of Rousseau's script is called qtMovieFromStills.js.  Like his, it is written in JScript and runs from a Windows command shell using WSH.  Here's how to use the script:
  1. Copy qtMovieFromStills.js from the bottom of this post, paste it into an editor (e.g. Notepad) and save it somewhere convenient -- I keep a "scripts" folder inside my project folder.
  2. Open a command window and run it using WSH, i.e. with the cscript command.
  3. qtMovieFromStills has two required and one optional argument:

    1. Windows file path to the first image in the sequence.
      E.g. "L:\Images\2010-01-19\MTV_0001.jpg"
    2. Windows file path to the desired output file.
      E.g. "L:\Images\MTV_2010-01-19.mov"
    3. Optionally, specify the framerate in frames-per-second.  E.g. "60".  It defaults to 30.  A list of supported frame rates appears in the QuickTime "Open A File" dialog.
  4. So, your shell command looks something like this.
cscript qtMovieFromStills.js L:\Images\01-19\MTV_0001.jpg L:\Images\MV01-19.mov 60

Difficulties

The first problem I ran into turned out to be very simple: I was using relative file paths in the arguments passed to QuickTime and this does not work well.  Apparently the QuickTime components do not recognize your current working directory, nor did the script provide useful error messages.  You must provide full paths (Like "C:\bletch\srcfile.jpg") rather than relative paths to the source and dest files.

The second problem is that the QuickTime export operation will fail beyond a certain number of frames.  And once again there is no useful error message.  The exact number depends on your machine resources -- on my larger machine the export was successful with 9,000 frames but failed with 12,000.  On my laptop it fails for 7,000 frames.  The workaround is to split them into two (or more) folders and generate separate movies.

The original script had little error detection and reporting.  I've added a lot more, but it doesn't yet work as well as I would like.

Automating the Conversion Process

Entering the script command itself is not easier than using the QuickTime GUI.  Where it pays off is in running it repeatedly over a batch of folders or automatically as part of a more general workflow automation.

Here's a little bash shell script I use with Cygwin to process  a series of directories. Note how the Windows-style paths must be escaped to the bash shell.
frameRate=60
basedir="L:\\Graphics\\Images\\SX110\\MTV-TL01"

dirs="2009-12-09 2009-12-10 2009-12-11 2009-12-12 2009-12-14"

script="$basedir\\scripts\\qtMovieFromStills.js"
for d in $dirs; do
  date 
  echo $d
  src="$basedir\\$d\\MTV-TL01_0001.JPG"
  dst="$basedir\\MTV-TL01_$d.mov"
  cscript $script $src $dst $frameRate
done

Internals of qtMovieFromStills.js

If you just want to use the script, you can stop reading here.

At this point, most of the script is error checking and reporting, and unraveling the QuickTime object hierarchy. The essentials are in only two lines.
qtControl.CreateNewMovieFromImages(sourcePath, frameRate, true);
This function does essentially what the QuickTime GUI does: makes a movie out of an image sequence.
qtExporter.BeginExport();
The Exporter takes QuickTime's internal representation of your movie and renders it to a file.  This takes the most time.

The one other piece that's worth some explanation is the way the script saves your export codec settings for reuse. They are saved as an opaque blob of binary data in the XML file:
C:\qtMovieFromStillsCodecInfo.xml
Whenever you want to change the settings, just delete the file.  Details on how this works are discussed by Cromie in a post to the quicktime-api mail list, as well as by Rousseau in his blog post.

References

Source Code: qtMovieFromStills.js

Copy the code and paste it into an editor. Save it as qtMovieFromStills.js.
// Windows Script Host JScript
// Create a QuickTime movie from a sequence of still images.
// Run from the command line on Windows using WSH:
//   cscript qtMovieFromStills.js sourcepath destpath [framerate]
//
// Authors:
//   Chip Chapin <cchapin@gmail.com> has extended a script by 
//   by Luc-Eric Rousseau (XSIBlog 2006, http://www.xsi-blog.com/archives/103)
//   which was in turn based on sample code by John Cromie that
//   accompanied his book "QuickTime for .NET and COM Developers"
//   (Elsevier 2006, http://www.skylark.ie/qt4.net/samplecode.asp).
//
//   My changes provide
//     -- Additional error checking/reporting of various kinds.
//     -- The script can be run in a loop without crashing.
//     -- Verbose progress reporting.

// Last Update: 2010-01-30

// Be sure to use FULLY-QUALIFIED Windows file paths, even from Cygwin.
// Relative paths don't work, and there is no error report from QT.
// TODO: can we get a progress bar during the initial rendering stage?

/*
 * Check existence of the parent folder of a file path.
 * @param {fso} fso
 * @param {string} fname
 * @return {boolean}
 */
function CheckFileFolder(fso, fname) {
  s = fso.GetParentFolderName(fname);
  return(fso.FolderExists(s));
}


// Get script arguments
if (WScript.Arguments.Length >= 2) {
  sourcePath = WScript.Arguments(0);
  destPath = WScript.Arguments(1);
  if (WScript.Arguments.Length >= 3) {
    frameRate = WScript.Arguments(2);
  } else {
    frameRate = 30;
  }
  WScript.Echo("FrameRate: " + frameRate);
} else {
  WScript.Echo("not enough parameters");
  WScript.Quit();
}

var fso =  WScript.CreateObject("Scripting.FileSystemObject");
if (!fso.FileExists(sourcePath)) {
  WScript.Echo("Missing source file '" + sourcePath +"'");
  WScript.Quit();
}
if (!CheckFileFolder(fso, destPath)) {
  WScript.Echo("Bad destination file path '" + destPath +"'");
  WScript.Quit();
}

// Launch QuickTime Player Application
var qtPlayerApp = WScript.CreateObject("QuickTimePlayerLib.QuickTimePlayerApp");
WScript.Sleep(8000);  // Give it time to launch.

if (!qtPlayerApp) {
  WScript.Echo("ERROR Failed to launch QuickTime Player App.");
  WScript.Quit();
}

// Get the QuickTime player and its associated controller.
var qtPlayer = qtPlayerApp.Players(1);
if (qtPlayer == null) {
  WScript.Echo("ERROR Failed to get QuickTime Player.");
  WScript.Quit();
}
WScript.Echo("Got player '" + qtPlayer.Caption + "'");
var qtControl = qtPlayer.QTControl;

// Now create the movie.
WScript.Echo("Creating new movie from stills at '" + sourcePath + "'...");
try {
  qtControl.CreateNewMovieFromImages(sourcePath,
                                     frameRate,
                                     true ); // rate is in frames per second
}
catch (e) {
  WScript.Echo("ERROR creating movie: " + e.number +
               " (" + (e.number>>16 & 0x1FFF) +
               "-" + (e.number & 0xffff) + ")");
  WScript.Echo(e.description);
  // TODO: Find more reliable error reporting.
  // The following doesn't work if the QTControl object is gone.
  WScript.Echo("QuickTime error " + qtControl.ErrorCode);
  var qte = qtControl.QuickTime.Error;
  WScript.Echo("  " + qte.ErrorCode + ", " + qte.Description);
  WScript.Echo("  " + qte.SourceReference);
  WScript.Quit();
}

var qtMovie = qtControl.Movie;
if (!qtMovie) {
  WScript.Echo("ERROR: No movie created (" + qtControl.ErrorCode + ")");
  WScript.Quit();
}
var duration = qtMovie.Duration;
if (duration == 0) {
  // This test isn't as helpful as I thought it would be.  I thought it
  // would catch the case where QT does not have a valid input file, but in
  // that case it seems to create a two-second empty movie 
  // (ie. duration = 20*framerate).
  WScript.Echo("ERROR: Movie has duration 0.");
  WScript.Quit();
}
// Duration appears to be the number of frames * 10.
WScript.Echo("Created new movie, duration " + duration);

// Set up the exporter.
var qt = qtPlayer.QTControl.QuickTime;
if (qt.Exporters.Count == 0) {
  // Only add an exporter if needed.
  qt.Exporters.Add();
  WScript.Echo("Adding new Exporter.");
} else {
  WScript.Echo("Using existing Exporter.");
}
var qtExporter = qt.Exporters(1);
if (!qtExporter) {
  WScript.Echo("ERROR: Unable to get Exporter.");
  WScript.Quit();
}
qtExporter.TypeName = "QuickTime Movie";
qtExporter.SetDataSource( qtMovie );

// Configure the exporter.
var CodecInfoFileName = "C:\\qtMovieFromStillsCodecInfo.xml";
var CodecFileInfo;
if (fso.FileExists(CodecInfoFileName)) {
  WScript.Echo("Reading codec config from '" + CodecInfoFileName + "'");
  CodecFileInfo =  fso.OpenTextFile( CodecInfoFileName );
}
if (CodecFileInfo) {
  var xmlCodecInfoText = CodecFileInfo.ReadAll();
  // cause the exporter to be reconfigured
  // http://developer.apple.com/technotes/tn2006/tn2120.html
  var tempSettings = qtExporter.Settings;
  tempSettings.XML = xmlCodecInfoText;
  qtExporter.Settings = tempSettings;
} else {
  qtExporter.ShowSettingsDialog();
  var xmlCodecInfoText = qtExporter.Settings.XML;
  CodecFileInfo = fso.CreateTextFile( CodecInfoFileName );
  if (CodecFileInfo) {
    CodecFileInfo.WriteLine(xmlCodecInfoText);
    CodecFileInfo.Close();
  } else {
    WScript.Echo("Warning: failed to save codec info to '"
                 + CodecInfoFileName + "'");
    WScript.Echo("continuing ...");
  }
}

// Export the movie.
WScript.Echo("Exporting ...");
try {
  qtExporter.DestinationFileName = destPath;
  qtExporter.ShowProgressDialog = true;
  // Uncomment this line if you want the export dialog box to appear.
  // qtExporter.ShowExportDialog();
  qtExporter.BeginExport();
  WScript.Echo("Exported to '" + destPath + "'");
}
catch (e) {
  WScript.Echo("ERROR " + (e.number>>16 & 0x1FFF) +
               "-" + (e.number & 0xffff) + 
               " exporting '" + destPath + "'");
  WScript.Echo(e.description);
  // JSON object only available in WSH 5.8+
  // WScript.Echo(JSON.stringify(e, null, 2));
  var qte = qt.Error;
  WScript.Echo("QuickTime Error " + qte.ErrorCode + ", " + qte.Description);
  //WScript.Echo(JSON.stringify(qte, null, 2));
}

// Close the player?
// Note: closing the player results in failures for subsequent invocations.
//qtPlayer.Close();

// End.

Using OLEView to Explore the QuickTime COM Interface

I'm currently working on some automation for my timelapse video project. As part of that effort I needed to explore the Windows scripting interface to QuickTime.  Here is a quick look at one of the tools I learned.

The QuickTime COM interface is poorly documented, but Rousseau mentioned the cool trick of using MicroSoft's OLEView tool to explore.  I had never used it before, so here is a quick walk-through.
  • Download oleview.exe if you don't already have it (see References section).
  • Open a command shell and run oleview.exe. It opens a window titled "OLE/COM Object Viewer".
  • Open the "Type Libraries" tab (it's near the bottom).  You'll be interested in everything that starts with "Apple QuickTime" as well as "QtMovExporter".
  • For example, double-click on "Apple QuickTime Player Library 1.0" to open the IDL viewer.  Look in the IDL panel and find the declaration "library QuickTimePlayerLib".  Now realize that this library is what a script refers to in a line such as  "WScript.CreateObject("QuickTimePlayerLib.QuickTimePlayerApp")".
  • Expand the "dispinterface" tabs for IQuickTimePlayerApp, IQuickTimePlayers, and IQuickTimePlayer.  Compare the listed methods with the ones used in our script.
  • Notice that the IQuickTimePlayer.QTControl method returns an "IQTControl" object.  It's not defined in QuickTimePlayerLib.  Close the IDL viewer window and return to the main OLE view window.  Double-click "Apple QuickTime Control 2.0" to view its interface and observe that it defines IQTControl.
This is a great way of checking the interface of any COM object.

References

I have lots more QuickTime references that will appear in an upcoming post about the automation itself.

Sunday, January 24, 2010

Timelapse Video: Mountain View Rainstorm 2009-01-19

Updated with more technical details.

My current office in Mountain View is on the 4th floor and has a wonderful view, even better than the previous building. One never knows how long these blessings will endure, and I've been itching to do some long-term timelapse out the window. Finally, starting in December, I got something going and have been capturing the view out my window almost every day.

Today I'm posting the first video from the project. I chose January 19th because of its interesting weather. I hope you enjoy it.



Hint: Click the "full screen" button on the viewer. This is the widescreen version of the video. I have also posted a copy with the original 4:3 aspect ratio.

Technical Notes

For a long-term timelapse project it is essential to use a camera that supports remote capture. I wanted to dedicate a good quality point-and-shoot camera to it rather than my usual practice of using a DSLR -- the mirror/shutter action is way too loud.

At first I thought I could use my Canon SD1100IS, but surprisingly it doesn't do remote capture. Turns out Canon has been phasing this feature out of their point-and-shoot cameras and I would have to go back to an older model. I finally purchased a used Canon SX110 on eBay (the newer SX120 won't work).

Processing

There are 8,723 individual still images, each of them 1600x1200 JPEG (4:3 aspect ratio) taken every 3 seconds over 8 hours, 25 minutes. The main considerations for processing them into a video are (a) how long a video do we want and (b) what aspect ratio will we present it at.

How Long?

Typical American video runs at 30 frames per second ("29.970"), the NTSC standard TV frame rate. At 30 fps our 8,723 images will run for 291 seconds, or 4:51. That's too long -- I think the video needs to run in around two minutes or else people will get bored. There are a variety of ways to speed things up, depending on your editing tools and your quality goals. I'll say a little more about this below.

What Aspect Ratio?

The original images are 4:3, which is fine for standard video but problematic for HD. Normal HD videos are 16:9. An HD-quality 4:3 video will look fine in, say, Quicktime but on YouTube it will have black bars on the sides.

The version shown above is 16:9. I originally posted the 4:3 version to YouTube but quickly decided it would look better in 16:9.

If I want to get rid of the black bars my alternatives are either to crop or to stretch. In this case I don't want to crop -- there's nothing in the frame I want to lose. On the other hand, the nature of the scene is such that I thought such stretching may not be noticeable.

Preserving Quality

One aspect of preserving quality is to avoid losing information during processing. There are two things that can easily get lost: you can lose frames and you can lose pixels.

Pixels -- It would be very natural to convert the 1600x1200 images into a 1440x1080 HD video stream, preserving the aspect ratio. But if you now decide to stretch it to 1920x1080 HD, you will have lost information in the horizontal dimension. So if that's a consideration it's better to keep the 1600x1200 format for as long as possible (even though it will slow your processing). Then you can resample this full-resolution video into whatever you want.

Frames -- Video editors slow to a crawl when manipulating 8,000 still images and the solution is to prerender them to video. In one version of this video I rendered them at 30fps, giving me a 4:50 video clip. Then I set my editor to playback at 2x normal speed. I thought that would be fine since all the frames are still there, but that's not how it works. Doubling the playback speed works by throwing away half the frames. Most of the time you don't notice, but when one slows the playback velocity during editing the video gets choppy. The solution, at least with my editor, is to prerender at 60fps so I can defer the decision on when to drop frames.

Pre-Render with QuickTime

My video editor (Sony Vegas) is quite capable of importing still image sequences, but doesn't handle more than a thousand of them very well.  I've found that it's much faster and way more convenient to use Apple QuickTime (Pro) to  prerender the images into a full-resolution video.  Then just add the video as a track in the editor.

Hints:
  • You must buy QuickTime Pro.  The free version won't do this.
  • From the File menu click "Open Image Sequence".  Then select just the first frame of your sequence.  It will automatically load the rest.
  • Be sure to select "60 frames per second".

Editing With Sony Vegas

Besides prerendering with QuickTime, there are a other things that took me some trial and error to figure out in the editing process with Sony Vegas.  If you want to do a widescreen (stretched) video:
  • Set your project properties to 1920x1080 and 60 (59.9...) fps.
  • After you add the QT video to a track, set its properties and uncheck "Maintain aspect ratio".  This allows the 4:3 camera frame to stretch to fill the 16:9 HD frame.  

Rendering the Results

I'm no expert on rendering but these are the best settings I've found so far to render a reasonably sized file that is usable both on YouTube and on the computer.  I used the Sony AVC codec with a custom template I call "YouTube 1920x1080".  I started with the default settings and made these changes:
  • Video rendering quality: Best
  • Frame size: "High definition (1920x1080)"
  • Frame rate: 29.970 (NTSC)
  • Pixel aspect ratio: 1.0
  • Video Bit rate (bps): 4,000,000 (i.e. 4 Mbps)
  • Audio sample rate: 44,100 (all my source material is sampled at 44.1)
The result is an MP4 file of about 80 MBytes that works fine as an upload for YouTube HD and also looks good on my computer.

How about you? Leave a comment, let me know what you've been doing to create Time-Lapse videos from still photos.

Monday, January 18, 2010

HP MediaSmart Server/MediaCollector: Making Custom Paths Work


This post applies to HP MediaSmart Server models EX490 and EX495 with HP software version 3.0.14.33083 8/11/2009.

The HP MediaSmart Server includes a tool called the Media Collector which scans your computers and copies media files onto the server. When you add a client (i.e. a computer) to the Media Collector configuration, it offers you the opportunity to specify custom paths to search. This is important for those of us who don't keep all our files under our home directory (folder).

Unfortunately the UI to specify a custom path is badly broken for many users at this time. On my system, if I'm persistent I can add a top-level path (e.g. "L:\") but can only rarely get even one level below that.

In this post I explain how to workaround the problem by manually editing the Media Collector configuration database. This technique is for advanced computer users. While I've tried to explain it thoroughly, it assumes that you have at least basic familiarity with such concepts as SQL databases and editing XML. I wish it were simpler, but it isn't ...

1. Log into your MediaSmart Server system using Remote Desktop Connection (RDC).
Hint: Find RDC under All Programs::Accessories.
Your MediaSmart Server is running a version of Windows called Windows Home Server (WHS) and is normally accessed through the WHS Console application. But to do what you need here, you need access to the full system.
NOTE: All remaining steps take place on the WHS system.

2. Open Explorer and locate the database folder used by the HP MediaSmart Server.
You can find it somewhere under "D:\folders" -- on my system it is "D:\folders\{2D2552D3-5884-4D18-9608-DC65BB76026A}" -- just look for the file "mssdb.db".
3. Open a command window and cd to the database folder.
...> cd D:\folders\{2D2552D3-5884-4D18-9608-DC65BB76026A}
...> d:
4. Now fire up the SQLite shell on the MSS database. Fortunately HP left one around for us to use, otherwise you could download it from sqlite.org.
...> "C:\Program Files\Hewlett-Packard\HP MediaSmart Server\sqlite3.exe" mssdb.db
Useful SQLite References:
5. Explore the database a little.
sqlite> .tables
McClient McConfig McFileEntry McMediaType
McClientVersion McDuplicates McFileEntryRequest McRequestNumber
The McConfig table is the one we want. Take a look at its schema:
sqlite> .schema McConfig
CREATE TABLE [McConfig](
[Target] [nvarchar] PRIMARY KEY NOT NULL,
[ConfigXml] [Text] NOT NULL);
This shows us that the table has two fields: "Target" and "ConfigXml". Now let's see how they are used. The following command displays the contents of all records in the table:
sqlite> select * from McConfig;
Note for the excessively curious: the full schemas for mssdb are found in its creation script,
C:\Program Files\Hewlett-Packard\HP MediaSmart Server\scripts\mssdb.sql


6. To make things simpler, you can look at just the configuration data for a single client. In this example the client is named "superfortress":
sqlite> select ConfigXml from McConfig where target = "superfortress";
Here's an example of what you might see:
<?xml version="1.0"?><MSS_MC_ClientConfig><Version>3.0.1</Version>
<ClientName>superfortress</ClientName><MediaRecord><MediaType>Photo</MediaType><
IsEnabled>true</IsEnabled><SourceLoc>Home</SourceLoc></MediaRecord><MediaRecord>
<MediaType>PhotoVideo</MediaType><IsEnabled>true</IsEnabled><SourceLoc>Home</Sou
rceLoc></MediaRecord><MediaRecord><MediaType>Music</MediaType><IsEnabled>true</I
sEnabled><SourceLoc>Home+iTunes+WMP</SourceLoc></MediaRecord><MediaRecord><Media
Type>AlbumArt</MediaType><IsEnabled>true</IsEnabled><SourceLoc>Home</SourceLoc><
/MediaRecord><MediaRecord><MediaType>Playlist</MediaType><IsEnabled>true</IsEnab
led><SourceLoc>Home+iTunes+WMP</SourceLoc></MediaRecord><MediaRecord><MediaType>
Video</MediaType><IsEnabled>true</IsEnabled><SourceLoc>Home</SourceLoc></MediaRe
cord></MSS_MC_ClientConfig>
As you can see, the configuration data is stored as XML text. It's hard to read this way -- here it is pretty-printed (well, it was pretty until Blogger banished the indenting):
<?xml version="1.0"?>
<MSS_MC_ClientConfig>
<Version>3.0.1</Version>
<ClientName>superfortress</ClientName>
<MediaRecord>
<MediaType>Photo</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>PhotoVideo</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>Music</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home+iTunes+WMP</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>AlbumArt</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>Playlist</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home+iTunes+WMP</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>Video</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
</MSS_MC_ClientConfig>
That is a very basic configuration with no custom paths. Since I can add partial custom paths using Console tool, I checked it again after adding as much of a custom path as I could. Now we can observe how the custom paths are represented.
<?xml version="1.0"?>
<MSS_MC_ClientConfig>
<Version>3.0.1</Version>
<ClientName>superfortress</ClientName>
<MediaRecord>
<MediaType>Photo</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>PhotoVideo</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>Music</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home+iTunes+WMP+Custom</SourceLoc>
<CustomLoc>L:\</CustomLoc>
</MediaRecord>
<MediaRecord>
<MediaType>AlbumArt</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
<MediaRecord>
<MediaType>Playlist</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home+iTunes+WMP+Custom</SourceLoc>
<CustomLoc>L:\</CustomLoc>
</MediaRecord>
<MediaRecord>
<MediaType>Video</MediaType>
<IsEnabled>true</IsEnabled>
<SourceLoc>Home</SourceLoc>
</MediaRecord>
</MSS_MC_ClientConfig>

We're going to edit the CustomLoc elements in the XML and make them point where we want.
7. Use the Administrative Tools:Services panel to stop the HPMediaSmartService (HPmssService.exe process).
If you don't stop the HPMediaSmartService, it may lock the database at inconvenient times.

8. Save a copy of your "mssdb.db" file. This will make it easy to start over if you need to.

9. In order to edit the existing XML configuration for your client you can copy and paste it into WordPad or NotePad, but you'll have to edit out the spurious newlines. Or you can use the ".output" command in SQLite to redirect to a file first.
I recommend using .output and just saving to a file -- I lost 30 minutes finding out I had accidentally deleted a "<" character while merging lines. I recommend you not try to pretty print it or break the lines for readability -- you should be editing a single block of text with no line breaks, same as what it was.
10. Edit the XML and update the database. In this example I edited it by changing the custom location to "L:\Audio\My Music\" (in two places). Your basic SQL command is this:
REPLACE INTO McConfig VALUES ('your client name', 'blob of xml');
You'll be inserting the name of your own client of course. We use single-quotes because there are double-quotes in the XML.
Hint: Construct the whole SQL command in the editor so you can just paste the whole thing into SQLite.
Note: Despite what you may see here, there should be no line breaks in the whole block of XML text.
sqlite>  REPLACE INTO McConfig VALUES ("superfortress",
'<?xml version="1.0"?><MSS_MC_ClientConfig><Version>3.0.1</Version><ClientName
>superfortress</ClientName><MediaRecord><MediaType>Photo</M
ediaType><IsEnabled>true</IsEnabled><SourceLoc>Home</SourceL
oc></MediaRecord><MediaRecord><MediaType>PhotoVideo</MediaTy
pe><IsEnabled>true</IsEnabled><SourceLoc>Home</SourceLoc>
</MediaRecord><MediaRecord><MediaType>Music</M
ediaType><IsEnabled>true</IsEnabled><SourceLoc>Home+iTunes+WMP+C
ustom</SourceLoc>
<CustomLoc>L:\Audio\My Music\</CustomLoc>
</MediaRecord><MediaRecord><MediaType>AlbumArt</MediaType>
<IsEnabled>true</IsEnabled><SourceLoc>Home</SourceLoc></Me
diaRecord><MediaRecord><MediaType>Playlist</Media
Type><IsEnabled>true</IsEnabled><SourceLoc>Home+iTunes+WMP+Cust
om</SourceLoc>
<CustomLoc>L:\Audio\My Music\</CustomLoc>
</MediaRecord><MediaRecord><MediaType>Video</MediaType><Is
Enabled>true</IsEnabled><SourceLoc>Home</SourceLoc></MediaRe
cord></MSS_MC_ClientConfig>');

Before moving on, check to see that your changes were accepted. This probably isn't necessary, but it will help you feel that you have accomplished something.
sqlite> SELECT * FROM McConfig WHERE target = "superfortress";

11. Sorry, but you're not done yet. You may have noticed that your client configuration also appears in the record for target=%Global% record, so we need to repeat steps 6-10 for that record. Here's what to do:
(a) Capture the existing XML for target="%Global%". Your command is similar to step 6:
sqlite> select ConfigXml from McConfig where target = "%Global%";
Use .output or Copy/Paste to capture the XML and edit the CustomLoc elements.

(b) Edit and update the %Global% entry. Your command is similar to step 10:
sqlite> REPLACE INTO McConfig VALUES ("%Global%", 'blob of XML');
(c) Check your results.
sqlite> SELECT * FROM McConfig WHERE target = "%Global%";
12. Now restart the HPMediaSmartService and check for errors.
Check the Event Viewer:Application log. If you made a mistake in the XML it will be reported there. (Hint: the Event Viewer is found under All Programs::Administrative Tools).

Use the WHS console to check the HP Media Collector status and configuration. You should see your new custom paths show up in the client configuration.
Good luck!