More Efficient Storage of Virtual Machine Backups

Here's a little trick I use to save substantial amounts of disk space. 

For testing our software, we heavily rely on virtual machines, mostly running various versions of Windows - we use VMware (both VMware Workstation on PC and VMware Fusion on Mac), Parallels, and VirtualBox extensively. 

VirtualBox is my current favorite - it's come a long way since the earlier versions, and to me it's now on a par with VMware and Parallels. Furthermore, it has a number of very handy features that the other two lack, and above all, it's free!

The nice thing with virtual machines that we can easily 'roll back' to a known state before starting a test session. All three of the tools mentioned above have a 'snapshotting' feature that makes 'rolling back' easy.

However, snapshots of virtual machines come at a cost: the more snapshots you pile up onto a single virtual machine, the slower that virtual machine becomes. 

To avoid the performance penalty, we instead often opt for a more primitive backup strategy: we regularly make compressed copies of a 'known good' virtual machine, and restore them as needed.

Here's the trick I devised to make the compressed copies substantially smaller without loss of data: 

1) Get rid of all snapshots before backing up (so the virtual hard disk becomes a single, large file).

2) Optionally, clear any temp files and so on.

3) Use a 'wiper' tool to erase any unused disk space in the virtual machine before compressing. 

If you don't do this, any unused disk space on the virtual hard disks will contain more or less random bit patterns, which don't compress very well.

Yet, if you zero out any unused disk space, the unused disk space on the virtual drive compresses extremely well. 

For example, I just have made a compressed backup of a heavily used virtual machine twice - first without erasing the unused space and then again after erasing unused space. The compressed file size fell from close to 80GB in size to 50GB in size. How's that for savings?

I mostly use a Windows-based tool called Eraser - it's free and open source.

The main caveat is that you have to make it erase any unused space with zeroes. 

Eraser's main purpose is to make erased data irretrievable for security-purposes, and by default it will fill up any unused disk space with random bit patterns, but that is not what we're after here.

To make Eraser work for me I had to add a 'custom' bit pattern first (which is simply a zero).

There are a few variants of Eraser available - don't use the most recent 6.x version that apparently won't allow you to fill unused space with zeroes, but instead insist on using random bit patterns. That is useless for our purpose. 

Instead, you have to download the older version (I used 5.8.8) which allows you to define your own erasure patterns:

http://sourceforge.net/projects/eraser/files/Eraser%205/

After installing, you need to select the Edit - Preferences - Erasing... menu. Then add a new method, call it 'Zero' and set it to use a zero as the bit pattern. 

Then erase the unused space with this pattern. Then, after shutting down, you can compress the virtual machine.

InDesign Scripting: Putting TextFrames in Perspective - for Visually Oriented Scripters

I admit it: I don't know the InDesign Scripting DOM by heart, so I spend a lot of time browsing around in the ExtendScript Toolkit Object Model Viewer.

I frequently work on very diverse projects - sometimes it's GWT/Java, sometimes it's ActionScript, sometimes it's PHP/WordPress, sometimes it's C++ or REALbasic, and so on... - and invariably, when I come back to a particular project, I need to spend some time refreshing basic knowledge.

It's like my brain 'switches' programming environment, and when I am deeply immersed in Java, I almost forget how to spell 'EkstentScript'.

The ExtendScript Toolkit Object Model Viewer (you find it under the 'Help' menu) is great for browsing the object model - but when it's been a while, it still feels like I am wading through molasses. One of the reasons for that is that I am a visually-oriented person. A picture is worth a few thousand mouse-clicks to me!

Lately, I've been revisiting BarredText - one of our latest scripted plug-ins, and to do that, I need to get back up to speed on TextFrames, Columns, Characters, Stories, and so on.
I decided I'd put in a little bit more effort, and put everything into a diagram - I've made a 2-page PDF file as a refresher I'll use next time I need to figure out how those buggers relate.

Because this might be useful to other people, I decided to release the PDF file under the 'Attribution-Share Alike' Creative Commons license:

http://www.rorohiko.com/downloads/rorohiko.blogspot.com-extendscript-textframe.pdf

I am not claiming this is error-free - if you find any errors, or can see ways to improve this, please let me know.

And if you like this - let me know too! Soon, I need to revisit tables and cells in ExtendScript - who knows, if I get enough positive feedback I might do another visual representation.

Easy Access to System Folders on Mac and Windows

Time for a few tidbits! On my list of 'good intentions' for 2011 is an entry 'write more blog posts'.

My idea is that I want to jot down a quick blog post each time I find myself using a trick or technique that might be useful to others, and that can be explained in a relatively short blog post.

(The qualifier 'relatively' in the previous sentence is necessary - I tend to have trouble keeping the number of words in a post down to a reasonable number. I'll try to keep it short and to the point!)

It dawned on me that a lot of tricks I have personally come to find mundane and obvious can be life-changing news to some people. I had forgotten how it was for me, how nothing was obvious, when I started out doing this job long, long time ago.

If you're a hardened, experienced software developer or computer tinkerer - my blog posts are not really for you. You might find the odd gem, but most of the stuff you'll go 'duh - knew that'. But if you're starting out on a new challenge, the wee tricks I'll be describing are things that made my life easier - and I hope they'll make yours easier too.

Ok, to make a living, I am writing software, in various environments, using C++, JavaScript, ExtendScript, REALbasic, PHP, you name it... on Mac, Windows and Linux. As a project grows, invariably I 'll need a spot to put down some 'rubbish' files - temporary files created by my program that are only needed for a little while, and then can get trashed.

Sometimes, while debugging, I want to access those rubbish files with my text editor to inspect their contents - for example, to verify all is well at a certain stage of the program execution.

Mac OS X and Linux (both close relatives of Unix) have a number of spots for storing temporary files - but as on most forms of Unix, you can access that location using the /tmp path. Sometimes /tmp is not a real directory but rather an alias or link to some other place - but that does not matter too much.

If you start a command line session (go to /Applications/Utilities, and start Terminal), you can type

cd /tmp
ls

to see what's in it.

On Windows, you go to the Start button, select the Run... option, and type cmd into the Open field.

Click Ok. That brings up a command line window.

In the command line window, type

cd %TEMP%
dir

to see what's in it.

As it goes, I tend to hop all over the place in my command line sessions - using the cd command to hop from directory to directory.

Occasionally, I find myself in a command line session in a deeply nested directory, and I wish I had a Finder or Explorer window to get a better overview of the directory (aka folder) contents.

That's easy: on Mac, you simply type

open .

And on Windows, you type

explorer .

in the command line (i.e. open<Space><Period><Enter>, or explorer<Space><Period><Enter>), and a Finder or Explorer window will pop up showing the contents of the directory you're in. Then you can start dragging icons of the files you see onto various apps to inspect them. For example, I often use BBEdit, TextWrangler, 0xED on Mac, and UltraEdit or UEStudio on Windows.

If you're on the command line, and you want to inspect a text file in the current directory, that's easy enough too. I am personally well-aquainted with vi, a crusty old Unix text editor, and I use it a lot, but vi is not exactly user-friendly.

No worries - on Mac, if you have TextWrangler installed (which is a free download from BareBones), it should have asked you whether you wanted to install the command line tools. If you're not sure whether you allowed it or not - go to the TextWrangler menu, and select Install Command Line Tools...

From now on, when you're in a command line session in Terminal, you can simply type something like:

edit whateverfilename.log

to inspect the contents of the file whateverfilename.log in TextWrangler, assuming you'd have a file called whateverfilename.log in your current directory.

If you have BBEdit (TextWrangler's big brother) installed, simply substitute bbedit for edit on the command line.

On Windows, it depends on the text editor you have - everyone has Notepad or Wordpad (called write on the command line). If you have UEStudio installed, you can use that too - you could type a command similar to any of the following:

notepad whateverfilename.txt
write anotherfilename.log
uestudio yetanother.out

Now, suppose I wanted to quickly open a Finder window on the /tmp directory without needing to start up the Terminal program. If you go and look on your startup drive, you won't see it - it's there, but the Finder hides system folders like /tmp.

To get there using the Finder, select the Go - Go To Folder... menu item, and type /tmp in the dialog. Click the Go button - and you should get a Finder window on the /tmp folder.

On Windows, to quickly access a hidden folder, you go to the Run... function in the Start menu, and type in the path name. Type %TEMP% to open a window onto the temporary folder. Or type %APPDATA% to get an Explorer window straight onto the current user's hidden application data folder.

On Mac, if you need to be in the /tmp folder a lot (or any other system folder like that - /var/log for example), simply drag the little folder icon of the window for /tmp to the side bar of the Finder window - from now on you can click an icon to have instant Finder access to /tmp

Another cool one to have in the sidebar is the Originals or Masters folder inside your Pictures/iPhoto Library folder - you can get at the original files without having to go through the whole right-click, Show Package Contents routine on the iPhoto Library every time. Be careful though: you should only look at, and not touch the contents of those folders.

Windows 7 has a very similar functionality - you can drag folder icons into the Favorites on the sidebar.

Going from Finder or Explorer to Terminal or the Windows command line is also easy. Suppose you have a Finder window open, and you want to get a Terminal window positioned in the same folder.

Go to the Terminal, and type cd<Space>. Then drag the little folder icon from the Finder window title bar into the Terminal window. It will automatically spell out the correct path name - hit the <Enter> key in the terminal window and you're there.

Windows knows the same trick: in the command line window, type cd<Space> and drag the icon from the Explorer address bar into the command line window.

I hope that was useful! There's more stuff like this I could blog about - let me know whether anyone is interested!

Cheers,

Kris