I’m happy to (sorta) announce I (Justin) will be giving a talk at Mongo Seattle 2011 on Dec 1st in downtown Seattle.

According my contacts at 10Gen the registration is going gang-busters and there are going to be some really great sessions to attend including one from our buddy Damon Cortesi at Simply Measured as well as a bunch of 10Gen engineers (the people that make MongoDB) plus MongoLab, Geek.net, VMWare and WordSquared (a super fun game!)

So, what are you waiting for?

Go ahead a register here  … and if you send me a DM or @Reply on Twitter (@learnmongo) I might just give you a pretty decent discount code!

A while back we wrote a post explaining how to compact MongoDB data files, that example shows how to use some server side javascript and a cron job to automatically compact the data files on a schedule … however this isn’t always ideal as it will compact all the collections in a database at one time.

If you have a very large Collection you didn’t want to run the operation on …or a number of Collections in your database you didn’t wish to compact you were a bit out of luck.

Also, using repairDatabase() requires the disk space for the current database plus the repaired copy (i.e. double the disk space.)

Enter MongoDB 2.0 … in 2.0+ you can now use the simple compact command to target the compact to a single collection.

Running compact has three major performance benefits:

• Compacts collection (less disk space.)
• Defragments a collection (data pages are aligned better.)
• Rebuilds and compacts the collection’s indexes (less RAM needed, and better perf.)

If  you have a lot of read/write/delete operations going on in your Collection this could possibly have a fairly noticeable performance impact.

There are two ways to run the compact command:

> db.yourCollection.runCommand("compact");

> db.runCommand({ compact : 'yourCollection' });

Now, that said there are two big downsides to using compact …

• The compact command blocks operations on the collection until it’s done compacting (so it’s best to run this off hours during scheduled maintenance.)
• It’s typically slower than repairDatabase in its actual operational time.

Those aside, as part of routine maintenance compact is a really helpful new feature and might justify an upgrade to 2.0 all on it’s own!

Since MongoDB will allow you to store more than just string and int values but also things like arrays … from time to time you might need to know how many items are in an array in your document.

For example, say you have a hosting business and to make a good profit you need to sell at least three upgrades to each hosting client.

To make sure you market to the correct people you want to find all the clients with two upgrades (and get them to sign up for just one more.)

Your general document structure looks something like this:

{
 customer_name : "Sam Taylor",
 email : "sam@widgetxyz.com",
 upgrades: ["ssl", "diskspace 3", "RAM 5"]
}

{
 customer_name : "Kyle Lopez",
 email : "klopez@buymybikes.com",
 upgrades: ["diskspace 3", "RAM 5"]
}

To get all the clients that have two upgrades we’ll use the $size operator:

> db.clients.find( { upgrades : { $size: 2 } } );

Now you’ll get back all the documents that have two upgrades in the “upgrades” array (or in this example, the document for “Kyle Lopez”.)

What about a range?

Unfortunately the $size operator doesn’t support ranges (like getting all documents with 2 to 4 array values.) To do that you’d need to create your own separate field to keep track of the count and query on that field.

Inspired by this Stackoverflow question I thought it would be worth while to post this quick tip …

If you ever need to copy (or “clone”) a MongoDB databse between two MongoDB servers its as easy as running the copyDatabase() command.

First, change into your admin db …

Then, run copyDatabase() supplying the needed information, the “from_db” (the database to copy) the “to_db” and the host name of the database you are copying …

> use admin;
> db.copyDatabase(…);

You’ll need to run the copyDatabase() command on the sever you want to “copy” the database to, aka the “destantion” server …

> db.copyDatabase(from_db, to_db, from_host);

If you are running MongoDB with accounts/password via –auth you’ll need to add that info on there too …

> db.copyDatabase(from_db, to_db, from_host, username, password);

A simple example for a database named “weyoun” on a host called “delta.quad.com” would be like so …

> db.copyDatabase('weyoun', 'weyoun', 'delta.quad.com');

There ya go, you should be good to go copying your MongoDB database!

Dear Readers,

Sorry it’s been so quiet lately … thing is LearnMongo is moving!

Well not moving sites or anything … but I’m moving … to the sunny North West (at least that’s what my new boss keeps telling me?) …

Seattle Bound (wait that sounds wrong)

Once all the moving stuff is taken care of LearnMongo will be back to regular posts (staring in March.)

I’m sure I’ll have plenty of coffee to fuel some awesome Mongoness.

Thanks for your support so far, and I’m looking forward to lots more Mongo in 2011.

~ Justin

MongoLA 2011

Here are the slides from my talk at MongoLA last week, hopefully the audio will be up soon (I will post it here.)

There is a nice write up of the event here… thanks to everyone that came!

Beyond Logging

Using MongoDB to Power a Private Social Network

(Oh, and log millions of e-mails too.)

StackOverflow Anwser Series

This is a new “concept” post based on StackOverflow Questions/Answers.

Feel free to comment below if this format is helpful & informative!

Overview

This Answer covers the following MongoDB topics:

• Data Files
• Compacting Data Files
• The db.repairDatabase(); command
• Capped Collections
• Server-side Javascript

Question

StackOverflow Link

Is there a way to auto compact MongoDB Data Files?

The mongodb documentation says that:

To compact this space, run db.repairDatabase() from the mongo shell (note this operation will block and is slow).

I’m wondering how to make the mongodb free deleted disk space automatically?

Answer

StackOverflow Link

In general if you don’t need to shrink your datafiles you shouldn’t shrink them at all. This is because “growing” your datafiles on disk is a fairly expensive operation and the more space that MongoDB can allocate in datafiles the less fragmentation you will have.

Also doing any sort of “shrink” will likely be a rather expensive operation and will likely lock your database while it’s running!

So, you should try avoid shrinking and provide as much disk-space as possible for the database.

However if you must shrink the database you should keep two things in mind.

• MongoDB grows it’s data files by doubling so the datafiles may be 64MB, then 128MB, etc up to 2GB (at which point it stops doubling to keep files until 2GB.)
• As with most any database … to do operations like shrinking you’ll need to schedule a separate job to do so, there is no “autoshrink” in MongoDB. In fact of the major noSQL databases (hate that name) only Riak will autoshrink. So, you’ll need to create a job using your OS’s scheduler to run a shrink. You could use an bash script, or have a job run a php script, etc.

Server-side Javascript

You can use server-side Javascript to do the shrink and run that JS via mongo’s shell on a regular bases via a job (like cron or the windows scheduling service) …

Assuming a collection called foo you would save the javascript below into a file called bar.js and run …

$ mongo foo bar.js

The javascript file would look something like …

bar.js

// Get a the current collection size.
var storage = db.foo.storageSize();
var total = db.foo.totalSize();

print('Storage Size: ' + tojson(storage));

print('TotalSize: ' + tojson(total));

print('-----------------------');
print('Running db.repairDatabase()');
print('-----------------------');

// Run repair
db.repairDatabase()

// Get new collection sizes.
var storage_a = db.foo.storageSize();
var total_a = db.foo.totalSize();

print('Storage Size: ' + tojson(storage_a));
print('TotalSize: ' + tojson(total_a));

This will run and return something like …

MongoDB shell version: 1.6.4
connecting to: foo
Storage Size: 51351
TotalSize: 79152
-----------------------
Running db.repairDatabase()
-----------------------
Storage Size: 40960
TotalSize: 65153

Run this on a schedule (during non-peak hours) and you are good to go.

Capped Collections

However there is one other option, capped collections.

Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). They are a bit like the “RRD” concept if you are familiar with that.

In addition, capped collections automatically, with high performance, maintain insertion order for the objects in the collection; this is very powerful for certain use cases such as logging.

Basically you can limit the size of (or number of documents in ) a collection to say .. 20GB and once that limit is reached MongoDB will start to throw out the oldest records and replace them with newer entries as they come in.

This is a great way to keep a large amount of data, discarding the older data as time goes by and keeping the same amount of disk-space used.

By default a MongoDB install does not use a username/password combination to access the database.

No Password, But Why?

This is down to the design philosophy of MongoDB which is to push much of the “logic” to the application level and keep the database doing what databases do best! Hence,  given the way that MongoDB is normally used it’s generally not necessary.

In a SQL environment you might have multiple users with multiple groups and schemas to lock down different levels of the database, tables, views, stored procedures, etc. …

The idea of this is of course to 1) protect data from unauthorized modification or deletion as well as 2)  limiting the tug ‘o war between users by clearly defining their rights.

So, you can give the accounting department different privileges then the marketing department or given root access to the developers (of course.)

Conversely MongoDB’s design philosophy would pass these (generally) over to the application itself to handle.

I Don’t Care, I Still Want Passwords!

No worries, if the idea of no users/passwords keeps you up at nights you can still have them!

MongoDB (currently) supports users on the database level with both “read/write” and “read only” options.

You can see all the users in a database in the system.users collection …

> use mydatabase
> db.system.users.find()

Creating MongoDB Users

To adding a new user is fairly straight forward …

Read/Write User

$ ./mongo
> use mydatabase
> db.addUser("admin", "Sup3rG00dP@azzword")

This creates a read/write user for the database mydatabase (you can choose any username you wish.)

This user will be “good for” this database alone.

Read Only User

> db.addUser("web", "prettyGoodPass", true)

The “true” parameter there makes the user read only (great for parts of the application code that you want to make sure never accidentally preform a write operation.)

Using A User: Authenticate

To “login” as a user you’ll need to authenticate, simply use …

> db.auth("admin", "Sup3rG00dP@azzword")

You can also authenticate via the command line using the mongo parameters below …

  -u [ --username ] arg username for authentication
  -p [ --password ] arg password for authentication

Changing Passwords

To change the password simply run the addUser command again with a new password.

> db.addUser("web", "wayGooderPass", true)

Delete a User

To remove a user you need to remove the corresponding document for that user in the system.users collection.

> db.system.users.remove({"user" : "web"});

Set MongoDB to Force Authentication

To force MongoDB to use authentication you’ll need to add the –auth parameter to MongoDB at startup (so you would need to restart.)

If you are using an unauthorized user you’ll get an error something like …

error: { “$err” : “unauthorized for db [mydatabase] lock type: -1 ” }

If you have a read only user and attempt a write option you’ll simply get back …

unauthorized

Lastly, if you try to use a user that doesn’t have read/write privlages on the admin database …

So, if you try to do something like list the avaible database on the server ( > show dbs ) you’ll get error too and you’ll need to make sure you switch to an account with read/write privlages in admin.

Really enjoyed MongoSV this last week … a few posts (in process) inspired by it!

In the meantime here are some pics, the first few are by yours truly.

(Note: I’m no photographer! Ha.)

Click here to view these pictures larger

One really useful built-in feature of MongoDB is it’s GridFS.

This filesystem within MongoDB was designed for … well, holding files, especially files over 4MB … why 4MB?

Well BSON objects are limited to 4MB in size (BSON is the format that MongoDB uses to store it’s database information) so GridFS helps store files across multiple chunks.

As Kristina Chodorow of 10Gen puts it …

GridFS breaks large files into manageable chunks. It saves the chunks to one collection (fs.chunks) and then metadata about the file to another collection (fs.files). When you query for the file, GridFS queries the chunks collection and returns the file one piece at a time.

Why would you want to break large files in to “chunks”? A lot of of comes down to efficient memory & disk usage.

Chunks ‘O Random-Access Memory

Gee, mister. You’re even hungrier than I am.

Say you want to store larger files (maybe a 2GB video) when you preform a query on that file all 2GB needs to be loaded into memory … say you have a bigger file, 10GB, 25GB etc … it’s quite likely you’d run out of usable RAM or not have that much RAM available at all!

So, GridFS solves this problem by streaming the data back (in chunks) to the client … this way you’d never need to use more than 4MB of RAM.

Other Reasons to Use GridFS

Some other nicities of GridFS are …

• If you are using replication or autosharding your GridFS files will be seamlessly sharded or replicated for you.
• Since MongoDB datafiles are broken into 2 GB chunks MongoDB will automatically break your files into OS manageable pieces.
• You won’t have to worry about OS limitations like ‘weird’ filenames or a large number of files in one directory, etc.
• MongoDB will auto generate the MD5 hash of your file and store it in the file’s document. This is useful to compare the stored file with it’s MD5 hash to see if it was uploaded correctly, or already exists your database.

Command Line: mongofiles

An easy way to get started and see how GridFS works is the use the mongofiles command line utility (if you downloaded the binaries of MongoDB you should file this tool in the bin directory.)

To make things easy, mongofiles accepts RESTful looking commands, for example …

$ ./mongofiles -d myfiles put 03-smbd-menu-screen.mp3
connected to: 127.0.0.1

added file: {
   _id: ObjectId('4ce9ddcb45d74ecaa7f5a029'),
   filename: "03-smbd-menu-screen.mp3",
   chunkSize: 262144,
   uploadDate: new Date(1290395084166),
   md5: "7872291d4e67ae8b8bf7aea489ab52c1",
   length: 1419631 }

done!

This uploaded (PUT) the 03-smbd-menu-screen.mp3 file to a database called myfiles (it could be any database.)

This file now resides in the myfiles DB in the fs.files Collection. We can confirm this by passing the list command.

$ ./mongofiles -d myfiles list
connected to: 127.0.0.1
03-smbd-menu-screen.mp3 1419631

Hurrah! We have our files in there …you can also query it via the MongoDB Shell like so …

> use myfiles;
> db.fs.files.find({});
{
   "_id" : ObjectId("4ce9ddcb45d74ecaa7f5a029"),
   "filename" : "03-smbd-menu-screen.mp3",
   "chunkSize" : 262144,
   "uploadDate" : "Mon Nov 22 2010 03:04:44 GMT+0000 (UTC)",
   "md5" : "7872291d4e67ae8b8bf7aea489ab52c1",
   "length" : 1419631
}

Note: the size, upload date & md5 are all produced for you which is pretty  handy.

Uploading a File (or Data) via MongoDB Driver

Likely a more realistic way of storing files in GridFS will be via one of the many available language drivers. Each driver handles GridFS a little differantly but the concepts are the same.

The first thing you need to sort is out is are you going to upload actual files or are you going to create files from strings of data?

For example an application that allows a user to upload a video directly from their computer to your application would use the file method … however an application that would take a profile image (for example) and compress and resize it for use in your application would likely use the string of data method.

The File Method

In this example we’ll assume the file is already in your filesystem in the /tmp/ dir, but the file could be wherever your web-server/PHP is configured to access.

To work with GridFS files in PHP you use MongoGridFS class, more information can be found in the documentation.

We will use MongoGridFS::storeFile but you could also use MongoGridFS::put (which works like the command line example.)

<?php

// Connect to Mongo and set DB and Collection
$mongo = new Mongo();
$db = $mongo->myfiles;

// GridFS
$grid = $db->getGridFS();

// The file's location in the File System
$path = "/tmp/";
$filename = "03-smbd-menu-screen.mp3";
// Note metadata field & filename field
$storedfile = $grid->storeFile($path . $filename,
                 array("metadata" => array("filename" => $filename),
                 "filename" => $filename));

// Return newly stored file's Document ID
echo $storedfile;

?>

The String of Data Method

String of data is very similar only we’ll pass a string instead of a file/path so use the code above but use storeBytes instead.

$storedfile = $grid->storeBytes("This is test file data!",
                 array("metadata" => array("filename" => $filename),
                 "filename" => $filename));

You could of course pass any string, string representation of a images (or an encoded file via a string) where we’ve put “This is test file data!” …

A Little About Metadata

For PHP it doesn’t really matter but since other drivers handle things slightly differently it’s best to write any metadata to it’s own metadata field as well as a separate filename field as we have done in the example above.

You can put any file metadata that makes sense for your use in the metadata field.

Stream Back Files

Now that our file or files are loaded into GridFS streaming back the file is farily simple …

• Connect to MongoDB
• Do a findOne() on the file
• Load it into memory using getBytes()
• Set the proper headers
• Stream the file back to the browser

So, here is how we’d stream back an image in PHP …

Stream an Image from GridFS to the Browser

Warning: this will load the file into memory. If the file is bigger than your memory, this will cause problems!

<?php
// Connect to Mongo and set DB and Collection
$mongo = new Mongo();
$db = $mongo->myfiles;     

// GridFS
$gridFS = $db->getGridFS();     

// Find image to stream
$image = $gridFS->findOne("chunk-screaming.jpg");

// Stream image to browser
header('Content-type: image/jpeg');
echo $image->getBytes();

?>

With a little adjustment you could stream back an mp3, or video, or prompt for a file download, etc.

Other Ways to Search for GridFD Files

You could also use the Document’s ID …

$image = $gridFS->findOne(
         array("_id" => new MongoId("4ceb167810f1d50a80e1c71c"))
         );

That will likely be how your application would look up a file in a real world system.

You can use any valid MongoDB findOne() query in it’s place as well, or use find() to get back a GridFS cursor of files, you can find out more about that here.

Deleting Files

You delete files in the same way, there are actually a couple ways to remove GridFS files, but we’ll just use one of the easiest …

Be really careful about passing the correct query to remove or you might just find yourself removing all your files! You can also use MongoGridFS::delete and pass the Document’s ID only.

<?php

// Connect to Mongo and set DB and Collection
$mongo = new Mongo();
$db = $mongo->myfiles;

// GridFS
$gridFS = $db->getGridFS();

// Find file to remove 
$removeFile = $gridFS->remove(
                array("_id" => new MongoId("4ceb167810f1d50a80e1c71c"))
              );

?>

Wrap Up

Hopefully you can now get started with GridFS and see if it will work well for your application … remember if you stream back (using the image example above) the files they will be loaded into memory and not streamed in 4MB chunks …so be careful!

Have fun.