| 1 | <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Chapter 33. Handling Large Directories</title><link rel="stylesheet" href="../samba.css" type="text/css"><meta name="generator" content="DocBook XSL Stylesheets V1.74.0"><link rel="home" href="index.html" title="The Official Samba 3.0.x HOWTO and Reference Guide"><link rel="up" href="optional.html" title="Part III. Advanced Configuration"><link rel="prev" href="SambaHA.html" title="Chapter 32. High Availability"><link rel="next" href="cfgsmarts.html" title="Chapter 34. Advanced Configuration Techniques"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 33. Handling Large Directories</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="SambaHA.html">Prev</a> </td><th width="60%" align="center">Part III. Advanced Configuration</th><td width="20%" align="right"> <a accesskey="n" href="cfgsmarts.html">Next</a></td></tr></table><hr></div><div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="largefile"></a>Chapter 33. Handling Large Directories</h2></div><div><div class="author"><h3 class="author"><span class="firstname">Jeremy</span> <span class="orgname">Samba Team</span> <span class="surname">Allison</span></h3><div class="affiliation"><span class="orgname">Samba Team<br></span><div class="address"><p><code class="email"><<a class="email" href="mailto:jra@samba.org">jra@samba.org</a>></code></p></div></div></div></div><div><div class="author"><h3 class="author"><span class="firstname">John</span> <span class="othername">H.</span> <span class="orgname">Samba Team</span> <span class="surname">Terpstra</span></h3><div class="affiliation"><span class="orgname">Samba Team<br></span><div class="address"><p><code class="email"><<a class="email" href="mailto:jht@samba.org">jht@samba.org</a>></code></p></div></div></div></div><div><p class="pubdate">March 5, 2005</p></div></div></div><p>
|
|---|
| 2 | <a class="indexterm" name="id2673338"></a>
|
|---|
| 3 | <a class="indexterm" name="id2673345"></a>
|
|---|
| 4 | <a class="indexterm" name="id2673352"></a>
|
|---|
| 5 | Samba-3.0.12 and later implements a solution for sites that have experienced performance degradation due to the
|
|---|
| 6 | problem of using Samba-3 with applications that need large numbers of files (100,000 or more) per directory.
|
|---|
| 7 | </p><p>
|
|---|
| 8 | <a class="indexterm" name="id2673366"></a>
|
|---|
| 9 | <a class="indexterm" name="id2673373"></a>
|
|---|
| 10 | The key was fixing the directory handling to read only the current list requested instead of the old
|
|---|
| 11 | (up to samba-3.0.11) behavior of reading the entire directory into memory before doling out names.
|
|---|
| 12 | Normally this would have broken OS/2 applications, which have very strange delete semantics, but by
|
|---|
| 13 | stealing logic from Samba4 (thanks, Tridge), the current code in 3.0.12 handles this correctly.
|
|---|
| 14 | </p><p>
|
|---|
| 15 | <a class="indexterm" name="id2673390"></a>
|
|---|
| 16 | <a class="indexterm" name="id2673397"></a>
|
|---|
| 17 | To set up an application that needs large numbers of files per directory in a way that does not
|
|---|
| 18 | damage performance unduly, follow these steps:
|
|---|
| 19 | </p><p>
|
|---|
| 20 | <a class="indexterm" name="id2673410"></a>
|
|---|
| 21 | First, you need to canonicalize all the files in the directory to have one case, upper or lower take your
|
|---|
| 22 | pick (I chose upper because all my files were already uppercase names). Then set up a new custom share for the
|
|---|
| 23 | application as follows:
|
|---|
| 24 | </p><table class="simplelist" border="0" summary="Simple list"><tr><td> </td></tr><tr><td><em class="parameter"><code>[bigshare]</code></em></td></tr><tr><td><a class="indexterm" name="id2673438"></a><em class="parameter"><code>path = /data/manyfilesdir</code></em></td></tr><tr><td><a class="indexterm" name="id2673449"></a><em class="parameter"><code>read only = no</code></em></td></tr><tr><td><a class="indexterm" name="id2673461"></a><em class="parameter"><code>case sensitive = True</code></em></td></tr><tr><td><a class="indexterm" name="id2673473"></a><em class="parameter"><code>default case = upper</code></em></td></tr><tr><td><a class="indexterm" name="id2673485"></a><em class="parameter"><code>preserve case = no</code></em></td></tr><tr><td><a class="indexterm" name="id2673496"></a><em class="parameter"><code>short preserve case = no</code></em></td></tr></table><p>
|
|---|
| 25 | </p><p>
|
|---|
| 26 | <a class="indexterm" name="id2673512"></a>
|
|---|
| 27 | <a class="indexterm" name="id2673518"></a>
|
|---|
| 28 | <a class="indexterm" name="id2673525"></a>
|
|---|
| 29 | Of course, use your own path and settings, but set the case options to match the case of all the files in your
|
|---|
| 30 | directory. The path should point at the large directory needed for the application any new files created in
|
|---|
| 31 | there and in any paths under it will be forced by smbd into uppercase, but smbd will no longer have to scan
|
|---|
| 32 | the directory for names: it knows that if a file does not exist in uppercase, then it doesn't exist at all.
|
|---|
| 33 | </p><p>
|
|---|
| 34 | <a class="indexterm" name="id2673546"></a>
|
|---|
| 35 | <a class="indexterm" name="id2673552"></a>
|
|---|
| 36 | <a class="indexterm" name="id2673559"></a>
|
|---|
| 37 | The secret to this is really in the <a class="link" href="smb.conf.5.html#CASESENSITIVE" target="_top">case sensitive = True</a>
|
|---|
| 38 | line. This tells smbd never to scan for case-insensitive versions of names. So if an application asks for a file
|
|---|
| 39 | called <code class="filename">FOO</code>, and it cannot be found by a simple stat call, then smbd will return file not
|
|---|
| 40 | found immediately without scanning the containing directory for a version of a different case. The other
|
|---|
| 41 | <code class="filename">xxx case xxx</code> lines make this work by forcing a consistent case on all files created by
|
|---|
| 42 | <span class="application">smbd</span>.
|
|---|
| 43 | </p><p>
|
|---|
| 44 | <a class="indexterm" name="id2673606"></a>
|
|---|
| 45 | <a class="indexterm" name="id2673612"></a>
|
|---|
| 46 | <a class="indexterm" name="id2673619"></a>
|
|---|
| 47 | Remember, all files and directories under the <em class="parameter"><code>path</code></em> directory must be in uppercase
|
|---|
| 48 | with this <code class="filename">smb.conf</code> stanza because <span class="application">smbd</span> will not be able to find lowercase filenames with these settings. Also
|
|---|
| 49 | note that this is done on a per-share basis, allowing this parameter to be set only for a share servicing an application with
|
|---|
| 50 | this problematic behavior (using large numbers of entries in a directory) the rest of your <span class="application">smbd</span> shares
|
|---|
| 51 | don't need to be affected.
|
|---|
| 52 | </p><p>
|
|---|
| 53 | This makes smbd much faster when dealing with large directories. My test case has over 100,000 files, and
|
|---|
| 54 | smbd now deals with this very efficiently.
|
|---|
| 55 | </p></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="SambaHA.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="optional.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="cfgsmarts.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 32. High Availability </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 34. Advanced Configuration Techniques</td></tr></table></div></body></html>
|
|---|
