1 |
|
---|
2 |
|
---|
3 | THIS IS INCOMPLETE! I'M ONLY COMMITING IT IN ORDER TO SOLICIT COMMENTS
|
---|
4 | FROM A FEW PEOPLE. DON'T TAKE THIS AS THE FINAL VERSION YET.
|
---|
5 |
|
---|
6 |
|
---|
7 | Samba4 Programming Guide
|
---|
8 | ========================
|
---|
9 |
|
---|
10 | .. contents::
|
---|
11 |
|
---|
12 | The internals of Samba4 are quite different from previous versions of
|
---|
13 | Samba, so even if you are an experienced Samba developer please take
|
---|
14 | the time to read through this document.
|
---|
15 |
|
---|
16 | This document will explain both the broad structure of Samba4, and
|
---|
17 | some of the common coding elements such as memory management and
|
---|
18 | dealing with macros.
|
---|
19 |
|
---|
20 |
|
---|
21 | Coding Style
|
---|
22 | ------------
|
---|
23 |
|
---|
24 | In past versions of Samba we have basically let each programmer choose
|
---|
25 | their own programming style. Unfortunately the result has often been
|
---|
26 | that code that other members of the team find difficult to read. For
|
---|
27 | Samba version 4 I would like to standardise on a common coding style
|
---|
28 | to make the whole tree more readable. For those of you who are
|
---|
29 | horrified at the idea of having to learn a new style, I can assure you
|
---|
30 | that it isn't as painful as you might think. I was forced to adopt a
|
---|
31 | new style when I started working on the Linux kernel, and after some
|
---|
32 | initial pain found it quite easy.
|
---|
33 |
|
---|
34 | That said, I don't want to invent a new style, instead I would like to
|
---|
35 | adopt the style used by the Linux kernel. It is a widely used style
|
---|
36 | with plenty of support tools available. See Documentation/CodingStyle
|
---|
37 | in the Linux source tree. This is the style that I have used to write
|
---|
38 | all of the core infrastructure for Samba4 and I think that we should
|
---|
39 | continue with that style.
|
---|
40 |
|
---|
41 | I also think that we should most definately *not* adopt an automatic
|
---|
42 | reformatting system in cvs (or whatever other source code system we
|
---|
43 | end up using in the future). Such automatic formatters are, in my
|
---|
44 | experience, incredibly error prone and don't understand the necessary
|
---|
45 | exceptions. I don't mind if people use automated tools to reformat
|
---|
46 | their own code before they commit it, but please do not run such
|
---|
47 | automated tools on large slabs of existing code without being willing
|
---|
48 | to spend a *lot* of time hand checking the results.
|
---|
49 |
|
---|
50 | Finally, I think that for code that is parsing or formatting protocol
|
---|
51 | packets the code layout should strongly reflect the packet
|
---|
52 | format. That means ordring the code so that it parses in the same
|
---|
53 | order as the packet is stored on the wire (where possible) and using
|
---|
54 | white space to align packet offsets so that a reader can immediately
|
---|
55 | map any line of the code to the corresponding place in the packet.
|
---|
56 |
|
---|
57 |
|
---|
58 | Static and Global Data
|
---|
59 | ----------------------
|
---|
60 |
|
---|
61 | The basic rule is "avoid static and global data like the plague". What
|
---|
62 | do I mean by static data? The way to tell if you have static data in a
|
---|
63 | file is to use the "size" utility in Linux. For example if we run::
|
---|
64 |
|
---|
65 | size libcli/raw/*.o
|
---|
66 |
|
---|
67 | in Samba4 then you get the following::
|
---|
68 |
|
---|
69 | text data bss dec hex filename
|
---|
70 | 2015 0 0 2015 7df libcli/raw/clikrb5.o
|
---|
71 | 202 0 0 202 ca libcli/raw/clioplock.o
|
---|
72 | 35 0 0 35 23 libcli/raw/clirewrite.o
|
---|
73 | 3891 0 0 3891 f33 libcli/raw/clisession.o
|
---|
74 | 869 0 0 869 365 libcli/raw/clisocket.o
|
---|
75 | 4962 0 0 4962 1362 libcli/raw/clispnego.o
|
---|
76 | 1223 0 0 1223 4c7 libcli/raw/clitransport.o
|
---|
77 | 2294 0 0 2294 8f6 libcli/raw/clitree.o
|
---|
78 | 1081 0 0 1081 439 libcli/raw/raweas.o
|
---|
79 | 6765 0 0 6765 1a6d libcli/raw/rawfile.o
|
---|
80 | 6824 0 0 6824 1aa8 libcli/raw/rawfileinfo.o
|
---|
81 | 2944 0 0 2944 b80 libcli/raw/rawfsinfo.o
|
---|
82 | 541 0 0 541 21d libcli/raw/rawioctl.o
|
---|
83 | 1728 0 0 1728 6c0 libcli/raw/rawnegotiate.o
|
---|
84 | 723 0 0 723 2d3 libcli/raw/rawnotify.o
|
---|
85 | 3779 0 0 3779 ec3 libcli/raw/rawreadwrite.o
|
---|
86 | 6597 0 0 6597 19c5 libcli/raw/rawrequest.o
|
---|
87 | 5580 0 0 5580 15cc libcli/raw/rawsearch.o
|
---|
88 | 3034 0 0 3034 bda libcli/raw/rawsetfileinfo.o
|
---|
89 | 5187 0 0 5187 1443 libcli/raw/rawtrans.o
|
---|
90 | 2033 0 0 2033 7f1 libcli/raw/smb_signing.o
|
---|
91 |
|
---|
92 | notice that the "data" and "bss" columns are all zero? That is
|
---|
93 | good. If there are any non-zero values in data or bss then that
|
---|
94 | indicates static data and is bad (as a rule of thumb).
|
---|
95 |
|
---|
96 | Lets compare that result to the equivalent in Samba3::
|
---|
97 |
|
---|
98 | text data bss dec hex filename
|
---|
99 | 3978 0 0 3978 f8a libsmb/asn1.o
|
---|
100 | 18963 0 288 19251 4b33 libsmb/cliconnect.o
|
---|
101 | 2815 0 1024 3839 eff libsmb/clidgram.o
|
---|
102 | 4038 0 0 4038 fc6 libsmb/clientgen.o
|
---|
103 | 3337 664 256 4257 10a1 libsmb/clierror.o
|
---|
104 | 10043 0 0 10043 273b libsmb/clifile.o
|
---|
105 | 332 0 0 332 14c libsmb/clifsinfo.o
|
---|
106 | 166 0 0 166 a6 libsmb/clikrb5.o
|
---|
107 | 5212 0 0 5212 145c libsmb/clilist.o
|
---|
108 | 1367 0 0 1367 557 libsmb/climessage.o
|
---|
109 | 259 0 0 259 103 libsmb/clioplock.o
|
---|
110 | 1584 0 0 1584 630 libsmb/cliprint.o
|
---|
111 | 7565 0 256 7821 1e8d libsmb/cliquota.o
|
---|
112 | 7694 0 0 7694 1e0e libsmb/clirap.o
|
---|
113 | 27440 0 0 27440 6b30 libsmb/clirap2.o
|
---|
114 | 2905 0 0 2905 b59 libsmb/clireadwrite.o
|
---|
115 | 1698 0 0 1698 6a2 libsmb/clisecdesc.o
|
---|
116 | 5517 0 0 5517 158d libsmb/clispnego.o
|
---|
117 | 485 0 0 485 1e5 libsmb/clistr.o
|
---|
118 | 8449 0 0 8449 2101 libsmb/clitrans.o
|
---|
119 | 2053 0 4 2057 809 libsmb/conncache.o
|
---|
120 | 3041 0 256 3297 ce1 libsmb/credentials.o
|
---|
121 | 1261 0 1024 2285 8ed libsmb/doserr.o
|
---|
122 | 14560 0 0 14560 38e0 libsmb/errormap.o
|
---|
123 | 3645 0 0 3645 e3d libsmb/namecache.o
|
---|
124 | 16815 0 8 16823 41b7 libsmb/namequery.o
|
---|
125 | 1626 0 0 1626 65a libsmb/namequery_dc.o
|
---|
126 | 14301 0 1076 15377 3c11 libsmb/nmblib.o
|
---|
127 | 24516 0 2048 26564 67c4 libsmb/nterr.o
|
---|
128 | 8661 0 8 8669 21dd libsmb/ntlmssp.o
|
---|
129 | 3188 0 0 3188 c74 libsmb/ntlmssp_parse.o
|
---|
130 | 4945 0 0 4945 1351 libsmb/ntlmssp_sign.o
|
---|
131 | 1303 0 0 1303 517 libsmb/passchange.o
|
---|
132 | 1221 0 0 1221 4c5 libsmb/pwd_cache.o
|
---|
133 | 2475 0 4 2479 9af libsmb/samlogon_cache.o
|
---|
134 | 10768 32 0 10800 2a30 libsmb/smb_signing.o
|
---|
135 | 4524 0 16 4540 11bc libsmb/smbdes.o
|
---|
136 | 5708 0 0 5708 164c libsmb/smbencrypt.o
|
---|
137 | 7049 0 3072 10121 2789 libsmb/smberr.o
|
---|
138 | 2995 0 0 2995 bb3 libsmb/spnego.o
|
---|
139 | 3186 0 0 3186 c72 libsmb/trustdom_cache.o
|
---|
140 | 1742 0 0 1742 6ce libsmb/trusts_util.o
|
---|
141 | 918 0 28 946 3b2 libsmb/unexpected.o
|
---|
142 |
|
---|
143 | notice all of the non-zero data and bss elements? Every bit of that
|
---|
144 | data is a bug waiting to happen.
|
---|
145 |
|
---|
146 | Static data is evil as it has the following consequences:
|
---|
147 | - it makes code much less likely to be thread-safe
|
---|
148 | - it makes code much less likely to be recursion-safe
|
---|
149 | - it leads to subtle side effects when the same code is called from multiple places
|
---|
150 | - doesn't play well with shared libraries or plugins
|
---|
151 |
|
---|
152 | Static data is particularly evil in library code (such as our internal
|
---|
153 | smb and rpc libraries). If you can get rid of all static data in
|
---|
154 | libraries then you can make some fairly strong guarantees about the
|
---|
155 | behaviour of functions in that library, which really helps.
|
---|
156 |
|
---|
157 | Of course, it is possible to write code that uses static data and is
|
---|
158 | safe, it's just much harder to do that than just avoid static data in
|
---|
159 | the first place. We have been tripped up countless times by subtle
|
---|
160 | bugs in Samba due to the use of static data, so I think it is time to
|
---|
161 | start avoiding it in new code. Much of the core infrastructure of
|
---|
162 | Samba4 was specifically written to avoid static data, so I'm going to
|
---|
163 | be really annoyed if everyone starts adding lots of static data back
|
---|
164 | in.
|
---|
165 |
|
---|
166 | So, how do we avoid static data? The basic method is to use context
|
---|
167 | pointers. When reading the Samba4 code you will notice that just about
|
---|
168 | every function takes a pointer to a context structure as its first
|
---|
169 | argument. Any data that the function needs that isn't an explicit
|
---|
170 | argument to the function can be found by traversing that context.
|
---|
171 |
|
---|
172 | Note that this includes all of the little caches that we have lying
|
---|
173 | all over the code in Samba3. I'm referring to the ones that generally
|
---|
174 | have a "static int initialised" and then some static string or integer
|
---|
175 | that remembers the last return value of the function. Get rid of them!
|
---|
176 | If you are *REALLY* absolutely completely certain that your personal
|
---|
177 | favourite mini-cache is needed then you should do it properly by
|
---|
178 | putting it into the appropriate context rather than doing it the lazy
|
---|
179 | way by putting it inside the target function. I would suggest however
|
---|
180 | that the vast majority of those little caches are useless - don't
|
---|
181 | stick it in unless you have really firm benchmarking results that show
|
---|
182 | that it is needed and helps by a significant amount.
|
---|
183 |
|
---|
184 | Note that Samba4 is not yet completely clean of static data like
|
---|
185 | this. I've gotten the smbd/ directory down to 24 bytes of static data,
|
---|
186 | and libcli/raw/ down to zero. I've also gotten the ntvfs layer and all
|
---|
187 | backends down to just 8 bytes in ntvfs_base.c. The rest still needs
|
---|
188 | some more work.
|
---|
189 |
|
---|
190 | Also note that truly constant data is OK, and will not in fact show up
|
---|
191 | in the data and bss columns in "size" anyway (it will be included in
|
---|
192 | "text"). So you can have constant tables of protocol data.
|
---|
193 |
|
---|
194 |
|
---|
195 | How to use talloc
|
---|
196 | -----------------
|
---|
197 |
|
---|
198 | Please see the separate document, lib/talloc/talloc_guide.txt
|
---|
199 | You _must_ read this if you want to program in Samba4.
|
---|
200 |
|
---|
201 |
|
---|
202 | Interface Structures
|
---|
203 | --------------------
|
---|
204 |
|
---|
205 | One of the biggest changes in Samba4 is the universal use of interface
|
---|
206 | structures. Go take a look through libcli/raw/interfaces.h now to get
|
---|
207 | an idea of what I am talking about.
|
---|
208 |
|
---|
209 | In Samba3 many of the core wire structures in the SMB protocol were
|
---|
210 | never explicitly defined in Samba. Instead, our parse and generation
|
---|
211 | functions just worked directly with wire buffers. The biggest problem
|
---|
212 | with this is that is tied our parse code with our "business logic"
|
---|
213 | much too closely, which meant the code got extremely confusing to
|
---|
214 | read.
|
---|
215 |
|
---|
216 | In Samba4 we have explicitly defined interface structures for
|
---|
217 | everything in the protocol. When we receive a buffer we always parse
|
---|
218 | it completely into one of these structures, then we pass a pointer to
|
---|
219 | that structure to a backend handler. What we must *not* do is make any
|
---|
220 | decisions about the data inside the parse functions. That is critical
|
---|
221 | as different backends will need different portions of the data. This
|
---|
222 | leads to a golden rule for Samba4:
|
---|
223 |
|
---|
224 | "don't design interfaces that lose information"
|
---|
225 |
|
---|
226 | In Samba3 our backends often received "condensed" versions of the
|
---|
227 | information sent from clients, but this inevitably meant that some
|
---|
228 | backends could not get at the data they needed to do what they wanted,
|
---|
229 | so from now on we should expose the backends to all of the available
|
---|
230 | information and let them choose which bits they want.
|
---|
231 |
|
---|
232 | Ok, so now some of you will be thinking "this sounds just like our
|
---|
233 | msrpc code from Samba3", and while to some extent this is true there
|
---|
234 | are extremely important differences in the approach that are worth
|
---|
235 | pointing out.
|
---|
236 |
|
---|
237 | In the Samba3 msrpc code we used explicit parse structures for all
|
---|
238 | msrpc functions. The problem is that we didn't just put all of the
|
---|
239 | real variables in these structures, we also put in all the artifacts
|
---|
240 | as well. A good example is the security descriptor strucrure that
|
---|
241 | looks like this in Samba3::
|
---|
242 |
|
---|
243 | typedef struct security_descriptor_info
|
---|
244 | {
|
---|
245 | uint16 revision;
|
---|
246 | uint16 type;
|
---|
247 |
|
---|
248 | uint32 off_owner_sid;
|
---|
249 | uint32 off_grp_sid;
|
---|
250 | uint32 off_sacl;
|
---|
251 | uint32 off_dacl;
|
---|
252 |
|
---|
253 | SEC_ACL *dacl;
|
---|
254 | SEC_ACL *sacl;
|
---|
255 | DOM_SID *owner_sid;
|
---|
256 | DOM_SID *grp_sid;
|
---|
257 | } SEC_DESC;
|
---|
258 |
|
---|
259 | The problem with this structure is all the off_* variables. Those are
|
---|
260 | not part of the interface, and do not appear in any real descriptions
|
---|
261 | of Microsoft security descriptors. They are parsing artifacts
|
---|
262 | generated by the IDL compiler that Microsoft use. That doesn't mean
|
---|
263 | they aren't needed on the wire - indeed they are as they tell the
|
---|
264 | parser where to find the following four variables, but they should
|
---|
265 | *NOT* be in the interface structure.
|
---|
266 |
|
---|
267 | In Samba3 there were unwritten rules about which variables in a
|
---|
268 | structure a high level caller has to fill in and which ones are filled
|
---|
269 | in by the marshalling code. In Samba4 those rules are gone, because
|
---|
270 | the redundant artifact variables are gone. The high level caller just
|
---|
271 | sets up the real variables and the marshalling code worries about
|
---|
272 | generating the right offsets.
|
---|
273 |
|
---|
274 | The same rule applies to strings. In many places in the SMB and MSRPC
|
---|
275 | protocols complex strings are used on the wire, with complex rules
|
---|
276 | about padding, format, alighment, termination etc. None of that
|
---|
277 | information is useful to a high level calling routine or to a backend - its
|
---|
278 | all just so much wire fluff. So, in Samba4 these strings are
|
---|
279 | just "char \*" and are always in our internal multi-byte format (which
|
---|
280 | is usually UTF8). It is up to the parse functions to worry about
|
---|
281 | translating the format and getting the padding right.
|
---|
282 |
|
---|
283 | The one exception to this is the use of the WIRE_STRING type, but that
|
---|
284 | has a very good justification in terms of regression testing. Go and
|
---|
285 | read the comment in smb_interfaces.h about that now.
|
---|
286 |
|
---|
287 | So, here is another rule to code by. When writing an interface
|
---|
288 | structure think carefully about what variables in the structure can be
|
---|
289 | left out as they are redundant. If some length is effectively defined
|
---|
290 | twice on the wire then only put it once in the packet. If a length can
|
---|
291 | be inferred from a null termination then do that and leave the length
|
---|
292 | out of the structure completely. Don't put redundant stuff in
|
---|
293 | structures!
|
---|
294 |
|
---|
295 |
|
---|
296 | Async Design
|
---|
297 | ------------
|
---|
298 |
|
---|
299 | Samba4 has an asynchronous design. That affects *lots* of the code,
|
---|
300 | and the implications of the asynchronous design needs to be considered
|
---|
301 | just about everywhere.
|
---|
302 |
|
---|
303 | The first aspect of the async design to look at is the SMB client
|
---|
304 | library. Lets take a look at the following three functions in
|
---|
305 | libcli/raw/rawfile.c::
|
---|
306 |
|
---|
307 | struct cli_request *smb_raw_seek_send(struct cli_tree *tree, struct smb_seek *parms);
|
---|
308 | NTSTATUS smb_raw_seek_recv(struct cli_request *req, struct smb_seek *parms);
|
---|
309 | NTSTATUS smb_raw_seek(struct cli_tree *tree, struct smb_seek *parms);
|
---|
310 |
|
---|
311 | Go and read them now then come back.
|
---|
312 |
|
---|
313 | Ok, first notice there there are 3 separate functions, whereas the
|
---|
314 | equivalent code in Samba3 had just one. Also note that the 3rd
|
---|
315 | function is extremely simple - its just a wrapper around calling the
|
---|
316 | first two in order.
|
---|
317 |
|
---|
318 | The three separate functions are needed because we need to be able to
|
---|
319 | generate SMB calls asynchronously. The first call, which for smb calls
|
---|
320 | is always called smb_raw_XXXX_send(), constructs and sends a SMB
|
---|
321 | request and returns a "struct cli_request" which acts as a handle for
|
---|
322 | the request. The caller is then free to do lots of other calls if it
|
---|
323 | wants to, then when it is ready it can call the smb_raw_XXX_recv()
|
---|
324 | function to receive the reply.
|
---|
325 |
|
---|
326 | If all you want is a synchronous call then call the 3rd interface, the
|
---|
327 | one called smb_raw_XXXX(). That just calls the first two in order, and
|
---|
328 | blocks waiting for the reply.
|
---|
329 |
|
---|
330 | But what if you want to be called when the reply comes in? Yes, thats
|
---|
331 | possible. You can do things like this::
|
---|
332 |
|
---|
333 | struct cli_request *req;
|
---|
334 |
|
---|
335 | req = smb_raw_XXX_send(tree, params);
|
---|
336 |
|
---|
337 | req->async.fn = my_callback;
|
---|
338 | req->async.private = my_private_data;
|
---|
339 |
|
---|
340 | then in your callback function you can call the smb_raw_XXXX_recv()
|
---|
341 | function to receive the reply. Your callback will receive the "req"
|
---|
342 | pointer, which you can use to retrieve your private data from
|
---|
343 | req->async.private.
|
---|
344 |
|
---|
345 | Then all you need to do is ensure that the main loop in the client
|
---|
346 | library gets called. You can either do that by polling the connection
|
---|
347 | using cli_transport_pending() and cli_request_receive_next() or you
|
---|
348 | can use transport->idle.func to setup an idle function handler to call
|
---|
349 | back to your main code. Either way, you can build a fully async
|
---|
350 | application.
|
---|
351 |
|
---|
352 | In order to support all of this we have to make sure that when we
|
---|
353 | write a piece of library code (SMB, MSRPC etc) that we build the
|
---|
354 | separate _send() and _recv() functions. It really is worth the effort.
|
---|
355 |
|
---|
356 | Now about async in smbd, a much more complex topic.
|
---|
357 |
|
---|
358 | The SMB protocol is inherently async. Some functions (such as change
|
---|
359 | notify) often don't return for hours, while hundreds of other
|
---|
360 | functions pass through the socket. Take a look at the RAW-MUX test in
|
---|
361 | the Samba4 smbtorture to see some really extreme examples of the sort
|
---|
362 | of async operations that Windows supports. I particularly like the
|
---|
363 | open/open/close sequence where the 2nd open (which conflicts with the
|
---|
364 | first) succeeds because the subsequent close is answered out of order.
|
---|
365 |
|
---|
366 | In Samba3 we handled this stuff very badly. We had awful "pending
|
---|
367 | request" queues that allocated full 128k packet buffers, and even with
|
---|
368 | all that crap we got the semantics wrong. In Samba4 I intend to make
|
---|
369 | sure we get this stuff right.
|
---|
370 |
|
---|
371 | So, how do we do this? We now have an async interface between smbd and
|
---|
372 | the NTVFS backends. Whenever smbd calls into a backend the backend has
|
---|
373 | an option of answer the request in a synchronous fashion if it wants
|
---|
374 | to just like in Samba3, but it also has the option of answering the
|
---|
375 | request asynchronously. The only backend that currently does this is
|
---|
376 | the CIFS backend, but I hope the other backends will soon do this to.
|
---|
377 |
|
---|
378 | To make this work you need to do things like this in the backend::
|
---|
379 |
|
---|
380 | req->control_flags |= REQ_CONTROL_ASYNC;
|
---|
381 |
|
---|
382 | that tells smbd that the backend has elected to reply later rather
|
---|
383 | than replying immediately. The backend must *only* do this if
|
---|
384 | req->async.send_fn is not NULL. If send_fn is NULL then it means that
|
---|
385 | the smbd front end cannot handle this function being replied to in an
|
---|
386 | async fashion.
|
---|
387 |
|
---|
388 | If the backend does this then it is up to the backend to call
|
---|
389 | req->async.send_fn() when it is ready to reply. It the meantime smbd
|
---|
390 | puts the call on hold and goes back to answering other requests on the
|
---|
391 | socket.
|
---|
392 |
|
---|
393 | Inside smbd you will find that there is code to support this. The most
|
---|
394 | obvious change is that smbd splits each SMB reply function into two
|
---|
395 | parts - just like the client library has a _send() and _recv()
|
---|
396 | function, so smbd has a _send() function and the parse function for
|
---|
397 | each SMB.
|
---|
398 |
|
---|
399 | As an example go and have a look at reply_getatr_send() and
|
---|
400 | reply_getatr() in smb_server/smb/reply.c. Read them? Good.
|
---|
401 |
|
---|
402 | Notice that reply_getatr() sets up the req->async structure to contain
|
---|
403 | the send function. Thats how the backend gets to do an async reply, it
|
---|
404 | calls this function when it is ready. Also notice that reply_getatr()
|
---|
405 | only does the parsing of the request, and does not do the reply
|
---|
406 | generation. That is done by the _send() function.
|
---|
407 |
|
---|
408 |
|
---|
409 | NTVFS
|
---|
410 | -----
|
---|
411 |
|
---|
412 | One of the most noticeable changes in Samba4 is the introduction of
|
---|
413 | the NTVFS layer. This provided the initial motivation for the design
|
---|
414 | of Samba4 and in many ways lies at the heart of the design.
|
---|
415 |
|
---|
416 | In Samba3 the main file serving process (smbd) combined the handling
|
---|
417 | of the SMB protocol with the mapping to POSIX semantics in the same
|
---|
418 | code. If you look in smbd/reply.c in Samba3 you see numerous places
|
---|
419 | where POSIX assumptions are mixed tightly with SMB parsing code. We
|
---|
420 | did have a VFS layer in Samba3, but it was a POSIX-like VFS layer, so
|
---|
421 | no matter how you wrote a plugin you could not bypass the POSIX
|
---|
422 | mapping decisions that had already been made before the VFS layer was
|
---|
423 | called.
|
---|
424 |
|
---|
425 | In Samba4 things are quite different. All SMB parsing is performed in
|
---|
426 | the smbd front end, then fully parsed requests are passed to the NTVFS
|
---|
427 | backend. That backend makes any semantic mapping decisions and fills
|
---|
428 | in the 'out' portion of the request. The front end is then responsible
|
---|
429 | for putting those results into wire format and sending them to the
|
---|
430 | client.
|
---|
431 |
|
---|
432 | Lets have a look at one of those request structures. Go and read the
|
---|
433 | definition of "union smb_write" and "enum write_level" in
|
---|
434 | libcli/raw/interfaces.h. (no, don't just skip reading it, really go
|
---|
435 | and read it. Yes, that means you!).
|
---|
436 |
|
---|
437 | Notice the union? That's how Samba4 allows a single NTVFS backend
|
---|
438 | interface to handle the several different ways of doing a write
|
---|
439 | operation in the SMB protocol. Now lets look at one section of that
|
---|
440 | union::
|
---|
441 |
|
---|
442 | /* SMBwriteX interface */
|
---|
443 | struct {
|
---|
444 | enum smb_write_level level;
|
---|
445 | struct {
|
---|
446 | union smb_handle file;
|
---|
447 | uint64_t offset;
|
---|
448 | uint16_t wmode;
|
---|
449 | uint16_t remaining;
|
---|
450 | uint32_t count;
|
---|
451 | const uint8_t *data;
|
---|
452 | } in;
|
---|
453 | struct {
|
---|
454 | uint32_t nwritten;
|
---|
455 | uint16_t remaining;
|
---|
456 | } out;
|
---|
457 | } writex, generic;
|
---|
458 |
|
---|
459 | see the "in" and "out" sections? The "in" section is for parameters
|
---|
460 | that the SMB client sends on the wire as part of the request. The smbd
|
---|
461 | front end parse code parses the wire request and fills in all those
|
---|
462 | parameters. It then calls the NTVFS interface which looks like this::
|
---|
463 |
|
---|
464 | NTSTATUS (*write)(struct request_context *req, union smb_write *io);
|
---|
465 |
|
---|
466 | and the NTVFS backend does the write request. The backend then fills
|
---|
467 | in the "out" section of the writex structure and gives the union back
|
---|
468 | to the front end (either by returning, or if done in an async fashion
|
---|
469 | then by calling the async send function. See the async discussion
|
---|
470 | elsewhere in this document).
|
---|
471 |
|
---|
472 | The NTVFS backend knows which particular function is being requested
|
---|
473 | by looking at io->generic.level. Notice that this enum is also
|
---|
474 | repeated inside each of the sub-structures in the union, so the
|
---|
475 | backend could just as easily look at io->writex.level and would get
|
---|
476 | the same variable.
|
---|
477 |
|
---|
478 | Notice also that some levels (such as splwrite) don't have an "out"
|
---|
479 | section. This happens because there is no return value apart from a
|
---|
480 | status code from those SMB calls.
|
---|
481 |
|
---|
482 | So what about status codes? The status code is returned directly by
|
---|
483 | the backend NTVFS interface when the call is performed
|
---|
484 | synchronously. When performed asynchronously then the status code is
|
---|
485 | put into req->async.status before the req->async.send_fn() callback is
|
---|
486 | called.
|
---|
487 |
|
---|
488 | Currently the most complete NTVFS backend is the CIFS backend. I don't
|
---|
489 | expect this backend will be used much in production, but it does
|
---|
490 | provide the ideal test case for our NTVFS design. As it offers the
|
---|
491 | full capabilities that are possible with a CIFS server we can be sure
|
---|
492 | that we don't have any gaping holes in our APIs, and that the front
|
---|
493 | end code is flexible enough to handle any advances in the NT style
|
---|
494 | feature sets of Unix filesystems that make come along.
|
---|
495 |
|
---|
496 |
|
---|
497 | Process Models
|
---|
498 | --------------
|
---|
499 |
|
---|
500 | In Samba3 we supported just one process model. It just so happens that
|
---|
501 | the process model that Samba3 supported is the "right" one for most
|
---|
502 | users, but there are situations where this model wasn't ideal.
|
---|
503 |
|
---|
504 | In Samba4 you can choose the smbd process model on the smbd command
|
---|
505 | line.
|
---|
506 |
|
---|
507 |
|
---|
508 | DCERPC binding strings
|
---|
509 | ----------------------
|
---|
510 |
|
---|
511 | When connecting to a dcerpc service you need to specify a binding
|
---|
512 | string.
|
---|
513 |
|
---|
514 | The format is:
|
---|
515 |
|
---|
516 | TRANSPORT:host[flags]
|
---|
517 |
|
---|
518 | where TRANSPORT is either ncacn_np for SMB or ncacn_ip_tcp for RPC/TCP
|
---|
519 |
|
---|
520 | "host" is an IP or hostname or netbios name. If the binding string
|
---|
521 | identifies the server side of an endpoint, "host" may be an empty
|
---|
522 | string.
|
---|
523 |
|
---|
524 | "flags" can include a SMB pipe name if using the ncacn_np transport or
|
---|
525 | a TCP port number if using the ncacn_ip_tcp transport, otherwise they
|
---|
526 | will be auto-determined.
|
---|
527 |
|
---|
528 | other recognised flags are:
|
---|
529 |
|
---|
530 | sign : enable ntlmssp signing
|
---|
531 | seal : enable ntlmssp sealing
|
---|
532 | spnego : use SPNEGO instead of NTLMSSP authentication
|
---|
533 | krb5 : use KRB5 instead of NTLMSSP authentication
|
---|
534 | connect : enable rpc connect level auth (auth, but no sign or seal)
|
---|
535 | validate : enable the NDR validator
|
---|
536 | print : enable debugging of the packets
|
---|
537 | bigendian : use bigendian RPC
|
---|
538 | padcheck : check reply data for non-zero pad bytes
|
---|
539 |
|
---|
540 |
|
---|
541 | Here are some examples:
|
---|
542 |
|
---|
543 | ncacn_np:myserver
|
---|
544 | ncacn_np:myserver[samr]
|
---|
545 | ncacn_np:myserver[\pipe\samr]
|
---|
546 | ncacn_np:myserver[/pipe/samr]
|
---|
547 | ncacn_np:myserver[samr,sign,print]
|
---|
548 | ncacn_np:myserver[sign,spnego]
|
---|
549 | ncacn_np:myserver[\pipe\samr,sign,seal,bigendian]
|
---|
550 | ncacn_np:myserver[/pipe/samr,seal,validate]
|
---|
551 | ncacn_np:
|
---|
552 | ncacn_np:[/pipe/samr]
|
---|
553 | ncacn_ip_tcp:myserver
|
---|
554 | ncacn_ip_tcp:myserver[1024]
|
---|
555 | ncacn_ip_tcp:myserver[sign,seal]
|
---|
556 | ncacn_ip_tcp:myserver[spnego,seal]
|
---|
557 |
|
---|
558 |
|
---|
559 | IDEA: Maybe extend UNC names like this?
|
---|
560 |
|
---|
561 | smbclient //server/share
|
---|
562 | smbclient //server/share[sign,seal,spnego]
|
---|
563 |
|
---|
564 | DCERPC Handles
|
---|
565 | --------------
|
---|
566 | The various handles that are used in the RPC servers should be created and
|
---|
567 | fetch using the dcesrv_handle_* functions.
|
---|
568 |
|
---|
569 | Use dcesrv_handle_new(struct dcesrv_connection \*, uint8 handle_type) to obtain
|
---|
570 | a new handle of the specified type. Handle types are unique within each
|
---|
571 | pipe.
|
---|
572 |
|
---|
573 | The handle can later be fetched again using::
|
---|
574 |
|
---|
575 | struct dcesrv_handle *dcesrv_handle_fetch(struct dcesrv_connection *dce_conn, struct policy_handle *p, uint8 handle_type)
|
---|
576 |
|
---|
577 | and destroyed by::
|
---|
578 |
|
---|
579 | dcesrv_handle_destroy(struct dcesrv_handle *).
|
---|
580 |
|
---|
581 | User data should be stored in the 'data' member of the dcesrv_handle struct.
|
---|
582 |
|
---|
583 |
|
---|
584 | MSRPC
|
---|
585 | -----
|
---|
586 |
|
---|
587 |
|
---|
588 |
|
---|
589 | - ntvfs
|
---|
590 | - testing
|
---|
591 | - command line handling
|
---|
592 | - libcli structure
|
---|
593 | - posix reliance
|
---|
594 | - uid/gid handling
|
---|
595 | - process models
|
---|
596 | - static data
|
---|
597 | - msrpc
|
---|
598 |
|
---|
599 |
|
---|
600 | don't zero structures! avoid ZERO_STRUCT() and talloc_zero()
|
---|
601 |
|
---|
602 |
|
---|
603 | GMT vs TZ in printout of QFILEINFO timezones
|
---|
604 |
|
---|
605 | put in full UNC path in tconx
|
---|
606 |
|
---|
607 | test timezone handling by using a server in different zone from client
|
---|
608 |
|
---|
609 | do {} while (0) system
|
---|
610 |
|
---|
611 | NT_STATUS_IS_OK() is NOT the opposite of NT_STATUS_IS_ERR()
|
---|
612 |
|
---|
613 | need to implement secondary parts of trans2 and nttrans in server and
|
---|
614 | client
|
---|
615 |
|
---|
616 | document access_mask in openx reply
|
---|
617 |
|
---|
618 | check all capabilities and flag1, flag2 fields (eg. EAs)
|
---|
619 |
|
---|
620 | large files -> pass thru levels
|
---|
621 |
|
---|
622 | setpathinfo is very fussy about null termination of the file name
|
---|
623 |
|
---|
624 | the overwrite flag doesn't seem to work on setpathinfo RENAME_INFORMATION
|
---|
625 |
|
---|
626 | END_OF_FILE_INFORMATION and ALLOCATION_INFORMATION don't seem to work
|
---|
627 | via setpathinfo
|
---|
628 |
|
---|
629 | on w2k3 setpathinfo DISPOSITION_INFORMATION fails, but does have an
|
---|
630 | effect. It leaves the file with SHARING_VIOLATION.
|
---|
631 |
|
---|
632 | on w2k3 trans2 setpathinfo with any invalid low numbered level causes
|
---|
633 | the file to get into a state where DELETE_PENDING is reported, and the
|
---|
634 | file cannot be deleted until you reboot
|
---|
635 |
|
---|
636 | trans2 qpathinfo doesn't see the delete_pending flag correctly, but
|
---|
637 | qfileinfo does!
|
---|
638 |
|
---|
639 | get rid of strtok
|
---|
640 |
|
---|
641 | add programming documentation note about lp_set_cmdline()
|
---|
642 |
|
---|
643 | need to add a wct checking function in all client parsing code,
|
---|
644 | similar to REQ_CHECK_WCT()
|
---|
645 |
|
---|
646 | need to make sure that NTTIME is a round number of seconds when
|
---|
647 | converted from time_t
|
---|
648 |
|
---|
649 | not using a zero next offset in SMB_FILE_STREAM_INFORMATION for last
|
---|
650 | entry causes explorer exception under win2000
|
---|
651 |
|
---|
652 |
|
---|
653 | if the server sets the session key the same for a second SMB socket as
|
---|
654 | an initial socket then the client will not re-authenticate, it will go
|
---|
655 | straight to a tconx, skipping session setup and will use all the
|
---|
656 | existing parameters! This allows two sockets with the same keys!?
|
---|
657 |
|
---|
658 |
|
---|
659 | removed blocking lock code, we now queue the whole request the same as
|
---|
660 | we queue any other pending request. This allows for things like a
|
---|
661 | close() while a pending blocking lock is being processed to operate
|
---|
662 | sanely.
|
---|
663 |
|
---|
664 | disabled change notify code
|
---|
665 |
|
---|
666 | disabled oplock code
|
---|
667 |
|
---|
668 |
|
---|
669 |
|
---|
670 | MILESTONES
|
---|
671 | ==========
|
---|
672 |
|
---|
673 |
|
---|
674 | client library and test code
|
---|
675 | ----------------------------
|
---|
676 |
|
---|
677 | convert client library to new structure
|
---|
678 | get smbtorture working
|
---|
679 | get smbclient working
|
---|
680 | expand client library for all requests
|
---|
681 | write per-request test suite
|
---|
682 | gentest randomised test suite
|
---|
683 | separate client code as a library for non-Samba use
|
---|
684 |
|
---|
685 | server code
|
---|
686 | -----------
|
---|
687 | add remaining core SMB requests
|
---|
688 | add IPC layer
|
---|
689 | add nttrans layer
|
---|
690 | add rpc layer
|
---|
691 | fix auth models (share, server, rpc)
|
---|
692 | get net command working
|
---|
693 | connect CIFS backend to server level auth
|
---|
694 | get nmbd working
|
---|
695 | get winbindd working
|
---|
696 | reconnect printing code
|
---|
697 | restore removed smbd options
|
---|
698 | add smb.conf macro substitution code
|
---|
699 | add async backend notification
|
---|
700 | add generic timer event mechanism
|
---|
701 |
|
---|
702 | clustering code
|
---|
703 | ---------------
|
---|
704 |
|
---|
705 | write CIFS backend
|
---|
706 | new server models (break 1-1)
|
---|
707 | test clustered models
|
---|
708 | add fulcrum statistics gathering
|
---|
709 |
|
---|
710 | docs
|
---|
711 | ----
|
---|
712 |
|
---|
713 | conference paper
|
---|
714 | developer docs
|
---|
715 |
|
---|
716 | svn instructions
|
---|
717 |
|
---|
718 | Ideas
|
---|
719 | -----
|
---|
720 |
|
---|
721 | - store all config in config.ldb
|
---|
722 |
|
---|
723 | - load from smb.conf if modtime changes
|
---|
724 |
|
---|
725 | - dump full system config with ldbsearch
|
---|
726 |
|
---|
727 | - will need the ability to form a ldif difference file
|
---|
728 |
|
---|
729 | - advanced web admin via a web ldb editor
|
---|
730 |
|
---|
731 | - normal web admin via web forms -> ldif
|
---|
732 |
|
---|
733 | - config.ldb will replace smb.conf, secrets.tdb, shares.tdb etc
|
---|
734 |
|
---|
735 | - subsystems in smbd will load config parameters for a share
|
---|
736 | using ldbsearch at tconx time
|
---|
737 |
|
---|
738 | - need a loadparm equivalent module that provides parameter defaults
|
---|
739 |
|
---|
740 | - start smbd like this: "smbd -C tdb://etc/samba/config.ldb" or
|
---|
741 | "smbd -C ldapi://var/run/ldapi"
|
---|
742 |
|
---|
743 | - write a tool that generates a template ldap schema from an existing
|
---|
744 | ldb+tdb file
|
---|
745 |
|
---|
746 | - no need to HUP smbd to reload config
|
---|
747 |
|
---|
748 | - how to handle configuration comments? same problem as SWAT
|
---|
749 |
|
---|
750 |
|
---|
751 | BUGS:
|
---|
752 | add a test case for last_entry_offset in trans2 find interfaces
|
---|
753 | conn refused
|
---|
754 | connect -> errno
|
---|
755 | no 137 resolution not possible
|
---|
756 | should not fallback to anon when pass supplied
|
---|
757 | should check pass-thu cap bit, and skip lots of tests
|
---|
758 | possibly allow the test suite to say "allow oversized replies" for trans2 and other calls
|
---|
759 | handle servers that don't have the setattre call in torture
|
---|
760 | add max file coponent length test and max path len test
|
---|
761 | check for alloc failure in all core reply.c and trans2.c code where allocation size depends on client parameter
|
---|
762 |
|
---|
763 | case-insenstive idea:
|
---|
764 | all filenames on disk lowercase
|
---|
765 | real case in extended attribute
|
---|
766 | keep cache of what dirs are all lowercase
|
---|
767 | when searching for name, don't search if dir is definately all lowercase
|
---|
768 | when creating file, use dnotify to tell if someone else creates at
|
---|
769 | same time
|
---|
770 |
|
---|
771 | solve del *.* idea:
|
---|
772 | make mangle cache dynamic size
|
---|
773 | fill during a dir scan
|
---|
774 | setup a timer
|
---|
775 | destroy cache after 30 sec
|
---|
776 | destroy if a 2nd dir scan happens on same dir
|
---|
777 |
|
---|