[988] | 1 |
|
---|
| 2 |
|
---|
| 3 | THIS IS INCOMPLETE! I'M ONLY COMMITING IT IN ORDER TO SOLICIT COMMENTS
|
---|
| 4 | FROM A FEW PEOPLE. DON'T TAKE THIS AS THE FINAL VERSION YET.
|
---|
| 5 |
|
---|
| 6 |
|
---|
| 7 | Samba4 Programming Guide
|
---|
| 8 | ========================
|
---|
| 9 |
|
---|
| 10 | .. contents::
|
---|
| 11 |
|
---|
| 12 | The internals of Samba4 are quite different from previous versions of
|
---|
| 13 | Samba, so even if you are an experienced Samba developer please take
|
---|
| 14 | the time to read through this document.
|
---|
| 15 |
|
---|
| 16 | This document will explain both the broad structure of Samba4, and
|
---|
| 17 | some of the common coding elements such as memory management and
|
---|
| 18 | dealing with macros.
|
---|
| 19 |
|
---|
| 20 |
|
---|
| 21 | Coding Style
|
---|
| 22 | ------------
|
---|
| 23 |
|
---|
| 24 | In past versions of Samba we have basically let each programmer choose
|
---|
| 25 | their own programming style. Unfortunately the result has often been
|
---|
| 26 | that code that other members of the team find difficult to read. For
|
---|
| 27 | Samba version 4 I would like to standardise on a common coding style
|
---|
| 28 | to make the whole tree more readable. For those of you who are
|
---|
| 29 | horrified at the idea of having to learn a new style, I can assure you
|
---|
| 30 | that it isn't as painful as you might think. I was forced to adopt a
|
---|
| 31 | new style when I started working on the Linux kernel, and after some
|
---|
| 32 | initial pain found it quite easy.
|
---|
| 33 |
|
---|
| 34 | That said, I don't want to invent a new style, instead I would like to
|
---|
| 35 | adopt the style used by the Linux kernel. It is a widely used style
|
---|
| 36 | with plenty of support tools available. See Documentation/CodingStyle
|
---|
| 37 | in the Linux source tree. This is the style that I have used to write
|
---|
| 38 | all of the core infrastructure for Samba4 and I think that we should
|
---|
| 39 | continue with that style.
|
---|
| 40 |
|
---|
| 41 | I also think that we should most definately *not* adopt an automatic
|
---|
| 42 | reformatting system in cvs (or whatever other source code system we
|
---|
| 43 | end up using in the future). Such automatic formatters are, in my
|
---|
| 44 | experience, incredibly error prone and don't understand the necessary
|
---|
| 45 | exceptions. I don't mind if people use automated tools to reformat
|
---|
| 46 | their own code before they commit it, but please do not run such
|
---|
| 47 | automated tools on large slabs of existing code without being willing
|
---|
| 48 | to spend a *lot* of time hand checking the results.
|
---|
| 49 |
|
---|
| 50 | Finally, I think that for code that is parsing or formatting protocol
|
---|
| 51 | packets the code layout should strongly reflect the packet
|
---|
| 52 | format. That means ordring the code so that it parses in the same
|
---|
| 53 | order as the packet is stored on the wire (where possible) and using
|
---|
| 54 | white space to align packet offsets so that a reader can immediately
|
---|
| 55 | map any line of the code to the corresponding place in the packet.
|
---|
| 56 |
|
---|
| 57 |
|
---|
| 58 | Static and Global Data
|
---|
| 59 | ----------------------
|
---|
| 60 |
|
---|
| 61 | The basic rule is "avoid static and global data like the plague". What
|
---|
| 62 | do I mean by static data? The way to tell if you have static data in a
|
---|
| 63 | file is to use the "size" utility in Linux. For example if we run::
|
---|
| 64 |
|
---|
| 65 | size libcli/raw/*.o
|
---|
| 66 |
|
---|
| 67 | in Samba4 then you get the following::
|
---|
| 68 |
|
---|
| 69 | text data bss dec hex filename
|
---|
| 70 | 2015 0 0 2015 7df libcli/raw/clikrb5.o
|
---|
| 71 | 202 0 0 202 ca libcli/raw/clioplock.o
|
---|
| 72 | 35 0 0 35 23 libcli/raw/clirewrite.o
|
---|
| 73 | 3891 0 0 3891 f33 libcli/raw/clisession.o
|
---|
| 74 | 869 0 0 869 365 libcli/raw/clisocket.o
|
---|
| 75 | 4962 0 0 4962 1362 libcli/raw/clispnego.o
|
---|
| 76 | 1223 0 0 1223 4c7 libcli/raw/clitransport.o
|
---|
| 77 | 2294 0 0 2294 8f6 libcli/raw/clitree.o
|
---|
| 78 | 1081 0 0 1081 439 libcli/raw/raweas.o
|
---|
| 79 | 6765 0 0 6765 1a6d libcli/raw/rawfile.o
|
---|
| 80 | 6824 0 0 6824 1aa8 libcli/raw/rawfileinfo.o
|
---|
| 81 | 2944 0 0 2944 b80 libcli/raw/rawfsinfo.o
|
---|
| 82 | 541 0 0 541 21d libcli/raw/rawioctl.o
|
---|
| 83 | 1728 0 0 1728 6c0 libcli/raw/rawnegotiate.o
|
---|
| 84 | 723 0 0 723 2d3 libcli/raw/rawnotify.o
|
---|
| 85 | 3779 0 0 3779 ec3 libcli/raw/rawreadwrite.o
|
---|
| 86 | 6597 0 0 6597 19c5 libcli/raw/rawrequest.o
|
---|
| 87 | 5580 0 0 5580 15cc libcli/raw/rawsearch.o
|
---|
| 88 | 3034 0 0 3034 bda libcli/raw/rawsetfileinfo.o
|
---|
| 89 | 5187 0 0 5187 1443 libcli/raw/rawtrans.o
|
---|
| 90 | 2033 0 0 2033 7f1 libcli/raw/smb_signing.o
|
---|
| 91 |
|
---|
| 92 | notice that the "data" and "bss" columns are all zero? That is
|
---|
| 93 | good. If there are any non-zero values in data or bss then that
|
---|
| 94 | indicates static data and is bad (as a rule of thumb).
|
---|
| 95 |
|
---|
| 96 | Lets compare that result to the equivalent in Samba3::
|
---|
| 97 |
|
---|
| 98 | text data bss dec hex filename
|
---|
| 99 | 3978 0 0 3978 f8a libsmb/asn1.o
|
---|
| 100 | 18963 0 288 19251 4b33 libsmb/cliconnect.o
|
---|
| 101 | 2815 0 1024 3839 eff libsmb/clidgram.o
|
---|
| 102 | 4038 0 0 4038 fc6 libsmb/clientgen.o
|
---|
| 103 | 3337 664 256 4257 10a1 libsmb/clierror.o
|
---|
| 104 | 10043 0 0 10043 273b libsmb/clifile.o
|
---|
| 105 | 332 0 0 332 14c libsmb/clifsinfo.o
|
---|
| 106 | 166 0 0 166 a6 libsmb/clikrb5.o
|
---|
| 107 | 5212 0 0 5212 145c libsmb/clilist.o
|
---|
| 108 | 1367 0 0 1367 557 libsmb/climessage.o
|
---|
| 109 | 259 0 0 259 103 libsmb/clioplock.o
|
---|
| 110 | 1584 0 0 1584 630 libsmb/cliprint.o
|
---|
| 111 | 7565 0 256 7821 1e8d libsmb/cliquota.o
|
---|
| 112 | 7694 0 0 7694 1e0e libsmb/clirap.o
|
---|
| 113 | 27440 0 0 27440 6b30 libsmb/clirap2.o
|
---|
| 114 | 2905 0 0 2905 b59 libsmb/clireadwrite.o
|
---|
| 115 | 1698 0 0 1698 6a2 libsmb/clisecdesc.o
|
---|
| 116 | 5517 0 0 5517 158d libsmb/clispnego.o
|
---|
| 117 | 485 0 0 485 1e5 libsmb/clistr.o
|
---|
| 118 | 8449 0 0 8449 2101 libsmb/clitrans.o
|
---|
| 119 | 2053 0 4 2057 809 libsmb/conncache.o
|
---|
| 120 | 3041 0 256 3297 ce1 libsmb/credentials.o
|
---|
| 121 | 1261 0 1024 2285 8ed libsmb/doserr.o
|
---|
| 122 | 14560 0 0 14560 38e0 libsmb/errormap.o
|
---|
| 123 | 3645 0 0 3645 e3d libsmb/namecache.o
|
---|
| 124 | 16815 0 8 16823 41b7 libsmb/namequery.o
|
---|
| 125 | 1626 0 0 1626 65a libsmb/namequery_dc.o
|
---|
| 126 | 14301 0 1076 15377 3c11 libsmb/nmblib.o
|
---|
| 127 | 24516 0 2048 26564 67c4 libsmb/nterr.o
|
---|
| 128 | 8661 0 8 8669 21dd libsmb/ntlmssp.o
|
---|
| 129 | 3188 0 0 3188 c74 libsmb/ntlmssp_parse.o
|
---|
| 130 | 4945 0 0 4945 1351 libsmb/ntlmssp_sign.o
|
---|
| 131 | 1303 0 0 1303 517 libsmb/passchange.o
|
---|
| 132 | 1221 0 0 1221 4c5 libsmb/pwd_cache.o
|
---|
| 133 | 2475 0 4 2479 9af libsmb/samlogon_cache.o
|
---|
| 134 | 10768 32 0 10800 2a30 libsmb/smb_signing.o
|
---|
| 135 | 4524 0 16 4540 11bc libsmb/smbdes.o
|
---|
| 136 | 5708 0 0 5708 164c libsmb/smbencrypt.o
|
---|
| 137 | 7049 0 3072 10121 2789 libsmb/smberr.o
|
---|
| 138 | 2995 0 0 2995 bb3 libsmb/spnego.o
|
---|
| 139 | 3186 0 0 3186 c72 libsmb/trustdom_cache.o
|
---|
| 140 | 1742 0 0 1742 6ce libsmb/trusts_util.o
|
---|
| 141 | 918 0 28 946 3b2 libsmb/unexpected.o
|
---|
| 142 |
|
---|
| 143 | notice all of the non-zero data and bss elements? Every bit of that
|
---|
| 144 | data is a bug waiting to happen.
|
---|
| 145 |
|
---|
| 146 | Static data is evil as it has the following consequences:
|
---|
| 147 | - it makes code much less likely to be thread-safe
|
---|
| 148 | - it makes code much less likely to be recursion-safe
|
---|
| 149 | - it leads to subtle side effects when the same code is called from multiple places
|
---|
| 150 | - doesn't play well with shared libraries or plugins
|
---|
| 151 |
|
---|
| 152 | Static data is particularly evil in library code (such as our internal
|
---|
| 153 | smb and rpc libraries). If you can get rid of all static data in
|
---|
| 154 | libraries then you can make some fairly strong guarantees about the
|
---|
| 155 | behaviour of functions in that library, which really helps.
|
---|
| 156 |
|
---|
| 157 | Of course, it is possible to write code that uses static data and is
|
---|
| 158 | safe, it's just much harder to do that than just avoid static data in
|
---|
| 159 | the first place. We have been tripped up countless times by subtle
|
---|
| 160 | bugs in Samba due to the use of static data, so I think it is time to
|
---|
| 161 | start avoiding it in new code. Much of the core infrastructure of
|
---|
| 162 | Samba4 was specifically written to avoid static data, so I'm going to
|
---|
| 163 | be really annoyed if everyone starts adding lots of static data back
|
---|
| 164 | in.
|
---|
| 165 |
|
---|
| 166 | So, how do we avoid static data? The basic method is to use context
|
---|
| 167 | pointers. When reading the Samba4 code you will notice that just about
|
---|
| 168 | every function takes a pointer to a context structure as its first
|
---|
| 169 | argument. Any data that the function needs that isn't an explicit
|
---|
| 170 | argument to the function can be found by traversing that context.
|
---|
| 171 |
|
---|
| 172 | Note that this includes all of the little caches that we have lying
|
---|
| 173 | all over the code in Samba3. I'm referring to the ones that generally
|
---|
| 174 | have a "static int initialised" and then some static string or integer
|
---|
| 175 | that remembers the last return value of the function. Get rid of them!
|
---|
| 176 | If you are *REALLY* absolutely completely certain that your personal
|
---|
| 177 | favourite mini-cache is needed then you should do it properly by
|
---|
| 178 | putting it into the appropriate context rather than doing it the lazy
|
---|
| 179 | way by putting it inside the target function. I would suggest however
|
---|
| 180 | that the vast majority of those little caches are useless - don't
|
---|
| 181 | stick it in unless you have really firm benchmarking results that show
|
---|
| 182 | that it is needed and helps by a significant amount.
|
---|
| 183 |
|
---|
| 184 | Note that Samba4 is not yet completely clean of static data like
|
---|
| 185 | this. I've gotten the smbd/ directory down to 24 bytes of static data,
|
---|
| 186 | and libcli/raw/ down to zero. I've also gotten the ntvfs layer and all
|
---|
| 187 | backends down to just 8 bytes in ntvfs_base.c. The rest still needs
|
---|
| 188 | some more work.
|
---|
| 189 |
|
---|
| 190 | Also note that truly constant data is OK, and will not in fact show up
|
---|
| 191 | in the data and bss columns in "size" anyway (it will be included in
|
---|
| 192 | "text"). So you can have constant tables of protocol data.
|
---|
| 193 |
|
---|
| 194 |
|
---|
| 195 | How to use talloc
|
---|
| 196 | -----------------
|
---|
| 197 |
|
---|
| 198 | Please see the separate document, lib/talloc/talloc_guide.txt
|
---|
| 199 | You _must_ read this if you want to program in Samba4.
|
---|
| 200 |
|
---|
| 201 |
|
---|
| 202 | Interface Structures
|
---|
| 203 | --------------------
|
---|
| 204 |
|
---|
| 205 | One of the biggest changes in Samba4 is the universal use of interface
|
---|
| 206 | structures. Go take a look through libcli/raw/interfaces.h now to get
|
---|
| 207 | an idea of what I am talking about.
|
---|
| 208 |
|
---|
| 209 | In Samba3 many of the core wire structures in the SMB protocol were
|
---|
| 210 | never explicitly defined in Samba. Instead, our parse and generation
|
---|
| 211 | functions just worked directly with wire buffers. The biggest problem
|
---|
| 212 | with this is that is tied our parse code with our "business logic"
|
---|
| 213 | much too closely, which meant the code got extremely confusing to
|
---|
| 214 | read.
|
---|
| 215 |
|
---|
| 216 | In Samba4 we have explicitly defined interface structures for
|
---|
| 217 | everything in the protocol. When we receive a buffer we always parse
|
---|
| 218 | it completely into one of these structures, then we pass a pointer to
|
---|
| 219 | that structure to a backend handler. What we must *not* do is make any
|
---|
| 220 | decisions about the data inside the parse functions. That is critical
|
---|
| 221 | as different backends will need different portions of the data. This
|
---|
| 222 | leads to a golden rule for Samba4:
|
---|
| 223 |
|
---|
| 224 | "don't design interfaces that lose information"
|
---|
| 225 |
|
---|
| 226 | In Samba3 our backends often received "condensed" versions of the
|
---|
| 227 | information sent from clients, but this inevitably meant that some
|
---|
| 228 | backends could not get at the data they needed to do what they wanted,
|
---|
| 229 | so from now on we should expose the backends to all of the available
|
---|
| 230 | information and let them choose which bits they want.
|
---|
| 231 |
|
---|
| 232 | Ok, so now some of you will be thinking "this sounds just like our
|
---|
| 233 | msrpc code from Samba3", and while to some extent this is true there
|
---|
| 234 | are extremely important differences in the approach that are worth
|
---|
| 235 | pointing out.
|
---|
| 236 |
|
---|
| 237 | In the Samba3 msrpc code we used explicit parse structures for all
|
---|
| 238 | msrpc functions. The problem is that we didn't just put all of the
|
---|
| 239 | real variables in these structures, we also put in all the artifacts
|
---|
| 240 | as well. A good example is the security descriptor strucrure that
|
---|
| 241 | looks like this in Samba3::
|
---|
| 242 |
|
---|
| 243 | typedef struct security_descriptor_info
|
---|
| 244 | {
|
---|
| 245 | uint16 revision;
|
---|
| 246 | uint16 type;
|
---|
| 247 |
|
---|
| 248 | uint32 off_owner_sid;
|
---|
| 249 | uint32 off_grp_sid;
|
---|
| 250 | uint32 off_sacl;
|
---|
| 251 | uint32 off_dacl;
|
---|
| 252 |
|
---|
| 253 | SEC_ACL *dacl;
|
---|
| 254 | SEC_ACL *sacl;
|
---|
| 255 | DOM_SID *owner_sid;
|
---|
| 256 | DOM_SID *grp_sid;
|
---|
| 257 | } SEC_DESC;
|
---|
| 258 |
|
---|
| 259 | The problem with this structure is all the off_* variables. Those are
|
---|
| 260 | not part of the interface, and do not appear in any real descriptions
|
---|
| 261 | of Microsoft security descriptors. They are parsing artifacts
|
---|
| 262 | generated by the IDL compiler that Microsoft use. That doesn't mean
|
---|
| 263 | they aren't needed on the wire - indeed they are as they tell the
|
---|
| 264 | parser where to find the following four variables, but they should
|
---|
| 265 | *NOT* be in the interface structure.
|
---|
| 266 |
|
---|
| 267 | In Samba3 there were unwritten rules about which variables in a
|
---|
| 268 | structure a high level caller has to fill in and which ones are filled
|
---|
| 269 | in by the marshalling code. In Samba4 those rules are gone, because
|
---|
| 270 | the redundant artifact variables are gone. The high level caller just
|
---|
| 271 | sets up the real variables and the marshalling code worries about
|
---|
| 272 | generating the right offsets.
|
---|
| 273 |
|
---|
| 274 | The same rule applies to strings. In many places in the SMB and MSRPC
|
---|
| 275 | protocols complex strings are used on the wire, with complex rules
|
---|
| 276 | about padding, format, alighment, termination etc. None of that
|
---|
| 277 | information is useful to a high level calling routine or to a backend - its
|
---|
| 278 | all just so much wire fluff. So, in Samba4 these strings are
|
---|
| 279 | just "char \*" and are always in our internal multi-byte format (which
|
---|
| 280 | is usually UTF8). It is up to the parse functions to worry about
|
---|
| 281 | translating the format and getting the padding right.
|
---|
| 282 |
|
---|
| 283 | The one exception to this is the use of the WIRE_STRING type, but that
|
---|
| 284 | has a very good justification in terms of regression testing. Go and
|
---|
| 285 | read the comment in smb_interfaces.h about that now.
|
---|
| 286 |
|
---|
| 287 | So, here is another rule to code by. When writing an interface
|
---|
| 288 | structure think carefully about what variables in the structure can be
|
---|
| 289 | left out as they are redundant. If some length is effectively defined
|
---|
| 290 | twice on the wire then only put it once in the packet. If a length can
|
---|
| 291 | be inferred from a null termination then do that and leave the length
|
---|
| 292 | out of the structure completely. Don't put redundant stuff in
|
---|
| 293 | structures!
|
---|
| 294 |
|
---|
| 295 |
|
---|
| 296 | Async Design
|
---|
| 297 | ------------
|
---|
| 298 |
|
---|
| 299 | Samba4 has an asynchronous design. That affects *lots* of the code,
|
---|
| 300 | and the implications of the asynchronous design needs to be considered
|
---|
| 301 | just about everywhere.
|
---|
| 302 |
|
---|
| 303 | The first aspect of the async design to look at is the SMB client
|
---|
| 304 | library. Lets take a look at the following three functions in
|
---|
| 305 | libcli/raw/rawfile.c::
|
---|
| 306 |
|
---|
| 307 | struct cli_request *smb_raw_seek_send(struct cli_tree *tree, struct smb_seek *parms);
|
---|
| 308 | NTSTATUS smb_raw_seek_recv(struct cli_request *req, struct smb_seek *parms);
|
---|
| 309 | NTSTATUS smb_raw_seek(struct cli_tree *tree, struct smb_seek *parms);
|
---|
| 310 |
|
---|
| 311 | Go and read them now then come back.
|
---|
| 312 |
|
---|
| 313 | Ok, first notice there there are 3 separate functions, whereas the
|
---|
| 314 | equivalent code in Samba3 had just one. Also note that the 3rd
|
---|
| 315 | function is extremely simple - its just a wrapper around calling the
|
---|
| 316 | first two in order.
|
---|
| 317 |
|
---|
| 318 | The three separate functions are needed because we need to be able to
|
---|
| 319 | generate SMB calls asynchronously. The first call, which for smb calls
|
---|
| 320 | is always called smb_raw_XXXX_send(), constructs and sends a SMB
|
---|
| 321 | request and returns a "struct cli_request" which acts as a handle for
|
---|
| 322 | the request. The caller is then free to do lots of other calls if it
|
---|
| 323 | wants to, then when it is ready it can call the smb_raw_XXX_recv()
|
---|
| 324 | function to receive the reply.
|
---|
| 325 |
|
---|
| 326 | If all you want is a synchronous call then call the 3rd interface, the
|
---|
| 327 | one called smb_raw_XXXX(). That just calls the first two in order, and
|
---|
| 328 | blocks waiting for the reply.
|
---|
| 329 |
|
---|
| 330 | But what if you want to be called when the reply comes in? Yes, thats
|
---|
| 331 | possible. You can do things like this::
|
---|
| 332 |
|
---|
| 333 | struct cli_request *req;
|
---|
| 334 |
|
---|
| 335 | req = smb_raw_XXX_send(tree, params);
|
---|
| 336 |
|
---|
| 337 | req->async.fn = my_callback;
|
---|
| 338 | req->async.private = my_private_data;
|
---|
| 339 |
|
---|
| 340 | then in your callback function you can call the smb_raw_XXXX_recv()
|
---|
| 341 | function to receive the reply. Your callback will receive the "req"
|
---|
| 342 | pointer, which you can use to retrieve your private data from
|
---|
| 343 | req->async.private.
|
---|
| 344 |
|
---|
| 345 | Then all you need to do is ensure that the main loop in the client
|
---|
| 346 | library gets called. You can either do that by polling the connection
|
---|
| 347 | using cli_transport_pending() and cli_request_receive_next() or you
|
---|
| 348 | can use transport->idle.func to setup an idle function handler to call
|
---|
| 349 | back to your main code. Either way, you can build a fully async
|
---|
| 350 | application.
|
---|
| 351 |
|
---|
| 352 | In order to support all of this we have to make sure that when we
|
---|
| 353 | write a piece of library code (SMB, MSRPC etc) that we build the
|
---|
| 354 | separate _send() and _recv() functions. It really is worth the effort.
|
---|
| 355 |
|
---|
| 356 | Now about async in smbd, a much more complex topic.
|
---|
| 357 |
|
---|
| 358 | The SMB protocol is inherently async. Some functions (such as change
|
---|
| 359 | notify) often don't return for hours, while hundreds of other
|
---|
| 360 | functions pass through the socket. Take a look at the RAW-MUX test in
|
---|
| 361 | the Samba4 smbtorture to see some really extreme examples of the sort
|
---|
| 362 | of async operations that Windows supports. I particularly like the
|
---|
| 363 | open/open/close sequence where the 2nd open (which conflicts with the
|
---|
| 364 | first) succeeds because the subsequent close is answered out of order.
|
---|
| 365 |
|
---|
| 366 | In Samba3 we handled this stuff very badly. We had awful "pending
|
---|
| 367 | request" queues that allocated full 128k packet buffers, and even with
|
---|
| 368 | all that crap we got the semantics wrong. In Samba4 I intend to make
|
---|
| 369 | sure we get this stuff right.
|
---|
| 370 |
|
---|
| 371 | So, how do we do this? We now have an async interface between smbd and
|
---|
| 372 | the NTVFS backends. Whenever smbd calls into a backend the backend has
|
---|
| 373 | an option of answer the request in a synchronous fashion if it wants
|
---|
| 374 | to just like in Samba3, but it also has the option of answering the
|
---|
| 375 | request asynchronously. The only backend that currently does this is
|
---|
| 376 | the CIFS backend, but I hope the other backends will soon do this to.
|
---|
| 377 |
|
---|
| 378 | To make this work you need to do things like this in the backend::
|
---|
| 379 |
|
---|
| 380 | req->control_flags |= REQ_CONTROL_ASYNC;
|
---|
| 381 |
|
---|
| 382 | that tells smbd that the backend has elected to reply later rather
|
---|
| 383 | than replying immediately. The backend must *only* do this if
|
---|
| 384 | req->async.send_fn is not NULL. If send_fn is NULL then it means that
|
---|
| 385 | the smbd front end cannot handle this function being replied to in an
|
---|
| 386 | async fashion.
|
---|
| 387 |
|
---|
| 388 | If the backend does this then it is up to the backend to call
|
---|
| 389 | req->async.send_fn() when it is ready to reply. It the meantime smbd
|
---|
| 390 | puts the call on hold and goes back to answering other requests on the
|
---|
| 391 | socket.
|
---|
| 392 |
|
---|
| 393 | Inside smbd you will find that there is code to support this. The most
|
---|
| 394 | obvious change is that smbd splits each SMB reply function into two
|
---|
| 395 | parts - just like the client library has a _send() and _recv()
|
---|
| 396 | function, so smbd has a _send() function and the parse function for
|
---|
| 397 | each SMB.
|
---|
| 398 |
|
---|
| 399 | As an example go and have a look at reply_getatr_send() and
|
---|
| 400 | reply_getatr() in smb_server/smb/reply.c. Read them? Good.
|
---|
| 401 |
|
---|
| 402 | Notice that reply_getatr() sets up the req->async structure to contain
|
---|
| 403 | the send function. Thats how the backend gets to do an async reply, it
|
---|
| 404 | calls this function when it is ready. Also notice that reply_getatr()
|
---|
| 405 | only does the parsing of the request, and does not do the reply
|
---|
| 406 | generation. That is done by the _send() function.
|
---|
| 407 |
|
---|
| 408 |
|
---|
| 409 | NTVFS
|
---|
| 410 | -----
|
---|
| 411 |
|
---|
| 412 | One of the most noticeable changes in Samba4 is the introduction of
|
---|
| 413 | the NTVFS layer. This provided the initial motivation for the design
|
---|
| 414 | of Samba4 and in many ways lies at the heart of the design.
|
---|
| 415 |
|
---|
| 416 | In Samba3 the main file serving process (smbd) combined the handling
|
---|
| 417 | of the SMB protocol with the mapping to POSIX semantics in the same
|
---|
| 418 | code. If you look in smbd/reply.c in Samba3 you see numerous places
|
---|
| 419 | where POSIX assumptions are mixed tightly with SMB parsing code. We
|
---|
| 420 | did have a VFS layer in Samba3, but it was a POSIX-like VFS layer, so
|
---|
| 421 | no matter how you wrote a plugin you could not bypass the POSIX
|
---|
| 422 | mapping decisions that had already been made before the VFS layer was
|
---|
| 423 | called.
|
---|
| 424 |
|
---|
| 425 | In Samba4 things are quite different. All SMB parsing is performed in
|
---|
| 426 | the smbd front end, then fully parsed requests are passed to the NTVFS
|
---|
| 427 | backend. That backend makes any semantic mapping decisions and fills
|
---|
| 428 | in the 'out' portion of the request. The front end is then responsible
|
---|
| 429 | for putting those results into wire format and sending them to the
|
---|
| 430 | client.
|
---|
| 431 |
|
---|
| 432 | Lets have a look at one of those request structures. Go and read the
|
---|
| 433 | definition of "union smb_write" and "enum write_level" in
|
---|
| 434 | libcli/raw/interfaces.h. (no, don't just skip reading it, really go
|
---|
| 435 | and read it. Yes, that means you!).
|
---|
| 436 |
|
---|
| 437 | Notice the union? That's how Samba4 allows a single NTVFS backend
|
---|
| 438 | interface to handle the several different ways of doing a write
|
---|
| 439 | operation in the SMB protocol. Now lets look at one section of that
|
---|
| 440 | union::
|
---|
| 441 |
|
---|
| 442 | /* SMBwriteX interface */
|
---|
| 443 | struct {
|
---|
| 444 | enum smb_write_level level;
|
---|
| 445 | struct {
|
---|
| 446 | union smb_handle file;
|
---|
| 447 | uint64_t offset;
|
---|
| 448 | uint16_t wmode;
|
---|
| 449 | uint16_t remaining;
|
---|
| 450 | uint32_t count;
|
---|
| 451 | const uint8_t *data;
|
---|
| 452 | } in;
|
---|
| 453 | struct {
|
---|
| 454 | uint32_t nwritten;
|
---|
| 455 | uint16_t remaining;
|
---|
| 456 | } out;
|
---|
| 457 | } writex, generic;
|
---|
| 458 |
|
---|
| 459 | see the "in" and "out" sections? The "in" section is for parameters
|
---|
| 460 | that the SMB client sends on the wire as part of the request. The smbd
|
---|
| 461 | front end parse code parses the wire request and fills in all those
|
---|
| 462 | parameters. It then calls the NTVFS interface which looks like this::
|
---|
| 463 |
|
---|
| 464 | NTSTATUS (*write)(struct request_context *req, union smb_write *io);
|
---|
| 465 |
|
---|
| 466 | and the NTVFS backend does the write request. The backend then fills
|
---|
| 467 | in the "out" section of the writex structure and gives the union back
|
---|
| 468 | to the front end (either by returning, or if done in an async fashion
|
---|
| 469 | then by calling the async send function. See the async discussion
|
---|
| 470 | elsewhere in this document).
|
---|
| 471 |
|
---|
| 472 | The NTVFS backend knows which particular function is being requested
|
---|
| 473 | by looking at io->generic.level. Notice that this enum is also
|
---|
| 474 | repeated inside each of the sub-structures in the union, so the
|
---|
| 475 | backend could just as easily look at io->writex.level and would get
|
---|
| 476 | the same variable.
|
---|
| 477 |
|
---|
| 478 | Notice also that some levels (such as splwrite) don't have an "out"
|
---|
| 479 | section. This happens because there is no return value apart from a
|
---|
| 480 | status code from those SMB calls.
|
---|
| 481 |
|
---|
| 482 | So what about status codes? The status code is returned directly by
|
---|
| 483 | the backend NTVFS interface when the call is performed
|
---|
| 484 | synchronously. When performed asynchronously then the status code is
|
---|
| 485 | put into req->async.status before the req->async.send_fn() callback is
|
---|
| 486 | called.
|
---|
| 487 |
|
---|
| 488 | Currently the most complete NTVFS backend is the CIFS backend. I don't
|
---|
| 489 | expect this backend will be used much in production, but it does
|
---|
| 490 | provide the ideal test case for our NTVFS design. As it offers the
|
---|
| 491 | full capabilities that are possible with a CIFS server we can be sure
|
---|
| 492 | that we don't have any gaping holes in our APIs, and that the front
|
---|
| 493 | end code is flexible enough to handle any advances in the NT style
|
---|
| 494 | feature sets of Unix filesystems that make come along.
|
---|
| 495 |
|
---|
| 496 |
|
---|
| 497 | Process Models
|
---|
| 498 | --------------
|
---|
| 499 |
|
---|
| 500 | In Samba3 we supported just one process model. It just so happens that
|
---|
| 501 | the process model that Samba3 supported is the "right" one for most
|
---|
| 502 | users, but there are situations where this model wasn't ideal.
|
---|
| 503 |
|
---|
| 504 | In Samba4 you can choose the smbd process model on the smbd command
|
---|
| 505 | line.
|
---|
| 506 |
|
---|
| 507 |
|
---|
| 508 | DCERPC binding strings
|
---|
| 509 | ----------------------
|
---|
| 510 |
|
---|
| 511 | When connecting to a dcerpc service you need to specify a binding
|
---|
| 512 | string.
|
---|
| 513 |
|
---|
| 514 | The format is:
|
---|
| 515 |
|
---|
| 516 | TRANSPORT:host[flags]
|
---|
| 517 |
|
---|
| 518 | where TRANSPORT is either ncacn_np for SMB or ncacn_ip_tcp for RPC/TCP
|
---|
| 519 |
|
---|
| 520 | "host" is an IP or hostname or netbios name. If the binding string
|
---|
| 521 | identifies the server side of an endpoint, "host" may be an empty
|
---|
| 522 | string.
|
---|
| 523 |
|
---|
| 524 | "flags" can include a SMB pipe name if using the ncacn_np transport or
|
---|
| 525 | a TCP port number if using the ncacn_ip_tcp transport, otherwise they
|
---|
| 526 | will be auto-determined.
|
---|
| 527 |
|
---|
| 528 | other recognised flags are:
|
---|
| 529 |
|
---|
| 530 | sign : enable ntlmssp signing
|
---|
| 531 | seal : enable ntlmssp sealing
|
---|
| 532 | spnego : use SPNEGO instead of NTLMSSP authentication
|
---|
| 533 | krb5 : use KRB5 instead of NTLMSSP authentication
|
---|
| 534 | connect : enable rpc connect level auth (auth, but no sign or seal)
|
---|
| 535 | validate : enable the NDR validator
|
---|
| 536 | print : enable debugging of the packets
|
---|
| 537 | bigendian : use bigendian RPC
|
---|
| 538 | padcheck : check reply data for non-zero pad bytes
|
---|
| 539 |
|
---|
| 540 |
|
---|
| 541 | Here are some examples:
|
---|
| 542 |
|
---|
| 543 | ncacn_np:myserver
|
---|
| 544 | ncacn_np:myserver[samr]
|
---|
| 545 | ncacn_np:myserver[\pipe\samr]
|
---|
| 546 | ncacn_np:myserver[/pipe/samr]
|
---|
| 547 | ncacn_np:myserver[samr,sign,print]
|
---|
| 548 | ncacn_np:myserver[sign,spnego]
|
---|
| 549 | ncacn_np:myserver[\pipe\samr,sign,seal,bigendian]
|
---|
| 550 | ncacn_np:myserver[/pipe/samr,seal,validate]
|
---|
| 551 | ncacn_np:
|
---|
| 552 | ncacn_np:[/pipe/samr]
|
---|
| 553 | ncacn_ip_tcp:myserver
|
---|
| 554 | ncacn_ip_tcp:myserver[1024]
|
---|
| 555 | ncacn_ip_tcp:myserver[sign,seal]
|
---|
| 556 | ncacn_ip_tcp:myserver[spnego,seal]
|
---|
| 557 |
|
---|
| 558 |
|
---|
| 559 | IDEA: Maybe extend UNC names like this?
|
---|
| 560 |
|
---|
| 561 | smbclient //server/share
|
---|
| 562 | smbclient //server/share[sign,seal,spnego]
|
---|
| 563 |
|
---|
| 564 | DCERPC Handles
|
---|
| 565 | --------------
|
---|
| 566 | The various handles that are used in the RPC servers should be created and
|
---|
| 567 | fetch using the dcesrv_handle_* functions.
|
---|
| 568 |
|
---|
| 569 | Use dcesrv_handle_new(struct dcesrv_connection \*, uint8 handle_type) to obtain
|
---|
| 570 | a new handle of the specified type. Handle types are unique within each
|
---|
| 571 | pipe.
|
---|
| 572 |
|
---|
| 573 | The handle can later be fetched again using::
|
---|
| 574 |
|
---|
| 575 | struct dcesrv_handle *dcesrv_handle_fetch(struct dcesrv_connection *dce_conn, struct policy_handle *p, uint8 handle_type)
|
---|
| 576 |
|
---|
| 577 | and destroyed by::
|
---|
| 578 |
|
---|
| 579 | dcesrv_handle_destroy(struct dcesrv_handle *).
|
---|
| 580 |
|
---|
| 581 | User data should be stored in the 'data' member of the dcesrv_handle struct.
|
---|
| 582 |
|
---|
| 583 |
|
---|
| 584 | MSRPC
|
---|
| 585 | -----
|
---|
| 586 |
|
---|
| 587 |
|
---|
| 588 |
|
---|
| 589 | - ntvfs
|
---|
| 590 | - testing
|
---|
| 591 | - command line handling
|
---|
| 592 | - libcli structure
|
---|
| 593 | - posix reliance
|
---|
| 594 | - uid/gid handling
|
---|
| 595 | - process models
|
---|
| 596 | - static data
|
---|
| 597 | - msrpc
|
---|
| 598 |
|
---|
| 599 |
|
---|
| 600 | don't zero structures! avoid ZERO_STRUCT() and talloc_zero()
|
---|
| 601 |
|
---|
| 602 |
|
---|
| 603 | GMT vs TZ in printout of QFILEINFO timezones
|
---|
| 604 |
|
---|
| 605 | put in full UNC path in tconx
|
---|
| 606 |
|
---|
| 607 | test timezone handling by using a server in different zone from client
|
---|
| 608 |
|
---|
| 609 | do {} while (0) system
|
---|
| 610 |
|
---|
| 611 | NT_STATUS_IS_OK() is NOT the opposite of NT_STATUS_IS_ERR()
|
---|
| 612 |
|
---|
| 613 | need to implement secondary parts of trans2 and nttrans in server and
|
---|
| 614 | client
|
---|
| 615 |
|
---|
| 616 | document access_mask in openx reply
|
---|
| 617 |
|
---|
| 618 | check all capabilities and flag1, flag2 fields (eg. EAs)
|
---|
| 619 |
|
---|
| 620 | large files -> pass thru levels
|
---|
| 621 |
|
---|
| 622 | setpathinfo is very fussy about null termination of the file name
|
---|
| 623 |
|
---|
| 624 | the overwrite flag doesn't seem to work on setpathinfo RENAME_INFORMATION
|
---|
| 625 |
|
---|
| 626 | END_OF_FILE_INFORMATION and ALLOCATION_INFORMATION don't seem to work
|
---|
| 627 | via setpathinfo
|
---|
| 628 |
|
---|
| 629 | on w2k3 setpathinfo DISPOSITION_INFORMATION fails, but does have an
|
---|
| 630 | effect. It leaves the file with SHARING_VIOLATION.
|
---|
| 631 |
|
---|
| 632 | on w2k3 trans2 setpathinfo with any invalid low numbered level causes
|
---|
| 633 | the file to get into a state where DELETE_PENDING is reported, and the
|
---|
| 634 | file cannot be deleted until you reboot
|
---|
| 635 |
|
---|
| 636 | trans2 qpathinfo doesn't see the delete_pending flag correctly, but
|
---|
| 637 | qfileinfo does!
|
---|
| 638 |
|
---|
| 639 | get rid of strtok
|
---|
| 640 |
|
---|
| 641 | add programming documentation note about lp_set_cmdline()
|
---|
| 642 |
|
---|
| 643 | need to add a wct checking function in all client parsing code,
|
---|
| 644 | similar to REQ_CHECK_WCT()
|
---|
| 645 |
|
---|
| 646 | need to make sure that NTTIME is a round number of seconds when
|
---|
| 647 | converted from time_t
|
---|
| 648 |
|
---|
| 649 | not using a zero next offset in SMB_FILE_STREAM_INFORMATION for last
|
---|
| 650 | entry causes explorer exception under win2000
|
---|
| 651 |
|
---|
| 652 |
|
---|
| 653 | if the server sets the session key the same for a second SMB socket as
|
---|
| 654 | an initial socket then the client will not re-authenticate, it will go
|
---|
| 655 | straight to a tconx, skipping session setup and will use all the
|
---|
| 656 | existing parameters! This allows two sockets with the same keys!?
|
---|
| 657 |
|
---|
| 658 |
|
---|
| 659 | removed blocking lock code, we now queue the whole request the same as
|
---|
| 660 | we queue any other pending request. This allows for things like a
|
---|
| 661 | close() while a pending blocking lock is being processed to operate
|
---|
| 662 | sanely.
|
---|
| 663 |
|
---|
| 664 | disabled change notify code
|
---|
| 665 |
|
---|
| 666 | disabled oplock code
|
---|
| 667 |
|
---|
| 668 |
|
---|
| 669 |
|
---|
| 670 | MILESTONES
|
---|
| 671 | ==========
|
---|
| 672 |
|
---|
| 673 |
|
---|
| 674 | client library and test code
|
---|
| 675 | ----------------------------
|
---|
| 676 |
|
---|
| 677 | convert client library to new structure
|
---|
| 678 | get smbtorture working
|
---|
| 679 | get smbclient working
|
---|
| 680 | expand client library for all requests
|
---|
| 681 | write per-request test suite
|
---|
| 682 | gentest randomised test suite
|
---|
| 683 | separate client code as a library for non-Samba use
|
---|
| 684 |
|
---|
| 685 | server code
|
---|
| 686 | -----------
|
---|
| 687 | add remaining core SMB requests
|
---|
| 688 | add IPC layer
|
---|
| 689 | add nttrans layer
|
---|
| 690 | add rpc layer
|
---|
| 691 | fix auth models (share, server, rpc)
|
---|
| 692 | get net command working
|
---|
| 693 | connect CIFS backend to server level auth
|
---|
| 694 | get nmbd working
|
---|
| 695 | get winbindd working
|
---|
| 696 | reconnect printing code
|
---|
| 697 | restore removed smbd options
|
---|
| 698 | add smb.conf macro substitution code
|
---|
| 699 | add async backend notification
|
---|
| 700 | add generic timer event mechanism
|
---|
| 701 |
|
---|
| 702 | clustering code
|
---|
| 703 | ---------------
|
---|
| 704 |
|
---|
| 705 | write CIFS backend
|
---|
| 706 | new server models (break 1-1)
|
---|
| 707 | test clustered models
|
---|
| 708 | add fulcrum statistics gathering
|
---|
| 709 |
|
---|
| 710 | docs
|
---|
| 711 | ----
|
---|
| 712 |
|
---|
| 713 | conference paper
|
---|
| 714 | developer docs
|
---|
| 715 |
|
---|
| 716 | svn instructions
|
---|
| 717 |
|
---|
| 718 | Ideas
|
---|
| 719 | -----
|
---|
| 720 |
|
---|
| 721 | - store all config in config.ldb
|
---|
| 722 |
|
---|
| 723 | - load from smb.conf if modtime changes
|
---|
| 724 |
|
---|
| 725 | - dump full system config with ldbsearch
|
---|
| 726 |
|
---|
| 727 | - will need the ability to form a ldif difference file
|
---|
| 728 |
|
---|
| 729 | - advanced web admin via a web ldb editor
|
---|
| 730 |
|
---|
| 731 | - normal web admin via web forms -> ldif
|
---|
| 732 |
|
---|
| 733 | - config.ldb will replace smb.conf, secrets.tdb, shares.tdb etc
|
---|
| 734 |
|
---|
| 735 | - subsystems in smbd will load config parameters for a share
|
---|
| 736 | using ldbsearch at tconx time
|
---|
| 737 |
|
---|
| 738 | - need a loadparm equivalent module that provides parameter defaults
|
---|
| 739 |
|
---|
| 740 | - start smbd like this: "smbd -C tdb://etc/samba/config.ldb" or
|
---|
| 741 | "smbd -C ldapi://var/run/ldapi"
|
---|
| 742 |
|
---|
| 743 | - write a tool that generates a template ldap schema from an existing
|
---|
| 744 | ldb+tdb file
|
---|
| 745 |
|
---|
| 746 | - no need to HUP smbd to reload config
|
---|
| 747 |
|
---|
| 748 | - how to handle configuration comments? same problem as SWAT
|
---|
| 749 |
|
---|
| 750 |
|
---|
| 751 | BUGS:
|
---|
| 752 | add a test case for last_entry_offset in trans2 find interfaces
|
---|
| 753 | conn refused
|
---|
| 754 | connect -> errno
|
---|
| 755 | no 137 resolution not possible
|
---|
| 756 | should not fallback to anon when pass supplied
|
---|
| 757 | should check pass-thu cap bit, and skip lots of tests
|
---|
| 758 | possibly allow the test suite to say "allow oversized replies" for trans2 and other calls
|
---|
| 759 | handle servers that don't have the setattre call in torture
|
---|
| 760 | add max file coponent length test and max path len test
|
---|
| 761 | check for alloc failure in all core reply.c and trans2.c code where allocation size depends on client parameter
|
---|
| 762 |
|
---|
| 763 | case-insenstive idea:
|
---|
| 764 | all filenames on disk lowercase
|
---|
| 765 | real case in extended attribute
|
---|
| 766 | keep cache of what dirs are all lowercase
|
---|
| 767 | when searching for name, don't search if dir is definately all lowercase
|
---|
| 768 | when creating file, use dnotify to tell if someone else creates at
|
---|
| 769 | same time
|
---|
| 770 |
|
---|
| 771 | solve del *.* idea:
|
---|
| 772 | make mangle cache dynamic size
|
---|
| 773 | fill during a dir scan
|
---|
| 774 | setup a timer
|
---|
| 775 | destroy cache after 30 sec
|
---|
| 776 | destroy if a 2nd dir scan happens on same dir
|
---|
| 777 |
|
---|