Well, we'll try to answer these witout writing a book, Paul. Incidently, if you can find a book titled "The Design and Behavior of Bolted Joints" by John H. Bickford (CRC Press) in the library, it can answer just about any question you can come up with; any edition after the first edition even references a couple of friction coefficient projects of mine in the reference list for Chapter 7 (Torque Control of Preload). I'd recommend you buy a copy, but it runs around $125 last I checked (4th edition).
First, there is no ideal length for thread engagement when torquing fasteners, just the caution that you don't use a thread engagement below the minimum engagement arrived at by calculations. With a simple nut and bolt, IF you aren't playing games with materials, the height dimension of the nut takes care of that for you. With a set stud and nut type joint, there is the necessity for someone to calculate the minimum length of engagement and assure that the set end engagement is over that value. A rule of thumb for most "normal" material combinations (does not work well for Grade 8 or higher strength steel head studs into an aluminum engine block) is that the set end length of engagement must be at least equal to the fastener nominal diameter. It's all done with computer programs that reduce the calculations to a couple minutes (to get all the stresses in the different parts of the joint, including bearing stress of the nut and the bolt head against the clamped material takes about 8 hours of number runching by hand, about 5 hours after you've done a few). The computer is about .2 seconds after about 2 minute's worth of inputs, then you have to evaluate the results and do another iteration if you're not where you need to be. In your example of a steel engine block and steel heads, the thread engagement at the set end for the set stud or the bolt into the blind hole, an engagement of 1 diameter would generally be adequate, though the actual engagement would probably be more - conservative design works in the long haul. Also worthy of note - nuts are stronger than bolts/studs, as are blind holes, so you can often get maximum allowable stress on a bolt or stud with a nut or set end material that is not as strong as the male fastener. Also, heavy hex nuts (being wider in the across the flats dimension) are stronger than regular hex nuts, and also have a larger bearing stress area, so they stress the clamped material less at the same torque, which also means they can be used at a higher torque if bearing stress under the nut is the limit, rather than the nut or bolt strength,
Yes, high initial preload and minimizing fastener to fastener preload variation are the two keys to establishing and maintaining joint integrity, and that involves exceeding some minimum value, which is where generic torque tables can sometimes lead one astray. When establishing the required preload for a joint, some target figure is usually used, expressed as a % of yield. The thing is, it ends up (when you use a properly designed calculation regimen) being X% of yield of the weakest element in the joint - the bolt or stud, the nut, and in the case of a set stud or a bolt driven into a blind hole, the set end material, OR 150% of yield of the clamped material for bearing stress. Depending on the materials you are dealing with, any one of these limits establishes your preload - for a "flat plate" type joint. And that assumes the joint design was produced with the proper size and number of fasteners to begin with. Then there are curve balls, like raised face flanges in piping systems, where, depending on the pressure class of the flanges, you may find that the preload limit is nowhere near what the fasteners can carry as the fastener load limits would cause bending of the flanges. Or like gasketed joints, where the driver is usually the desired (or maximum allowable) gasket compression.
Permissable/allowable bolt stretch is a function of fastener length for any chosen diameter, regardless of what % of the length of the fastener is threaded; longer fasteners can be stretched more than shorter ones, which is why the target preload is usually expressed as a % of yield rather than as an increase in length due to fastener stretch. HOWEVER, IF you have the proper equipment AND unimpeded access to both ends of the fastener, a stretch keyed to that diameter, length, and material of fastener can be calculated for that preload, and stretch measured to establish preload. It's expensive and labor intensive and highly dependent on the skill of the technician, but it is an excellent method sometimes used in assembly of the crankshaft end of connecting rods, for instance.
Lubing the threads is desirable in most joints; whether the threaded fastener goes into a softer/weaker material isn't important - that gets back to establishing the target preload based on X% of yield strength of the weakest element of the joint - the torque to be applied would be limited to the yield strength of the aluminum the the Grade 8 fastener was driven into, as an example. What is important is knowing what thread lube was used as the basis for establishing the torque, especially if you are using a generic torque table - if the supplied torque was based on dry threads and you use a Molybdenum Disulfide based thread lube (Molykote 77 of GN, say), you would need to reduce the listed torque by over 50% or you would drastically overstress the weakest element in the joint because the friction coeficient of MoS2 lubed fastener is drastically less than dry threads.
What you are talking about in relation to the effort required to drive a threaded fastener into a threaded hole or threaded nut onto it is called running torque. In a normal situation, it is ignored. If the threads are so dirty or buggered up that real resistance is a problem, they need to be further cleaned or chased with the appropriate tap or die. If that isn't possible, you can measure the running torque with a torque wrench and add it to the specified torque. Where the joint uses nylock self locking nuts, the running torque should be added to a specified torque from a table for fasteners smaller than 5/8" - beyond that, the running torque is too small a fraction to worry about, but there is no reason not to add it if it makes the installer feel good, it's just extra work to determine the running torque. A way around that is there are specifications for reuse of nylock self locking fasteners which list a MINIMUM acceptable running torque. Just assume that value and don't worry about induced error - again, they're down in the weeds of fastener to fastener preload variation and don't matter.
You want to have more than just the minimum amount of torque necessary to get into the minimum stretch - we'd write another chapter if we get into short term preload loss, long term preload loss and fatigue issues - suffice it to say that when you are given a torque specification for a specific joint, it was arrived at with the intent to give both satisfiactory initial clamping load AND long term maintenance of sufficient load; it is well over the minum that would be required to make the joint work in the short term.
At installation, definitely clean both male fasteners and nuts or blind holes, then lube with the specified lube - if you reduce the friction coefficient too much, you run a risk of overstressing some part of the joint or overcompressing a gasket. If the torque specified is a dry torque, you can use motor oil or axle/bearing grease (NOT grease with added Molybdenum Disulfide as found in some bearing greases, which will announce that presence proudly - teflon additives are OK) as a thread lube and use the dry torque without any change (I have about a quarter of a million $$ worth of friction coefficient research a previous employer was kind enough to fund over about a 9 year period to prove that one). Thread lubes that actually reduce the dry lube friction coefficient way down there come from the anti-seize family of compounds; the pressure additives in motor oils and bearing greases are for a different type of pressure loading and don't make things any slipprier once you start loading the threads significantly.
If you would like some further insights into the quirks of bolted joints, check out this article in ARC Racing's web site (disclaimer - I wrote it
).
http://arcracing.blogspot.com/2005/04/on-rod-bolts-by-mike-gifford.html