Recently, I have been reading Stephen Wolfram's, The Second Law, and might now better comment on entropy and how it is related to system identification. Note that the statistical form of entropy is as the expectation of the logarithm of probabilities of parts of some partition. In thermodynamics, the change in entropy is the change in heat of a system normalized by absolute temperature. This heat is distributed over a partition of molecules. Heat in a vacuum cannot be held--simply transmitted by radiation.
Molecules hold heat by the excitation of their kinematics which is perceivable as temperature. Energy can additionally be held by elevation of states of electron shells which collapse and emit photons that carry heat by radiation; convection is the mass movement of energetic molecules and conduction is the transfer of energy between molecules caused by collision. Below the level of the molecule, there is no sense in the thermodynamic understanding of entropy. Subatomic particles do not all have mass and do not all act as a molecule does. Not everything in the universe consists of molecules in contact with each other and therefore, the idea of heat death of the universe is nonsense. Please do not worry about it. It makes a person dour and prone to nihilism (q.v. Boltzmann).
The point here is that whether it is used in information theory or thermodynamics, the concept of entropy is simply a mathematical relation for a system characterized by multiplicative interactions that can be decomposed by additive parts via the logarithm--it is a model. Where the model is appropriate, as in the case of heat among molecules, it is a useful quantification of the state. Where it is not, it is not used. Thus, the "Second Law" has limited applicability. It is not even applicable as we generally believe it to be. Energy lost by radiation makes a process irreversible, not by some miracle of the creation of greater "entropy" but rather by the simple loss of energy under the First Law. Heat is a catch-all term for perceptible energy in matter that tracks to temperature. Heat is simply a form of energy--it is not every form. Thermodynamic entropy only applies to heat.
Interestingly, many also confuse the phenomenon of mixing as also being entropic--but this is not properly understood either. Mixing of molecules in three dimensions occurs because the mean free travel between molecules tends to a maximum simply by the physics of motion and the fact that the energy transmitted by collisions is not unidirectional. Entropic models may fit these systems only because the mathematics fits. There is nothing special about the physics that links one entropically-described process to another. Entropy is a model, not a law of nature applicable to everything.
So, while I do not share Stephen's excitement, I may share in some of his insights. Entropy is a useful mathematical concept for the decomposition of a multiplicative system into constituent parts. That is all.
