Donald Knuth, a mathematician (his adviser was Marshall Hall, his thesis on algebraic structures related to projective planes) turned computer scientist, became dissatisfied with typesetting while producing his multi-volume The Art of Computer Programming. So he took time off to produce a computer typesetting system which would satisfy his high standards. The result was 
TeX has remained almost unchanged since the late 1970s, and still produces much higher-quality mathematical typesetting than more recent word-processors. It and its derivatives such as LaTeX are now so standard in mathematical publishing that journals specify that manuscripts should be in LaTeX, and often provide style files for the purpose. The inventor of LaTeX, Leslie Lamport, neglected to give instructions on how it should be pronounced, with the result that this is somewhat controversial.
TeX is not WYSIWYG. It gives users the possibility to “create masterpieces of the publishing art”, as its creator said; but also allows various horrors, since the computer cannot divine the operator’s intentions.
One of my pet aversions is the use of “less than” and “greater than” for angle brackets. Suppose I have a group G generated by two elements a and b. If I say, G=\langle a,b\rangle in a mathematical formula, TeX gives me 
I was once asked to review a new mathematics journal for a librarians’ journal. I had to point out that, since the authors effectively typeset their own papers, they were able to produce horrors like complicated fractions in exponents, which are very hard to read and parse. (Incidentally, in traditional publishing, copy-editors were there to save us from this; but publishers have given up on this important function.)
Other problems with mathematics are not entirely the fault of the typesetting system.
A t-(v,k,λ) design (or, for short, a t-design, consists of a set of v points, with a collection B of k-element subsets called blocks, such that each set of t points is contained in a unique block. The concept was introduced by Dan Hughes in the early 1960s (though the case t = 2 was familiar to statisticians much earlier). Dan credits Donald Higman with inventing the terminology. I once asked Dan about the correct way to typeset this. His reply was, more or less, $t$-$(v,k,\lambda)$, producing 
(Incidentally, Dan Hughes and Don Higman were early mentors of mine, to whom I owe a great debt. Don was my doctoral examiner in Oxford, and as I student I took and wrote up notes of his lectures on coherent configurations. Dan twice offered me a job, and also persuaded me to write my first book, with Jack van Lint. They both organised regular Oberwolfach programs at which I was a regular attendee and learned a lot.)
There is another problem with the notation, the same as for the group theorists’ usage of p-groups. A 2-design is a t-design with t = 2. But other concepts have been introduced, such as Ryser’s λ-designs; should a 2-design be a λ-design with λ = 2? It gets worse. Arnold Neumaier allowed t to be an integer plus a half; the title of his paper was t½-designs, or more precisely 
There is another example of this. To most of the combinatorial world, a k-graph is something a bit like a graph, except that an edge contains k vertices rather than just two. (This has one of the problems I just discussed, if a number is substituted for k.) But the notion of two-graph was defined by Graham Higman, as a 2-cocycle (mod 2) on the simplex, that is, a collection of 3-element sets with the property that any 4-element set contains an even number of them. These are very important objects (another story), and I think that Higman wrote the word “two” to discourage such substitutions. In this he was not completely successful, and in any case, this is now a source of some confusion …
Peter Cameron's Blog 