You have two basic choices, wire rope and synthetic rope. Both will do the job and each have plusses and minuses. They both are rated for strength via diameter and the tensile strength of their materials. In the common 6,000-15,000 recreational 4x4 winch, cable diameters of 5/16, 3/8 and 7/16-inch are seen in both wire and synthetic ropes. Tensile strength, the point at which the rope breaks (a.k.a. "parts") varies according to type, characteristics and diameter, but is higher than the pull rating of the winch. Their ratings have some reserve for a safety factor to account for wear and minor damage.
Wire Rope: Wire rope is the traditional choice. It's made up of strands of carbon steel wire, usually seven bundles of 19 strands (a.k.a. 7x19). Less common is 6x37, which is more flexible and resistant to fatigue but less resistant to abrasion than 7x19 because of the thinner wires. With just a few exceptions, 7x19 wire rope is what comes with winches from the factory.
On the plus side, wire rope is more resistant to abrasion than synthetic. It's very heat resistant and relatively inexpensive. On the downside, it vulnerable to crushing when not properly respooled. It's relatively heavy, with a 120 foot roll of 5/16 cable weighing about 28 pounds with a hook. Wire rope can be spliced but not easily or by novices. Individual wire strands routinely break and create small hooks that rend flesh, so gloves are vital. Wire rope can store large amounts of kinetic energy and can whip with deadly effect if something breaks loose.
Synthetic Winch Rope: Synthetic winch ropes are the new kid on the block and there's lots to talk about. As few premium winch packages come new with synthetic rope. Most winch manufacturer's are offering a synthetic rope upgrade and you can buy the rope separately to upgrade most types of older winches.
On the plus side, synthetic is light; a 120 foot roll with a hook weighs about 5 pounds. Synthetic generally drops dead with minimal "snap" when something breaks. It's often stronger than an equivalent diameter wire rope and has appendage-friendly surfaces. It floats and in an emergency you can tie a knot into it. Knots do weaken it but if you perform a long splice at a break, it's as good as new. Splicing 12 strand rope is lots easier than wire rope.
Synthetic rope also has a few disadvantages. It's more susceptible to chafing than wire rope. It has a high initial cost, though it's superior performance makes it a good value overall. The most significant potential operational problem has to do with synthetic's ability to withstand heat. The common planetary winch has a one-way brake built into the center of the winch drum. When spooling out under power, i.e. lowering the vehicle under a load, that brake will generate some serious heat. Winch manufacturers caution against this lowering practice and advise doing it only in short 20 foot increments with long cooling periods. Rope damage from heat can go unseen on that first layer and damage may start occurring at temperatures as low as 150 degrees.
There are several ways around the temperature dilemma. Some winch rope has a higher temperature resistance but comes with lower tensile strength, so you would have to use a larger diameter rope to maintain the same tensile strength. An interesting cure is to have enough of the high temp rope spliced onto a stronger rope to cover the first later of the drum. Larger diameter rope can be splices to smaller to compensate for the strength difference. An even easier solution is to place an insulating sheath of nylon over that first layer. The easiest solution is to follow the winch manufacturer's recommendation and not power out in long stretches. Remember that most worm or spur gear winches do not have this problem.
