A security traded in some context other than on a formal exchange such as the NYSE, TSX, AMEX, etc. The phrase "over-the-counter" can be used to refer to stocks that trade via a dealer network as opposed to on a centralized exchange. It also refers to debt securities and other financial instruments such as derivatives, which are traded through a dealer network.
Taobiz explains Over-The-Counter - OTC
In general, the reason for which a stock is traded over-the-counter is usually because the company is small, making it unable to meet exchange listing requirements. Also known as "unlisted stock", these securities are traded by broker-dealers who negotiate directly with one another over computer networks and by phone.
Although Nasdaq operates as a dealer network, Nasdaq stocks are generally not classified as OTC because the Nasdaq is considered a stock exchange. As such, OTC stocks are generally unlisted stocks which trade on the Over the Counter Bulletin Board or on the pink sheets. Be very wary of some OTC stocks, however; the OTCBB stocks are either penny stocks or are offered by companies with bad credit records.
Instruments such as bonds do not trade on a formal exchange and are, therefore, also considered OTC securities. Most debt instruments are traded by investment banks making markets for specific issues. If an investor wants to buy or sell a bond, he or she must call the bank that makes the market in that bond and asks for quotes.