Advanced Message Queuing Protocol

From Wikipedia, the free encyclopedia

Jump to: navigation, search
 This article needs additional citations for verification. Please help improve this article by adding reliable references (ideally, using inline citations). Unsourced material may be challenged and removed. (April 2009)
The Internet Protocol Suite
Application Layer
BGP · DHCP · DNS · FTP · GTP · HTTP · IMAP · IRC · Megaco · MGCP · NNTP · NTP · POP · RIP · RPC · RTP · RTSP · SDP · SIP · SMTP · SNMP · SOAP · SSH · Telnet · TLS/SSL · XMPP · (more)
Transport Layer
TCP · UDP · DCCP · SCTP · RSVP · ECN · (more)
Internet Layer
IP (IPv4, IPv6) · ICMP · ICMPv6 · IGMP · IPsec · (more)
Link Layer
ARP · RARP · NDP · OSPF · Tunnels (L2TP) · PPP · Media Access Control (Ethernet, MPLS, DSL, ISDN, FDDI) · Device Drivers · (more)
This box: view • talk • edit

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for Message Oriented Middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security[1].

AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardise middleware have happened at the API level (e.g. JMS) and this did not create interoperability[2]. Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.

Contents

[edit] Overview

AMQP was originally designed to provide a vendor-neutral (i.e. interoperable across multiple vendors) protocol for managing the flow of messages across an enterprise's business systems.

AMQP is middleware to provide a point of rendezvous between backend systems, such as data stores and services and front end systems such as end user applications. The first applications happen to have been in the financial industry, i.e. trading desks, where the message flows were real time order flows and market data. Though originally used inside of enterprises AMQP can, easily, be used to move messages between organizations.

AMQP lets system architects built common messaging patterns out of a simpler underlying model. Typical messaging patterns are: request-response, in which messages are sent to or from specific recipients, publish-and-subscribe, in which information is distributed to a set of recipients according to various subscription criteria, and round-robin, in which tasks are distributed fairly among a set of recipients. Realistic applications combine these, e.g. round-robin for distributing work plus request-response for sending back responses.

The protocol specification defines a binary wire protocol used between a client and server (also known as a broker). In addition the specification outlines a messaging queuing model and services that an implementation provides.

The queuing model of AMQP provides for a wide range of messaging use-cases and further refines the functions of the clients and brokers. The function of brokers can be usefully broken into two kinds: exchanges and message queues. Message queues store messages, and various implementations can achieve various quality of service. For example a slow but tornado-proof message queue would keep redundant copies in multiple geographic regions while a fast but fragile message queue might keep everything in a single process's RAM. To help improve interoperablity some of these aspects of the message queues are specified in the protocol, e.g. you can state what you need asking a message queue implementing broker to create a new queue.

The standard AMQP exchanges have no semantics for storing messages. They route them to queues, which store them on behalf of recipients. Exchanges implement a range of message routing techniques: one-to-one message passing (like email to one recipient), one-to-N (like an email list), one-to-one-of-N (like a queue for the next open checkout), and so on. Since all exchanges accept messages from N senders, AMQP allows all one-to-any routing to be N-to-any. The rules that configure an exchange, known as bindings, can range from very simple (pass everything input this message Q) to procedural inspections of message content. AMQP allows arbitrary exchange semantics through custom exchanges (which can queue, generate, consume, and route messages in any way desired by the implementation).

Messages consist of an envelope of properties used in routing and by applications and a content, of any size. AMQP message contents are opaque binary blobs. Messages are passed between brokers and clients using the protocol commands Basic.Publish and Basic.Deliver. These commands are asynchronous so that conditions that arise from a command's evaluation are signalled by sending additional commands back on the channel that carried the command originally. AMQP also provides a synchronous message delivery command, Basic.Get/Get-Ok.

Examples of error conditions include signalling by an exchange that it could not route a message because no route was found, or signalling that a message queue declined to accept a message (say because it was full). Message brokers may be configured to handle exceptions in different ways. For example: routing the associated message to a dead letter queue or bring even the broker to a hard stop.

[edit] Development

AMQP was developed from mid-2004 to mid-2006 by JPMorgan Chase & Co. and iMatix Corporation who also developed implementations in C/C++ and Java. JPMorgan Chase & Co. and iMatix documented the protocol as an interoperable specification and assigned to a working group that included Red Hat, Cisco Systems, TWIST, IONA, and iMatix. As of February 2009, the working group consists of Cisco Systems, Credit Suisse, Deutsche Börse Systems, Envoy Technologies, Inc., Goldman Sachs, IONA Technologies PLC, iMatix Corporation, JPMorgan Chase Bank Inc. N.A, Microsoft Corporation,Novell, Rabbit Technologies Ltd., Red Hat, Inc., Solace Systems, Tervela Inc., TWIST Process Innovations ltd and 29West Inc.

A notable design goal of AMQP was to enable the creation of open standard protocol stacks for business messaging both within and between firms by combining AMQP with one of the many open standards describing business transactions, such as FpML or more generically as a reliable transport for SOAP.

Whilst AMQP originated in the financial services industry, it has general applicability to a broad range of middleware problems.

[edit] The AMQP model

AMQP defines a number of entities. From a connection perspective the relevant ones are:

  • Message broker: a server to which AMQ clients connect using the AMQ protocol. Message brokers can run in a cluster but these details are implementation specific and are not covered by the specification.
  • User: a user is an entity that, by providing credentials in form of a password, may or may not be authorized to connect to a broker.
  • Connection: a physical connection e.g. using TCP/IP or SCTP. A connection is bound to a user.
  • Channel: a logical connection that is tied to connection. Hence communication over a channel is stateful clients that perform concurrent operations on a connection should maintain a distinct channel for each of those. Client that use a threaded model of concurrency e.g. encapsulate the channel declaration in a thread-local variable.
Entities in the AMQP model used for message transfer

The entities used for the actual sending and receiving of messages are all declared on a channel. A declaration assures the issuing client that the entity exists (or was previously declared by another client). Any attempt to declare a named entity with different properties than it was declared before will result in an error. In order to change the properties of such an entity it must be deleted prior to a re-declaration (with changed properties).

Some of these entities are named. The naming must be unique within the scope of the entity and its broker. Since clients usually (at least no such operations are defined in the AMQP specification) do not have the means to get a list of all available named entities, the knowledge of an entity name is what allows the client to perform operations on it.

Names are encoded in UTF-8, must be between 1 and 255 characters in length and must start with a digit, a letter or an underscore character.

[edit] Exchanges

Exchanges are the entities to which messages are sent. They are named and have a type as well as properties such as:

  • passive: the exchange will not get declared but an error will be thrown if it does not exist.
  • durable: the exchange will survive a broker restart.
  • auto-delete: the exchange will get deleted as soon as there are no more queues bound to it. Exchanges to which queues have never been bound will never get auto deleted.

Note that exchanges are scheduled for removal in AMQP/1.0.

[edit] Queues

Queues are the entities which receive messages. They are named and have properties but not a type. Clients can subscribe to queues to the effect that the message broker delivers the contents of the queue to the client. Alternatively clients can pop messages from the queue as they see fit.

Message are guaranteed to be delivered in order when are delivered for the first time to a queue. If some kind of rerouting operations (e.g. due to failures) such an order is not guaranteed.

The properties of queues are:

  • alternate-exchange: when message are rejected by a subscriber or orphaned by queue deletion message get routed to the this exchange and get removed from the queue.
  • passive: the exchange will not get declared but an error will be thrown if it does not exist.
  • durable: the queue will survive a broker restart.
  • exclusive: there can only be one client for this specific queue.
  • auto-delete: the queue will get deleted as soon as no more subscriptions are active on it. This shares the same constraint as the auto-delete property for exchanges: if no subscription has been ever active on the queue it will not get auto-deleted. An exclusive queue however will always get auto-deleted when the client terminates its session.

Note that queues are scheduled to replace exchanges in AMQP/1.0.

[edit] Messages

Message are unnamed and are published to an exchange. They consist of a header and a content body. While the body is opaque data the header contains a number of optional properties:

  • routing-key: this field is used in ways dependent on the type of the exchange.
  • immediate: the message will get handled as unroutable if at least one of the queues which would receive the message has no subscription on it.
  • delivery-mode: indicates that a message might need persistence. Only for such messages the broker makes a best-effort the prevent a loss of the message before consumption. If there is uncertainty on the brokers end about the successful delivery of a message (e.g. in case of errors) it might deliver a message more than once. Non persistent delivery modes do not show this kind of behavior.
  • priority: an indicator (a range between 0 and 9) that a message has higher precedence than others.
  • expiration: the duration in milliseconds before the broker may handle the message as unroutable.

[edit] Bindings

A binding is a relationship between one queue and one exchange that specifies how messages flow from the exchange to the queue. The binding properties match the routing algorithm used in exchanges. Bindings (and exchange algorithms) can be placed on a curve of increasing complexity:

  • Unconditional - the binding has no properties and requests "all" messages from the exchange.
  • Conditional on a fixed string - the binding has one property, the routing key and requests all messages that have an identical routing key.
  • Conditional on a pattern match - the binding has one property, the routing key and requests all messages that match the routing key using a pattern-matching algorithm. Arbitrary pattern syntaxes could be used. AMQP implements topic matching.
  • Conditional on multiple fixed strings - the binding has a table of properties, the arguments and requests all messages whose headers match these arguments, using logical ANDs or ORs to combine matches.
  • Conditional on multiple patterns - the binding has a table of properties, the arguments and requests all messages whose headers match these arguments, using a pattern matching algorithm and logical combinations.
  • Conditional on algorithmic comparison - the binding has an algorithmic expression (like an SQL SELECT WHERE clause) and requests all messages whose headers match that expression.
  • Conditional on content inspection - the binding specifies arbitrary criteria that are resolved by inspection of the actual message content.

Not all these are implemented as standard, or by all implementations.

[edit] Exchange types and the effect of bindings

These four entities form the basic model of the AMQP. The key to understand how a message is passed to a queue lies in the relationship between the type of an exchange and the resulting interpretation of the routing key.

An exchange will deliver up to one copy of a message to a queue if the routing key in the message matches a binding (subsequent semantically identical bindings will not lead to duplicate copies). What constitutes a match however is solely dependent on the type of an exchange:

  • a direct exchange matches when the routing key property of a message and the key of the binding are identical.
  • a fanout exchange always matches, even on bindings without a key.
  • a topic exchange matches the routing key property of a message on binding key words. Words are strings which are separated by dots. Two additional characters are also valid: the *, which matches 1 word and the #, which matches 0..N words. Example: *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq.
  • a headers exchange matches on the presence of keys as well as key–value pairs which can be concatenated with logical and–or connections in a messages header. In this case the routing key is not a criterion for matching that is considered by the exchange. Neither does the binding carry a single routing key but special format which contains header keys and / or key-value-pairs which match on the header key being present or the header key being present and the value being the same respectively.

Other e.g. vendor-specific exchanges are explicitly permitted in the specification.

The concept of binding named queues to named exchanges has powerful properties (with binding making those two entities independent of each other). It is, for instance, possible to bind a single queue with multiple bindings to the same or to different exchanges. Or multiple consumer can share the name of a queue and bind to it with the same parameters and will therefore get only message that the other consumers did not consume. Or multiple consumers can declared independent queues but share the bindings and get all the message every other consumer would get on the bound exchange with these bindings.

[edit] Specification revisions and the future of AMQP

The following specifications of the AMQ protocol have been published, in chronological order:

  • 0-8 in June 2006
  • 0-9 in December 2006
  • 0-10 (documents are undated)
  • 0-9-1 in November 2008

AMQP/0-10 is the closest to the forthcoming AMQP/1.0. The main technology introduced in 0-10 is the session which abstracts a bilateral conversation across failures. 0-10 also introduced a flow control mechanism to prevent consumer congestion, distributed transactions and more rigorous support for JMS.

The release 1.0 of the specification has currently draft status. The release is likely to change the model illustrated above by changing the role of exchanges and bindings, and replacing these with queues and a new entity called links.

This change aims to remedy two problems with the previous approach:

  1. The publisher needs to know too much about the receivers topology (what exchanges and exchange types are available).
  2. Producer flow control is challenging - if an Exchange is routing a message to 2 different queues, one empty and the other nearly full, what flow control information should be relayed to the producer and how would that be determined?

Other changes include the introduction of a queue addressing schema similar to E-mail and XMPP. This raises addresses to first-class entities, and allows for the publication of service location records using the DNS.

[edit] Implementations

These are the known publicly available AMQP implementations:

[edit] Comparative specifications

These are the known open specifications that cover the same or similar space as AMQP:

  • Stomp, a text-based pub-sub protocol developed by Codehaus, uses the JMS-like semantics of 'destination'.
  • RestMS, an HTTP-based message routing and queuing protocol that provides AMQP interoperability through an optional profile.
  • XMPP, the Extensible Messaging and Presence Protocol.

There has not as yet been a formal comparison of these and other protocols in the same space. While JMS, the Java Messaging service, is often compared to AMQP, JMS is an API specification (which does not guarantee interoperability between implementations) while AMQP is a wire-level protocol specification (which in theory does). Note that AMQP does not have a standard API.

[edit] External links

[edit] References

  1. ^ O'Hara, J. (2007). "Toward a commodity enterprise middleware". Acm Queue 5: 48–55. doi:10.1145/1255421.1255424. http://www.acm.org/acmqueue/digital/Queuevol5no4_May2007.pdf. 
  2. ^ Vinoski, S. (2006). "Advanced Message Queuing Protocol". Ieee Internet Computing 10: 87–89. doi:10.1109/MIC.2006.116. http://dsonline.computer.org/portal/site/dsonline/menuitem.6dd2a408dbe4a94be487e0606bcd45f3/index.jsp?&pName=dso_level1_article&TheCat=1001&path=dsonline/2006/12&file=w6tow.xml&;jsessionid=LR6hp4hBgfJRWv1vwFDlgvLv471rG9VXMpwLNmjL0mfnYNkTnGrs!-288318448. 
Retrieved from "http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol"
Personal tools
Languages