You know, I've had thoughts along similar lines in the email space (SMTP). HTTP is such a fantastic protocol and an amazing amount of engineering effort has gone into it compared to SMTP. I've wondered whether there would be any interest in defining a translation from SMTP into HTTP, with an eye toward eventually deprecating SMTP in the fullness of time.
For example, to send an email, perhaps you just send an HTTP POST request to a canonical endpoint (email.example.com), instead of all the rigamarole that SMTP servers require with a unique text protocol requiring multiple round trips. Have you seen the number of SMTP commands involved in sending a single email? Here's an abbreviated transcript of what it's like to send an email using `telnet`:
# Wait for banner from server (RT #1)
220 email-inbound-relay-1234.example.com ESMTP Sendmail 1.0.0; Thu, 29 Sep 2016 19:22:12 GMT
# Send EHLO and wait for reply (RT #2)
EHLO example.com
250-email-inbound-relay-1234.example.com Hello ws-1.example.com [1.2.3.4], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
...
250 HELP
# At this phase you should really send STARTTLS and negotiate a TLS connection,
# but we'll just ignore that for now and proceed plaintext.
# Specify sender (RT #3)
MAIL FROM: jcrites@example.com
250 2.1.0 jcrites@example.com... Sender ok
# Specify recipient (RT #4)
RCPT TO: jcrites@example.net
250 2.1.5 jcrites@example.net... Recipient ok
# Specify message headers and content (RT #5)
DATA
354 Enter mail, end with "." on a line by itself
Subject: Hello, world!
Fun stuff
.
# Wait for reply (RT #6)
250 2.0.0 u8U1LC1l022963 Message accepted for delivery
Furthermore, if you skip these steps or front-run them, some servers will consider that suspicious or spammy behavior. (RFC 2920 properly allows this as an extension called pipelining, advertised in the EHLO reply above.)
With full use of SMTP extensions, things are a bit better than I imply but still frustratingly suboptimal. For example, I've run across ISPs who purely for their own load management reasons want to close an SMTP session at the TCP level after an arbitrary number of emails have been sent (N < 100)! Why would they desire that? If we're going to exchange more messages, then it's certainly less efficient for us both to negotiate a new TCP session and TLS session, rather than reuse the one we already have, but such is the practice of email. So message sending often can be as inefficient as this. When sending to some ISPs worldwide it's not uncommon for a single message to take seconds to deliver under normal network conditions.
How about we replace all of that with an HTTP POST to email.example.com, specifying the email headers and content with the POST body, and the sender and recipient as headers or querystring parameters? I think it'd be nice to get there eventually rather than drag SMTP on forever. All of the effort that goes into HTTP clients, servers, and security could benefit the email community as well.
Proper TLS security is still nascent in SMTP -- only because of Google's actions with Gmail and their Safer Email [1] initiative has TLS really come into widespread adoption at all. Today, although a lot of email is nominally taking place over TLS, most clients are not involving any sort of path validation and the connections are susceptible to MITM; and email clients don't specify client TLS certificates nor do servers examine them. If we were to employ it, TLS client certificate authentication could be an effective way to prevent email forgery, e.g., require email from example.com to be sent from a client with a TLS certificate for that domain. This kind of thing would be much easier to achieve in the HTTP world than in the SMTP world. We could also take advantage of HTTP/2 pipelining to efficiently deliver a lot of traffic across just one TCP connection.
We'd still need most of the effort invested into email, such as all of the effort fighting abuse, and mail servers would still need to buffer outbound messages and authenticate inbound ones, etc. (and we'd still need SPF, DKIM, DMARC) but at least it would simplify the foundational and protocol-level work, like what's involved in bootstrapping a new email client or server from scratch. You could write basic code to send an email in a few minutes using an HTTP library in any language. SMTP is pretty well entrenched, however, and the incremental benefit is probably not large enough, so I don't have my hopes up.
FastMail has been working on something called JMAP [1] for quite some time. It's an HTTP-based replacement for IMAP. Perhaps it could be extended to replace SMTP as well. Then we would have a single, HTTP-based API for all of our email needs.
I'm not convinced that's a useful generalization. Beyond that IMAP and SMTP do something vaguely related to email, there's very little overlap between the two protocols.
Yes, that's an apt comparison. Proprietary APIs for email exist today between parties that trust each other (clients and their service providers). What I'm proposing is to take it further and devise a standard HTTP-based protocol for message transmission between equal parties like ISPs, and for scenarios where there isn't preexisting trust.
For example, today if you use an HTTP API to submit a message to SendGrid or Mailgun or Amazon SES, that's a trusted relationship based on an account you have with the service, typically a paid relationship. Each provider has its own unique API, which is incompatible with other providers.
In the next step of that process, your service provider's Mail Transfer Agent (MTA) communicates with the final destination mail server (`example.com MX`), and that part is a peer relationship between ISPs (quasi-trusted or untrusted). This communication is all SMTP today, and I'm proposing the idea of a standard way to transmit emails over HTTP in this layer too, in such a way as that it would, in the fullness of time, obsolete SMTP.
I'm recently thinking a lot about email because of project[0], and yes, I do agree that it would be great to tunnel SMTP inside HTTP(S). (But yes, I'd go with a SRV entry, not a constant name.)
But no, it's not a great idea to let HTTP replace SMTP. SMTP is a stateful protocol, and while that brings some problems, it also brings some gains. For example, backend servers can keep connections open between them, peers can negotiate resource usage in real time, and the entire extension model is only possible because of connection state.
You'd lose all of those (or replace them with ugly hacks) by tunneling over stateless HTTP. It's a worth trade-off on some situations (like when you are behind a bad proxy), but not always.
For example, to send an email, perhaps you just send an HTTP POST request to a canonical endpoint (email.example.com), instead of all the rigamarole that SMTP servers require with a unique text protocol requiring multiple round trips. Have you seen the number of SMTP commands involved in sending a single email? Here's an abbreviated transcript of what it's like to send an email using `telnet`:
Furthermore, if you skip these steps or front-run them, some servers will consider that suspicious or spammy behavior. (RFC 2920 properly allows this as an extension called pipelining, advertised in the EHLO reply above.)With full use of SMTP extensions, things are a bit better than I imply but still frustratingly suboptimal. For example, I've run across ISPs who purely for their own load management reasons want to close an SMTP session at the TCP level after an arbitrary number of emails have been sent (N < 100)! Why would they desire that? If we're going to exchange more messages, then it's certainly less efficient for us both to negotiate a new TCP session and TLS session, rather than reuse the one we already have, but such is the practice of email. So message sending often can be as inefficient as this. When sending to some ISPs worldwide it's not uncommon for a single message to take seconds to deliver under normal network conditions.
How about we replace all of that with an HTTP POST to email.example.com, specifying the email headers and content with the POST body, and the sender and recipient as headers or querystring parameters? I think it'd be nice to get there eventually rather than drag SMTP on forever. All of the effort that goes into HTTP clients, servers, and security could benefit the email community as well.
Proper TLS security is still nascent in SMTP -- only because of Google's actions with Gmail and their Safer Email [1] initiative has TLS really come into widespread adoption at all. Today, although a lot of email is nominally taking place over TLS, most clients are not involving any sort of path validation and the connections are susceptible to MITM; and email clients don't specify client TLS certificates nor do servers examine them. If we were to employ it, TLS client certificate authentication could be an effective way to prevent email forgery, e.g., require email from example.com to be sent from a client with a TLS certificate for that domain. This kind of thing would be much easier to achieve in the HTTP world than in the SMTP world. We could also take advantage of HTTP/2 pipelining to efficiently deliver a lot of traffic across just one TCP connection.
We'd still need most of the effort invested into email, such as all of the effort fighting abuse, and mail servers would still need to buffer outbound messages and authenticate inbound ones, etc. (and we'd still need SPF, DKIM, DMARC) but at least it would simplify the foundational and protocol-level work, like what's involved in bootstrapping a new email client or server from scratch. You could write basic code to send an email in a few minutes using an HTTP library in any language. SMTP is pretty well entrenched, however, and the incremental benefit is probably not large enough, so I don't have my hopes up.
[1] https://www.google.com/transparencyreport/saferemail/