Annotated Obby Session

Below is the traffic generated by a simple session between Ebby and Gobby. I've annotated it so that others who may try to implement Obby support in other editors will have an easy starting point and hints on the protocol. Lines in italics are sent by Ebby, plain lines are sent by Gobby.

Everything the server sends consists of a command and parameters. Some commands also have subcommands, such as obby_document having subcommands like subscribe, sync_line, and record. The record subcommand is further divided up into ins and del.

All numbers given are in hexadecimal.

Note that this pertains to version 0.3 of the protocol. (Gobby 0.3 is rather temperamental in a few areas. Sending incorrect operation counts (see below) or positions beyond the end of the document will cause crashes. This has been fixed in 0.4.)

Sample Session

Logging In

net6_client_login:phil:88ff44

The client logs on by sending a net6_client_login command, followed by their username and a hexadecimal representation of the color they want to be represented with.

obby_welcome:5:2q3zg3il03:pm6vo8c81hi8a28j6ifqfhwmnwhsku7meo7gl2lyjg81ep0qwqojc6i6lygfb2el5lxr8smhg2lufjxxgd0iyno9vn88naikemz:4d

The server responds with an obby_welcome command, the version of the protocol it supports, and (I believe) an RSA authentication.

obby_sync_init:5

The server begins the initial sync by specifying how many lines of data the sync will include.

net6_client_join:8:phil:2:88ff44

This is the client's own user information, in the format net6_user_id:username:obby_user_id:color

net6_client_join:1:puyo:1:ccccff

Any other users will be listed here with the same format. This information is necessary to get proper coloring support in the client.

obby_sync_usertable_user:3:philh:88ff44

I believe this represents other users who may have edited existing documents, but are no longer connected.

obby_sync_doclist_document:1:1:a:1
obby_sync_doclist_document:1:2:b:1

The server then sends a list of every open document. Each document is identified by a pair of numbers: one for the obby user id of the creator, and one for the per-user index of the document. (User 2's first document would be "2 1", their second "2 2", etc.) The fields here are creator's user id, creator document index, document name, and a list of users currently subscribed to the document. The server may also notify the client of new documents created by sending obby_document_create with the same parameters.

obby_sync_final

This simply indicates that the server is finished syncing with the client.

Subscribing

obby_document:1 1:subscribe:2

The client sends this to subscribe to one of the documents listed in obby_sync_doclist_document above using the document ID made up of the creator's ID and the document index.

obby_document:1 1:sync_init:3

The server tells the client how many lines there are in the document.

obby_document:1 1:sync_line:jibbery jabbery:0:1
obby_document:1 1:sync_line: this is also cool. :0:3:7:1
obby_document:1 1:sync_line:another line here.:0:1

Each line is sent. The numbers after the line indicate who authored it. On the second line, it says :0:3:7:1. This indicates that the first chunk of the line (starting at position 0) was authored by user 3, while the second chunk of the line (starting at position 7) was authored by user 1. Note that this synchronization process has been changed in 0.4: You now get sync_chunk:text:0 where the user 0 has written text. Multiple chunks make up the whole document, newlines are escaped by \n.

obby_document:1 1:subscribe:2

Changes to the document

The server is done sending lines, and the client is considered subscribed.

obby_document:1 1:record:0:0:ins:37:s
obby_document:1 1:record:1:0:ins:38:e
obby_document:1 1:record:2:0:ins:39:n
obby_document:1 1:record:3:0:ins:3a:t
obby_document:1 1:record:4:0:ins:3b: 

Here is an instance of the record subcommand. Our client has just entered the word 'sent ' and transmitted it to the server. The numbers in between record and ins are the local and remote operation counts. Every time the client makes a modification to the document, the local operation count should increase. Every time the server transmits a change, the client should increase its remote operation count. The number right after ins is the position at which the character should be inserted. Note that you can also insert more than just one character, a command like obby_document:1 1:record:0 0:sent is also valid.

obby_document:1 1:record:1:0:5:ins:3c:r
obby_document:1 1:record:1:1:5:ins:3d:e
obby_document:1 1:record:1:2:5:ins:3e:c
obby_document:1 1:record:1:3:5:ins:3f:e
obby_document:1 1:record:1:4:5:ins:40:i
obby_document:1 1:record:1:5:5:ins:41:v
obby_document:1 1:record:1:6:5:ins:42:e
obby_document:1 1:record:1:7:5:ins:43:d

Note that when the server sends insertions, the position of the local operation count and remote operation count is reversed (so the first number is the server's local operation count and the second number the server's remote operation count). Here a remote user just typed the word 'received' at a position right after our client typed 'sent '. Note that if the client were to transmit something at this point, it should sent 8 as the remote operation count, not 7. This is because every received modification should increase the remote operation count to one greater than what the message indicating the change shows.

obby_document:1 1:record:1:8:15:del:23:1

Here the server informs the client that a deletion has occurred with a del. Apart from the operation count information directly proceeding the record subcommand, the final two parameters indicate the position the deletion began as well as the length of the deletion.

obby_document:1 1:record:1:9:15:del:3c:8

Here a whole 8 characters have been deleted, from 0x3c to 0x44.

Transforming operations

Ok, lets say we are sending

obby_document:1 1:record:0:0:ins:0:a

Later, we receive

obby_document:1 1:record:0:0:ins:1:b

One could assume that this means that we have to insert 'b' at position 1 in the buffer, but this is not correct. When we look at the server's remote operation count we note that it is still zero. This means that inserting the 'b' at position 1 is incorrect because the server did not know about our 'a' when he sent us its operation. If the document, say, was foobar, we made afoobar out of it. The server inserted 'b' at position 1 which results in fboobar. When we would apply the 'b' at position 1 in our document we would get abfoobar which is not what the server intended since it inserted the 'b' between the 'f' and the 'o' and not before the 'f'. The solution seems obvious: We have to insert the 'b' at position 2 because we already inserted a character before the 'b' that the server did not know about.

This process is called inclusion transformation: We include the effect of the operation "insert a at position 0" (lets call it ins(a@0)) to the operation "insert b at position 1" (ins(b@1)). This resulted in "insert b at position 2" (ins(b@2)). So we have to cache every operation that we sent to the server in some kind of list with the local operation count included. Whenever we get an operation from the server we remove all operations from that list that have a smaller local operation count as the server's remote operation count (because these are operations the server already knows about). If some operations remain, we have to perform an inclusion transformation of every operation that is still in the list with the operation the server sent us before applying it to the document.

Now, imagine the above scenario the other way around. We send

obby_document:1 1:record:0:0:ins:2:a

and get

obby_document:1 1:record:0:0:ins:0:bar

This time, we must include the effect of ins(a@2) into ins(bar@0). Since 'a' was inserted behind 'bar', nothing has to be done. But imagine we now get a second record:

obby_document:1 1:record:1:0:ins:3:baz

The remote operation count is still zero, so when the server wrote 'baz', he did still not get our 'a' and intended to insert 'baz' right behind the 'bar' he inserted beforehand. But when we include the effect of ins(a@2) into ins(baz@3), we get ins(baz@4). So, when the document's initial content was 'foo', we first made 'foao' out of it, then ins(bar@0) is applied resulting in 'barfoao' and after this ins(baz@4), leading to a final content of 'barfbazao'. It is easy to see that this is not what the server intended since it inserted 'baz' right after 'bar' without the 'f' inbetween. So what went wrong here?

So, when the server sent the first record ins(bar@0), we did not transform anything, which was right, because our 'a' has been inserted behind the server's 'bar'. However, this caused the 'a' to move 3 characters forward. So we also should have included the effect of ins(bar@0) into ins(a@2), and not only the other way around. This way, ins(a@2) turns into ins(a@5). As soon as the second record ins(baz@3) arrives and we include the effect of ins(a@5) into ins(baz@3), it still remains ins(baz@3) because ins(a@5) is now behind it. But notice that, again, we have also to include the effect of ins(baz@3) into ins(a@5), resulting in ins(a@8), to be prepared for the case when the server sends a third record with a zero remote operation count.

Why ins and del is not enough

In fact, ins and del are not the only required operations. There are two other operations known as NoOperation? (noop) and SplitOperation? (split). These are not generated directly in response of user input but might result from inclusion transformations. A NoOperation? is just an operation that does nothing and therefore has no parameters. A SplitOperation? is simply a wrapper around two operations.

Obviously the NoOperation? is the simpler case, so I will begin to explain this one. Let's say, we have a document whose content is 'foobar'. We now delete some characters to make 'far' out of it by sending

obby_document:1 1:record:0:0:del:1:3

to the server. The server decided to delete only 'oo' and sent

obby_document:1 1:record:0:0:del:1:2

When we get this operation we notice that the server deleted 'oo' before he got our deletion of 'oob'. Therefore, we have to include the effect of del(1,3) into del(1,2). When we delete three characters starting at index 1, the first to characters starting at index 1 are already deleted, so we have to do nothing, because we already deleted ourselves what the server deleted. The result of this transformation is therefore a noop. This happens every time when we receive a delete operation that deletes a range that is already deleted.

The SplitOperation? is required when one inserts text into a range of text that has to deleted. Consider the following example (again, the initial content is 'foobar'):

obby_doucment:1 1:record:0:0:ins:3:bal
obby_document:1 1:record:0:0:del:0:6

so the server deletes all six characters in the document, resulting in an empty document while we inserted 'bal' at position 3 which turns to 'foobalbar'. Now we receive the delete record from the server. Obviously the server wants to delete 'foobar', so we cannot just perform del(0,6) because it would be 'bar' what remains and not 'bal'. When transforming the incoming operation, it has to be splitted up into two delete operations: del(0,3) and del(6,9) which deletes the original 'foobar'.

Disconnecting

obby_document:1 1:unsubscribe

When you're done with a document, send an unsubscribe message to the server.

obby_document:1 1:unsubscribe:2

The server confirms you have unsubscribed. The same type of message will be used to inform you if other users unsubscribe from a document you have subscribed to.

To disconnect from the Obby session, simply close the socket.