close
  • Mo chevron_right

    End to end encryption in Movim - OMEMO is (finally) there!

    pubsub.movim.eu / Movim · Wednesday, 15 December - 22:01 edit · 7 minutes

A few days ago I finally closed the OMEMO encryption ticket on Github. Opened in 2015 it had many twists and turns along the years but I finally found a proper way of integrating it in Movim.

In this article I'll explain why adding #E2EE (End to End Encryption) was not as easy as with the other #XMPP clients (and more generaly all the chat clients that are using a similar encryption protocol) and how I addressed the issue.

But before going in the details I'd like to thanks the NLNet Fundation for its financial support in this project. With their help I was able to free-up some time to work on the problem and propose a proper architecture (detailled bellow) for it.

NLNet Foundation logo

The result of this work will be released with the upcoming 0.20 version of #Movim. There is still some quirks and whims about it but the base is there and works pretty well.

End to End encryption in XMPP, a quick overview

The introduction of Signal in 2015 brought a small revolution into the encryption protocols in the IM ecosystem. The Double Ratchet Algorythm (see the dedicated technical documentation on the Signal website) allowed users to exchange messages between different clients in an “end to end encrypted” way (only user devices themselves know how to encrypt and decrypt messages) with some technical improvements (not detailed here) that made the new protocol a “must have” for all the others IM solutions.

Today the Double Ratchet Algorithm is used in applications such as WhatsApp or Matrix.

In the XMPP ecosystem it was primarily pushed by Daniel Gultsch in the Conversations.im client and standardized along the way in the OMEMO XMPP Extension XEP-0384: OMEMO Encryption. Throughout the years many XMPP clients implemented OMEMO, their status can be tracked on the following website Are we OMEMO yet?.

The OMEMO architecture

Without going too deep into the technical details the general idea about OMEMO is to generate some keys on each of the user's devices and publish the public ones on their account server.

Using the keys published on the XMPP user's servers, anyone can then start an encrypted session at any time (the servers are always available) and start to send messages to the desired contact without having to wait.

Publishing keys and building sessions with OMEMO

If one of the user's contacts wants to start an encrypted discussion they will first start to get those keys, then build sessions with their secret one and encrypt a message using the freshly built sessions.

If a user receives a new encrypted message and doesn't have an encryption session to that device, their device will then retrieve the contact keys, build the encryption sessions and start decrypting messages.

This can be done automatically if the contact trusts blindly the key used or in a more “trusted” way by accepting manually each keys to build the encryption sessions on.

All the existing XMPP clients are using this simple architecture. XMPP servers are storing their users' #OMEMO public keys and the users are connecting directly using their different devices to build their encrypted sessions.

The Movim particularity

But Movim is kind of special. The XMPP connection is actually not maintained on user devices but by the Movim server (built in PHP and running behind a web server such as Apache or nginx, see Movim General architecture on the Wiki). Movim is then processing everything server side, saving the information (articles, contacts and messages) in a SQL Database (PostgreSQL or MySQL) and then showing the result to the Movim users through a lightweight website.

If a user is connecting on the same Movim instance through several browsers using the same XMPP account all the browsers are then “merged” into one unique XMPP session (called "resource") and all the browsers are synchronized in real time by the Movim server. This is pretty useful to save memory and to prevent Movim to maintain several XMPP connections at the same time for a unique user. This also allows quick disconnection/reconnections, the users can close and reopen their tabs without having to reload the whole XMPP state when they come back after a while (Movim is closing the XMPP session after a day of inactivity).

End to end encryption actually requires to encrypt and decrypt messages on the user device, this brings several issues:

  • For Movim, the user device is actually a “dumb” browser that only display the messages pre-processed by the Movim server, there is no logic whatsoever browser side
  • A user can use simultaneously several browsers with the same XMPP connection on the same Movim instance
  • All the message processing logic is done server side

This unique architecture requires a very unique way of adressing the E2EE situation. Hopefully OMEMO offers all the tools needed to handle those cases.

Split the logic

The OMEMO extension is actually talking about devices, for a large majority of the XMPP clients a device is connected through a unique XMPP session (one device equal one current XMPP resource in those cases).

Publishing keys with Movim

The fact that Movim is sharing a unique session (resource) with several devices (browsers) is actually not an issue in the end. Each browser will then be considered as a unique device on its own, with its own key and its own OMEMO encrypted sessions.

Building encrypted sessions with Movim

This brings some interesing results. When a user is connected using the same XMPP account using two different browsers on the same Movim server (also called instance, or pod), an encrypted message sent by the browser Firefox will then directly be decrypted by the browser Chrome without even having to travel through the XMPP network.

The term “browser” is also defining more than actual browsers (like Firefox, Chrome or Opera). Since we can have private navigation or containers (in Firefox) each time it is seen as a different “browser” on the Movim side (because each context is separated, with a different cookie and different local data).

So the global idea is to continue to handle the messages server side, push the encrypted message object to the browser, and then implement only the key handling and message encryption-decryption flow browser side. When doing this implementation I actually looked at the Converse.js and JSXC OMEMO implementations, the Movim implementation is really close to the one done on those two clients (I am also re-using the libsignal JavaScript implementation).

This architecture actually works for the current version of OMEMO (0.3.0) where only the body is encrypted. The upcoming versions are looking to encrypt a larger part of the XML stanza. This will be way more difficult to handle for Movim, as it will require to decrypt messages browser side and then implement a second parser, this time in JavaScript (everything is parsed in PHP using libxml at the moment).

if (textarea.dataset.encryptedstate == 'yes') {
    // Try to encrypt the message
    let omemo = ChatOmemo.encrypt(jid, text, Boolean(textarea.dataset.muc));
    if (omemo) {
        omemo.then(omemoheader => {
            ...
            xhr = Chat_ajaxHttpDaemonSendMessage(jid, tempId, muc, null, replyMid, mucReceipts, omemoheader);
            ...
        });
    }
} else {
    xhr = Chat_ajaxHttpDaemonSendMessage(jid, text, muc, null, replyMid, mucReceipts);
    ...
}

This little JavaScript Movim code extract presents the differences in handling encrypted and unencrypted messages. The text variable is containing the clear text version of the message. When the body is encrypted it is then calling the same method as for a clear text message.

This method is actually a wrapper generated by the RPC (Remote Procedure Call) Movim server core. Once this function is called an Ajax called is made and the rest of the flow is handled server side. The encrypted body, and generated OMEMO headers passed will be injected in a freshly generated XMPP XML <message>.

Keep the messages in the local database

With the separation of the logic it was then required to keep a copy of the decrypted messages browser side.

To do that an IndexedDB database is used. This database is quite simple and only contains a key-value store, where the key is the message id (the same as the one in the Movim server SQL server databse) and the value the plaintext message.

  • When a message is decrypted the plaintext body is then stored in this database.
  • When the user sends an encrypted message, the original text is also saved in this database.
  • If a message cannot be decrypted, the message key is still saved in the browser database with a false value. This prevents Movim to try to decrypt several times a message, knowing that the decryption will fail each time in the end.

Using this database, when a chat is loaded, all the messages are then sent chronologically from the server, passed trough a little bit of code that will lookup the state of all messages and then decrypt the ones that are not decrypted yet, the already decrypted messages are then shown, or an error is displayed for those that cannot be decrypted.

To sum up

In this article I tried to present you what limitations I faced when trying to implement end to end encryption within Movim and what architectural and technical solutions were used to address them.

The current solution seems to fit and bring all the desired features to Movim without too much downsides. The feature can now be considered as done and will be released soon. And as always, lots of small fixes and adjustments will be integrated to polish it afterward.

That's all folks!

edhelas

  • wifi_tethering open_in_new

    This post is public

    mov.im

  • favorite

    8 Like

    jorgeluis, meta, matlag, marzanna, kill, sthaan, niklas, eyome

  • 2 Comments

  • person

    23 December sthaan

    Great work! Does OMEMO also work for group chats?

    I'm however a little worried if the Javascript handling the keys is served by the same server which e2ee protect against. If I understand correctly, a malicious server operator could easily extract my private key by (temporarily) serving a modified, malicious version of the Movim frontend.

  • person

    31 December kill

    Bon boulot

  • Mo chevron_right

    A little update from the Movim project

    pubsub.movim.eu / Movim · Tuesday, 16 November - 21:29 edit · 1 minute

It's been a while since I have told you about Movim. I was quite busy the past few months with many other things in my personal life, but this doesn't mean that I haven't worked on Movim!

So in short, here is a few things that have been changed recently:

api.movim.eu

api.movim.eu has been upgraded and refreshed with a new Laravel backend, Google ReCAPTCHA has been replaced with hCaptcha

The Legals page will be soon upgraded. A specific ticket has been opened to allow you to report and bring ideas on how to improve it.

End of the Android application

Following our previous blog post about the state of the Android application I decided to officially archive the Android app and remove it from F-Droid.

Some work has been done to integrate Movim better as a #PWA (Progressive Web App) to still allow you to enjoy it from your mobile phone.

Security Audit

We had an extensive #SecurityAudit by Radically Open Security that did an amazing job.

The audit covered Movim itself but also api.movim.eu and all the related projects, including movim/movim_docker. The result of the audit is currently restricted but we are actively using the finding to improve the general project security:

An finally on Movim itself

  • I added a basic support of OMEMO for MUC Groups, you can already try it out on the official pod
  • And as always, bugs fixes!

That's all folks!

edhelas

  • wifi_tethering open_in_new

    This post is public

    mov.im

  • favorite

    10 Like

    chrisalzo, sthaan, odomar, marzanna, drone839, samuel, joergd, debacle, matlag, blue

  • Mo chevron_right

    Element and Movim Messengers Comparison Made Simple - ubuntubuzz.com

    pubsub.movim.eu / Movim · Sunday, 3 October - 20:05 edit

A nice #comparison between #Movim (XMPP) and Element (Matrix) from similar features. Thanks !

  • wifi_tethering open_in_new

    This post is public

    mov.im

  • favorite

    5 Like

    debacle, purplesky, matlag, blue, eyome

  • 2 Comments

  • person

    3 October slavko

    No worth to link it...

  • person

    4 October neko_teko

    When we can expect inline replies?

  • Mo chevron_right

    We are looking for some help for the Movim Android application

    pubsub.movim.eu / Movim · Monday, 31 May, 2021 - 20:17 edit · 2 minutes

Hi,

I am writing this little post today to ask for (a bit) of help on the official Movim #Android application that is currently available there.

I am maintaining #Movim for more than 10 years now, mostly without asking for any help from the hundreds of daily users that we have now (on the servers we're aware of, at least). I am also maintaining and updating the infrastructure hosting the website, the official pod and the #XMPP server and related services.

However Movim is a little more than that, there is also an Android app, and a desktop app (currently abandoned).

I am maintaining the Android app with the really thin knowledge that I have on this platform. Most of the code of the application is basically copy/pasted from StackOverflow and a bit hacky. Fixing and developing this app is always really time consuming as the environment, and the way of dealing with it, is really different than the knowledge that I have on the web development side.

So basically today I'm asking from some help for the official Movim Android application. If you have a little bit of experience in Android development (or if you want to learn !) and if you're willing to help me to improve the application do not hesitate to contact me through our official chatroom, directly on Github or as a comment of this post.

The app is quite simple. It is basically a WebView that shows one of the configured pods. And… that's mostly it.

What would be really helpful would be to have (non exhaustive list):

  • Another pair of eye to check and maybe refactor the few classes that are contained in the application
  • Create two flavors of the app, one "Play Store ready" and one "free of Google Play" integration (the current app cannot be published on F-Droid because there is some strong dependencies there)
  • Fix the mic/camera support and authorizations
  • See if the current notification system can be improved
  • Send an event to Movim when the app is "put in background" in a chat conversation to ensure that the notifications are re-enabled in this specific case
  • And any other nice feature that you would like to see integrated

There is no pressure or deadline, any pull requests that could help improving or fixing things in the application would be really appreciated.

I am currently dealing with Google to re-enable the application on the Play Store as well, I will keep you updated about that.

Regards,

edhelas

  • Mo chevron_right

    Movim 0.19 - Ikeya is out!

    pubsub.movim.eu / Movim · Saturday, 20 February, 2021 - 21:58 edit · 7 minutes

This version is a major step for Movim with a lot of improvements on the performances and many (many!) small fixes and new features.

Chat and chatrooms

A criticism that was coming once in a while is how difficult it was to join and/or create a chatroom in Movim. The 0.19 version follow the Modern XMPP Multi-user Chats recommendations that is already implemented in several other XMPP clients. This standard is bringing a common naming convention and flow to create and configure chatrooms in XMPP.

Add chatroom dialog

On top of that the join/create chatroom flow was completely redesigned. The new one is inspired by the Telegram one, with one unique "+" button to start a new one-to-one or one-to-many conversation. The global chatroom search was also directly integrated in the flow to allow you to quickly search and join one of the few thousands already listed ones.

The one-to-one chat conversations views are now cached server side. This simple trick is saving some precious milliseconds during the page load. The performance boost allowed us to simply remove the chat list placeholder (you don't need a placeholder if the content is loading instantly).

Conversations improvements

The encrypted messages are now displayed in the bubbles.

An encrypted message

You can receive messages read confirmation in group chats and in small channels (when there are less than 10 persons connected).

XEP-0201: "Best Practices for Message Threads" has been implemented. Now Movim handles replies on received messages. Useful to follow multiple topics during a conversation.

A replied message

Like images before, now videos are also embedded in conversations with the support of the WebM and H264 video format.

With this awesome feature, the Tenor API was also integrated. You can now search in millions of GIFs and publish them instantly in your conversations. The Tenor API is disabled by default, the server administrator can enable it by adding its custom API key in the Movim admin panel.

The GIF picker

All the published message URLs are also now embedded in the discussions. Now you can also easily browse the URL related pictures directly in the Movim picture preview gallery widget.

An embeded URL

Chat rooms with unread messages are put on top of the chat rooms list now.

A package of ~1500 new emojis were added to the emoji picker.

A couple of new emojis

Communities and blogs

The explore page was reorganized with a few new features. You can now filter the explore panel to return only the communities or blog articles (the same way you can filter the articles on the news page). A "More" button is also now allowing you to retrieve older articles.

The new discover page

The communities on the servers are now ordered by last updated.

But also…

The preview widget is now having a "copy to clipboard" button. Useful!

Preview and copy easily the picture link

There is a first integration of the Firebase Push notifications in Movim. This is currently in test with the Android app and will require some more improvements and adjustments before being released as a stable feature.

You can now directly drag & drop files or copy & paste pictures in Movim to trigger the upload dialog box.

The drag and drop box

Performances

Batch insertion

Lets now dive in the more technical aspects with the performances improvements.

Most of the performances bump in Ikeya are actually regarding how he database is requested. In the previous Movim version, a presence buffer was introduced to save "batches" of incoming requests in one database query and not hundred of them.

The presence is a core concept of XMPP (eXtensible Messaging and Presence Protocol). During the connection, you will receive a presence from each online contact from your contact list, similarly, when you join a chatroom, you will as well receive a presence for each member connected to the room.

So now imagine that you have a big account, with hundreds of contacts and a few dozen chat rooms with hundreds of connected users in each of them. During the login, your XMPP client will then receive thousands of presences. The presence buffer is there to "stack them" when they are received and save them in batches in the database and then notifies the UI (your browser) only once that "all those presences were saved".

This presence buffer element was greatly improved and will not only save the presences in batches now, but also handle some related data like capabilities requests (those information are used to know what your contacts XMPP clients are capable of) as well as vcard requests (your contact personal information and avatars).

In a similar aspect, the retrieved bookmarked conferences and community subscriptions are also now saved in batches in the database, saving dozens of requests.

Preload

Another important optimisation that can be done when dealing with databases is to preload some information when retrieving list of items. Lets imagine that you need to retrieve 50 messages from the database. Those messages might have related data, such as reactions, replies or attached files. It is often good to load in parallel all those data "in batches".

The wrong way
1. Get the messages A, B, E and F
2. Process the messages
  - Check if A has an attached file
  - Check if B has an attached file
  …

The good way
1. Get the messages A, B, E and F
2. Try to get files for the A, B, E and F messages
3. Once the two lists are retrieved, see if some files matches the messages

This was already done in many places in Movim (hopefully), but even after years we are still finding some small improvements that were missing. All those small improvements are now allowing Movim to retrieve full conversations and scroll the history in a few hundred milliseconds, even if the database is filled with millions of messages.

You should know that each time you open a discussion in Movim, your browser is actually asking the server about all the contact that will be displayed, there is no local cache. If you have a backend that is fast enough, you don't need complex Javascript frontends to process and store things ;)

Database connection auto-close

Each connected user on a Movim instance is having it's own specific process launched on the server. This is bringing some nice isolation and performances distribution, especially on servers with many parallel available threads (this also brings some memory consumption issues that will be improved in the upcoming versions).

For each of those launched "user-session-processes" a database connection was opened to allow them to perform database queries quickly.

The main issue there is that database servers (such as MySQL and PostgreSQL, the two supported servers for Movim) can only handle a certain limit of parallel connections. Movim is then now automatically closing unused connections after a few seconds and resume them once a new query is sent. This allows Movim to handle hundreds of parallel sessions without overloading the database server anymore.

Translations cache

The languages translations are now cached when the daemon is launched once for all. This brings a few milliseconds gain during page load. It's not much but it's an easy win!

The translations can also be manually recompiled using the following command

php daemon.php compileLanguages

Picture proxyfying and caching

Most of the Movim pictures are now proxyfied by Movim. This way the Movim pod will hide its users personal IPs and browser information when the pictures are retrieved. The proxy is also moving all the pictures URL under a specific picture/ directory. This simple trick allows server admins to create a nice caching system directly on the web-server level.

For example, if a friend in a chatroom is sharing a nice picture that is a bit heavy. The first person that will receive the picture will ask Movim, through the Movim web-server to download it and return an optimized version of it. The web-server will then keep a copy of the optimized picture. All the other users will then ask the same resource, and the web-server will simply return the cached version, without even asking anything to Movim anymore.

A specific section in the Movim setup tutorial was added to explain you how to setup and configure this cache system during the Movim deployment.

To conclude

This was quite a big version indeed, we hope that you will enjoy it. The upcoming one will be a special release and will focus on one specific feature. We will come back to it in the upcoming weeks, be patient.

A clue about the upcoming release?

As always, if you like Movim please share it around. You can always help us by directly contributing to the project and help funding us on our Patreon.

That's all folks!

  • Mo chevron_right

    Movim | Basic Review & Beginner's Guide

    pubsub.movim.eu / Movim · Sunday, 14 February, 2021 - 11:17 edit

Thanks for this awesome guide by UbuntuBuzz. With a lot of details and some ideas of improvements for the future. #Movim #review #guide

  • Mo chevron_right

    Movim is moving!

    pubsub.movim.eu / Movim · Wednesday, 20 January, 2021 - 06:47 edit

Following our previous article, with the sudden subscription of hundred of new users both on our XMPP #services movim.eu and jappix.com and on #Movim itself it seems that we are starting to be quite limited by our current servers capacity.

You might have noticed some connection issues and downtimes the past few days that are the direct result in this recent gain of interest for Movim.

In the upcoming days we are planning to migrate the whole #infrastructure to a new server to give a bit more space for the project to grow and to allow many new users to join us.

This will involve some extra server costs. If you want to help us covering our expenses you can always join our Patreon.

By the way, we also bumped the size limit from 3Mb to 10Mb on our XMPP upload service, enjoy!